Skip to content

Store and Forward

What store and forward does

Ordinary collection is a live stream. If the link between two nodes drops, the values produced while it was down are gone.

Store and forward changes that for the tags you choose. The sending node writes each of those values to a durable buffer on its own disk, called a spool, before it counts as sent. The receiving node acknowledges what it has stored, and only then does the sender release its copy. A link outage, a node restart or a power loss does not lose what is already in the spool.

This is used primarily for MQTT connections to the cloud or into Historians which can accept historical data.

The spool is always in the path

Buffering does not switch on when a link drops. Every selected value goes to disk first, whether the link is up or down. What an outage changes is only the draining side: the spool stops emptying and its depth grows.

There are two durable buffers in every store and forward stream, and they cover different failures:

Buffer Lives on Covers
Uplink spool The sending node The link to the receiving node being down
Receiver buffer The receiving node The final destination, such as a broker, being down

What gets buffered

Store and forward is selective. Only the tags named by a data route with Store & forward turned on are spooled. Everything else keeps flowing on the ordinary live path and is not buffered.

Setting it up takes two things:

  1. On Connections, a Store & Forward Uplink on the sending node: a named durable push destination with an Endpoint pointing at the receiving node, plus its spool sizing. On the receiving node, a Store & Forward Receiver to accept the pushes.
  2. On Data Routing, a route with Store & forward enabled. The wizard describes the checkbox as "Buffer this route's data durably (survives a WAN outage)."

An uplink with no route naming it carries nothing. The uplink card shows Selected tags for exactly this reason: 0 selected tags means nothing will spool through that uplink.

Each tag rides exactly one durable path. A tag already designated to one uplink cannot also be designated to another.

Buffered values keep their own timestamps

A spooled value carries the full record: its tag, value, type, quality, unit, source and the timestamp of the measurement. Replaying it later does not restamp it with the replay time.

Separate store and forward streams

One pair of nodes is not limited to one durable path. A sending node can hold up to 8 uplinks, and a receiving node can hold several receivers. Each uplink is its own complete stream, with its own spool, its own backlog and its own limits: a stalled or slow destination fills that stream's spool and no other.

Three named pieces wire a stream end to end, and each choice feeds the next:

  1. A receiver on the receiving node is a named landing buffer. An egress delivers everything its receiver accepts, so data bound for different destinations belongs in different receivers.
  2. An uplink on the sending node names the receiver its pushes land in, with its Target receiver field. Blank means the receiving node's default receiver.
  3. A data route names the uplink its stream rides. The route wizard always shows that uplink; a node with more than one requires a pick.

The receiver rides the uplink

A route cannot name a receiver of its own. Everything pushed on an uplink lands in that uplink's target receiver, whichever routes the records came from. To land a route's data in a different receiver, create another uplink targeting that receiver and put the route on it.

A worked example, two independent streams between a sending node and a receiving node:

Cloud stream Historian stream
Data route, on the sending node line1-to-cloud, rides uplink to-cloud line2-history, rides uplink to-history
Uplink, on the sending node to-cloud, Target receiver cloud-buffer to-history, Target receiver history
Receiver, on the receiving node cloud-buffer, egress to the cloud connector history, egress to the historian connector

If the historian goes down, only the history side backs up, and each stream reports its own backlog state. The cloud stream never queues behind it.

The one-path rule above applies across streams too: two routes may ride different uplinks, but they must not select the same tag. A route whose selection would put a tag on a second uplink is refused with an error naming the tag and both uplinks.

The sending node redials on its own. It does not need a restart.

Replay is oldest first, in the order the values were captured, and the whole backlog drains before the stream catches up to real time. Live values are not jumped ahead of the backlog: there is one queue, new values join the back of it, and the drain reads from the front.

The link reports its progress as one of four states:

State Meaning
disconnected The link is down.
connecting The node is dialing.
flushing The link is up and the backlog is being replayed. Every reconnect passes through this.
connected The link is up and caught up. Steady state.

A long outage therefore shows flushing for as long as the backlog takes to clear.

Duplicates and ordering

Delivery between two Flowgate nodes is at-least-once on the wire. A reconnect or a restart can resend records the receiver already has. The receiver recognizes them by their record id and discards the repeats, so the effect at the receiving node is a single copy of each value.

Order is preserved per path: values replay in the order the sending node captured them. Values from different tags are not re-sorted into a single chronological sequence, so if you need strict cross-tag time order, sort by the timestamp each value carries.

The last hop to an external system is different

Once data leaves the receiving node for an external destination such as an MQTT broker, delivery is plain at-least-once and there is no per-value id to deduplicate on. A reconnect or a restart can redeliver a boundary message. Consumers that cannot tolerate a duplicate should deduplicate on the tag and timestamp.

When the buffer fills

Each spool has two independent limits, both set on the uplink or receiver:

  • Max size (MiB): how much live data the buffer may hold. Leave it blank for the node default.
  • Min disk free (MiB): a floor of free disk space the buffer will not eat into. Set it to 0 to turn the floor off.

When full decides what happens at the size limit:

The node deletes the oldest record that has not been acknowledged, counts it, and accepts the new one. You lose the oldest data and keep the newest flowing. Losses are counted and reported, never silent.

The node stops accepting new records and applies back-pressure. Nothing already in the spool is lost, but the loss moves upstream to the point where the data is collected.

The disk floor behaves differently from both. If free disk falls below Min disk free, the node refuses every new record regardless of the overflow setting, because deleting old records does not give the space back to the operating system. Clearing that condition needs free disk.

Spool files do not shrink

A spool file grows toward its configured maximum and stays there. Size the disk for the maximum you configure, not for the depth you normally expect.

Records are compressed on the way in. Compression is zstd by default, and can be set to none.

What an operator sees

Every store and forward path reports its own condition. The diagnostics for a node show these rows:

Row What it tells you
Store & forward The path's backlog state.
S&F backlog How many records are waiting.
S&F ingest The rate of records arriving into the spool.
S&F headroom How long before the oldest data starts dropping.
S&F holding How old the oldest record still held is.
S&F dropped How many records have been discarded.

The backlog state is one of six values, from healthiest to worst:

State Meaning
idle Nothing waiting.
filling The spool is growing.
draining The backlog is being worked off.
stalled The link is up and the far end is taking nothing.
dropping Records are being discarded right now.
blocked New records are being refused.

Treat stalled as seriously as dropping

A stalled path looks connected but is moving no data. One common cause is an uplink naming a receiver the far node does not have. Those pushes are held rather than dropped, so nothing is lost yet, but the spool fills toward its limit while the condition lasts.

The S&F headroom row reads stalled, dropping now, blocked · disk low or blocked · spool full when one of those conditions applies, and otherwise estimates the time remaining at the current rate. A path that is dropping or blocked, or that is stalled, is raised as a critical issue on Troubleshooting.