Simple IoT - Blog

Sync is now stream replication

2026-08-18T00:00:00+00:00

The previous post covered moving the Simple IoT store to NATS JetStream. This post covers how we use streams to synchronize offline/catch-up data. There is also a demo of offline synchronization on the Simple IoT YouTube channel.

Offline synchronization provides several key benefits:

Nothing is lost during network disruptions, which are a fact of life in distributed systems, especially edge systems.
We can take down cloud/upstream services (including databases) at any time and no data is lost. This drastically simplifies operations and maintenance if we don’t need always-on, highly available infrastructure.

The old way

In the past, SIOT has synchronized cloud and edge instances using a Merkle-style hash tree: hash every node, compare hashes up the tree, and push the subtrees that differ. It worked, but there was a lot of overhead in calculating the hashes and updating every upstream node every time a point changed.

The new way

Since the SIOT store is now on JetStream, we already have history data available. The only tricky parts are:

How do we handle multiple writers (downstream and upstream)?
Where do we draw the stream boundaries?

I originally planned one stream per node, but ended up with streams at the device boundaries, which is simpler and has less overhead.

Because both the downstream and upstream can write data, there needs to be a stream for each writer. Sync means each side holds a replica of the other’s stream, driven by durable NATS consumers: a push pump copies the device’s stream up, a pull pump copies the hub’s configuration stream down. Current state on either side is the merge of the two, newest timestamp winning.

Because no instance ever writes remote data into its own streams, points cannot echo back and forth between instances. There is no loop to suppress, which removes an entire class of sync bug by construction.

A durable consumer remembers its position across disconnects, so a device that has been offline for two weeks reconnects and receives exactly the backlog, in order, with original timestamps. No rescan, no comparison, no sync period to configure. Replication is continuous: a write on one side shows up on the other as fast as the connection carries it.

SIOT drastically simplifies stream-based replication

With SIOT, the developer rarely has to be concerned with the low-level stream details. SIOT handles all of those details under the hood, and all the developer needs to do is create nodes, points, and the clients that respond to and send points. The rest happens automatically.

Distributed systems are hard, especially systems where you can’t assume the connection is solid and all services will be running all the time. With SIOT, a lot of this complexity is eliminated. NATS JetStream provides the building blocks (streams, durable consumers, etc.), and SIOT manages them in a way that is transparent to the developer.

The full design is in the sync reference and ADR-7.

The store is now a stream

2026-08-13T00:00:00+00:00

Simple IoT 0.23 replaces the SQLite store with NATS JetStream. This is the biggest architectural change in the project’s history, and it is the first of a series of posts on what we’ve been building this year. This one covers the store itself; the next covers what it does for synchronization.

Two shapes of the same data

SIOT has always been a NATS application. Every component, whether a Modbus client, a rule, or the UI, communicates by publishing points as messages. The store’s job was to catch those messages and keep them.

For years it did that by translating each point into rows in SQLite (and before that, Bolt and Genji). SQLite served us well, and I would still reach for it in plenty of applications. But it meant we maintained two shapes of the same data: the message in flight and the row at rest, with translation code in between, and a Merkle-style hash tree bolted on top so two instances could figure out how their rows differed.

The real problem was never the database. It was that we were translating between two data models when one of them already did everything we needed.

Points are already messages

A JetStream stream is an append-only log of messages with sequence numbers and per-subject indexing. That shape matches IoT data unusually well:

Persisting a point means capturing it. The same message that moves between components lands in a stream. No translation layer, no second data model.
History is the natural byproduct. A stream retains every point written to a subject, so time-series history lives in the same place as current state. The current value of a point is simply the last message on its subject.
Sequence numbers replace the hash tree. Streams are ordered, so another instance can replicate one and know exactly what it has and has not seen. More on this in the next post.
It is already there. JetStream runs inside the NATS server every SIOT instance ships, on cloud machines and small edge devices alike. There is no separate database process to install or operate.

Reads never touch JetStream directly. The store keeps the merged current state in in-memory caches, populated from the subject tips at startup, so queries stay fast.

Retention that understands configuration

Retention is per subject: each point type on each node keeps its own most recent 20,000 messages by default. Per-subject limits have a property that time- or size-based retention cannot give you: configuration points, written a handful of times, are effectively kept forever, while a sensor reporting every minute wraps after about two weeks. Current state is always preserved.

The default works out to about four months of history for 10-minute data, bounded on disk even for a device nobody visits. A device with little flash can lower it with SIOT_STORE_MAX_MSGS_PER_SUBJECT.

Compression pays exactly where it matters

Streams are compressed with S2 by default. Point data compresses unusually well, because the same type repeats in every message, keys come from a small set, and timestamps march forward. In our testing, a store of 100,000 scraped points went from 33.4 MB to 11.7 MB. A small store gives up nothing, and compression starts paying exactly where disk begins to matter.

Migrating

Existing SQLite databases migrate with siot export on v0.22.0 followed by siot import on 0.23. The full design analysis is recorded in ADR-7, and the store reference covers retention, compression, and durability tuning in detail.

This is the kind of simplification that feels right. Stay tuned for how synchronization works, and how streams simplify so many things.

Hello

2020-06-23T00:00:00+00:00

The Simple IoT project now has a web site. Stay tuned for more updates!