跳转到内容
简体中文

Replication

此内容尚不支持你的语言。

DistributedData replicates state by gossip:

  • Every gossipInterval, each node picks one random peer.
  • The node sends a full snapshot of every key it currently holds — or as much of it as fits one frame, resuming next tick (see What gets gossiped).
  • The peer merges each entry; if its state changed, notify local subscribers.

This means writes propagate eventually — usually within 1-2 gossip rounds (1-2 seconds default). For most workloads this is fine; for cases where it isn’t, see Quorum reads/writes.

node-Bnode-Anode-Bnode-Alocal update applied — immediatemerge into local stateper-CRDT merge functionif local value changednotify subscribersgossip tick — full snapshot arrives

Local writes are immediate. Gossip is for propagation.

const distributedDataOptions = DistributedDataOptions.create().withGossipInterval(1_000);
const dd = system.extension(DistributedDataId).start(
cluster,
distributedDataOptions, // default 1s
);

The default gossipInterval is 1 second — chosen as a balance between propagation speed and gossip-bandwidth cost. Lower intervals → faster convergence + more chatter. Higher → less chatter, slower convergence.

For typical applications:

WorkloadInterval
Latency-sensitive shared state (online presence)250-500 ms
Default for most apps1 s
Counters / flags that change rarely2-5 s
Very large clusters where bandwidth matters5-10 s

The same knob is settable from application.conf, so a deployment can tune it without a rebuild — the builder still wins where both are set:

actor-ts.distributed-data.gossip-interval = 2s

See Configuration for the rest of the block.

The full state, sliced to fit one frame — each tick, DistributedData serializes as many keys as fit max-gossip-bytes and pushes them to one random peer, resuming next tick where this one stopped. There’s no delta tracking and no per-peer receipt history, so a store that fits the budget travels whole on every tick.

This makes gossip volume proportional to total state size, not update rate. A million-entry map costs its full serialized size every time it’s gossiped, whether or not it changed; a small counter costs almost nothing no matter how often it increments.

Slicing is safe because merging is per key, and a key absent from a frame means “no information about it this round” — not “deleted”. CRDT merge is idempotent and commutative, so a key that arrives a tick later, or twice, changes nothing about where the cluster converges; it only changes when.

max-gossip-bytes defaults to 1 MiB and is always clamped down to remote.max-frame-bytes (16 MiB by default). Without it a large store produced one frame past the wire cap, and that frame is rejected by the receiver on its 4-byte length prefix — before a single key is parsed. The receiving transport answers that by dropping the connection, which carries heartbeats, membership gossip and every cross-node tell as well. So the store did not converge slowly; it did not converge at all, while killing one peer link per tick.

A single key whose own encoding exceeds the budget cannot be sliced any further — the unit is one key’s full state. It is skipped, and the replica warns (rate-limited, naming the key and both sizes) and increments distributed_data_gossip_skipped_keys_total. That key does not converge. Split the value, or raise both max-gossip-bytes and remote.max-frame-bytes.

If your store grows large, raising gossip-interval spreads the cost over time; lowering max-gossip-bytes spreads one tick’s cost over more ticks.

The early warning: log-data-size-exceeding

Section titled “The early warning: log-data-size-exceeding”

The budget only says something once a key has outgrown a whole frame — by which point that key has already stopped converging. A key at a large fraction of the budget is invisible by comparison and still expensive: it gossips fine, but it eats that fraction of every tick it appears in, so everything else in the store sweeps proportionally slower and nothing says why.

log-data-size-exceeding (default 100K, roughly a tenth of the budget) is that early warning. A key past it is named, not skipped — the packer treats it exactly as it would have:

const distributedDataOptions = DistributedDataOptions.create()
.withMaxGossipBytes(1024 * 1024)
.withLogDataSizeExceeding(100 * 1024);

The warning is rate-limited to one line a minute and carries a running total, and it has its own quiet period rather than sharing the skip warning’s. That matters in the case where both fire: raising max-gossip-bytes is the fix for “this key cannot be gossiped at all”, and it makes “this key costs a tenth of every tick” more relevant, not less. 0 switches the reporting off.

Because the packer only visits keys from its cursor until the budget fills, a key in a large store is measured once per sweep rather than once per tick — the right cadence for something already rate-limited to a minute.

// Every gossip tick, the node picks ONE random reachable peer.

Round-robin gossip would be more predictable but creates synchronized waves; random-peer-per-tick disperses load and typically converges in O(log N) rounds for N peers.

After K rounds, the probability that any specific peer hasn’t received the update is (1 - 1/N)^K — for N=10 nodes and K=5 rounds, that’s < 60 % chance a peer hasn’t seen it; after K=20 rounds, < 13 %.

In practice, convergence is much faster because gossip is multi-hop: peers re-gossip what they received.

import { Actor } from 'actor-ts';
import { DistributedDataChanged, DistributedDataId, GCounter } from 'actor-ts/crdt';
// An actor subscribes itself, and is told on its own turn:
class HitsLog extends Actor<DistributedDataChanged<GCounter>> {
override preStart(): void {
this.system.extension(DistributedDataId).get().subscribe<GCounter>('hits', this.self);
}
override onReceive(changed: DistributedDataChanged<GCounter>): void {
console.log(`hits is now ${changed.value.value()}`);
}
}
// A callback runs on the replicator's turn:
const unsubscribe = dd.subscribe<GCounter>('hits', (counter) => {
console.log(`hits is now ${counter.value()}`);
});
// Later:
unsubscribe();

A subscriber is either an actor, which is told a DistributedDataChanged and does its work on its own turn, or a callback, which runs synchronously on the replicator’s turn. Subscribe an actor for anything that does work or touches actor state — dashboard widgets, real-time UI updates, business logic that should react to changes anywhere in the cluster. Keep a callback for a cheap synchronous read.

Both are told after every successful merge that changes the local value (deep-equal check via the CRDT’s toJSON):

  • Local updates → when the replicator applies the update; update() itself only queues it.
  • Remote updates → when gossip arrives and the merge changes the local value — for every key the round changed.
  • Quorum writes, quorum reads and identity repairs → the same: every merge that changes a key notifies its subscribers, whatever produced it.
  • Idempotent updates → nobody is told (no change).

An actor is told through a latest-value slot, one per subscriber and key:

  • One notification at a time → at most one DistributedDataChanged per key waits in the actor’s mailbox. A change while it waits replaces the value it carries; the next one is posted only once the actor has been handed the last. So the actor handles the latest value as of its own dispatch, and a burst of two hundred updates behind a busy handler arrives as one notification.
  • Paced by its own mailbox → there is no timer, and no notify-subscribers-interval to configure: the actor is told as often as it takes the notifications, and never more often than the key changes.
  • Not shed by a bound → the notification is admitted past a bounded mailbox’s capacity, like a death-watch Terminated. Dropped, it would leave the slot waiting for a hand-over that never comes, and the actor would never be told about the key again.
  • Replayed through the mailbox → subscribing to a key that already holds a value tells the actor that value too — asynchronously, once the replicator has handled the subscription, not before subscribe returns.
  • One subscription per actor and key → subscribing again adds no second subscription. The returned function unsubscribes; a notification already queued still arrives.
  • On this node, and watched → the actor has to live on the replica’s node: a ref to an actor elsewhere is refused with a TypeError, because a notification carries this replica’s live CRDT. The replicator watches the actor, so its subscriptions end when it stops, and a restart of the replicator keeps them. A TestProbe, which has no mailbox to pace, is told every change directly.

A callback runs synchronously on the replicator’s own turn, once per changed key per merge:

  • Subscribing to a key that already holds a value → the new callback is called with it right away, on the caller’s stack, before subscribe returns. One that subscribes from inside another subscriber’s notification gets that value this way, once.
  • A callback that throws → logged as a warning (DistributedData: subscriber for "<key>" threw, with the error); the other subscribers are still called, and the one that threw stays subscribed.

Because a callback runs on the replicator’s turn, its time is the replicator’s time: while one runs, no merge, gossip round or quorum reply is processed — for any key. Twenty calls of a 10 ms callback held a local read queued behind them back by 203 ms. See Callbacks fire on every change below.

When MemberRemoved fires for a cluster member, DistributedData does nothing — there’s no per-peer state to clean up. Gossip holds no version vector and no per-peer receipt history; each tick just picks a random peer from the current up-members, so a departed node simply stops being a gossip target.

State the peer contributed stays in the local replica, merged in like any other update. CRDT tombstones (a removed ORSet element, a null register in an LWWMap) are never garbage collected — they’re kept permanently so a stale gossip can’t resurrect a removed key. Over a long-lived store with heavy churn, tombstone accumulation is the practical cost to watch.

A replica that rejoins under the same identity later (stable pod names, persistent volumes) needs no special handling — it’s just another up-member again, and gossip reconverges its state over the next few ticks.

Rough shape for a 10-node cluster, default 1-second gossip:

  • Per tick, per node — one message carrying up to max-gossip-bytes of serialized store goes to a single random peer. Its size is set by total state, not by how much changed since the last tick, so an idle store still pays this each tick; the cost just doesn’t grow with write rate.
  • Scaling knobs — store size drives the per-message cost, gossip-interval drives how often you pay it, and max-gossip-bytes caps any single payment. A large store on a fast interval is the expensive combination; a large store with a small budget is the slow-convergence one.

For very-large clusters (50+ nodes), gossip volume can become significant. The framework’s gossip is anti-entropy style — all-to-all over time — which scales O(N) per node per round. Bigger clusters may want larger gossipInterval or a different gossip topology (not currently configurable; an issue if needed).

// Node-A writes:
dd.update('hits', ..., (c) => c.increment('a', 1));
// Node-B reads, but the value doesn't reflect:
console.log(dd.get('hits')?.value()); // ← shows stale value

If this surprises you:

  1. Wait a gossip round (1 s default). Most apparent staleness resolves within 1-2 gossip cycles.
  2. Check gossipInterval — if set higher than default, the wait is proportional.
  3. Use getAsync with consistency: 'majority' for reads that must reflect the latest known state across replicas.
  4. Check cluster membership — if node-B is unreachable from node-A, no gossip is flowing.