Перейти к содержимому
Русский

Cluster overview

Это содержимое пока не доступно на вашем языке.

A cluster is a group of ActorSystems — typically one per node — that know about each other. Nodes gossip their membership state, detect each other’s failures, and route messages across the network. Once joined, code on any node can tell an actor on any other node; the cluster transport hides the wire.

Two ways to think about it:

  • From the application’s perspective: one logical actor system spread across N nodes. An ActorRef can point at an actor on any node; the same code that worked single-node still works.
  • From the runtime’s perspective: N independent systems exchanging gossip + heartbeats, each tracking who’s alive, routing messages through a chosen transport.

Cluster.bootstrap packages the four steps you’d otherwise wire by hand — create the ActorSystem, resolve seeds, Cluster.join, hook up SIGTERM/SIGINT — into a single call:

import { Cluster, ClusterBootstrapOptions } from 'actor-ts/cluster';
// Local dev — no env vars, no seeds: single-node cluster.
const { system, cluster, shutdown } = await Cluster.bootstrap(ClusterBootstrapOptions.create('my-app'));
// With explicit overrides:
const { system, cluster } = await Cluster.bootstrap(
ClusterBootstrapOptions.create('my-app')
.withHost('0.0.0.0')
.withPort(2552)
.withSeeds(['node-a:2552', 'node-b:2552'])
.withRoles(['compute']),
);
// Once joined:
cluster.upMembers(); // → ReadonlyArray<Member> of up-status nodes
cluster.subscribe((evt) => { /* MemberUp, MemberDown, ... */ });

For full control (custom dispatcher, manual signal handling, own discovery loop) the lower-level pair still works:

import { ActorSystem } from 'actor-ts';
import { Cluster, ClusterOptions } from 'actor-ts/cluster';
const system = ActorSystem.create('my-app');
const clusterOptions = ClusterOptions.create()
.withHost('0.0.0.0')
.withPort(2552)
.withSeeds(['node-a:2552', 'node-b:2552']);
const cluster = await Cluster.join(
system,
clusterOptions,
);

A few settings are doing most of the work:

SettingPurpose
host + portThe interface and port this node binds. 0.0.0.0 means “every interface” and is the default.
advertisedHost + advertisedPortThe address peers dial — this node’s identity. Unset, the host is derived from host when that is routable, then from CLUSTER_HOST / POD_IP / HOSTNAME, then loopback; the port is simply port.
seedsOther nodes’ addresses. At startup, this node contacts them to join the existing cluster. An empty list = “I’m the first one” (auto-promotes to leader).
rolesTags this node carries. Routers and shard regions can filter on these (role: 'compute' skips nodes without the tag).
appVersionThe application version this node states to its peers — a label for operators, shown on /cluster/members and wherever a member is printed, so the mixed window of a rolling update can be read off the cluster. Nothing places, routes or downs on it. Usually set by the deployment: actor-ts.cluster.app-version = ${?APP_VERSION} — quote a numeric-looking version inside the variable (APP_VERSION='"1.10"'): an environment value is read as JSON first, so an unquoted 1.10 reaches the peers as 1.1.

After Cluster.join resolves, the node is in the cluster (though possibly still in joining state for a few seconds until convergence).

Cluster.join also publishes the instance onto the ActorSystem, so you do not have to thread the handle through every actor that needs it:

system.cluster; // Option<Cluster> — None if this system never joined
this.context.cluster; // the same Option, inside an actor
this.cluster; // unwrapped, inside an actor — throws if there is none

The unwrapped form is the one to use in code that only ever runs clustered, which includes every sharded entity and every singleton — those are constructed by the framework, so there is no call site to inject a cluster at:

class CartEntity extends Actor<CartMessage> {
override preStart(): void {
// No constructor argument, no closure, no options field.
this.log.info(`cart ${this.entityId} on ${this.cluster.selfAddress}`);
}
private onCheckout(): void {
if (this.cluster.isLeader()) { /* ... */ }
}
}

this.cluster.sharding and this.cluster.singleton come along with it, so an actor can start a region or a singleton from the inside.

Reach for this.context.cluster instead when the actor must also run on a plain, unclustered system — it answers None rather than throwing. Both read through to system.cluster on each access, so an actor that outlived the join, or a system that rejoined after leave(), always sees the current instance.

Every member goes through a small state machine:

optional

leader confirms

failure detector

heartbeat resumes

strategy, operator or auto-down

cluster.leave()

tombstone TTL, at once under auto-down

joining

weaklyUp

up

unreachable

down

leaving

removed

  • joining — just announced itself. Other peers know about it but it isn’t routable yet.
  • weakly-up (optional) — gossip-visible to peers but the leader hasn’t confirmed it. Useful for partition-tolerant joins; see Weakly-up.
  • up — fully in the cluster, routable. This is the steady-state status.
  • unreachable — the failure detector flagged this peer as not heartbeating. Still officially a member, but routing avoids it. Flips back to up when it heartbeats this node again — after a pause, though not after a symmetric partition, which leaves neither side heartbeating the other (issue #930). If it stays silent, the failure detector removes it at down-after — the default with no strategy configured — or it stays here until something removes it — see down.
  • leaving — the member is gracefully leaving (via cluster.leave()). Its peers record it as removed as soon as its leave reaches them.
  • down — removed by a decision: a downing strategy’s, an operator’s (Cluster.down, POST /cluster/down), or — with no strategy configured, under the default failure-detector.auto-down = on — the failure detector’s own. Passes straight on to removed.
  • removed — out of the cluster. A leave, a strategy’s decision and an operator’s down keep it as a tombstone for a TTL (default 24 h), so stale gossip from a slow peer can’t accidentally resurrect it. The auto-down route deletes the entry instead, which is why a peer that still lists the member can bring it back — see the failure detector.

The cluster’s events stream surfaces every transition — subscribe to MemberUp / MemberRemoved / MemberUnreachable etc. and react.

Subscribing, and what the replay tells you

Section titled “Subscribing, and what the replay tells you”

cluster.subscribe(listener) starts by replaying the membership that already exists, so a listener that attaches ten minutes into a run still learns the world it joined instead of waiting for the next change. Two forms, chosen per subscription:

import { CurrentClusterState } from 'actor-ts/cluster';
// 'events' (the default) — the membership as the events that built it:
// MemberJoined per member, then the status event that member has reached,
// then LeaderChanged. The live stream's own handler covers the replay.
cluster.subscribe((evt) => { /* … */ });
// 'snapshot' — one CurrentClusterState, whatever the cluster's size.
cluster.subscribe(
(evt) => {
if (evt instanceof CurrentClusterState) {
console.log(`${evt.members.length} members, ${evt.unreachable.length} unreachable`);
}
},
{ replayMode: 'snapshot' },
);

Pick 'snapshot' when the listener wants to know where things stand rather than to re-live how they got there: it is one callback instead of one per member, and it marks where the replay ends — which the event form cannot. unreachable is a subset of members, not a set beside it, and CurrentClusterState is never published on the event stream: it describes one subscriber’s starting point, not something that happened to the cluster.

The replay always reflects getMembers(): tombstones are excluded, and every member is announced in the status it actually holds — an unreachable peer replays as unreachable, not as a fresh join.

Live events follow three rules:

  • An event goes to the listeners subscribed when it was emitted and still subscribed when their turn comes (#1294). A listener that subscribes from inside a callback starts with the next event — its replay already reflects the change in flight — and one whose unsubscribe has run is not called again, not even with the event in flight.
  • Each status transition is announced once (#1428). Four paths used to announce it twice — the failure detector’s unreachable and down verdicts, an applied downing decision and Cluster.down — so two nodes could count one change differently.
  • Dispatch is synchronous and depth-first. A listener that changes the membership inside its callback — downs a peer, leaves the cluster — has the nested events delivered to every listener before the outer event reaches the listeners subscribed after it. With one listener that downs a peer on MemberUnreachable and a second subscribed behind it, the second receives ReachabilityChanged, MemberDown, MemberRemoved and only then MemberUnreachable, still carrying the status unreachable.

When order matters, subscribe to the cluster’s events on system.eventStream instead. The cluster publishes each event there before it calls any listener, so an actor subscribed there receives them in the order they were emitted, and a Cluster.subscribe listener that changes the membership cannot reorder them.

Every event above states a change. If what you want is “tell me where the membership stands every N seconds”, set a cadence and subscribe to ClusterStatsPublished:

actor-ts.cluster.publish-stats-interval = 30s # 0 (the default) arms no timer
import { ClusterStatsPublished } from 'actor-ts/cluster';
cluster.subscribe((evt) => {
if (evt instanceof ClusterStatsPublished) {
console.log(`${evt.up}/${evt.members} up, ${evt.unreachable} unreachable`);
}
});

It carries the member count, the up count, the unreachable count, the leader as an Option<Member> and this node’s own address — every one of them already reachable through the Cluster API, which is the point: what the key buys is the cadence, so a dashboard does not have to run a polling loop of its own.

Three properties worth knowing:

  • Node-local, like every other cluster event, and reached through system.eventStream as well as cluster.subscribe. It is deliberately not on cluster.eventStream: a per-node periodic sample on the cluster-wide bus costs N frames per node per interval to say what each node can already read locally. Bridge it yourself if you want a fleet-wide view — selfAddress is there so the samples stay distinguishable when you do.
  • Never replayed. A subscriber that attaches gets the membership replay described above and no stats sample: replaying a measurement taken before you were listening is just a stale number.
  • Off by default. 0 arms no timer at all, which is what every release before the key did.

It is not the only place these figures appear, and the three do not answer quite the same question. The DevTools dashboard samples them on its own ticker and additionally counts nodes that recently left, so a cluster of three that lost one still reads “2 / 3 up” — this event counts the live member map instead. The cluster_members_up metric is a pull surface with no callback. This one is the framework-native push.

cluster.leader() returns the lowest-addressed up member — whichever member’s host:port sorts first. It is emphatically not the oldest member: address order and join order are unrelated, so a node that joins last leads immediately if its address sorts lowest, and it takes over whatever the leader hosts (a cluster singleton, shard allocation).

That is deliberate. The one property the leader has to have is that every node names the same one, and address order gives that from gossip every node already has — no monotonic join sequence has to travel on the wire. The cost is that “who leads” is decided by addressing rather than uptime: stable across a restart of the same pod, not stable across a re-address.

Gossip is how members agree on who’s in the cluster. Every gossipIntervalMs (default 1000 ms), each member picks a random reachable peer and exchanges its view of the cluster. Over a few rounds, every peer converges on the same state — without any central coordinator.

The protocol carries:

  • Membership table — every member’s address, status, roles, and version vector.
  • Reachability observations — “I haven’t heard from X recently.”

Two peers exchanging gossip merge their tables by picking the higher version for each member. This is what makes convergence work without a leader-elected coordinator: every concurrent update is eventually seen by every peer.

For the deep dive, see Joining and seeds.

The default failure detector is a simple, deterministic one based on elapsed-time thresholds. Every node sends each reachable member a heartbeat on a fixed cadence and tracks, per peer, when it last heard from it: any frame counts as a sign of life, and a peer that stays silent past a threshold is flagged. (The opt-in φ-accrual detector reads the same liveness, but learns its timing from heartbeat frames only — see liveness is any frame.)

  • unreachableAfterMs — once a peer has been silent this long, the member is marked unreachable.
  • downAfterMs — if the silence persists this long, the detector’s verdict is down. With no downing strategy that verdict removes the member — failure-detector.auto-down is on by default — and the surviving nodes take over its shards and singletons; under the opt-out auto-down = off it stays parked at unreachable until an operator or a strategy removes it — see Split-brain and downing.

Plain elapsed-time limits, no statistical variance tracking — sufficient for LAN-scale clusters. See Failure detector for tuning, the φ-accrual alternative and the WAN caveats.

When the network partitions, two halves of the cluster may both remain operational but lose contact with each other. Without intervention, both sides keep running and accept conflicting writes — the classic split-brain problem.

actor-ts ships several downing strategies:

StrategyWhat it does
KeepMajorityThe side with more nodes wins; the smaller side downs itself.
KeepOldestThe side containing the lowest-addressed member wins — “oldest” is address order, not join order.
KeepRefereeA designated referee node’s view wins.
StaticQuorumThe side that meets a configured quorum size wins; a side below it downs itself.
LeaseMajorityThe majority side wins, but only while it holds a coordination lease.

See Downing strategies for the full set. Pick deliberately. With no strategy — the default — the failure detector removes an unreachable member itself once its verdict is down (actor-ts.cluster.failure-detector.auto-down ships on), and the last available node takes over the dead member’s shards and singletons. That is right for a crashed node and forks the cluster under a partition, because every side removes the others at the same moment and both halves run on as two clusters that do not merge again when the network heals (issue #930). So for production, configure lease-majority or another strategy. With two or more survivors, one survivor’s gossip can bring a member the other has just removed back as unreachable until it is removed again — see the failure detector. auto-down = off is the opt-out: the detector parks the member at unreachable, its shards stay homed on it and its remote watchers get no Terminated, until an operator downs it (Cluster.down, or POST /cluster/down with actor-ts.management.enable-down-endpoint = on), a strategy is configured, or it restarts on the same address. Neither setting keeps the leader in place: the leader is the first up member in a node’s view, so the members that can still reach each other elect another one, and a singleton without a role moves with it.

Once two nodes share a cluster, ref.tell(message) to a foreign-node ref just works:

const remote = await system.actorSelection(
'actor-ts://my-app@10.0.0.5:2552/user/api/sessions/user-42',
).resolveOne();
remote.tell({ kind: 'whatever' });
// → serialized, sent over the transport, delivered to the foreign actor's mailbox

The cluster transport frames the message as a tagged JSON tree — the same format the persistence stores write, so a Date, Map, Set, bigint or Uint8Array in the body arrives as itself (#450) — includes the routing path, and the destination’s transport delivers it. Reply-to refs serialize cleanly — the receiving node re-attaches a remote-routable handle so replies flow back over the same transport.

See Refs across nodes for the wire-format details.

The cluster module is the foundation; everything interesting about distributed actor systems comes from the extensions that build on it:

ExtensionWhat it adds
ShardingOne actor per “entity key,” distributed across nodes, with automatic rebalancing on membership changes.
SingletonOne actor cluster-wide. Re-spawned elsewhere if the host node leaves.
DistributedPubSubTopic-based fan-out across the cluster.
DistributedDataCRDT-based shared state with eventual consistency.
Cluster routerRoutes messages across cluster-up-members at a well-known path.
ReceptionistService registry — actors register, others look up by key.

You don’t enable these by default; you reach for them when needed. This page covers the foundation they all share — once you understand membership, gossip, and failure detection, the extensions follow.

A “cluster” of one node is valid. Calling Cluster.join with no seeds (or seeds that are unreachable) gives you a singleton cluster — the local node auto-promotes to leader, becomes up, and every extension that depends on cluster (sharding, singleton, pubsub) works as if it were in a larger cluster.

This means cluster code can be developed and tested with a single node; you don’t need a Docker Compose setup to get started. Add more nodes later by giving them seeds pointing at the first.

Three primary motivations:

  1. Scale-out: more actors than one node’s memory or CPU can handle. Sharding distributes them.
  2. Fault tolerance: if one node dies, the work moves to another. Singleton and sharding handle the failover.
  3. Geographic distribution: actors near their users or data, coordinating with the rest of the cluster over the WAN.

For a single-process app, you don’t need the cluster module. For a multi-process app where each process holds independent state, you don’t need it either — just use process boundaries. Reach for it when you genuinely need shared logical state across multiple machines.

The Cluster API reference covers the join/leave/subscribe surface.