MultiNodeSpec
이 콘텐츠는 아직 번역되지 않았습니다.
MultiNodeSpec runs N cluster nodes inside one test process.
Each is a real ActorSystem; they communicate via an
in-process transport (no TCP). Useful for testing cluster
behavior — sharding, singletons, gossip convergence,
failover — without Docker.
import { MultiNodeSpec, TestProbe } from 'actor-ts/testkit';
// Each role name is one node — and doubles as that node's system name.const spec = new MultiNodeSpec({ roles: ['a', 'b', 'c'] });await spec.start();await spec.awaitMembers('a', 3); // wait for the cluster to converge
// Reach any node's ActorSystem with systemFor(role):const node2 = spec.systemFor('b');
// Spawn actors on a specific node and assert with a probe:const probe = new TestProbe(spec.systemFor('a'));const remote = node2.spawnAnonymous(Worker);
remote.tell({ kind: 'do', replyTo: probe });await probe.expectMessage({ kind: 'done' });
await spec.stop();When to use it
Section titled “When to use it”Three primary cases:
- Testing cluster behavior — sharding rebalances, singleton failover, gossip convergence — verifying these without real network setup.
- Reproducing distributed bugs — easier to isolate when the entire cluster runs in one process.
- CI-friendly cluster tests — fast (sub-second), no Docker, no ports.
For tests that need true parallelism (real OS threads — concurrent journal writes, scheduler interleaving), use ParallelMultiNodeSpec. For real TCP/TLS or cross-host latency, use an external Docker-Compose cluster.
Configuration
Section titled “Configuration”type MultiNodeSpecOptionsType = { roles: ReadonlyArray<string>; // one node per role; must be unique seedRoles?: ReadonlyArray<string>; // bootstrap seeds — defaults to [roles[0]] addresses?: Record<string, { host: string; port: number }>; failureDetector?: ClusterOptionsType['failureDetector']; gossipIntervalMs?: number; // default 100 (vs. 1 s in production) awaitTimeoutMs?: number; // default 10_000 — how long await* helpers wait logLevel?: LogLevel; // defaults to a quiet NoopLogger downing?: (role: string) => DowningProvider | undefined; failureDetectorAutoDown?: boolean; // default unset = the shipped true stableAfterMs?: number; // default 100 (vs. 20 s in production)};| Field | Purpose |
|---|---|
roles | Node list — each entry is one node, and the string doubles as its system name. Must be unique. |
seedRoles | Which roles act as bootstrap seeds. Defaults to the first role. |
addresses | Per-role host/port overrides. Auto-allocated if omitted. |
failureDetector | Failure-detector overrides — tests usually tighten these so crashes are detected quickly. |
gossipIntervalMs | Gossip round interval. Default 100 ms. |
awaitTimeoutMs | Default timeout for the await* helpers. Default 10 s. |
logLevel | Log level — quiet by default. |
downing | Per-role split-brain resolver factory. |
failureDetectorAutoDown | Whether every node’s failure detector removes a peer it declares down. Unset keeps the shipped true: a crashed role is removed at down-after, so a spec that exercises crash failover with no downing strategy needs nothing more. false is the opt-out a spec sets to watch a crashed role parked at unreachable, where it stays a member. A role’s downing provider decides that role’s removals whatever this says. |
stableAfterMs | How long the view must hold still before downing is consulted. Default 100 ms — the shipped 20 s is twice awaitTimeoutMs, so a spec that inherited it would time out before its resolver was asked anything. |
Pass a plain object as shown, or build it fluently with
MultiNodeSpecOptions.create().withRoles(['a', 'b', 'c']).
Roles, seeds, and addresses
Section titled “Roles, seeds, and addresses”Each entry in roles is a distinct node — the string is that
node’s identity and its ActorSystem name, so it must be unique
(the constructor throws on duplicates or an empty list). By
default the first role is the sole bootstrap seed; override
that with seedRoles:
const spec = new MultiNodeSpec({ roles: ['seed', 'worker-1', 'worker-2'], seedRoles: ['seed'],});await spec.start();Ports are auto-allocated on 127.0.0.1; pin specific ones with
addresses when a test needs fixed endpoints:
const spec = new MultiNodeSpec({ roles: ['a', 'b'], addresses: { a: { host: '127.0.0.1', port: 2551 } },});What’s shared
Section titled “What’s shared”All nodes share:
- A shared in-process message bus — they can talk to each other.
- The same gossip protocol — membership converges.
- Independent actor systems — separate dispatchers, schedulers, supervisor trees.
This means:
- Real cluster semantics — members come up, gossip converges, failure detector observes, sharding rebalances.
- No serialization — messages between “nodes” pass by reference (in-process). Not a fit for testing serialization-dependent behavior.
Failover testing
Section titled “Failover testing”Two ways to take a node down: crash(role) yanks its transport
out abruptly (peers detect it via the failure detector), and
leave(role) performs a graceful cluster exit.
// Verify singleton failover:const spec = new MultiNodeSpec({ roles: ['a', 'b', 'c'], failureDetector: { heartbeatIntervalMs: 50, unreachableAfterMs: 200, downAfterMs: 400 }, // No downing strategy: the shipped auto-down removes the crashed node at downAfterMs.});await spec.start();await Promise.all([ spec.awaitMembers('a', 3), spec.awaitMembers('b', 3), spec.awaitMembers('c', 3),]);
// ... start the singleton on every node ...
// Identify the current host, then crash it:const host = spec.clusterFor('a').leader().toNullable();await spec.crash('a');
// Wait for the survivors to re-converge without 'a':await spec.awaitMembers('b', 2);
// ... assert the singleton has moved to a survivor ...
await spec.stop();crash(role) removes the node abruptly; the others’ detectors mark
it unreachable after unreachableAfterMs and reach down at
downAfterMs. What that verdict does is the spec’s choice: by
default the detector removes the node, as above —
failureDetectorAutoDown unset keeps the shipped true; a role’s
downing strategy decides instead; and with
failureDetectorAutoDown: false and no strategy, the node stays a
member at unreachable — no shard is reallocated and no remote
watcher gets Terminated. The await* helpers replace
fragile setTimeout sleeps — they poll until the condition holds (or
throw after awaitTimeoutMs).
Restarting a node
Section titled “Restarting a node”restart(role) brings a role back at the same address as a new process —
the shape of a pod replacement. The old system is crashed and terminated, a
fresh system and cluster join in its place with a new incarnation, and the
other members see the old run end before the new one joins. withNodeSetup
runs on every node after its join and again on a restarted one, so the
sharded types, singletons and subscriptions a test starts exist on the new
node too.
const spec = new MultiNodeSpec(MultiNodeSpecOptions.create() .withRoles(['a', 'b', 'c']) // Runs on every node once it has joined — and again on a restarted one. .withNodeSetup((role, system, cluster) => { cluster.sharding.start(shardingOptions); }));await spec.start();
const oldRun = spec.clusterFor('a').selfAddress.incarnation;await spec.restart('a'); // same address, new processawait spec.awaitMembers('b', 3);expect(spec.clusterFor('a').selfAddress.incarnation).not.toBe(oldRun);The restarted node’s seeds are the other running roles, so a restarted seed
rejoins the cluster it founded. Pass restart(role, { seeds: [] }) to have it
found a cluster of its own instead — the fork
issue #1176 is about.
References taken from systemFor / clusterFor before the restart name the
old node; ask again afterwards.
A network that works badly
Section titled “A network that works badly”partition and crash describe a network that is perfect or severed. Between
them lies every condition a cluster actually runs in: a fifth of the gossip
pushes lost, frames overtaking each other, a peer that is slow rather than
dead. degrade(roleA, roleB, profile) is that middle, and restore(roleA, roleB) undoes it.
A partition that outlasts downAfterMs behaves as it does in production:
with no downing strategy, each side removes the other and runs on alone, and
a later heal does not bring the two back together
(issue #930). A spec that
wants one side to win gives the roles a strategy through downing.
const spec = new MultiNodeSpec(MultiNodeSpecOptions.create() .withRoles(['a', 'b', 'c']) .withScheduler(new ManualScheduler()) .withFaultSeed(20_231));
await spec.start();
// The cluster forms on a clean network, and the network then gets worse.spec.degrade('a', 'b', { dropProbability: 0.2, // a fifth of the frames never arrive duplicateProbability: 0.1, // a tenth arrive twice reorderWindow: 4, // up to four frames may overtake each other latencyMs: 50, // measured on the spec's scheduler});
await spec.advanceUntil( () => spec.clusterFor('a').getMembers().filter((m) => m.status === 'up').length === 3, { budgetMs: 60_000, description: 'the view converges despite the loss' },);
spec.restore('a', 'b');Every field is optional and defaults to “no fault”, so a profile names only what
it changes. degrade is bidirectional like partition; for a one-way
degradation reach for spec.faultsFor(role).degradeTo(address, profile).
The faults are deterministic
Section titled “The faults are deterministic”Every decision comes from one seeded generator per node, drawn in send order —
and under a shared ManualScheduler that order is itself deterministic. So a
run that goes red reproduces exactly, and the spec’s timeout messages name the
seed that does it:
MultiNodeSpec: timeout after 10000 ms — awaitMembers(a, expected=3) — current: 2 Faults were injected on: a, b. Reproduce with withFaultSeed(20231).That is not a detail. An unreproducible chaos test is a flake generator, and a
fixed seed is what makes the difference — the trade being that the suite stops
finding new networks on its own. Vary the seed deliberately (a loop over
withFaultSeed) when the question is “does this hold for many networks” rather
than “does this hold for the network that broke”.
latencyMs needs a scheduler
Section titled “latencyMs needs a scheduler”A delay taken from the wall clock would reintroduce exactly the timing
dependence ManualScheduler exists to remove, so
it is refused rather than quietly taken: a non-zero latencyMs without a
scheduler throws at construction. The other three faults need no clock.
reorderWindow is a depth, not a shuffle
Section titled “reorderWindow is a depth, not a shuffle”A window of n holds up to n frames and releases them as one permuted batch,
so a frame is displaced by fewer than n positions. That bound is the
point: a shuffle could permute a frame to the end of the run and leave a test
waiting for something that is never coming. The last partial batch is released
by restore, by stop, and by the next batch filling up — continuous gossip
does the last of those on its own.
It works on any transport
Section titled “It works on any transport”FaultyTransport is a decorator, so InMemoryTransport, MultiNodeTransport
and MessageChannelTransport all gain the same controls without any of them
growing fault logic. Outside a spec:
const link = new FaultyTransport(new InMemoryTransport(address), { dropProbability: 0.2 });Per-node TestProbe
Section titled “Per-node TestProbe”MultiNodeSpec has no probe factory — construct a TestProbe
against whichever node’s system you want to observe:
const probeA = new TestProbe(spec.systemFor('a')); // probe on node 'a'const probeB = new TestProbe(spec.systemFor('b')); // probe on node 'b'Each probe is bound to one node’s actor system. Useful when testing routing — verify that a message ends up on the expected node.
Sharding tests
Section titled “Sharding tests”Start a sharding region on every node via its Cluster — reach
each one with clusterFor(role):
const spec = new MultiNodeSpec({ roles: ['a', 'b', 'c'] });await spec.start();await spec.awaitMembers('a', 3);
const startShardingOptions = StartShardingOptions.create<Command>() .withTypeName('entity') .withEntityActor(Entity) .withExtractEntityId((message) => message.id);const regions = spec.allRoles().map((role) => spec.clusterFor(role).sharding.start(startShardingOptions));
// Spawn entities; verify they spread across nodes:for (const id of ['e1', 'e2', 'e3', 'e4', 'e5']) { regions[0].tell({ id, kind: 'wake-up' });}
await spec.stop();The framework’s coordinator distributes shards across the nodes just like a real cluster.
What MultiNodeSpec doesn’t test
Section titled “What MultiNodeSpec doesn’t test”Where to next
Section titled “Where to next”- Testing overview — the bigger picture.
- TestKit — single-system testing.
- ParallelMultiNodeSpec — worker-thread variant.
- Cluster overview — what MultiNodeSpec simulates.
