Pular para o conteúdo
Português (BR)

MultiNodeSpec

Este conteúdo não está disponível em sua língua ainda.

MultiNodeSpec runs N cluster nodes inside one test process. Each is a real ActorSystem; they communicate via an in-process transport (no TCP). Useful for testing cluster behavior — sharding, singletons, gossip convergence, failover — without Docker.

import { MultiNodeSpec, TestProbe } from 'actor-ts/testkit';
// Each role name is one node — and doubles as that node's system name.
const spec = new MultiNodeSpec({ roles: ['a', 'b', 'c'] });
await spec.start();
await spec.awaitMembers('a', 3); // wait for the cluster to converge
// Reach any node's ActorSystem with systemFor(role):
const node2 = spec.systemFor('b');
// Spawn actors on a specific node and assert with a probe:
const probe = new TestProbe(spec.systemFor('a'));
const remote = node2.spawnAnonymous(Worker);
remote.tell({ kind: 'do', replyTo: probe });
await probe.expectMessage({ kind: 'done' });
await spec.stop();

Three primary cases:

  1. Testing cluster behavior — sharding rebalances, singleton failover, gossip convergence — verifying these without real network setup.
  2. Reproducing distributed bugs — easier to isolate when the entire cluster runs in one process.
  3. CI-friendly cluster tests — fast (sub-second), no Docker, no ports.

For tests that need true parallelism (real OS threads — concurrent journal writes, scheduler interleaving), use ParallelMultiNodeSpec. For real TCP/TLS or cross-host latency, use an external Docker-Compose cluster.

type MultiNodeSpecOptionsType = {
roles: ReadonlyArray<string>; // one node per role; must be unique
seedRoles?: ReadonlyArray<string>; // bootstrap seeds — defaults to [roles[0]]
addresses?: Record<string, { host: string; port: number }>;
failureDetector?: ClusterOptionsType['failureDetector'];
gossipIntervalMs?: number; // default 100 (vs. 1 s in production)
awaitTimeoutMs?: number; // default 10_000 — how long await* helpers wait
logLevel?: LogLevel; // defaults to a quiet NoopLogger
downing?: (role: string) => DowningProvider | undefined;
failureDetectorAutoDown?: boolean; // default unset = the shipped true
stableAfterMs?: number; // default 100 (vs. 20 s in production)
};
FieldPurpose
rolesNode list — each entry is one node, and the string doubles as its system name. Must be unique.
seedRolesWhich roles act as bootstrap seeds. Defaults to the first role.
addressesPer-role host/port overrides. Auto-allocated if omitted.
failureDetectorFailure-detector overrides — tests usually tighten these so crashes are detected quickly.
gossipIntervalMsGossip round interval. Default 100 ms.
awaitTimeoutMsDefault timeout for the await* helpers. Default 10 s.
logLevelLog level — quiet by default.
downingPer-role split-brain resolver factory.
failureDetectorAutoDownWhether every node’s failure detector removes a peer it declares down. Unset keeps the shipped true: a crashed role is removed at down-after, so a spec that exercises crash failover with no downing strategy needs nothing more. false is the opt-out a spec sets to watch a crashed role parked at unreachable, where it stays a member. A role’s downing provider decides that role’s removals whatever this says.
stableAfterMsHow long the view must hold still before downing is consulted. Default 100 ms — the shipped 20 s is twice awaitTimeoutMs, so a spec that inherited it would time out before its resolver was asked anything.

Pass a plain object as shown, or build it fluently with MultiNodeSpecOptions.create().withRoles(['a', 'b', 'c']).

Each entry in roles is a distinct node — the string is that node’s identity and its ActorSystem name, so it must be unique (the constructor throws on duplicates or an empty list). By default the first role is the sole bootstrap seed; override that with seedRoles:

const spec = new MultiNodeSpec({
roles: ['seed', 'worker-1', 'worker-2'],
seedRoles: ['seed'],
});
await spec.start();

Ports are auto-allocated on 127.0.0.1; pin specific ones with addresses when a test needs fixed endpoints:

const spec = new MultiNodeSpec({
roles: ['a', 'b'],
addresses: { a: { host: '127.0.0.1', port: 2551 } },
});

All nodes share:

  • A shared in-process message bus — they can talk to each other.
  • The same gossip protocol — membership converges.
  • Independent actor systems — separate dispatchers, schedulers, supervisor trees.

This means:

  • Real cluster semantics — members come up, gossip converges, failure detector observes, sharding rebalances.
  • No serialization — messages between “nodes” pass by reference (in-process). Not a fit for testing serialization-dependent behavior.

Two ways to take a node down: crash(role) yanks its transport out abruptly (peers detect it via the failure detector), and leave(role) performs a graceful cluster exit.

// Verify singleton failover:
const spec = new MultiNodeSpec({
roles: ['a', 'b', 'c'],
failureDetector: { heartbeatIntervalMs: 50, unreachableAfterMs: 200, downAfterMs: 400 },
// No downing strategy: the shipped auto-down removes the crashed node at downAfterMs.
});
await spec.start();
await Promise.all([
spec.awaitMembers('a', 3),
spec.awaitMembers('b', 3),
spec.awaitMembers('c', 3),
]);
// ... start the singleton on every node ...
// Identify the current host, then crash it:
const host = spec.clusterFor('a').leader().toNullable();
await spec.crash('a');
// Wait for the survivors to re-converge without 'a':
await spec.awaitMembers('b', 2);
// ... assert the singleton has moved to a survivor ...
await spec.stop();

crash(role) removes the node abruptly; the others’ detectors mark it unreachable after unreachableAfterMs and reach down at downAfterMs. What that verdict does is the spec’s choice: by default the detector removes the node, as above — failureDetectorAutoDown unset keeps the shipped true; a role’s downing strategy decides instead; and with failureDetectorAutoDown: false and no strategy, the node stays a member at unreachable — no shard is reallocated and no remote watcher gets Terminated. The await* helpers replace fragile setTimeout sleeps — they poll until the condition holds (or throw after awaitTimeoutMs).

restart(role) brings a role back at the same address as a new process — the shape of a pod replacement. The old system is crashed and terminated, a fresh system and cluster join in its place with a new incarnation, and the other members see the old run end before the new one joins. withNodeSetup runs on every node after its join and again on a restarted one, so the sharded types, singletons and subscriptions a test starts exist on the new node too.

const spec = new MultiNodeSpec(MultiNodeSpecOptions.create()
.withRoles(['a', 'b', 'c'])
// Runs on every node once it has joined — and again on a restarted one.
.withNodeSetup((role, system, cluster) => {
cluster.sharding.start(shardingOptions);
}));
await spec.start();
const oldRun = spec.clusterFor('a').selfAddress.incarnation;
await spec.restart('a'); // same address, new process
await spec.awaitMembers('b', 3);
expect(spec.clusterFor('a').selfAddress.incarnation).not.toBe(oldRun);

The restarted node’s seeds are the other running roles, so a restarted seed rejoins the cluster it founded. Pass restart(role, { seeds: [] }) to have it found a cluster of its own instead — the fork issue #1176 is about. References taken from systemFor / clusterFor before the restart name the old node; ask again afterwards.

partition and crash describe a network that is perfect or severed. Between them lies every condition a cluster actually runs in: a fifth of the gossip pushes lost, frames overtaking each other, a peer that is slow rather than dead. degrade(roleA, roleB, profile) is that middle, and restore(roleA, roleB) undoes it.

A partition that outlasts downAfterMs behaves as it does in production: with no downing strategy, each side removes the other and runs on alone, and a later heal does not bring the two back together (issue #930). A spec that wants one side to win gives the roles a strategy through downing.

const spec = new MultiNodeSpec(MultiNodeSpecOptions.create()
.withRoles(['a', 'b', 'c'])
.withScheduler(new ManualScheduler())
.withFaultSeed(20_231));
await spec.start();
// The cluster forms on a clean network, and the network then gets worse.
spec.degrade('a', 'b', {
dropProbability: 0.2, // a fifth of the frames never arrive
duplicateProbability: 0.1, // a tenth arrive twice
reorderWindow: 4, // up to four frames may overtake each other
latencyMs: 50, // measured on the spec's scheduler
});
await spec.advanceUntil(
() => spec.clusterFor('a').getMembers().filter((m) => m.status === 'up').length === 3,
{ budgetMs: 60_000, description: 'the view converges despite the loss' },
);
spec.restore('a', 'b');

Every field is optional and defaults to “no fault”, so a profile names only what it changes. degrade is bidirectional like partition; for a one-way degradation reach for spec.faultsFor(role).degradeTo(address, profile).

Every decision comes from one seeded generator per node, drawn in send order — and under a shared ManualScheduler that order is itself deterministic. So a run that goes red reproduces exactly, and the spec’s timeout messages name the seed that does it:

MultiNodeSpec: timeout after 10000 ms — awaitMembers(a, expected=3) — current: 2
Faults were injected on: a, b. Reproduce with withFaultSeed(20231).

That is not a detail. An unreproducible chaos test is a flake generator, and a fixed seed is what makes the difference — the trade being that the suite stops finding new networks on its own. Vary the seed deliberately (a loop over withFaultSeed) when the question is “does this hold for many networks” rather than “does this hold for the network that broke”.

A delay taken from the wall clock would reintroduce exactly the timing dependence ManualScheduler exists to remove, so it is refused rather than quietly taken: a non-zero latencyMs without a scheduler throws at construction. The other three faults need no clock.

A window of n holds up to n frames and releases them as one permuted batch, so a frame is displaced by fewer than n positions. That bound is the point: a shuffle could permute a frame to the end of the run and leave a test waiting for something that is never coming. The last partial batch is released by restore, by stop, and by the next batch filling up — continuous gossip does the last of those on its own.

FaultyTransport is a decorator, so InMemoryTransport, MultiNodeTransport and MessageChannelTransport all gain the same controls without any of them growing fault logic. Outside a spec:

const link = new FaultyTransport(new InMemoryTransport(address), { dropProbability: 0.2 });

MultiNodeSpec has no probe factory — construct a TestProbe against whichever node’s system you want to observe:

const probeA = new TestProbe(spec.systemFor('a')); // probe on node 'a'
const probeB = new TestProbe(spec.systemFor('b')); // probe on node 'b'

Each probe is bound to one node’s actor system. Useful when testing routing — verify that a message ends up on the expected node.

Start a sharding region on every node via its Cluster — reach each one with clusterFor(role):

const spec = new MultiNodeSpec({ roles: ['a', 'b', 'c'] });
await spec.start();
await spec.awaitMembers('a', 3);
const startShardingOptions = StartShardingOptions.create<Command>()
.withTypeName('entity')
.withEntityActor(Entity)
.withExtractEntityId((message) => message.id);
const regions = spec.allRoles().map((role) =>
spec.clusterFor(role).sharding.start(startShardingOptions)
);
// Spawn entities; verify they spread across nodes:
for (const id of ['e1', 'e2', 'e3', 'e4', 'e5']) {
regions[0].tell({ id, kind: 'wake-up' });
}
await spec.stop();

The framework’s coordinator distributes shards across the nodes just like a real cluster.