コンテンツにスキップ
日本語

Joining and seeds

このコンテンツはまだ日本語訳がありません。

A node enters a cluster by contacting a seed node. The seed gossips back its current membership view; the joiner is added as joining, propagates through gossip, and once the leader sees it (plus convergence), transitions to up. The leader sends its view straight to every member it has just promoted, so a joiner learns it is up one hop after the promotion rather than whenever a random gossip round happens to pick it — which is what keeps Cluster.bootstrap’s readiness wait independent of the cluster’s size (#1747).

clusterseed nodesjoining nodeclusterseed nodesjoining nodejoining → weakly-up? → upJoin announcementgossip JoinGossip — current viewGossip from the leader, sent on promotion

This page covers the mechanics of that handshake, plus the seed-discovery layer on top.

import { ActorSystem } from 'actor-ts';
import { Cluster, ClusterOptions } from 'actor-ts/cluster';
const system = ActorSystem.create('my-app');
const clusterOptions = ClusterOptions.create()
.withHost('10.0.0.5')
.withPort(2552)
.withSeeds(['10.0.0.5:2552', '10.0.0.6:2552', '10.0.0.7:2552']);
const cluster = await Cluster.join(
system,
clusterOptions,
);

Three seeds. The joiner sends its join to all three, and again every seedRetryIntervalMs until it is up. Once any seed accepts, the cluster’s gossip propagates the new member; convergence to up happens within a few seconds on a healthy network.

The seed list is just a bootstrap hint — once joined, the node learns about every other peer via gossip. Seeds don’t have to be special after the join.

type ClusterOptionsType = {
host: string; // interface to bind (wildcard ok)
advertisedHost?: string; // address peers dial (never a wildcard)
port: number; // port to bind
advertisedPort?: number; // port peers dial (defaults to port)
seeds?: string[]; // peer addresses for bootstrap
roles?: string[]; // role tags
failureDetector?: Partial<...>;
transport?: Transport;
gossipIntervalMs?: number;
seedRetryIntervalMs?: number; // re-contact the seeds until up
shutdownAfterUnsuccessfulJoinMs?: number; // give the join up (0 = no deadline)
// ...
};

The seed-related knobs:

SettingDefaultWhat
seeds[]List of "host:port" strings. Empty = “I’m the first.”
seedRetryIntervalMs3000Retry the seed list this often until this node is up — whether or not a seed has answered, and through weakly-up too.
selfElection'immediate'When this node may form a cluster alone: 'immediate' (only on an empty seed list), 'never', or a millisecond grace. Set by cluster bootstrap.
shutdownAfterUnsuccessfulJoinMs0Give the join up after this long if nothing can promote this node by then: one warning, leave, then the coordinated-shutdown pipeline. 0 sets no deadline; a value shorter than seedRetryIntervalMs is refused — see When no seed ever answers.

seeds and roles also have a config form, so a deployment can move them out of the code:

actor-ts.cluster {
seed-nodes = ["billing@10.0.0.5:2552"]
roles = ["backend"]
}

Precedence is the framework’s usual one, per field: an explicit .withSeeds(…) wins, an unset one falls through to the file. What makes one file work for the whole deployment is that a node removes its own address before dialling — so the recipe below (one designated first node, the rest pointing at it) is a single list that is correct on every node: the named node reads its own address, filters it out, and self-elects.

Both keys are read as lists and only as lists, so the tempting seed-nodes = ${?SEED_NODES} is refused at start-up rather than split on commas. One optional substitution per entry is the form that works — unset ones drop out of the array:

actor-ts.cluster.seed-nodes = [ ${?SEED_1}, ${?SEED_2}, ${?SEED_3} ]
actor-ts.cluster.roles = [ ${?NODE_ROLE} ]

For a single comma-separated variable, use seedsFromEnv(…) in code instead. A malformed entry — a host with no port, say — is refused by Cluster.join before the transport binds, with the offending string in the message.

const clusterOptions = ClusterOptions.create()
.withHost('0.0.0.0')
.withPort(2552)
.withSeeds([]);
const cluster = await Cluster.join(
system,
clusterOptions,
);

0.0.0.0 there is the bind address. What this node gossips is resolved separately and is never a wildcard — with nothing else configured it advertises 127.0.0.1, which is exactly right for a single-node development run and is why the pair needs no attention until there is a second machine. See Cluster overview.

An empty seeds list (or one that’s all-unreachable) means this node bootstraps the cluster by itself. It auto-promotes to leader; future joiners contact it.

This makes single-node development trivial — no seed list to maintain. Add a second node later by giving it the first’s address as a seed.

For production, designate one node with an empty seed list and give the rest that node’s address — or, better, use cluster bootstrap, which removes the need for a designated first node entirely.

Forming a cluster of one is exactly what you want in development and often not what you want in production: the founder is up on its own, so singletons start on it and shards are placed on it, before any of its replicas exist. minimumMembersBeforeUp holds every up transition — the leader’s promotions and this node’s own self-election — until that many members are present:

const clusterOptions = ClusterOptions.create()
.withHost('0.0.0.0')
.withPort(2552)
.withSeeds([])
.withMinimumMembersBeforeUp(3);

The founder now stays joining until three members are in its member map, then self-elects and promotes the other two in the same pass. Every node should carry the same value — it is a property of the cluster, not of the node.

Counted are members in joining, weakly-up and up. A leaving member does not hold the threshold open for its replacement, so a rolling restart closes the gate for exactly as long as the replacement takes to appear. The threshold is never evaluated in reverse: there is no up → joining transition, so a cluster that later shrinks keeps the members it promoted and holds only new ones.

Per-role thresholds compose with it as an AND, which is how you say “not until there is a backend to place work on”:

const clusterOptions = ClusterOptions.create()
.withHost('0.0.0.0')
.withPort(2552)
.withSeeds([])
.withMinimumMembersBeforeUp(3)
.withMinimumMembersBeforeUpPerRole({ backend: 2 });

This is not a quorum and not a security control: any node may announce itself up, so it bounds accidents — a partitioned half forming its own cluster of one — rather than adversaries. Split-brain arbitration is downing, and a threshold above 1 widens the window in which a cluster is smaller than a static-quorum strategy expects, so check the two together.

It is also not bootstrap.minimum-members, which waits without changing anything: see awaitReady. Setting this one largely removes the reason to set that one.

The symmetric seed list does not cold-start

Section titled “The symmetric seed list does not cold-start”

It is tempting to hand every node the same list containing every node. That configuration never forms a cluster:

n3n2n1n3n2n1All three fresh, all given seed list [n1, n2, n3]filters itself out → seeds = [n2, n3] → not emptysame on both — everyone waits to be let inno member is `up`, so there is no leader,so nobody is ever promotedjoin announcementjoin announcement

Cluster removes this node’s own address from its seed list, so a symmetric list leaves no node with the empty list that selfElection: 'immediate' requires. Every node stays joining forever; there is no leader, and the leader is what promotes joining → up.

Two ways out:

  • One designated first node with seeds: [], the rest pointing at it. Simple, but that node is special at start-up, which containers and autoscalers make awkward.
  • Cluster bootstrap — every node gets the same configuration and the framework elects exactly one to break the tie. This is the option to reach for whenever nodes start simultaneously.

The trap is the same when the list comes from a shared application.conf. Self-exclusion is what makes one file safe, and it only helps if the file names one node: seed-nodes = ["app@n1:2552"] shipped everywhere is the first way out written once, while seed-nodes listing every node is this deadlock written once. Cluster bootstrap is the answer whenever the members are not known in advance.

The seedRetryIntervalMs retry loop still matters, but it solves a different problem: a seed that is reachable eventually. It cannot manufacture a first member.

It does, however, carry the diagnosis. A node still joining or weakly-up after a few contact rounds, with no up member in sight and no self-election due, logs one WARN naming its status, what it can see and what to change — the unanswered seed addresses when nothing has replied, or the seed list and selfElection pairing above when the peers are all present and all waiting. It is reported once, not per round. A join deadline, when one is set, acts on the same verdict.

weakly-up is in that sentence because it is no closer to up: it is the node’s own weaklyUpAfterMs timer moving it on, not a leader admitting it. So the loop does not stop there — it keeps dialling until the node is up, which is also what lets a seed that starts after the promotion still find the node.

That line is worth knowing about because the symptom is nowhere near the cause: a cluster stuck this way has no up member, a cluster singleton picks its host from the up members, and so the first thing most applications notice is an AskTimeoutError from a singleton proxy.

The retry loop has no end of its own. A node whose seeds never answer — wrong addresses, a firewall, seeds that never started — logs the stall warning above after three rounds and then goes on dialling for as long as the process lives. On an orchestrator that is a pod that never joins and never exits, so nothing ever restarts it.

shutdownAfterUnsuccessfulJoinMs gives the join a deadline:

const clusterOptions = ClusterOptions.create()
.withHost('10.0.0.8')
.withPort(2552)
.withSeeds(['10.0.0.5:2552', '10.0.0.6:2552'])
.withShutdownAfterUnsuccessfulJoinMs(60_000);
const cluster = await Cluster.join(system, clusterOptions);

or, for every node of a deployment at once:

actor-ts.cluster.shutdown-after-unsuccessful-join = 60s

The deadline is when the node decides, not what decides. When it passes, the node asks the question the stall warning answers — can anything still promote it? Only if nothing can does it give up: one WARN naming the key, the deadline and the same remedy, then leave(), then the coordinated-shutdown pipeline with ClusterJoinUnsuccessfulReason. If something can, it asks again every seedRetryIntervalMs until the node is up or the answer changes.

At the deadlineGives up?
No seed has answered, selfElection: 'immediate'yes
selfElection: 'never', no member upyes
Peers known, none up, 'immediate' everywhere — the symmetric list aboveyes
weakly-up by its own timer, nothing else changedyes — weakly-up is not joined
minimum-members-before-up unmet, and nobody answeringyes
A member is upno — that leader promotes the node
A deferred selfElection still pendingno — and when it fires, the node forms a cluster of one
minimum-members-before-up unmet, live peers presentno — the cluster is waiting for members, not stuck
minimum-members-before-up unmet, the peers that answered have since diedyes, once they are gone — removed at down-after by the default, or parked there under failure-detector.auto-down = off, which counts as gone; a peer only unreachable, short of down-after, still counts as live

A few details settle the edges:

  • 0, the default, sets no deadline — what every release before the key did. A value shorter than seedRetryIntervalMs is refused at Cluster.join: the node would give up before it had dialled twice.
  • The clock starts at the join, not at discovery. Under cluster bootstrap the seed lookup and any stable observation run first, and the key composes with awaitReady there — see Waiting for readiness.
  • leave() comes first, so the seed loop stops even with coordinated-shutdown.auto-register-tasks = off. What the run ends in is the pipeline’s to decide: the actor system terminates by default, and the process exits only with coordinated-shutdown.exit-process = on, and then never with status 0 — see Task semantics.
  • A readiness wait answers to it too: an awaitReady() still pending when the join gives up rejects at once with a ClusterReadyTimeoutError that names the key, rather than waiting out its own deadline.
  • A founder’s pending election is invisible to the others. A node whose seeds answer but are themselves still joining — a founder waiting out a deferred selfElection of its own — sees nothing that can promote it: that timer runs on the founder, and nothing gossips it. So it gives up at its deadline on a cluster about to form, and the deadline has to exceed the founder’s grace. Behind stable observation cluster bootstrap refuses a shorter one; a plain Cluster.join does not.

A hard-coded seed list works for tests and small clusters. For production where nodes have dynamic IPs (containers, K8s pods), use a seed provider:

ProviderWhen
ConfigStatic list (the case above).
DNSResolves _actor-ts._tcp.example.com SRV records.
Kubernetes APILists pods matching a label selector.
AggregateFalls through multiple providers (e.g. K8s, then DNS).
import { KubernetesApiSeedProvider, KubernetesApiSeedProviderOptions } from 'actor-ts/discovery';
const kubernetesApiSeedProviderOptions = KubernetesApiSeedProviderOptions.create()
.withNamespace('default')
.withServiceName('actor-ts')
.withPort(2552);
const seedProvider = new KubernetesApiSeedProvider(
kubernetesApiSeedProviderOptions,
);
const seeds = await seedProvider.discover();
const clusterOptions = ClusterOptions.create()
.withHost(process.env.POD_IP!)
.withPort(2552)
.withSeeds(seeds);
const cluster = await Cluster.join(
system,
clusterOptions,
);

The provider returns a snapshot of seed addresses; the framework uses them to bootstrap the join. See Discovery overview for the seed provider model.

The one-call gate first — awaitReady resolves once this node is up and, with minimumMembers, once the cluster has reached its expected size; isReady is the synchronous probe:

await cluster.awaitReady({ minimumMembers: 3, timeoutMs: 30_000 });
cluster.isReady(); // probe the same predicate without waiting
cluster.selfMember(); // this node's own record — status included

With timeoutMs set it rejects with ClusterReadyTimeoutError instead of leaving you to guess; without it, it waits indefinitely. See Cluster bootstrap → Waiting for readiness for how Cluster.bootstrap applies the same wait by default.

For logic that reacts to individual transitions, subscribe to the events:

import { SelfUp, MemberUp } from 'actor-ts/cluster';
cluster.subscribe((evt) => {
if (evt instanceof SelfUp) {
console.log(`this node is now Up`);
} else if (evt instanceof MemberUp) {
console.log(`peer ${evt.member.address} reached Up`);
}
});

Two key events:

  • SelfUp fires once when this node transitions to up.
  • MemberUp fires every time any member reaches up.

Counting MemberUps by hand to answer “are at least 3 nodes up?” is exactly what awaitReady({ minimumMembers: 3 }) replaces — the subscription is for reacting to changes, not for gating startup.

A node that joins with a different value for something the cluster assumes is uniform does not fail. It joins, it runs, and the disagreement surfaces later as behaviour nobody can account for. actor-ts.cluster.configuration-compatibility-check makes it say so at the moment the two nodes meet instead.

Each node publishes the effective value of every setting in checked-paths on the ordinary gossip lane and compares what its peers publish back. Effective, not configured: precedence is **explicit options

HOCON > built-in defaults**, so two nodes whose config files are identical and whose builders are not still compare unequal — which is the case that matters, since the builder is the documented way to set most of these.

const clusterOptions = ClusterOptions.create()
.withHost('n1')
.withPort(2552)
.withConfigurationCompatibilityCheckedPaths([
'actor-ts.remote.max-frame-bytes',
'actor-ts.cluster.failure-detector.heartbeat-interval',
]);

Only a setting something publishes can be compared, and the framework publishes three:

SettingWhy a divergence mattersIn checked-paths by default
actor-ts.remote.max-frame-bytesA frame over the peer’s lower cap kills the whole association, and the sender is never told why. There is no protocol negotiation to catch it.yes
actor-ts.cluster.failure-detector.heartbeat-intervalA peer that beats more slowly than this node’s down-after tolerates is judged down for being configured differently, not for being dead — and removed by whatever acts on that verdict: a downing strategy, or the detector itself under failure-detector.auto-down = on. With neither, it is parked at unreachable.no
actor-ts.cluster.tombstone.time-to-liveOne side prunes a tombstone the other still enforces, so a removed address is re-admitted on one side only.no

The last two are off by default because staging a new value one node at a time is an ordinary operation, and both divergences heal. The first has no detection anywhere else, which is the bar for being seeded.

Publish one of your own from anywhere that has the resolved value:

cluster.publishConfigurationFact('acme.orders.partition-count', String(partitions));

It travels only if checked-paths lists the name — that list is an allow-list of what leaves the node, which is why there is no second list of paths to redact. A name outside the character set a HOCON path uses (lower-case, dots and hyphens) is refused when the options are validated, rather than published and silently dropped by every peer.

Any point in a node’s life is a valid moment to call it, and that is the point of the call existing at all: the three settings above are resolved before the first gossip frame moves, but a partition count that comes back from a migration or a broker is not. A fact published after every peer’s record has settled still reaches every peer’s comparison — each node’s own gossip carries its current facts, and a peer replaces what it holds for a member when that member itself restates them. A copy relayed by a third node fills a gap but never overwrites, so a stale copy doing the rounds cannot flap the comparison. Publishing the same value again changes nothing and costs nothing.

When two nodes disagree, each logs one line naming the setting and both values, and publishes a MemberConfigurationMismatch carrying the same. Once per setting per peer, not once per gossip round. Absence on either side is silence: a peer running an older version, or one whose checked-paths does not list that setting, reports nothing rather than reporting everything during a rolling upgrade.

enforce = on adds exactly one thing — the diverging peer stops being one of this node’s placement candidates, so a cluster router will not route to it. It never downs or quarantines anyone, and nothing that elects a role reads that set: a misconfigured majority would otherwise evict the correctly-configured minority, and an asymmetric candidate view would elect two singleton hosts rather than one. Off by default, because an enforcing default turns any rolling deploy that changes a checked value into a self-inflicted outage.

actor-ts.sharding.number-of-shards is deliberately not one of the published settings. A region whose shard count disagrees with the coordinator’s is already refused outright, with an error naming the key and a ShardRegionRegistrationRefused event — see Sharding introspection.