Joining and seeds
このコンテンツはまだ日本語訳がありません。
A node enters a cluster by contacting a seed node. The seed
gossips back its current membership view; the joiner is added as
joining, propagates through gossip, and once the leader sees it
(plus convergence), transitions to up. The leader sends its view
straight to every member it has just promoted, so a joiner learns it is
up one hop after the promotion rather than whenever a random gossip
round happens to pick it — which is what keeps Cluster.bootstrap’s
readiness wait independent of the cluster’s size (#1747).
This page covers the mechanics of that handshake, plus the seed-discovery layer on top.
The simplest case — explicit seeds
Section titled “The simplest case — explicit seeds”import { ActorSystem } from 'actor-ts';import { Cluster, ClusterOptions } from 'actor-ts/cluster';
const system = ActorSystem.create('my-app');
const clusterOptions = ClusterOptions.create() .withHost('10.0.0.5') .withPort(2552) .withSeeds(['10.0.0.5:2552', '10.0.0.6:2552', '10.0.0.7:2552']);const cluster = await Cluster.join( system, clusterOptions,);Three seeds. The joiner sends its join to all three, and again
every seedRetryIntervalMs until it is up. Once any seed
accepts, the cluster’s gossip propagates the new member;
convergence to up happens within a few seconds on a healthy
network.
The seed list is just a bootstrap hint — once joined, the node learns about every other peer via gossip. Seeds don’t have to be special after the join.
Configuration
Section titled “Configuration”type ClusterOptionsType = { host: string; // interface to bind (wildcard ok) advertisedHost?: string; // address peers dial (never a wildcard) port: number; // port to bind advertisedPort?: number; // port peers dial (defaults to port) seeds?: string[]; // peer addresses for bootstrap roles?: string[]; // role tags failureDetector?: Partial<...>; transport?: Transport; gossipIntervalMs?: number; seedRetryIntervalMs?: number; // re-contact the seeds until up shutdownAfterUnsuccessfulJoinMs?: number; // give the join up (0 = no deadline) // ...};The seed-related knobs:
| Setting | Default | What |
|---|---|---|
seeds | [] | List of "host:port" strings. Empty = “I’m the first.” |
seedRetryIntervalMs | 3000 | Retry the seed list this often until this node is up — whether or not a seed has answered, and through weakly-up too. |
selfElection | 'immediate' | When this node may form a cluster alone: 'immediate' (only on an empty seed list), 'never', or a millisecond grace. Set by cluster bootstrap. |
shutdownAfterUnsuccessfulJoinMs | 0 | Give the join up after this long if nothing can promote this node by then: one warning, leave, then the coordinated-shutdown pipeline. 0 sets no deadline; a value shorter than seedRetryIntervalMs is refused — see When no seed ever answers. |
The same two settings in HOCON
Section titled “The same two settings in HOCON”seeds and roles also have a config form, so a deployment can move them
out of the code:
actor-ts.cluster { seed-nodes = ["billing@10.0.0.5:2552"] roles = ["backend"]}Precedence is the framework’s usual one, per field: an explicit
.withSeeds(…) wins, an unset one falls through to the file. What makes
one file work for the whole deployment is that a node removes its own
address before dialling — so the recipe below (one designated first node,
the rest pointing at it) is a single list that is correct on every node:
the named node reads its own address, filters it out, and self-elects.
Both keys are read as lists and only as lists, so the tempting
seed-nodes = ${?SEED_NODES} is refused at start-up rather than split on
commas. One optional substitution per entry is the form that works — unset
ones drop out of the array:
actor-ts.cluster.seed-nodes = [ ${?SEED_1}, ${?SEED_2}, ${?SEED_3} ]actor-ts.cluster.roles = [ ${?NODE_ROLE} ]For a single comma-separated variable, use
seedsFromEnv(…) in code instead. A
malformed entry — a host with no port, say — is refused by Cluster.join
before the transport binds, with the offending string in the message.
The first node
Section titled “The first node”const clusterOptions = ClusterOptions.create() .withHost('0.0.0.0') .withPort(2552) .withSeeds([]);const cluster = await Cluster.join( system, clusterOptions,);0.0.0.0 there is the bind address. What this node gossips is
resolved separately and is never a wildcard — with nothing else
configured it advertises 127.0.0.1, which is exactly right for a
single-node development run and is why the pair needs no attention
until there is a second machine. See
Cluster overview.
An empty seeds list (or one that’s all-unreachable) means this
node bootstraps the cluster by itself. It auto-promotes to
leader; future joiners contact it.
This makes single-node development trivial — no seed list to maintain. Add a second node later by giving it the first’s address as a seed.
For production, designate one node with an empty seed list and give the rest that node’s address — or, better, use cluster bootstrap, which removes the need for a designated first node entirely.
Holding the first node back
Section titled “Holding the first node back”Forming a cluster of one is exactly what you want in development and
often not what you want in production: the founder is up on its own,
so singletons start on it and shards are placed on it, before any of
its replicas exist. minimumMembersBeforeUp holds every up
transition — the leader’s promotions and this node’s own
self-election — until that many members are present:
const clusterOptions = ClusterOptions.create() .withHost('0.0.0.0') .withPort(2552) .withSeeds([]) .withMinimumMembersBeforeUp(3);The founder now stays joining until three members are in its member
map, then self-elects and promotes the other two in the same pass.
Every node should carry the same value — it is a property of the
cluster, not of the node.
Counted are members in joining, weakly-up and up. A leaving
member does not hold the threshold open for its replacement, so a
rolling restart closes the gate for exactly as long as the replacement
takes to appear. The threshold is never evaluated in reverse: there is
no up → joining transition, so a cluster that later shrinks keeps
the members it promoted and holds only new ones.
Per-role thresholds compose with it as an AND, which is how you say “not until there is a backend to place work on”:
const clusterOptions = ClusterOptions.create() .withHost('0.0.0.0') .withPort(2552) .withSeeds([]) .withMinimumMembersBeforeUp(3) .withMinimumMembersBeforeUpPerRole({ backend: 2 });This is not a quorum and not a security control: any node may
announce itself up, so it bounds accidents — a partitioned half
forming its own cluster of one — rather than adversaries. Split-brain
arbitration is downing, and a threshold
above 1 widens the window in which a cluster is smaller than a
static-quorum strategy expects, so check the two together.
It is also not bootstrap.minimum-members, which waits without
changing anything: see
awaitReady. Setting this one largely
removes the reason to set that one.
The symmetric seed list does not cold-start
Section titled “The symmetric seed list does not cold-start”It is tempting to hand every node the same list containing every node. That configuration never forms a cluster:
Cluster removes this node’s own address from its seed list, so a
symmetric list leaves no node with the empty list that
selfElection: 'immediate' requires. Every node stays joining
forever; there is no leader, and the leader is what promotes
joining → up.
Two ways out:
- One designated first node with
seeds: [], the rest pointing at it. Simple, but that node is special at start-up, which containers and autoscalers make awkward. - Cluster bootstrap — every node gets the same configuration and the framework elects exactly one to break the tie. This is the option to reach for whenever nodes start simultaneously.
The trap is the same when the list comes from a shared application.conf.
Self-exclusion is what makes one file safe, and it only helps if the file
names one node: seed-nodes = ["app@n1:2552"] shipped everywhere is the
first way out written once, while seed-nodes listing every node is this
deadlock written once. Cluster bootstrap is the answer whenever the members
are not known in advance.
The seedRetryIntervalMs retry loop still matters, but it solves a
different problem: a seed that is reachable eventually. It cannot
manufacture a first member.
It does, however, carry the diagnosis. A node still joining or
weakly-up after a few contact rounds, with no up member in sight and
no self-election due, logs one WARN naming its status, what it can see
and what to change — the unanswered seed addresses when nothing has
replied, or the seed list and selfElection pairing above when the peers
are all present and all waiting. It is reported once, not per round. A
join deadline, when one is set, acts on the
same verdict.
weakly-up is in that sentence because it is no closer to up: it is
the node’s own weaklyUpAfterMs timer moving it
on, not a leader admitting it. So the loop does not stop there — it
keeps dialling until the node is up, which is also what lets a seed
that starts after the promotion still find the node.
That line is worth knowing about because the symptom is nowhere near
the cause: a cluster stuck this way has no up member, a
cluster singleton picks its host from
the up members, and so the first thing most applications notice is
an AskTimeoutError from a singleton proxy.
When no seed ever answers
Section titled “When no seed ever answers”The retry loop has no end of its own. A node whose seeds never answer — wrong addresses, a firewall, seeds that never started — logs the stall warning above after three rounds and then goes on dialling for as long as the process lives. On an orchestrator that is a pod that never joins and never exits, so nothing ever restarts it.
shutdownAfterUnsuccessfulJoinMs gives the join a deadline:
const clusterOptions = ClusterOptions.create() .withHost('10.0.0.8') .withPort(2552) .withSeeds(['10.0.0.5:2552', '10.0.0.6:2552']) .withShutdownAfterUnsuccessfulJoinMs(60_000);const cluster = await Cluster.join(system, clusterOptions);or, for every node of a deployment at once:
actor-ts.cluster.shutdown-after-unsuccessful-join = 60sThe deadline is when the node decides, not what decides. When it passes,
the node asks the question the stall warning answers — can anything still
promote it? Only if nothing can does it give up: one WARN naming the key,
the deadline and the same remedy, then leave(), then the
coordinated-shutdown pipeline with
ClusterJoinUnsuccessfulReason. If something can, it asks again every
seedRetryIntervalMs until the node is up or the answer changes.
| At the deadline | Gives up? |
|---|---|
No seed has answered, selfElection: 'immediate' | yes |
selfElection: 'never', no member up | yes |
Peers known, none up, 'immediate' everywhere — the symmetric list above | yes |
weakly-up by its own timer, nothing else changed | yes — weakly-up is not joined |
minimum-members-before-up unmet, and nobody answering | yes |
A member is up | no — that leader promotes the node |
A deferred selfElection still pending | no — and when it fires, the node forms a cluster of one |
minimum-members-before-up unmet, live peers present | no — the cluster is waiting for members, not stuck |
minimum-members-before-up unmet, the peers that answered have since died | yes, once they are gone — removed at down-after by the default, or parked there under failure-detector.auto-down = off, which counts as gone; a peer only unreachable, short of down-after, still counts as live |
A few details settle the edges:
0, the default, sets no deadline — what every release before the key did. A value shorter thanseedRetryIntervalMsis refused atCluster.join: the node would give up before it had dialled twice.- The clock starts at the join, not at discovery. Under
cluster bootstrap the seed lookup and any
stable observation run first, and the key composes with
awaitReadythere — see Waiting for readiness. leave()comes first, so the seed loop stops even withcoordinated-shutdown.auto-register-tasks = off. What the run ends in is the pipeline’s to decide: the actor system terminates by default, and the process exits only withcoordinated-shutdown.exit-process = on, and then never with status0— see Task semantics.- A readiness wait answers to it too: an
awaitReady()still pending when the join gives up rejects at once with aClusterReadyTimeoutErrorthat names the key, rather than waiting out its own deadline. - A founder’s pending election is invisible to the others. A node
whose seeds answer but are themselves still
joining— a founder waiting out a deferredselfElectionof its own — sees nothing that can promote it: that timer runs on the founder, and nothing gossips it. So it gives up at its deadline on a cluster about to form, and the deadline has to exceed the founder’s grace. Behind stable observation cluster bootstrap refuses a shorter one; a plainCluster.joindoes not.
Seed discovery — beyond a static list
Section titled “Seed discovery — beyond a static list”A hard-coded seed list works for tests and small clusters. For production where nodes have dynamic IPs (containers, K8s pods), use a seed provider:
| Provider | When |
|---|---|
| Config | Static list (the case above). |
| DNS | Resolves _actor-ts._tcp.example.com SRV records. |
| Kubernetes API | Lists pods matching a label selector. |
| Aggregate | Falls through multiple providers (e.g. K8s, then DNS). |
import { KubernetesApiSeedProvider, KubernetesApiSeedProviderOptions } from 'actor-ts/discovery';
const kubernetesApiSeedProviderOptions = KubernetesApiSeedProviderOptions.create() .withNamespace('default') .withServiceName('actor-ts') .withPort(2552);const seedProvider = new KubernetesApiSeedProvider( kubernetesApiSeedProviderOptions,);
const seeds = await seedProvider.discover();
const clusterOptions = ClusterOptions.create() .withHost(process.env.POD_IP!) .withPort(2552) .withSeeds(seeds);const cluster = await Cluster.join( system, clusterOptions,);The provider returns a snapshot of seed addresses; the framework uses them to bootstrap the join. See Discovery overview for the seed provider model.
Watching the join progress
Section titled “Watching the join progress”The one-call gate first — awaitReady resolves once this node is up
and, with minimumMembers, once the cluster has reached its expected
size; isReady is the synchronous probe:
await cluster.awaitReady({ minimumMembers: 3, timeoutMs: 30_000 });cluster.isReady(); // probe the same predicate without waitingcluster.selfMember(); // this node's own record — status includedWith timeoutMs set it rejects with ClusterReadyTimeoutError instead
of leaving you to guess; without it, it waits indefinitely. See
Cluster bootstrap → Waiting for readiness
for how Cluster.bootstrap applies the same wait by default.
For logic that reacts to individual transitions, subscribe to the events:
import { SelfUp, MemberUp } from 'actor-ts/cluster';
cluster.subscribe((evt) => { if (evt instanceof SelfUp) { console.log(`this node is now Up`); } else if (evt instanceof MemberUp) { console.log(`peer ${evt.member.address} reached Up`); }});Two key events:
SelfUpfires once when this node transitions toup.MemberUpfires every time any member reachesup.
Counting MemberUps by hand to answer “are at least 3 nodes up?” is
exactly what awaitReady({ minimumMembers: 3 }) replaces — the
subscription is for reacting to changes, not for gating startup.
Settings that have to agree
Section titled “Settings that have to agree”A node that joins with a different value for something the cluster assumes
is uniform does not fail. It joins, it runs, and the disagreement surfaces
later as behaviour nobody can account for.
actor-ts.cluster.configuration-compatibility-check makes it say so at the
moment the two nodes meet instead.
Each node publishes the effective value of every setting in
checked-paths on the ordinary gossip lane and compares what its peers
publish back. Effective, not configured: precedence is **explicit options
HOCON > built-in defaults**, so two nodes whose config files are identical and whose builders are not still compare unequal — which is the case that matters, since the builder is the documented way to set most of these.
const clusterOptions = ClusterOptions.create() .withHost('n1') .withPort(2552) .withConfigurationCompatibilityCheckedPaths([ 'actor-ts.remote.max-frame-bytes', 'actor-ts.cluster.failure-detector.heartbeat-interval', ]);Only a setting something publishes can be compared, and the framework publishes three:
| Setting | Why a divergence matters | In checked-paths by default |
|---|---|---|
actor-ts.remote.max-frame-bytes | A frame over the peer’s lower cap kills the whole association, and the sender is never told why. There is no protocol negotiation to catch it. | yes |
actor-ts.cluster.failure-detector.heartbeat-interval | A peer that beats more slowly than this node’s down-after tolerates is judged down for being configured differently, not for being dead — and removed by whatever acts on that verdict: a downing strategy, or the detector itself under failure-detector.auto-down = on. With neither, it is parked at unreachable. | no |
actor-ts.cluster.tombstone.time-to-live | One side prunes a tombstone the other still enforces, so a removed address is re-admitted on one side only. | no |
The last two are off by default because staging a new value one node at a time is an ordinary operation, and both divergences heal. The first has no detection anywhere else, which is the bar for being seeded.
Publish one of your own from anywhere that has the resolved value:
cluster.publishConfigurationFact('acme.orders.partition-count', String(partitions));It travels only if checked-paths lists the name — that list is an
allow-list of what leaves the node, which is why there is no second list
of paths to redact. A name outside the character set a HOCON path uses
(lower-case, dots and hyphens) is refused when the options are validated,
rather than published and silently dropped by every peer.
Any point in a node’s life is a valid moment to call it, and that is the point of the call existing at all: the three settings above are resolved before the first gossip frame moves, but a partition count that comes back from a migration or a broker is not. A fact published after every peer’s record has settled still reaches every peer’s comparison — each node’s own gossip carries its current facts, and a peer replaces what it holds for a member when that member itself restates them. A copy relayed by a third node fills a gap but never overwrites, so a stale copy doing the rounds cannot flap the comparison. Publishing the same value again changes nothing and costs nothing.
When two nodes disagree, each logs one line naming the setting and both
values, and publishes a MemberConfigurationMismatch carrying the same.
Once per setting per peer, not once per gossip round. Absence on either
side is silence: a peer running an older version, or one whose
checked-paths does not list that setting, reports nothing rather than
reporting everything during a rolling upgrade.
enforce = on adds exactly one thing — the diverging peer stops being one
of this node’s placement candidates, so a
cluster router
will not route to it. It never downs or quarantines anyone, and nothing
that elects a role
reads that set: a misconfigured majority would otherwise evict the
correctly-configured minority, and an asymmetric candidate view would elect
two singleton hosts rather than one. Off by default, because an enforcing
default turns any rolling deploy that changes a checked value into a
self-inflicted outage.
actor-ts.sharding.number-of-shards is deliberately not one of the
published settings. A region whose shard count disagrees with the
coordinator’s is already refused outright, with an error naming the key and
a ShardRegionRegistrationRefused event — see
Sharding introspection.
What can go wrong
Section titled “What can go wrong”Where to next
Section titled “Where to next”- Cluster overview — the bigger picture.
- Cluster bootstrap — stable observation and initial-seed election for simultaneous starts.
- Weakly-up — gradual-join semantics for slow convergence.
- Failure detector — how heartbeats keep the membership view fresh after join.
- Discovery overview — seed providers for dynamic environments.
- Downing strategies — split-brain resolution after the cluster forms.
