Cluster bootstrap

A cold start asks every node the same question at the same moment — “is there a cluster yet?” — and discovery answers it differently on each of them. DNS has not fully propagated; a pod is Ready before its IP is in the headless service; the Kubernetes API returns a partial pod list. Act on the first answer and each node forms a cluster out of the subset it happened to see.

Gossip never repairs that. The results are separate clusters with the same name, each with its own leader, its own singletons and its own shard allocations.

Cluster bootstrap is the phase that runs before Cluster.join and makes sure at most one cluster comes out of a simultaneous start.

The two rules

1 — Stable observation. Poll the seed provider on pollIntervalMs and require the returned set (plus this node) to be byte-identical for stableMarginMs before acting on it. Any change restarts the margin. A failed lookup is not an empty set — it does not count as an observation at all, so a DNS outage can never be mistaken for “I am alone”.

2 — Deferred self-election. Order the settled set by address. The lowest is the initial seed — but it does not form a cluster on the spot. Every node, winner included, joins with the other contact points as its seeds; only the winner also gets a deadline (selfElectionGraceMs) after which it will form a cluster if nobody has promoted it.

Why the winner still dials its seeds

Addresses are handed out by the platform, so a node that starts later can perfectly well sort first — a scaled-up pod, or a restarted one with a recycled IP. Letting the election winner self-elect immediately would have it form a second cluster beside the running one. Because self-election is only a deadline, the running cluster’s leader promotes it within a gossip round or two and the deadline never fires.

Turning it on

The one-line path — bootstrapCluster runs the phase for you:

import { bootstrapCluster, ClusterBootstrapOptions } from 'actor-ts/cluster';

const bootstrapOptions = ClusterBootstrapOptions.create('my-app')
  .withHost(process.env.POD_IP!)
  .withPort(2552)
  .withDiscovery('kubernetes')
  .withStableObservation(true);
const { cluster, shutdown } = await bootstrapCluster(bootstrapOptions);

Pass an object instead of true to override the timings:

const bootstrapOptions = ClusterBootstrapOptions.create('my-app')
  .withHost(process.env.POD_IP!)
  .withDiscovery('kubernetes')
  .withStableObservation({ requiredContactPoints: 3, stableMarginMs: 8_000 });

Driving it yourself

If you call Cluster.join directly, run the observation and feed both of its outputs into the options — the seed list and the selfElection policy:

import {
  Cluster,
  ClusterOptions,
  NodeAddress,
  StableObservation,
  StableObservationOptions,
} from 'actor-ts/cluster';

const selfAddress = new NodeAddress('my-app', process.env.POD_IP!, 2552);

const observationOptions = StableObservationOptions.create()
  .withSeedProvider(seedProvider)
  .withSelfAddress(selfAddress)
  .withRequiredContactPoints(3);
const observation = new StableObservation(observationOptions);
const targets = await observation.resolveJoinTargets();

const clusterOptions = ClusterOptions.create()
  .withHost(selfAddress.host)
  .withPort(selfAddress.port)
  .withSeeds(targets.seeds.map((address) => address.toString()))
  .withSelfElection(targets.selfElection);
const cluster = await Cluster.join(system, clusterOptions);

resolveJoinTargets() returns the settled set, whether this node won (isInitialSeed), and the selfElection value to pass on. Deriving that value yourself is the one mistake that reintroduces the split brain, so the observation does it for you.

Settings

Setting	Default	What
stableMarginMs	5000	How long the contact-point set must stay unchanged.
pollIntervalMs	1000	How often the seed provider is polled.
maxWaitMs	60000	Total budget; exceeding it throws.
requiredContactPoints	1	Fewest contact points a settled observation may contain.
selfElectionGraceMs	10000	How long the elected node waits before forming a cluster.

Same keys under actor-ts.cluster.bootstrap.*, with the usual precedence — explicit options > HOCON > built-in defaults:

actor-ts.cluster.bootstrap {
  stable-margin           = 5s
  poll-interval           = 1s
  max-wait                = 60s
  required-contact-points = 3
  self-election-grace     = 10s
}

requiredContactPoints is the one worth changing. The margin catches discovery that is slow; only a required count catches discovery that is stably wrong — a resolver that consistently returns two of your three pods will settle happily on the wrong set. The default of 1 exists so single-node development needs no configuration; in production set it to the replica count you expect.

Waiting for readiness

A resolved bootstrapCluster means a ready cluster: this node is a full member (up) and at least minimum-members members are up. When that is not reached within the budget, the bootstrap runs the coordinated-shutdown pipeline and rejects with a ClusterReadyTimeoutError naming self’s status, the up count and the bar — it never resolves for a node that is still joining.

const bootstrapOptions = ClusterBootstrapOptions.create('my-app')
  .withHost(process.env.POD_IP!)
  .withDiscovery('kubernetes')
  .withStableObservation(true)
  .withAwaitReady({ minimumMembers: 3, timeoutMs: 30_000 });
const node = await bootstrapCluster(bootstrapOptions);
node.formedNewCluster; // false — joined an existing cluster rather than founding one

awaitReady takes true (the default — wait with the computed budget), false or 0 (skip the wait), a number (the budget in ms), or the options bag above. Unset fields fall through to HOCON and then to the computed budget:

Field	Default	What
minimumMembers	1	Fewest up members — self included — before the cluster counts as ready. Size it like requiredContactPoints: to the replica count.
timeoutMs	derived	await-ready when set; else self-election-grace + 5 s behind stable observation — the budget every node’s readiness actually hangs on, winner or not — else a flat 5 s.

actor-ts.cluster.bootstrap {
  minimum-members = 3
  # await-ready   = 30s   # unset selects the grace-aware computed default
}

The same wait exists standalone — after a hand-wired Cluster.join, or later in the process before a rebalancing-sensitive step:

await cluster.awaitReady({ minimumMembers: 3, timeoutMs: 30_000 });
cluster.isReady();      // synchronous probe of the same predicate
cluster.selfMember();   // this node's own record, tombstone included
cluster.selfElected;    // true — this node founded its cluster

Without timeoutMs the promise waits indefinitely, like system.whenTerminated() — pair it with a deadline wherever the cluster may legitimately never form; a node that has left or been removed never becomes ready. Readiness here deliberately means membership: app-registered readiness checks (the /ready endpoint’s aggregate) may depend on initialisation that runs after bootstrap, so waiting on them could deadlock the very call that comes first. /ready stays the load balancer’s view; awaitReady is the in-process one.

To restore the old fire-and-forget shape, pass awaitReady: false and watch membership yourself via cluster.awaitReady().catch(…).

What it refuses to do

It throws rather than joining anyway

If the set never settles within maxWaitMs, resolveJoinTargets() rejects with a StableObservationError naming the last observation and this node’s advertised address. There is no degraded fallback to a direct join: that would be indistinguishable from the failure the phase exists to prevent — a node that could not agree with anyone about who is out there, joining anyway. A start that fails loudly is recoverable by a restart or an operator; a second cluster is not.

It refuses a wildcard advertised host

The election is ordered on this node’s address, so that address has to be the one peers dial. 0.0.0.0 (and ::, and an empty string) is a bind address: every node that claimed it would sort identically, every node would believe it won, and every node would form its own cluster. The phase rejects it up front.

Getting there takes deliberate effort, because a wildcard bind host no longer becomes a wildcard identity: with withHost('0.0.0.0') and nothing else, the advertised address resolves through CLUSTER_HOST / POD_IP / HOSTNAME and then to loopback. What this guard catches is a withAdvertisedHost('0.0.0.0') written on purpose — and a hand-built NodeAddress handed straight to StableObservation, which never passes through that chain.

In production, name it: bind 0.0.0.0, advertise POD_IP. Loopback keeps a single-node run working; it does not make a second node reachable.

Bootstrap or a plain seed join?

Situation	Use
Fixed, known addresses; one designated first node	Plain seed join.
Local development, single node	Plain seed join — the empty seed list already means “I am first”.
Nodes start simultaneously with dynamic addresses (K8s Deployment, autoscaling group)	Bootstrap.
Every node is given the same seed list	Bootstrap — see below.

That last row is easy to miss. Cluster filters this node out of its own seed list, so if every node lists every node then no node is left with the empty list that ordinary self-election needs. Nobody becomes up, nobody becomes leader, and nobody is ever promoted — the cluster deadlocks in joining. The election is what breaks the symmetry.

Costs

• Startup latency. At least stableMarginMs, plus selfElectionGraceMs on a genuine cold start (paid once, by one node). Joining an existing cluster is unaffected — the grace never fires.
• Discovery load. One lookup() per pollIntervalMs per node until the set settles. The poll cadence is deliberately constant rather than backing off: a growing interval would sample the stable margin at drifting points, and the load it saves — at most a few dozen lookups per node — is not worth weakening the guarantee.

Where to next

• Joining and seeds — the join handshake this phase feeds.
• Discovery overview — the seed providers the observation polls.
• Downing strategies — split-brain resolution after the cluster has formed.
• Weakly-up — partial progress while convergence is pending.