StableObservation

Defined in: src/cluster/bootstrap/StableObservation.ts:142

Stable-observation bootstrapping (#148) — the step that decides who starts a cluster before anybody does.

What it is for

A cold start hands every node the same question (“is there a cluster yet?”) at the same moment, and discovery answers it differently on each of them: DNS has not fully propagated, a pod is Ready before its IP is in the headless service, the K8s API returns a partial pod list. Acting on the first answer gives each node a different world, and each of them forms a cluster out of the subset it can see. Gossip never bridges the results — they are separate clusters with the same name.

The fix is to stop treating the first answer as the answer. This class polls the provider until the set it returns has been unchanged for stableMarginMs, then orders that set by address and elects the first entry as the only node permitted to form a cluster from nothing.

Why the election alone would not be enough

Electing the lowest address and having it self-elect immediately — the obvious reading — trades one split brain for another. Addresses are assigned by the platform, so a node started later can perfectly well sort first: a scaled-up pod, or a restarted one with a recycled IP. It would win the election against a cluster that is already running and form a second one beside it.

So the election does not decide whether to join, only who may give up waiting. Every node — winner included — joins with the full contact-point set as its seeds; the winner additionally gets a ClusterOptionsType.selfElection deadline, and the losers get 'never'. If a cluster exists, its leader promotes the winner long before the deadline and nothing is formed. If none exists, the deadline expires exactly once, on exactly one node.

That is also why this needs no contact-point probe endpoint of its own: “does a cluster already exist?” is answered by the join path itself, which is authoritative, rather than by a second read-only view that can disagree with it.

Usage

const observationOptions = StableObservationOptions.create()
  .withSeedProvider(seedProvider)
  .withSelfAddress(selfAddress)
  .withRequiredContactPoints(3);
const observation = new StableObservation(observationOptions);
const targets = await observation.resolveJoinTargets();

const clusterOptions = ClusterOptions.create()
  .withHost(selfAddress.host)
  .withPort(selfAddress.port)
  .withSeeds(targets.seeds.map((address) => address.toString()))
  .withSelfElection(targets.selfElection);
const cluster = await Cluster.join(system, clusterOptions);

bootstrapCluster does all of the above when its stableObservation option is set; reach for this class directly only when you drive Cluster.join yourself.

Constructors

Constructor

new StableObservation(options): StableObservation

Defined in: src/cluster/bootstrap/StableObservation.ts:145

Parameters

options

StableObservationOptions

Returns

StableObservation

Methods

resolveJoinTargets()

resolveJoinTargets(): Promise<JoinTargets>

Defined in: src/cluster/bootstrap/StableObservation.ts:179

Poll until the contact-point set settles, then elect. Rejects with a StableObservationError once maxWaitMs is spent.

Throwing rather than falling back to a direct join is deliberate (the issue’s open question 3). The fallback is friendlier exactly once — the first time somebody’s discovery is misconfigured — and after that it is indistinguishable from the failure this class exists to prevent: a node that could not agree with anyone about who is out there, joining anyway. A start that fails loudly is recoverable by a restart or an operator; a second cluster is not.

Returns

Promise<JoinTargets>