콘텐츠로 이동
한국어

Downing strategies

이 콘텐츠는 아직 번역되지 않았습니다.

When the cluster partitions, both halves stay running — and both think the other half has failed. Without intervention:

  • Two singletons would exist (one per side).
  • Sharded entities for the same key could spawn on both sides.
  • DistributedData replicas would diverge until reconciliation.

A downing strategy picks a winning side and forcibly downs the losing side. Each losing node leaves the cluster; the winning side continues as the cluster. Whether a losing node’s actors stop as well is actor-ts.cluster.run-coordinated-shutdown-when-down. With it on, the shipped default, the node runs its coordinated-shutdown pipeline, which ends its actor system unless coordinated-shutdown.terminate-actor-system is off. With it off it only leaves, and its actors keep running beside the winners — a second writer for every entity and singleton it hosted. Either way a node acts only on its own strategy’s verdict. The winning side cannot down it, but it refuses its heartbeats, and a node every peer refuses has its own strategy asked at once — so a node that was paused or cut off while the others removed it stands down once it can hear them again (issue #940).

network split

During partition (KeepMajority decides)

n1 · n2 · n3
majority → continues

n4 · n5
minority → downs itself

Before partition

n1 · n2 · n3 · n4 · n5

Without a strategy, the failure detector removes an unreachable peer itself at down-after — right for a crashed node, whose shards and singletons the survivors take over, down to the last available node. Under a partition, though, both halves remove each other and keep running. When the partition heals, you have two separate clusters with the same name and no automatic merge (issue #930). Always configure a strategy for production — lease-majority, or another of the five below.

StrategyWins on…Trade-off
KeepMajorityThe side with > N/2 members.Simple; ties (exactly N/2-N/2 split) means both sides down.
KeepOldestThe side containing the lowest-addressed member (see the caveat below).Works for even-sized clusters where majority is undefined.
KeepRefereeThe side containing a designated referee node.Predictable but creates a single point of failure (the referee).
StaticQuorumThe side that meets a configured quorum size.Stricter than majority; the quorum-size config is the operator’s choice.
LeaseMajorityMajority + must hold a coordination lease.Paranoid safety; requires a Lease provider (K8s, etc.).

Pick by your cluster’s topology and operational constraints (see the picking section below).

import { Cluster, ClusterOptions, KeepMajority } from 'actor-ts/cluster';
const clusterOptions = ClusterOptions.create()
.withHost(host)
.withPort(port)
.withSeeds(seeds)
.withDowning(new KeepMajority());
const cluster = await Cluster.join(
system,
clusterOptions,
);

Pass the provider as downing. Every cluster event (member unreachable, member reachable, etc.) re-runs the provider with the current view; it returns the set of addresses to forcibly down. Returning an empty set means “no decision yet — wait.”

A strategy is asked nothing until the member view has been unchanged for stable-after, which defaults to 20 s:

const clusterOptions = ClusterOptions.create()
.withDowning(new KeepMajority())
.withSplitBrainResolver({ stableAfterMs: 20_000 });
actor-ts.cluster.split-brain-resolver.stable-after = 20s

This is not a flap filter, or not only one. A partition is not one event the cluster observes; it is a sequence of them. The failure detector marks peers unreachable one at a time, as each crosses unreachable-after, and downing used to be evaluated at the end of that same tick — so a 2/2 partition whose two remote peers were detected on different ticks was resolved as two successive majority decisions rather than one equal-split decision:

tick 1: a=up b=up c=unreachable d=up 3 reachable of 4 -> down c
tick 2: a=up b=up c=removed d=unreachable 2 reachable of 3 -> down d

The tombstone closes the trap: a force-down marks the member removed, and every strategy filters its candidates to up | leaving | unreachable, so the denominator shrinks along with the numerator and what is left of the side reads as a majority of it. Both halves run the identical computation over their mirror image and both survive — and with LeaseMajority the lease is never contended for at all, because the equal-split branch that reaches it is never taken.

Measured, that was not a corner case: the framework’s own four-node partition test failed 8 of 15 runs with the window disabled and 0 of 30 with a 1 s window, on the same machine within the same minute.

Two consequences worth knowing:

  • The window must outlast the spread between the first and last detection of one partition, plus the gossip that follows — not merely the duration of a flap. A window shorter than the spread lets the sequence through again.
  • It costs failover latency. Nothing is downed until the view has been still for the window, so a genuine node loss takes that much longer to resolve. The 20 s default is the established value for this knob and is a deliberate trade of availability for integrity; a cluster that needs faster failover should lower it knowingly rather than by omission.

stable-after = 0 restores the pre-window behaviour, and the defect above with it. It exists so a deployment can ask for that explicitly.

All five strategies can be selected without touching the code, which is what lets one image run unarbitrated in development and keep-majority in production:

actor-ts.cluster.split-brain-resolver {
active-strategy = keep-majority # off | keep-majority | keep-oldest | keep-referee | static-quorum | lease-majority
# Per-strategy options, each read only by the strategy it belongs to.
keep-majority.role = backend
static-quorum.quorum-size = 3
keep-referee.referee-address = "my-app@10.0.0.1:2552"
keep-referee.down-all-if-below-quorum = 2
}

active-strategy builds the same object withDowning(…) takes, so the ordinary precedence applies: an explicit withDowning(…) wins, and a Cluster.join that names none falls through to the file. off is the shipped default and means no strategy at all — see the caution at the bottom of this page for what that leaves you with.

static-quorum.quorum-size and the two keep-referee keys are absent from reference.conf rather than shipped with a default: each is required or bounded, and quorum-size = 0 / referee-address = "" are refused by the strategies’ own validators, so any value that could ship would be one that stops the node. Naming static-quorum or keep-referee without its key is refused at startup, with the missing key in the message.

lease-majority needs a lease backend as well. Its lease is built from actor-ts.coordination.lease, named <system>-split-brain-resolver and owned by this node, so a deployment selects it with both blocks:

actor-ts.coordination.lease {
backend = kubernetes
ttl = 15s
}
actor-ts.cluster.split-brain-resolver.active-strategy = lease-majority

With the shipped backend = none the strategy is refused at startup rather than run with nothing to arbitrate through. Its role and acquire timeout stay in code; build it as shown below when either matters.

A strategy is not asked the moment a peer goes silent. The membership and reachability view has to hold still for stable-after first — 20 s by default — and only then is the configured provider consulted:

actor-ts.cluster.split-brain-resolver {
active-strategy = keep-majority
# The view must be unchanged this long before the strategy is asked.
stable-after = 20s
# Down everyone, self included, when it never once is.
down-all-when-unstable = off
}

Or in code, where the same two settings are one nested block:

import { ClusterOptions, SplitBrainResolverOptions } from 'actor-ts/cluster';
const resolverPolicy = SplitBrainResolverOptions.create()
.withStableAfterMs(30_000)
.withDownAllWhenUnstable(true);
const clusterOptions = ClusterOptions.create()
.withHost(host)
.withPort(port)
.withDowning(new KeepMajority())
.withSplitBrainResolver(resolverPolicy);

Why a mid-churn decision is the wrong decision. Every bundled strategy is a pure function of one view — KeepMajority counts it, KeepOldest orders it, StaticQuorum measures it. A partition does not arrive as one event: peers cross unreachable-after a few hundred milliseconds apart, and a view sampled in the middle of that shows a cluster that existed at no instant. Counting a majority against it is how a node concludes it is in the minority a second before the rest of its own side is marked unreachable too. The window is what makes the input a settled fact rather than a snapshot of a transition.

The clock is the failure-detection tick, so the window’s resolution is actor-ts.cluster.failure-detector.heartbeat-interval — 500 ms by default. That is irrelevant at twenty seconds and worth knowing if you tune the window down to a few hundred milliseconds.

There is no upper bound against failure-detector.down-after. Once a strategy is configured the detector stops evicting entirely: it parks an unreachable peer and leaves the down/removed transition to the resolver, so the window has no deadline to out-run and may be as long as the deployment wants. With no strategy configured, the two settings do nothing at all — there is no decision for them to delay. The detector removes the peer at down-after itself under the default failure-detector.auto-down = on, or parks it there under auto-down = off.

stable-after is a preference, not a precondition, and the reason is the same sentence read the other way round. The view the window watches is every member’s address and status, so any join, leave or status transition anywhere in the cluster restarts it. A deployment whose membership moves more often than stable-after would therefore never arbitrate a partition at all — and at the shipped twenty seconds that is an ordinary autoscaling group, a rolling deploy, or one flapping node. It would not be a safe failure either: with a strategy configured the failure detector stops evicting, so a partition nobody arbitrates simply stays.

So a peer that has been continuously unreachable for 2 × stable-after is put to the strategy whatever else is moving. That one peer’s silence is a settled fact on its own; the churn around it is about other members, and a verdict is owed about the partition rather than about the churn.

The multiple sits deliberately between the two thresholds either side of it: above one window, so a view that does settle is always arbitrated through the ordinary path first and the ceiling never becomes the normal case; and below the 3 × stable-after escalation deadline below, so a strategy is asked before down-all-when-unstable can stop the cluster.

A peer that recovers loses its place: the clock measures one uninterrupted run of silence, so a peer that flaps never accumulates its way to the ceiling out of intervals none of which meant anything on its own.

down-all-when-unstable is the escalation for a cluster that will not converge: if the view has not once held still for a whole stable-after across 3 × stable-after of continuous change — and something is still unreachable — every member is downed, this node included, and the reason is logged at warn. No strategy is consulted; a view that unsettled is not evidence any of them could decide on.

The margin is derived from the window rather than configured beside it. It only ever means “several times however long this deployment thinks a view needs to settle”, so a second key would be one more number to keep consistent — and a margin below the window could never observe one quiet window before firing.

import { KeepMajority } from 'actor-ts/cluster';
new KeepMajority();

The classic strategy. Counts members; the side with more wins.

Picks:

  • Most general default.
  • No external dependencies.
  • Predictable when N is odd.

Doesn’t:

  • Handle exact N/2-N/2 splits — both sides down themselves.
  • Distinguish between “5 members healthy” and “5 members all on the same machine, the rack burns” — majority by count is blind to physical topology.

Right for clusters with an odd number of nodes and no special-purpose deployments.

import { KeepOldest, KeepOldestOptions } from 'actor-ts/cluster';
const keepOldestOptions = KeepOldestOptions.create().withRole('backend');
new KeepOldest(keepOldestOptions);

The side containing the “oldest” member wins — unless the oldest is alone (see downIfAlone below).

Useful when:

  • You have an even number of nodes, where KeepMajority is ambiguous on 50/50 splits.
  • You can pin the address of the node that should survive — then the strategy is deterministic and needs no quorum arithmetic.

downIfAlone — on by default, keep-oldest.down-if-alone in the config file — decides what happens when the oldest is the only member on its side of the split, against two or more on the other:

  • On: the oldest loses and the group carries on. That is the same answer from both sides: survivors that find a crashed oldest unreachable down it, and an oldest cut off on its own downs itself.
  • Off: the side holding the oldest survives and nothing else does. A crashed oldest is on nobody’s side, so every survivor downs itself and the crash of one node — the lowest-addressed, which is also the leader — stops the cluster (#932). Choose it only if you would rather stop than run without the node you pinned.

Two sides of one member each are not “alone against a group”: a two-node cluster keeps the oldest either way.

import { KeepReferee, KeepRefereeOptions } from 'actor-ts/cluster';
const keepRefereeOptions = KeepRefereeOptions.create()
.withRefereeAddress('my-app@10.0.0.1:2552')
.withDownAllIfBelowQuorum(3);
new KeepReferee(
keepRefereeOptions,
);

A designated referee node is the deciding member. The side containing the referee wins; the other side downs itself.

Picks:

  • Most predictable strategy — there’s no ambiguity about which side wins.
  • Works for any cluster size, including even-sized.

Doesn’t:

  • Survive the referee itself going down. If the referee disappears, neither side has it, and the strategy can’t decide. Hence downAllIfBelowQuorum — “if the cluster is below this size, down everyone and let operators rebuild.”

Useful for two-DC clusters with a tie-breaker node in a neutral location (a third DC, a control-plane K8s namespace).

import { StaticQuorum, StaticQuorumOptions } from 'actor-ts/cluster';
const staticQuorumOptions = StaticQuorumOptions.create().withQuorumSize(3);
new StaticQuorum(staticQuorumOptions);

When some member is unreachable, a side wins only if it has at least quorumSize reachable members; below the quorum, the side downs itself. With nothing unreachable it decides nothing, however small the cluster is.

Picks:

  • Stricter than majority — protects against minority-survivor scenarios where the minority would otherwise continue.
  • Configurable based on your operator’s confidence threshold.

Doesn’t:

  • Recover automatically. If multiple sub-quorum partitions form, none wins; the operator has to manually rebuild.

Right when you’d rather fail-stop than risk wrong-side survival.

import { LeaseMajority, LeaseMajorityOptions } from 'actor-ts/cluster';
const leaseMajorityOptions = LeaseMajorityOptions.create().withLease(someLeaseImplementation);
new LeaseMajority(
leaseMajorityOptions,
);

Majority plus a coordination lease. Wrap any other strategy: the winning side must additionally acquire a lease (e.g. a K8s Lease resource) before considering itself authoritative.

Picks:

  • Belt-and-braces safety. Two-fold check.
  • Useful when the network is unpredictable (e.g. cloud cross-zone).

Doesn’t:

  • Help if the lease provider itself is partitioned away.

Right for paranoid production scenarios where you’d rather double the cost of split-brain protection.

From a config file the strategy gets its lease from the coordination block instead of from withLease(…) — see from the config file. That lease is built at the strategy’s first decision, from the address the decision carries, because the strategy is constructed before the node’s address exists.

The interesting case is not “the lease said no” but “the lease said nothing”. Two things leave an acquire() unwanted while it is still on the wire: it has not resolved within acquireTimeoutMs (5 s by default), or the partition view moved on under it — the partition healed, or the unreachable set changed. The second is the likelier of the two, since a membership change inside a 5 s window needs no stall at all. Either way the strategy stops waiting for the attempt, and then has to assume it may still succeed on the wire, leaving the lease claimed by a node that no longer believes it won. The same three rules cover both:

  • The late result is dropped. Every attempt carries an epoch; a result arriving after its epoch was retired cannot write a decision. So a slow acquire() → true can never make this side claim survival after the fact.
  • A late win is released, and nothing new starts until it is. Ownership only begins when the acquire resolves, so the undo waits for it to report back — releasing earlier would be a no-op. While that is outstanding, decide() keeps returning an empty decision rather than starting a second attempt: two attempts overlapping is precisely how the cleanup could delete a lease this side is actively claiming.
  • If the release fails, the strategy stops deciding. The lease is then in an unknown state, so it enters fail-safe and returns an empty decision for this partition view until the partition heals. Deciding nothing is always safe; deciding wrongly is not.

The visible symptom of a lease provider having a bad day is therefore a cluster that waits, not one that splits.

A win is only as good as the lease behind it. Once the view has been stable for stable-after, the cluster asks the strategy again on every failure-detector tick, so a cached win can be returned long after the acquire that produced it. A node that stalled past the lease’s TTL in between — a long GC pause, a starved container — has lost the lease to the other side, which is by then downing this one. Replaying the old win is how both halves end up acting as the survivor.

So before a won decision is returned again, the strategy asks lease.checkAlive(). If the lease has run out, the win is dropped and the side arbitrates afresh: the new acquire loses if the other side holds the lease now — and this side then downs itself — and wins it back if nobody took it. A lost arbitration is not re-asked; it does not rest on the lease. This relies on the backend answering checkAlive() against its TTL rather than from a held flag, which both built-in backends do (#937).

An acquire that takes longer than the lease’s TTL can never produce a usable win: the strategy keeps arbitrating and decides nothing, the same posture as while an acquire is pending.

A won arbitration keeps the lease, and the backend renews it for as long as the strategy holds it: decide() never gives a win back, since a cached win is only as good as the lease under it. Leaving does. cluster.leave(), and the leave at once of a node its own decision downed or that gave its join up, call the strategy’s shutdown() once the failure-detector tick that asks it has stopped. A won lease is released there and then, so the members left behind can take it at their next even split instead of finding it held by a node that is gone, and nothing renews it once the node has left. The leave also makes an acquire() still on the wire unwanted — a third case beside the two above, and their first two rules cover it: a win the attempt reports afterwards is dropped and released where it lands. A release that fails is logged and the leave goes on; the record then runs out at its TTL.

The strategy decides nothing after it, so a node that joins again needs a fresh one. A strategy selected in configuration is built anew by every Cluster.join; one built in code has to be built again rather than handed to a second join.

Three questions in order:

  1. Odd or even cluster size?

    • Odd → KeepMajority.
    • Even → KeepOldest or KeepReferee (avoid ties).
  2. Do you have a stable tie-breaker node?

    • Yes → KeepReferee (most predictable).
    • No → KeepMajority or KeepOldest.
  3. Is fail-stop preferable to potential wrong-winner?

    • Yes → StaticQuorum (errs on the side of stopping).
    • No → one of the above.

For a typical 3-node K8s deployment: KeepMajority. For a 2-DC setup with a third-region tie-breaker: KeepReferee. For a 5-node cluster that should never go below 3: StaticQuorum(3).

The provider interface is tiny:

interface DowningProvider {
decide(view: ClusterPartitionView): DowningDecision;
shutdown?(): Promise<void>; // optional: give back what it holds
}
type ClusterPartitionView = {
allMembers: ReadonlyArray<Member>;
unreachable: ReadonlySet<string>; // address strings
self: NodeAddress;
};

Implement decide(view) => Set<string> and return the address strings to down. Empty set means “no decision.” shutdown() is for a strategy that holds something it has to give back: the cluster calls it once when this node leaves, after the last decide(), and LeaseMajority releases its lease there. A shutdown() that throws or rejects is logged, and the leave goes on.

Useful for app-specific rules — e.g. “always keep the node hosting role=primary,” or “if the partition includes the DB-master node, that side wins.”

There is no separate quarantine. What keeps a removed member out is its tombstone: the removed record the cluster keeps for the address, which refuses any record for it at the same or an older version. Whether a removal leaves one — and so whether the process that was removed can carry on talking and be let back in — depends on the route that removed it. The same run below means that process: the same incarnation, still announcing the version it had when it was removed.

Removed byLeaves behindThe same run speaks again
Cluster.down() or POST /cluster/downA tombstone.Refused until the tombstone is pruned.
A strategy’s decision, or down-all-when-unstableA tombstone.Refused until the tombstone is pruned.
The peer’s own leaveA tombstone.Refused until the tombstone is pruned.
The failure detector with no strategy — failure-detector.auto-down = on, the defaultNothing — the entry is deleted.Readmitted on its next gossip frame; a heartbeat alone finds no entry to bring back.
Nobody: no strategy and auto-down = off, the opt-outNothing is removed — the peer is parked at unreachable.Back to up on its next heartbeat to this node.

The last row’s way back needs the peer to send this node a heartbeat, which ends a pause, not a partition: after a symmetric partition heals, neither side heartbeats or gossips a peer it holds unreachable, so the parked members stay parked and the cluster stays split until an operator acts (issue #930).

A tombstone refuses for the larger of tombstone.time-to-live and tombstone.min-retention — 24 h at the defaults — and goes on the first prune pass after that. time-to-live is the operator’s knob; there is no quarantine setting beside it. min-retention is a floor under it, six times failure-detector.down-after when left at 0s, so that a short time-to-live cannot drop a tombstone before peers that have not converged yet have seen it:

actor-ts.cluster.tombstone {
time-to-live = 24h # how long a removed address is refused
prune-interval = 5m # how often expired tombstones are dropped
min-retention = 0s # floor under time-to-live; 0s = 6 x down-after
}

The refusal is a membership one. Nothing refuses the connection — no handshake asks membership — so the downed node’s frames still arrive and the messages it sends are still delivered. What is refused is its record: it is not a member, and nothing that places work by membership — sharding, singletons, cluster-aware routing, downing quorums — counts it or its roles. After the prune the same run is back on its next gossip frame, at its old version and with the roles it had: the window is a delay, not a verdict. A node downed while still running is not told about a downing decided elsewhere (see the top of this page), so unless something stops it, it is a member again once the window has passed.

The detector’s own route — the default — is the one with no refusal. It deletes the entry instead of tombstoning it, so the same run is readmitted on its next gossip frame — after MemberRemoved was announced for it and its remote watchers were sent a node-loss Terminated, with nothing to tell them it is back. Nothing is left to refuse a survivor’s stale record of it either, which is why a crashed member keeps coming back under auto-down (what the detector decides has the measurement). Nor does deleting help a healed partition: neither side heartbeats or gossips a peer it holds unreachable, and after the delete there is no entry to address at all (issue #930).

A restart is readmitted by version. A process seeds its version from its own wall clock when it joins, which out-versions the tombstone its previous run left behind, so a node that restarts rejoins at once whichever route removed it — nobody waits out the window, and nothing has to be cleared. If its previous run was never removed — parked under auto-down = off, held for a strategy, or restarted inside down-after — the new run’s record replaces a live one, and the node ends the old run first: MemberDown and MemberRemoved for the old incarnation, so that its remote watchers get their node-loss Terminated and its shards and singletons move, then MemberJoined for the new one. That takes both records carrying an incarnation; a record without one replaces the old run silently, as every record did before (a restart ends the previous run).

The tombstone refuses the run it was written for, not only its version: since cluster wire v2 every address on a frame names its incarnation — which process answers there, minted once per Cluster.join — and a record for the incarnation a tombstone holds is refused whatever version it carries (issue #823, issue #940). A process that was downed while still running cannot come back by restamping its version; it stands down instead, and only a restart joins again.

One limit remains, because the floor a restart has to clear is still a version:

  • A keep-out lever for a peer with standing. A member this node holds active may gossip records about other addresses, and a removed record dated inside the version-skew budget (maxVersionSkewMs, five minutes by default) out-versions the fresh version a legitimate restart starts from. The restarted node stays out until it restarts again past that version, or the tombstone expires, counted from a removedAt the same peer chose.

A restart carries a new incarnation, so it could be admitted over any tombstone of a previous run regardless of version; it is not yet, which is what keeps this lever open.

The DowningProvider API reference covers the strategy interface.