Skip to content
English

Sharding with lease

The sharding coordinator runs as a cluster singleton — only the leader’s coordinator is active. With a downing strategy + healthy network, that’s enough.

During a network partition + buggy downing config, both halves might briefly think they’re leader → two coordinators → conflicting shard allocations → entities possibly spawned on both sides.

The single-writer lease prevents this:

import { ClusterSharding, StartShardingOptions } from 'actor-ts/cluster';
import { KubernetesLease, KubernetesLeaseOptions } from 'actor-ts/coordination';
const kubernetesLeaseOptions = KubernetesLeaseOptions.create()
.withName('order-sharding-coordinator')
.withOwner(process.env.POD_NAME!)
.withTtlMs(30_000)
.withNamespace(process.env.K8S_NAMESPACE!);
const startShardingOptions = StartShardingOptions.create<Command>()
.withTypeName('order')
.withEntityActor(OrderEntity)
.withExtractEntityId((message) => message.id)
.withLease(new KubernetesLease(
kubernetesLeaseOptions,
));
const sharding = cluster.sharding.start(
startShardingOptions,
);

Now even if two nodes both think they’re leader, only one holds the lease. Only the lease-holder’s coordinator processes shard allocations.

Lease backendnode Bnode ALease backendnode Bnode ABoth think they're cluster leaderpartition + insufficient downing configatomic CAS — exactly one winsTTL expiry or crashCoordinator transitions cleanlyallocations resumelease.acquirelease.acquiresuccess — A is coordinatorfail — B's coordinator stays passiverenewing acquire firessuccess — B becomes coordinator

The lease backend (K8s API server, etcd) provides the atomic exactly-one-holder guarantee — it’s the source of truth beyond gossip.

The lease fields, beside the base sharding settings:

type StartShardingOptionsType<TMessage> = {
lease?: Lease;
useLease?: boolean;
acquireRetryIntervalMs?: number;
};
FieldPurpose
leaseThe Lease instance — typically KubernetesLease. Wins over useLease.
useLeaseBuild the coordinator’s lease from actor-ts.coordination.lease instead. Default false; also settable node-wide as actor-ts.sharding.use-lease.
acquireRetryIntervalMsRetry cadence on failed acquire, default 5000 ms. Also settable node-wide as actor-ts.sharding.acquire-retry-interval.
actor-ts.coordination.lease {
backend = kubernetes
ttl = 30s
}
actor-ts.sharding {
use-lease = on
acquire-retry-interval = 2s
}

use-lease = on gives every sharded type this node starts a coordinator lease the framework builds from the coordination block — named <system>-sharding-<type> and owned by this node, its cluster address and incarnation — so the file picks the backend and nothing about identity. Name and owner are the values a file every node shares cannot carry: one name for the block would put every type on one record, and one owner on every node would let each believe it held the lease. With backend = none the switch is refused when the type starts, rather than advertising split-brain protection that is not there. A withLease(…) in code still wins, and withUseLease(false) opts one type out. See leases the framework builds.

The retry interval is worth tuning because it is the recovery latency of a partition that heals: until an acquire succeeds the coordinator issues no AllocateShard, and everything routed for the type waits in the regions’ buffers meanwhile.

Same Lease abstraction as singleton with lease — see Coordination for the interface.

The lease gates coordinator state writes:

  • Shard allocations — assigning shards to regions.
  • Rebalance decisions — moving shards between regions.
  • Handoff coordination — orchestrating shard handoffs.

The lease doesn’t gate:

  • Per-region entity hosting (regions are tied to actual cluster members, not the coordinator).
  • Entity messaging (messages route via the coordinator’s last-known allocation; the lease isn’t in the message path).

So the worst-case during a partition is slightly stale shard assignments — entities continue running, just no new allocations until lease ownership stabilizes.

Three good fits:

  1. Production multi-region clusters where partitions are plausible.
  2. Financial / inventory entities where dual-allocation would cause real damage.
  3. Compliance requiring “no possibility of split-brain in any single-tenant production system.”

For typical single-region K8s deployments with a downing strategy, the lease is paranoid-safe — adds operational complexity for protection against rare edge cases.

K8s API server outage → no replica can acquire → no shard allocations

The lease backend becomes a SPOF. For most setups, K8s API availability is much higher than the cluster itself — but plan for the rare case.

Old coordinator loses lease (TTL expiry: 30s)
→ new coordinator acquires (typically sub-second after TTL)
→ new coordinator rebuilds state from gossip + journal

The failover window is the lease TTL — ~30 s typical. During that window:

  • No new shard allocations happen.
  • Existing entities continue receiving messages.
  • New entity IDs that need allocation queue up; processed after failover.

Acceptable for most workloads.

{
downingProvider: new KeepMajority(),
// + the sharding lease
lease: ...,
}

Both layers active. Downing handles normal cluster convergence; the lease guarantees the coordinator-uniqueness invariant.

SetupCoordinator-uniqueness guarantee
No downing, no leaseBest-effort. Partitions cause dual coordinators.
Downing strategy onlyStrong on stable networks.
Downing + leaseParanoid-safe. Both invariants enforced.

For singleton + sharding production setups in critical-data scenarios, both is the recommended pattern.