Skip to content
English

KubernetesLease

KubernetesLease implements the Lease interface against Kubernetes’s built-in Lease resource (the coordination.k8s.io/v1 API). Production-grade: backed by etcd, strongly consistent, RBAC-controlled.

import { KubernetesLease, KubernetesLeaseOptions } from 'actor-ts/coordination';
const kubernetesLeaseOptions = KubernetesLeaseOptions.create()
.withName('my-singleton-lease')
.withOwner(process.env.POD_NAME!)
.withTtlMs(30_000)
.withRenewalIntervalMs(10_000)
.withNamespace(process.env.K8S_NAMESPACE!);
const lease = new KubernetesLease(
kubernetesLeaseOptions,
);

The K8s API server’s etcd-backed store provides the single-holder guarantee. Two pods concurrently calling acquire() produce exactly one winner, regardless of pod scheduling, network partition between pods, etc.

type KubernetesLeaseOptionsType = {
// From LeaseOptionsType:
name: string;
owner: string;
ttlMs: number;
renewalIntervalMs?: number;
acquireRetries?: number;
acquireRetryDelayMs?: number;
// K8s-specific:
namespace?: string; // unset: read from the mount
namespacePath?: string;
tokenPath?: string;
caPath?: string;
apiServerUrl?: string; // all three together, or none of them
authToken?: string; // all three together, or none of them
caCert?: string; // all three together, or none of them
tokenReloadIntervalMs?: number;
operationTimeoutMs?: number;
leaseNameMaxLength?: number;
};
K8s fieldDefaultWhat
namespacefrom the mountK8s namespace where the Lease resource lives. Unset, it is the one the Pod’s own ServiceAccount mount reports.
namespacePath/var/run/…/serviceaccount/namespaceFile the namespace is read from.
tokenPath/var/run/…/serviceaccount/tokenFile the bearer token is read from — a path, never the token.
caPath/var/run/…/serviceaccount/ca.crtFile the CA cert pinning the API server’s TLS is read from.
apiServerUrlin-clusterThe K8s API server URL — https://kubernetes.default.svc when unset. Requires authToken + caCert.
authTokenin-clusterBearer token for the API server — read from tokenPath when unset. Requires apiServerUrl + caCert.
caCertin-clusterPEM-encoded CA cert for the API server’s TLS — read from caPath when unset. Requires apiServerUrl + authToken.
tokenReloadIntervalMs60000How long a credential read from the ServiceAccount mount is reused before the token file is checked again. No effect on an explicit authToken.
operationTimeoutMs10000Ceiling on one API-server request — wall-clock, over the whole exchange, so a TLS handshake the server never completes is bounded too.
leaseNameMaxLength253Longest object name sent unchanged; a longer name is truncated deterministically. See below.

For pods running in-cluster you only need name — the framework reads the API URL, token, CA cert and namespace from the standard locations. Set namespace explicitly to address a Lease object in a different namespace than the one the Pod runs in.

For tests / dev pointing at a local K8s API (kind, minikube), override apiServerUrl + authToken + caCert.

Every field above except the three credential values and the two test seams can come from configuration instead, under actor-ts.coordination:

actor-ts.coordination.lease {
ttl = 30s
kubernetes {
namespace = "actors"
operation-timeout = 10s
}
}

Explicit options still win, then this block, then the built-in defaults. A lease built in code may have no ActorSystem in scope, so the constructor loads the config chain itself — the files the ACTOR_TS_* variables name or the working directory holds, and a .env, are honoured exactly as ActorSystem.create honours them — unless you hand it a Config: new KubernetesLease(options, system.config) reads that one and nothing else, overrides included.

The framework can also build this lease itself. With backend = kubernetes in the same block, a cluster singleton, a sharded type’s coordinator or lease-majority asks for one by switch, and gets a lease named after the consumer and owned by the node, tuned by this block as read from the node’s own configuration — see leases the framework builds.

There is deliberately no key for authToken or caCert: a bearer token does not belong in a file that gets committed, logged, or rendered in the DevTools config panel. token-path / ca-path / namespace-path are the supported way to move that half out of code — indirection to a file, never the secret itself — and they name the in-cluster source only, so redirecting one while also supplying an explicit apiServerUrl is rejected.

The three connection fields are one credential

Section titled “The three connection fields are one credential”

They are all-or-nothing: supply all three, or none of them. A partial set throws OptionsError at construction.

const partialOptions = KubernetesLeaseOptions.create()
.withName('my-singleton-lease')
.withOwner(process.env.POD_NAME!)
.withTtlMs(30_000)
.withNamespace('my-app')
.withApiServerUrl('https://k8s.example.internal');
new KubernetesLease(partialOptions);
// OptionsError: KubernetesLeaseOptions: authToken + caCert must be supplied
// together with apiServerUrl — explicit API-server credentials are
// all-or-nothing

Each field used to fall back to the in-cluster mount on its own, so naming an apiServerUrl and nothing else sent the pod’s own ServiceAccount token to that host. The TLS pin limited the damage — the target still had to present a chain to the cluster CA — but a cluster credential should never travel to an address it was not issued for.

apiServerUrl must use https. The client dials node:https regardless of what the URL says, so an http:// URL never produced a plaintext connection; it produced a confusing one.

The two sources are also treated differently after they are read, because only one of them can go stale.

An explicit authToken is read once and reused for the process lifetime. There is nowhere to re-read it from — it came from the constructor — so ageing it out would only replace it with itself.

The pod’s mounted ServiceAccount token is bounded: the projected token carries an expiry, and the kubelet rewrites the file as it rotates. A mounted credential is therefore reused for at most tokenReloadIntervalMs, after which the token file’s mtime decides between another interval on the same bytes and a fresh read — so the steady state costs one stat per interval per lease rather than three file reads.

const kubernetesLeaseOptions = KubernetesLeaseOptions.create()
.withName('my-singleton-lease')
.withOwner(process.env.POD_NAME!)
.withTtlMs(30_000)
.withNamespace('my-app')
.withTokenReloadIntervalMs(30_000);

On top of the interval, a 401 or 403 from the API server invalidates the cached credential: the mount is read again and the request retried exactly once before anything is reported as lease loss. An explicit token is never retried that way — re-sending it would only double the traffic on a path that is already failing.

Memoising the mounted token for the process lifetime is what this replaces, and the failure mode was not subtle. The first 401 fired onLost, ClusterSingleton stopped the child and re-acquired on the same lease instance every five seconds, and every attempt replayed the same dead bearer token — so the singleton stayed down until the pod was restarted, and so did every replica old enough to have hit the same expiry.

The API server validates a Lease object’s name as a DNS-1123 subdomain, so a lease named after something the application owns — an entity id, a tenant key, a composed singleton path — can be longer than it will accept. A name past leaseNameMaxLength keeps as much of its head as fits and gains a hash of the whole original:

// 'singleton-<300 chars>' becomes 'singleton-xxx…-1f4k9zq'

The hash is a plain FNV-1a of the full name, so it is a function of the name and nothing else: two pods must derive the same object name from the same lease, or each would acquire its own record and both believe they held it. It also means two names that differ only past the cut land on different objects, which a plain slice would not.

Before this the over-long name was simply sent, and came back as an opaque K8sLeaseError from the first GET — at the moment the singleton was trying to start.

Required fields are enforced at construction

Section titled “Required fields are enforced at construction”

name, owner and ttlMs have no default. The constructor throws OptionsError when one of them is missing, before a single request reaches the API server:

const incompleteOptions = KubernetesLeaseOptions.create()
.withName('my-singleton-lease')
.withNamespace('my-app');
new KubernetesLease(incompleteOptions);
// OptionsError: KubernetesLeaseOptions: owner is required

The check is not cosmetic. Without owner the Lease object is written with no spec.holderIdentity — the undefined key simply drops out of the JSON body — and an unowned lease reads as free to every pod: every acquire() returns true, and the single-holder guarantee is gone without one error being logged. A missing ttlMs produces the same outcome by a different route, since the expiry it computes is NaN and NaN is never later than now.

namespace is checked one step later, at the first API call, because unset now means “read it from the Pod’s ServiceAccount mount” and that read is asynchronous. The guard did not go away — it moved:

// Error: KubernetesLease: no namespace available. Either set `namespace`
// (in code or as actor-ts.coordination.lease.kubernetes.namespace), or run
// inside a Pod whose ServiceAccount mount carries one.

The distinction from the three above is the one that matters. Those three, left unset, disable mutual exclusion silently, so the constructor is the only place they can be caught. A missing namespace stops every request with the error above.

The pod’s ServiceAccount needs permission to manage Lease resources:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: actor-ts-lease-holder
namespace: my-app
rules:
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: actor-ts-lease-holder
namespace: my-app
subjects:
- kind: ServiceAccount
name: actor-ts
roleBinding:
kind: Role
name: actor-ts-lease-holder
apiGroup: rbac.authorization.k8s.io

Without these, acquire() rejects with 403 (forbidden).

Without delete, release() works but leaves the Lease object behind after release (harmless; the next acquire reuses it).

The first acquire() call creates a Lease object:

Terminal window
$ kubectl get lease -n my-app
NAME HOLDER AGE
my-singleton-lease pod-abc-1 30s

The framework writes:

  • metadata.name — the lease name.
  • spec.holderIdentity — the owner.
  • spec.acquireTime — when this owner took it.
  • spec.renewTime — last renewal (updated every renewalIntervalMs).
  • spec.leaseDurationSeconds — derived from ttlMs.

Other holders check renewTime + leaseDurationSeconds < now() to decide whether the current holder is stale — but never at face value, because both fields were written by the holder they are being used to judge:

  • leaseDurationSeconds counts for at most 4 × the challenger’s own ttlMs. The factor is deliberately generous rather than a straight clamp to ttlMs: during a rolling upgrade that raises the TTL, a node still running the smaller value would otherwise declare a live holder expired and take the lease from it.
  • A renewTime further ahead than one ttlMs is not credible from a holder with a working clock, and counts as expired. Believing it is what lets one write wedge the lease for good.
  • A missing or unparseable renewTime counts as live. For an owned record with no usable timestamp, “someone holds this” is the safe reading.

So a corrupt or hostile Lease object costs at most 4 × ttlMs of unavailability instead of pinning the lease indefinitely. Configure the same ttlMs on every pod that competes for one lease, and the tolerance never comes into play.

no

yes

this owner already holds

another holder, still fresh

another holder, stale

acquire

GET the lease object

exists?

CREATE with this owner
if 409 conflict — retry

check holder + renewTime

who holds it?

return true — idempotent

return false — contention

CAS — replace owner if
renewTime matches

The atomicity comes from K8s’s optimistic-concurrency CAS via resourceVersion — two simultaneous attempts to claim a stale lease produce one winner.

While holding, the framework re-PUTs the whole Lease object with a bumped spec.renewTime every renewalIntervalMs:

PUT /apis/coordination.k8s.io/v1/namespaces/<ns>/leases/<name>
{
metadata: { resourceVersion: "148302", ... }, // echoed back for the CAS
spec: { holderIdentity: "pod-abc-1", leaseDurationSeconds: 30,
renewTime: "2025-05-13T12:00:00.000Z" }
}

The resourceVersion from the last read turns the write into an optimistic-concurrency compare-and-set — K8s rejects with 409 if anyone else mutated the object since.

At most one renewal PUT is ever on the wire. renewalIntervalMs is a wall-clock cadence, not a per-request budget: the default is ttlMs / 3 — 5 s at the 15 s TTL recommended above — while the HTTP client’s own timeout is 10 s, so a single request may legitimately span two ticks. That timeout is a wall-clock deadline held by an abort signal, not a socket timeout: Bun never arms the socket one while a TLS handshake is pending, and a holder whose API server stalled mid-handshake used to keep its renewal wedged on one request instead of losing the lease (#1529). A tick that finds one still outstanding is skipped, not queued: a renewal carries nothing the next one will not carry. Without that guard the two writes are built from the same snapshot and carry the same resourceVersion, and the API server rejects one of the holder’s own writes — see below (#761).

If the write fails, renewal gives up immediately and fires onLost — there is no retry budget inside the loop, with two exceptions:

  • Transient (5xx, connection refused, timeout) → fire onLost.
  • CAS conflict (409) or 404 → the object is re-read before ownership is given up, because a 409 only says the resourceVersion moved, not that anyone else took the lease. What the re-read finds decides:
    • gone → someone deleted the object; fire onLost.
    • a different holderIdentity → a real takeover; fire onLost.
    • still this owner → nothing was lost. The server’s resourceVersion is adopted so the next tick’s CAS matches, and the loop keeps holding.
  • 401 / 403 → the credential was rejected, not the lease. For a mounted ServiceAccount token the mount is read again and the PUT retried exactly once; only a second rejection fires onLost. See Credential lifetime depends on the source above.

A re-read that itself fails falls through to the transient case and fires onLost: ownership that cannot be confirmed may not be assumed.

onLost fires when:

  • A renewal PUT is rejected and the re-read shows a different holderIdentity, or no lease object at all.
  • A renewal tick finds ttlMs already gone by since the last write the API server accepted was sent — the event loop stalled, a network partition kept renewals out, or the previous renewal PUT is still hanging on the wire. No PUT is sent: another pod may hold the lease by now (#937).
  • A renewal fails outright (network error, a credential the API server keeps rejecting).

onLost does not fire on a CAS conflict whose re-read still names this owner. That is the holder conflicting with one of its own writes — another controller bumping the object, or a re-acquire() racing a renewal — and the record is still ours.

checkAlive() turns false at the TTL deadline itself, without waiting for a tick. The deadline is judged on two clocks and passes when either says so: the monotonic one, which NTP cannot step back, and the wall clock, which still counts the time a suspended host lost.

The handler should drop ownership state immediately — see Lease API for the contract.

Each lease holder generates:

  • 1 GET + (potentially) 1 CREATE on acquire.
  • 1 PUT every renewalIntervalMs while holding — and 1 extra GET on the rare tick whose PUT is rejected, to find out whether the lease was actually lost.
  • 1 DELETE on release.

For a 30-second TTL with 10-second renewal, that’s ~6 API calls per minute per lease. Pennies on any modest K8s deployment.

For clusters with many leases (e.g., one per sharded entity type

  • one per singleton + one per coordinator), the API server load is still negligible — K8s easily handles thousands of Lease writes per second.

For integration tests with a real K8s API (kind, minikube, ephemeral CI clusters):

import { randomUuid } from 'actor-ts';
const kubernetesLeaseOptions = KubernetesLeaseOptions.create()
.withName('test-lease-' + randomUuid())
.withOwner('test-runner')
.withTtlMs(5_000)
.withApiServerUrl('https://localhost:8443')
.withAuthToken(fs.readFileSync('./test-token', 'utf-8'))
.withCaCert(fs.readFileSync('./test-ca.crt', 'utf-8'))
.withNamespace('test');
const lease = new KubernetesLease(
kubernetesLeaseOptions,
);
await lease.acquire();
expect(lease.checkAlive()).toBe(true);
await lease.release();

Use unique lease names per test (random UUID suffix) so parallel tests don’t fight. Tear down with release() + a final delete sweep in test teardown.