Skip to content
English

Cluster security

The cluster transport defaults to plain TCP without authentication — fast, simple, fine for a private network. Not fine for any cluster that crosses an untrusted boundary (the public internet, multi-tenant kubernetes, cross-region links without VPN).

This page covers the production-security setup for the cluster transport.

ConcernHow to address
Eavesdropping — peer-to-peer traffic readable by anyone on the wire.TLS on the cluster transport.
Unauthorized joins — a malicious node connects + becomes a cluster member.Mutual TLS (mTLS) — peer certificate verification.

Enable TLS with mutual authentication (mTLS) for any external-facing cluster — it addresses both.

import { TcpTransport, NodeAddress, Cluster, ClusterOptions } from 'actor-ts/cluster';
import fs from 'node:fs';
const transport = new TcpTransport(
NodeAddress.parse('my-app@10.0.0.5:2552'),
system.log,
{
tls: {
cert: fs.readFileSync('./tls/cluster.crt'),
key: fs.readFileSync('./tls/cluster.key'),
ca: fs.readFileSync('./tls/ca.crt'),
rejectUnauthorized: true, // verify peer certs
},
},
);
const clusterOptions = ClusterOptions.create()
.withHost('10.0.0.5')
.withPort(2552)
.withSeeds([...])
.withTransport(transport);
await Cluster.join(system, clusterOptions);

The TLS settings:

  • cert + key — this node’s certificate + private key. Both carry the material itself, not a path to it: nothing in the transport reads from disk, and none of the three runtimes accepts a filename in these fields either. Load it yourself, as the example above does. Both halves are required together on a listener — see the refusals below.
  • ca — trusted CA bundle. Use to verify peers’ certificates.
  • rejectUnauthorized: true — fail handshakes where the peer’s cert isn’t signed by ca.
  • requestClientCert — whether the listener demands a certificate from whoever connects to it. You will not normally set this: it defaults to true whenever ca is present, since a trust bundle on a cluster listener has no other purpose.

With a shared ca, the cluster is mutually authenticated — every connection requires a peer cert signed by the trusted CA, in both directions.

mTLS answers “may this peer be in the cluster”. On its own it never answered “is this peer the node it claims to be”: the hello frame carries an address and no credential, so one CA-signed node could announce itself under another member’s address. That matters because the gossip-authority rules below all key off the connection’s peer — they are exactly as strong as the identity underneath them.

So when a peer presents a certificate, the address it claims in hello must be one the certificate vouches for. A claim is accepted when the CN or a SAN covers either:

  • the address’s host — the ordinary case, where each node’s certificate carries its own hostname or IP, or
  • the full systemName@host — for deployments that mint a per-node identity and want the tighter binding.

Wildcards are honoured for the host, in the leftmost label only, exactly as TLS hostname verification does.

Nothing changes for a cluster with no peer certificate to read: plain TCP, one-way TLS, and a Deno listener (which cannot report one at all) behave exactly as before. The check strengthens mTLS deployments rather than adding a switch that can be left off.

Four configurations are refused at bind time rather than started in a weaker state than they read as:

  • An incomplete server credential — cert without key, key without cert, or a tls object carrying neither (a ca alone says which peers to trust when dialling; it gives the listener nothing to present). Empty counts as absent, which is what an unset environment variable or a mis-mounted secret looks like by the time it arrives.
  • requestClientCert: true with no ca — there would be nothing to validate peer certificates against.
  • A ca that holds no certificate — an empty or blank string, an empty byte array or array, null — whatever requestClientCert says. That is what a missing secret looks like by the time it arrives, not a way to ask for one-way TLS — requestClientCert: false is that. The HTTP server holds its ca to the same rule, stated once under Client certificates.
  • An mTLS listener on Deno — Deno.listenTls takes only a cert and a key, with no way to request or verify a client certificate, so the listener would authenticate nobody.

A Deno node can still join an mTLS cluster: it presents its own cert / key when dialling, so the listener (on Node.js or Bun) authenticates it like any other peer. Only hosting the listener is unavailable — in a mixed deployment, keep the seed nodes on Node.js or Bun.

One Deno difference worth knowing: rejectUnauthorized has no equivalent there and is not mapped. Deno always validates the chain, so reaching a self-signed peer means supplying its signing CA in ca — which is the shape recommended here anyway.

Three approaches:

Terminal window
openssl req -x509 -newkey rsa:4096 -keyout cluster.key -out cluster.crt -days 365 -nodes -subj "/CN=actor-ts"

Use the same cert + key on every node. Fine for dev / staging. Don’t use in production.

Terminal window
# Create a CA once:
openssl req -x509 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 3650 -nodes -subj "/CN=actor-ts-ca"
# Per-node certs signed by the CA:
openssl req -newkey rsa:4096 -keyout node-1.key -out node-1.csr -nodes -subj "/CN=node-1"
openssl x509 -req -in node-1.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out node-1.crt -days 365

Each node gets its own cert; everyone trusts the CA. Rotating certs is per-node and doesn’t require touching the CA.

For K8s deployments, use cert-manager with an internal CA or HashiCorp Vault. Certs are mounted as volume secrets; rotation handled by the cert manager.

# Example cert-manager Certificate spec:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: actor-ts-cluster
spec:
secretName: actor-ts-cluster-tls
issuerRef:
name: actor-ts-ca
kind: ClusterIssuer
commonName: actor-ts
dnsNames:
- actor-ts-cluster.svc
duration: 8760h
renewBefore: 720h

The pod mounts the secret as files; the actor reads them.

Cluster port (2552) — internal-only:
- pods can talk to pods on 2552
- not exposed via Service / Ingress
- LoadBalancer never sees it

Even with TLS + auth, expose the cluster port narrowly. A NetworkPolicy in K8s:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: actor-ts-cluster-internal-only
spec:
podSelector:
matchLabels:
app: actor-ts
ingress:
- from:
- podSelector:
matchLabels:
app: actor-ts
ports:
- protocol: TCP
port: 2552

Only app=actor-ts pods can reach port 2552 on each other.

ThreatMitigation
Network eavesdroppingTLS
Man-in-the-middleTLS + cert verification
Unauthorized cluster joinmTLS (CA-signed peer certs)
Insider with stolen certCert rotation + revocation
Malformed or hostile wire frameShape validation at the decode boundary
Frame split into thousands of chunks to multiply decode costLinear decode buffer — every byte copied once, on arrival
Socket that opens, sends half a frame and goes quietStall deadline on an incomplete frame, plus a cap on inbound connections
Sockets opened in bulk that send nothing at all, to fill the inbound capHandshake deadline on every accepted socket, so no slot is held open-ended
Peer growing a node’s registries without boundPer-registry caps on pub-sub and the receptionist
Address claimed before the node that owns it existsTight version-skew cap on every gossiped member version
Gossip frame captured off the wire and played back laterPer-sender frame sequence — a frame must out-number the last one accepted from that sender (partial: only once one has been)
Spoofed discovery answer steering the bootstrapmTLS, plus pinnedAddresses on the seed provider
A ref in a message body naming an internal host, to make this node dial itFrames leave only for members and nodes this node already holds a connection to (partial: without mTLS a peer still becomes a member under an address its own hello names)
Compromised pod inside the clusterApplication-level auth (out of scope here)

The cluster transport handles transport-level security. Application-level concerns (auth between specific actors, per-tenant isolation) remain your job — the cluster transport is trusted within itself.

A seed provider asks an outside party — DNS, the K8s API — which addresses to talk to, and that answer arrives before any certificate has been checked. It cannot let an attacker join the cluster: the address a peer claims must be vouched for by its certificate, so a spoofed seed produces a connection that dies at the handshake.

That guarantee is configuration-dependent, which is the part worth stating plainly. A node only demands a peer certificate if its listener was configured to request one; where TLS is off, or where a half-configured listener never asks, there is nothing to check the claimed address against and a poisoned discovery answer is followed as given.

So pin what the resolver is allowed to say:

const dnsSeedProviderOptions = DnsSeedProviderOptions.create()
.withHostname('actor-ts.example.com')
.withSystemName('my-app')
.withPort(2552)
.withPinnedAddresses(['10.0.0.0/8'])
.withLog((message) => logger.warn(message));

Addresses outside the list never become seeds. Details, including the SRV-mode caveat (targets are hostnames, so pins are suffixes there and the eventual A lookup stays unpinned), are on the DNS seed provider and Kubernetes API seed provider pages.

This is defence in depth: it lowers what a DNS or Endpoints compromise buys, and it is the only discovery-layer control left standing when mTLS is not in place.

Until #860 the snippet above was the only way to reach it: Cluster.bootstrap({ discovery: 'dns' }) and the env-driven ladder built their providers with no pin at all, so the documented mitigation was unreachable on the path most deployments take. It is now a config key, and it ships in the same block as the keys that decide which provider runs:

actor-ts.discovery {
dns.pinned-addresses = ["10.0.0.0/8"]
kubernetes.pinned-addresses = ["10.0.0.0/8"]
}

Set it wherever actor-ts.cluster.bootstrap.discovery.method or service-name is set, and for the same reason both were added together: a config file is a wider input surface than a code call, so making the destination of discovery settable without making the pin settable would have widened the attack surface and left the guard where it was. A list that cannot match in the configured mode — CIDRs only in SRV mode, a host suffix under Kubernetes — is refused at startup rather than silently discarding every answer, because an empty seed set reads as “we are alone” and ends in a one-node cluster.

Frames arrive as JSON and used to be cast to the protocol type rather than checked against it, so a peer could hand a node a value the code then read as if the type were true. Every frame is now validated before any handler sees it, on every transport, the worker mesh included:

  • The frame must be an object carrying a string kind. null, a bare string and a number are refused — previously null alone was an eight-byte remote process kill, and it needed no completed handshake.
  • Node addresses must have a non-empty systemName and host plus a positive integer port. A port that arrives as the string "2552" is the case worth knowing about: it renders identically in every log line and keys every map the same way, but never compares equal — a node that merged its own address in that shape stopped recognising itself.
  • A gossiped member’s status must be one of the seven legal values. An unknown one used to reach an exhaustive match that threw after the member had been stored, so the node crashed and re-gossiped the poisoned entry to its peers.
  • A gossip batch is validated whole. One malformed member refuses the frame, so a bad entry cannot ride in behind a good one.

Two tiers, deliberately: a frame that fails validation is dropped and the connection stays up, because one bad frame should not cost a healthy peer its link. A handler that throws drops the connection — that is the case nobody understands, and it must not escape into the runtime’s socket callback.

The worker mesh takes the first tier and neither half of the second. MessageChannelTransport — the transport a worker thread and the main thread speak over a MessagePort — checks the envelope’s own shape and then runs the payload through the same validateWireFrame the socket transports use, so a malformed frame from a worker never reaches a handler either. WorkerBroker, which relays between threads, checks only the envelope’s two address fields — the payload is the transport’s job — and rewrites from to the port the frame arrived on, so a worker cannot speak for a sibling. The broker also reports what it refuses: a malformed envelope or a port that refuses a frame is a folded WARN — one line per reason per 30 seconds, carrying the count — that names the registered address of the port the most recent frame arrived on and never the payload, and a frame to an address nobody registered is the same line at DEBUG, since that is what every healthy sibling sends for a moment after a crash (#1276). The transport half reports too, on the same pattern: a malformed envelope, a payload the wire checks refuse and a handler that throws each bump transport.dropped() and reach the log as one folded WARN per reason per 30 seconds, carrying the count. Two of the three carry the sender as well; the envelope one does not, and cannot: when the envelope itself is what was refused, no address was read far enough to name, so that line leaves most recently from off entirely — which is what the mesh page says of the same line. The payload one names the reason validateWireFrame gave — which is what tells a version skew from a hostile peer — and none of the three carries the frame (#1515). Before that the transport was the last silent edge on this path: a worker running a skewed version had its frames black-holed with no signal at all, which is worse than before the guard existed, when the frame at least arrived.

The sender is escaped on the way to the log, which the broker’s line does not need to do. Through a broker from is rewritten to the arriving port, so it is host-minted; wired port-to-port there is no broker to do that, and a node address only has to carry non-empty strings — so a host with a newline in it is a well-formed address that would otherwise forge whole log lines.

Frame kinds registered by extensions (sharding, pub-sub, the receptionist, DistributedData, DevTools) pass this layer and validate their own payloads. So does replicated event sourcing, whose event envelopes ride on pub-sub rather than on a frame kind of their own — see what a replica trusts from its peers.

Validation is only half of it. A frame can be perfectly well-formed and still speak for a node that did not send it, and that class of defect is closed the same way everywhere in this codebase: the identity is taken from the connection, not from the payload. A leave may only retire the node that sent it; gossip replaces the sending peer’s contribution and no one else’s; a sharding or singleton directive is honoured only when it arrived wrapped by the per-path envelope handler, which is the one place the authenticated address exists; and a replicated event’s replica is held against the node that published it. A frame that reaches an actor by generic path resolution instead — which strips nothing but does carry no sender — is treated as unauthenticated rather than as trusted.

A CRDT’s own state is the one place that rule does not reach. DistributedData validates a decoded payload for shape and plausibility: a counter slot must be a non-negative integer at or under MAX_COUNTER_SLOT (≈ 2.2e12 — the largest per-slot value for which a full counter’s value() is still an exact integer), a register timestamp may be no more than five minutes ahead of local time, a collection is capped at 4 096 entries, and a __proto__ key is refused. It does not check that the sending peer had any business writing the slot it wrote. Grow-only counters merge by maximum, so any peer able to gossip can raise any replica’s slot inside those bounds — the owning node cannot lower it again, the value is written through to the durable record, and delete(key) is best-effort because a peer still holding the key re-gossips it. A DistributedData counter is therefore trustworthy exactly as far as the set of peers you let onto the wire, which makes the mTLS setup above a prerequisite for a quota or billing counter rather than a hardening step.

On the socket transports every rejection is logged at WARN naming the peer and the offending field, so a dropped frame is diagnosable — a version mismatch and a hostile peer look different in the log. On the worker path both halves of the channel do the same, folded: the broker logs a WARN per reason for what it will not route, and the transport logs one for what it will not deliver; see above.

Validation runs on a whole frame, and a whole frame has to be reassembled from however many TCP chunks the sender chose to split it into. The peer chooses that split, which used to be the expensive half: the decoder rebuilt its accumulator on every chunk, so assembling one frame cost work quadratic in the chunk count. A frame just under the 16 MiB cap, delivered in TCP-sized ~1400-byte writes, is about 12 000 chunks and ≈ 100 GB of memory copying — roughly 6000× amplification on bytes the attacker never had to send, and reachable before the hello gate, so no membership was needed. The decoder now appends into a buffer it grows by doubling, so each byte is copied once on arrival and the amplification is 1×.

Three bounds cover what an unauthenticated socket may still hold while it is connected:

  • An incomplete frame may sit for 30 seconds without another byte arriving before the connection is closed. It is a stall bound, not a budget for the frame: the deadline is re-armed on every chunk, so a peer shipping a large frame over a congested link is never punished for being slow — only for going silent.
  • The handshake itself has 5 seconds, from the moment the socket is accepted, and an accepted socket that has not sent its hello by then is closed. This is the one an incomplete frame does not cover: a socket that sends no bytes at all is not stalled mid-frame, so nothing about it is being tracked. It is the same 5 seconds the dialling side already gives itself, and that clock starts earlier — before the TCP connect and the TLS handshake — so a peer that is still trying has always given up first. Once the hello lands the deadline is gone: an established peer between two gossip rounds is idle by design and is never dropped for it.
  • Inbound connections are capped at 1024. A fully-meshed cluster needs one per peer, so the cap is set far above any realistic topology; what it removes is the ability to multiply the per-connection cost by opening sockets in a loop. The newest connection is refused rather than an established one evicted — eviction would let an attacker push real peers off the node.

The second and third are a pair. A cap is only worth what its slots’ turnover is worth, and without a handshake deadline the cap becomes the attack: 1024 connections that send nothing hold every slot for the life of the process, and every peer and ClusterClient after them is refused. On Bun that is reachable even under mTLS, because a socket’s open callback fires before the TLS handshake completes — the slot would be taken while there is still no certificate to check.

The first and third are what bound inbound decode memory at all: the product of the two, with the per-connection term already capped by actor-ts.remote.max-frame-bytes. Lowering that key is also the lever for a deployment that never sends large envelopes — the resident cost per connection follows it directly.

Bytes and work are not the same bound, and this is where the two part company. An accepted frame goes to JSON.parse and only then to the tagged-tree walker, so the depth ceiling (actor-ts.serialization.read-constraints.max-nesting-depth) reports an over-deep payload after the parse has already built it. Measured on Bun 1.4, one 16 MiB frame of balanced [ costs about 1.0 s of CPU and 260 MiB of heap before the refusal — roughly four times what a legitimate frame of the same size costs, and repeatable by any peer that completed the handshake. Two keys bound it and neither is on by default: remote.max-frame-bytes, and serialization.read-constraints.max-document-bytes, which is checked against the same 4-byte length prefix and refuses the frame before its payload is buffered at all. A pre-parse scan of the raw buffer would close it with neither set; it was measured at 18–28 % of every frame decode and not shipped, for a fourfold amplification (#880).

What a well-formed frame still cannot grow

Section titled “What a well-formed frame still cannot grow”

Shape validation says a frame is readable, not that acting on it is free. Pub-sub gossip is the clean example: a peer naming 100 000 topics it claims subscribers for sends one perfectly legal frame, and the receiver used to allocate 100 000 map entries for it — no local Subscribe, no malformed field, no log line. The receptionist had the same shape on its own gossip path.

Membership itself had the same shape, and it is the one registry nobody has to opt into: every clustered node keeps a map of members, and gossip is what fills it. The rules below decide whether a claim is believable; none of them bounded how many believable claims one peer may make. A sender announcing its own address is waved through by design — refusing that would mean no node could ever join — so naming a fresh address per frame allocated an entry per name.

These registries are capped now, and the caps apply to the gossip path as much as to local calls:

RegistryCapped byDefault
Live cluster memberscluster.max-members1000
removed tombstonescluster.max-tombstones10000
Pub-sub subscribers per topiccluster.pub-sub.max-subscribers-per-topic10000
Pub-sub topicscluster.pub-sub.max-topics10000
Pub-sub remote claimants per topiccluster.pub-sub.max-remote-nodes-per-topic1000
Receptionist subscribers per keycluster.receptionist.max-subscribers-per-key1000
Receptionist subscriptions totalcluster.receptionist.max-subscriptions-total10000

A claim over a cap is dropped and logged rather than refused on the wire — the frame is well-formed, and dropping the connection over it would let one noisy peer cost a healthy link. A local Subscribe over a cap is answered with SubscribeRejected instead, because the caller is in a position to do something about it. Both pub-sub registries also watch their subscribers, so the other half of the old growth — refs that stopped without ever unsubscribing — is reclaimed rather than capped.

Lower the defaults for a deployment where the legitimate numbers are far below them: a cap is only a bound on the damage, and one set far above real usage bounds very little.

Splitting members from tombstones is not tidiness: the two kinds of phantom have different lifetimes. A phantom member in up / joining / unreachable that arrives as a record in a gossip frame is a member the failure detector watches from the moment it is merged, and nothing ever heartbeats for it, so both detectors declare it down — φ measuring the silence from that moment. What that verdict does is the downing configuration’s call. Under the default — no strategy, failure-detector.auto-down = on — the detector removes it, down-after after the attacker stops feeding it, and for good only once no other node still carries it, because that route deletes without a tombstone and a node that has not deleted the phantom yet gossips it back. With a strategy configured, or under auto-down = off, the phantom is parked at unreachable like any silent peer and holds its live slot until a strategy or an operator removes it. A frame’s sender field is no longer a route at all. A from naming an address this node held no record of used to be filed as joining with no authority check and promoted like any joining member — this node’s leader, if its address sorted first, and an address the heartbeat and gossip ticks then dialled. That door is closed (issue #939): a sender is introduced by its own record, merged like any other, and every gossip frame carries one. max-members is therefore the one bound on live phantoms, and the merge the one way in. A record gossiped as removed is watched by nothing: only the tombstone prune reclaims it, once the larger of cluster.tombstone.time-to-live and tombstone.min-retention has passed — a day at the defaults. Two residues with two lifetimes, so two caps. Refusing a tombstone costs nothing, because a tombstone for an address this node holds no record of suppresses nothing that exists — while a refused live record costs a legitimate member one gossip round, and comes with a WARN naming the cap.

Both caps are charged on the bucket a record moves into, not on whether it creates an entry. A gossiped record that keeps an entry where it already is — up → unreachable, or a newer tombstone over an older one — is a free in-place update. One that crosses between the buckets is charged to the bucket it enters: a tombstone re-incarnated as up needs room among the live members, a live member gossiped as removed needs room among the tombstones. Charging only entry creation let the two caps trade headroom with each other for free — a re-incarnation vacated the tombstone bucket without giving up a map slot, so the next flood of tombstones was admitted too, and alternating the two grew the map without bound while respecting both caps at every individual step. A refused conversion leaves the member live, where it goes the way any silent member does: removed by the detector at down-after under the default, or — with a strategy configured or under auto-down = off — parked at unreachable until a strategy or an operator removes it.

Tombstones this node mints itself — a peer’s leave, a downing decision, an operator down() — convert a record it already holds and are never subject to the cap. Capping its own bookkeeping would drop the suppression that stops stale gossip resurrecting an evicted address, which is a liveness bug wearing a security fix’s clothes.

The ceiling worth knowing is not the heap. Gossip carries the whole member list, so at roughly 110 000 entries a node’s own frame outgrows remote.max-frame-bytes and every peer terminates the connection on the length prefix — the node evicts itself from the cluster while still running, long before anything runs out of memory.

The failure detector is held to the same bound. It keeps state per peer, and it used to create that state for every connection identity a frame arrived under — without mTLS, a name the peer picks in its hello. A peer reconnecting under fresh names grew it by an entry per name while the member map held at its cap: 500 names left 502 entries beside 3 members, each carrying a sample window under the φ-accrual detector. Liveness now reaches the detector only for a member that is not a tombstone, or a configured seed that is not one either. Every route that admits a member registers it, and every route that takes it out of the live set forgets its entry — a tombstone this node mints, a tombstone a gossiped record brings, and a deletion from the member map alike. So the detector holds an entry per live member and at most one per seed, bounded by max-members and the seed list (#1103).

Passing the shape check does not make a frame believable. Gossip merges used to be decided purely by version magnitude, and versions are seeded from Date.now() — so an attacker could always pick a winning number and rewrite any member’s status, including the receiving node’s own. Two rules now sit in front of the merge:

  • Nobody downgrades us. A claim about this node’s own address is refused. The one exception is promotion out of joining/weakly-up into up, which has to come from outside because it is the leader’s decision — and which is harmless to accept, since a joining node is already trying to become up. What bounds it is how little it carries: the status, and a version limited only by the maxVersionSkewMs cap (5 minutes by default, see A version cannot claim an address in advance below), so a stranger can date this node’s record up to that far ahead — clamping the version is tracked in #1797. Nothing else crosses over. The promotion asks nothing of its sender, which is why this node has to keep its own roles, incarnation, storage identities, configuration facts and app version over whatever the record says. Before #1104 the roles came from the record — and so did the storage identities of a node that had resolved none of its own — so one frame from a stranger during the joining window chose what this node hosts and places. Everything else about our own record we decide ourselves.
  • Third-party claims need a sender with standing. Saying something about another node requires the connection’s peer to be a member this node already considers active. A sender may always announce its own record — that is how joining works.

Both rules key on the connection’s peer, not on the payload’s from field, since that field is the one thing an attacker fully controls. The same reasoning covers three neighbours. A leave is only accepted from the node that is leaving. A heartbeat refreshes the failure detector for the peer that sent it rather than the address it names — that one previously let a peer keep a dead node looking healthy, and made the receiver dial an attacker-chosen host. And a ClusterClient ask is answered down the connection the envelope arrived on: routing on the payload let any connected party choose the address this node replied to, and opened a connection to.

The client envelope no longer carries a sender field at all — after the rule above it had exactly one correct value, which is the one the receiver already holds. An envelope that still names an address other than its own connection is counted on cluster_envelope_from_mismatch_total{frame} and otherwise ignored; see ClusterClient.

Unreachability is deliberately not covered by these rules: “I cannot reach C” is inherently a third-party observation, and every node must converge on the same view before a downing provider decides. Refusing those claims would leave each node with only its own reachability picture.

The first rule means no peer can tell this node it was removed — and a node the rest of the cluster downed while it was paused or cut off used to go on running everything it hosted. So a peer that holds this node’s run as removed answers its heartbeats with a membership-refused frame instead of an acknowledgment, naming that run with its incarnation (#940). The frame is taken as a statement about the sender only: “I will not have you back”. The receiver parks that peer at unreachable and stops letting its heartbeats bring it back; its own record stays untouched, and the verdict stays with its own downing strategy. Once every peer has refused it, the view counts as settled and the strategy is asked at once rather than after stable-after.

So a forged refusal buys what dropping that peer’s traffic already buys — the peer looks unreachable — and no more: the real peer’s gossip, which lists this node as a member, withdraws it, and a refusal naming another run at this address, or none, is ignored. Forging one for every peer makes the node’s strategy decide on a view in which it is alone; that is the reach of cutting all of its traffic, and the reason the caution below applies here too.

The rules above bound what a frame may claim. One more bounds what it may reach: an envelope carries the target path the sender chose, and the node resolves it against its own actor tree. That resolution used to walk from the root, so /system/… was exactly as reachable as /user/… and any peer that completed a hello could deliver straight to a shard coordinator, a singleton manager or the pub-sub mediator — bypassing the per-path handler each of them registers, and with it the sender attribution the rules above are built on.

The /system subtree is now refused on that path, in every configuration and at both seams — the cluster wire and the ClusterClientReceptionist. Framework actors stay reachable through their registered handlers, which is the door that knows which connection a frame came in on. Nothing an operator can configure opens the old route back up.

The /user half is a deployment’s own decision, because a peer naming an application actor is how an ActorRef works across nodes. Narrow it where the cluster network crosses a boundary you do not control:

actor-ts.remote {
untrusted-mode = true
trusted-selection-paths = ["/user/orders/*", "/user/reporting/intake"]
}

An entry is an exact path unless it ends in /*, which makes it that path and its subtree — segment-anchored, so /user/orders/* does not admit /user/orders-archive. Refusals are counted on cluster_envelope_refusals_total{reason, frame} and folded into at most one WARN per reason every 30 seconds, naming the connection’s peer and never the requested path; a line per refusal would hand the peer log amplification instead. Watch reason="not-allow-listed" on the day you turn the mode on — it is where an incomplete allow-list shows up. See configuration.

The same policy guards the one other frame in which a peer names a path of its choosing: a watch. A refused path is answered like a missing one — Terminated with existenceConfirmed = false — so a watcher never hangs and the answer reveals nothing the refusal did not already; the refusal is counted under frame="watch".

A ref in a message cannot choose where this node dials

Section titled “A ref in a message cannot choose where this node dials”

The other direction has the same shape. An ActorRef in a message body crosses the wire as a marker naming its node — system, host, port — and the receiver rebuilds a remote ref from it. That address is the sender’s claim. The marker used to be rebuilt without even the validation every other wire address gets, and whatever node a ref named was dialled the moment it was used: an actor answering a decoded replyTo, the idiom every handler follows, opened a connection to an address of the sender’s choosing and wrote a frame to it — a cloud metadata endpoint, an internal admin port. Watching the ref did the same with a watch frame (#936).

A marker is now validated as a node address, and one that is not valid decodes to Nobody. And a frame that carries a user payload or a death-watch message — every send that is not the membership protocol’s own — leaves only for:

  • a member that is not a tombstone, or
  • a node the transport already holds a connection to, acked or still in its handshake — sending over an existing connection dials nothing. It is how a reply reaches a node that has joined but that this node has not heard of yet, and how a tell to a seed straight after Cluster.join resolves is queued on the connection the join opened instead of dropped. A custom transport that does not implement the optional holdsConnectionTo is asked through peers() instead, which knows acked connections only, so on it that tell is refused until the handshake lands.

Anything else is dropped, as a frame to an unreachable node is, counted on cluster_outbound_refusals_total{frame} (envelope or wire) and folded into at most one WARN every 30 seconds that names the most recent destination, escaped. The commonest cause on a healthy cluster is a node that has just left. A ref is rebuilt lazily, so one to a node this node learns of a moment later still works when it is used. Watching a ref whose node is refused is answered at once with Terminated and existenceConfirmed false, as a refused watch on the far side is, so the watcher does not wait for an answer that cannot come.

The rule is only as strong as membership. Since #939 a gossip frame’s payload from no longer files a member: a sender is introduced only by its own record, merged as a self-announcement held to the identity its connection presented. Without mTLS that identity is still the peer’s own choice, so a peer that can connect can announce itself under an address it picks — and from then on this gate lets frames through to it, and the membership protocol’s own gossip and heartbeats dial it every interval with no ref involved at all. So this closes the route through a message body, not the minting of a member. mTLS is what binds a member’s address to a certificate.

A sharding directive needs the coordinator behind it

Section titled “A sharding directive needs the coordinator behind it”

The same reasoning reaches one layer up, into the extensions whose frames pass the wire edge and validate their own payloads. A ShardRegion used to treat any message whose kind began with sharding. as a framework directive — including the five only the coordinator may issue. HandOff stops every entity under a shard; ShardHome moves ownership; RememberedEntities pre-creates entities; ShardMapUpdate publishes an allocation map to every local subscriber, DevTools panel and application listener included. One small frame from anyone who had completed hello did any of that, repeatably.

The region could not have checked: sharding registered no per-path envelope handler, so an inbound frame reached the actor through generic path resolution, which delivers with no sender at all. It now claims its own path on the envelope router, which hands the handler the connection’s peer, and wraps the frame in a class instance before putting it in its own mailbox — a shape a JSON wire body cannot mint, so the directive arms can tell a routed frame from a forged one. A tagged { kind } object would not have been enough: it is reproducible verbatim from a payload.

Both halves are load-bearing. Without the wrapper check, a frame addressed non-canonically — a trailing slash, a doubled separator — misses the exact-string handler lookup, still resolves to the same region through the actor tree, and arrives unwrapped. And without the origin check, an authenticated peer is any cluster member, so every member could issue directives to every region.

Refused frames are dropped and logged at WARN. The coordinator’s own local leg builds the same wrapper, so a single-node cluster rebalances exactly as before.

The same gap ran in the other direction, and it was the more expensive one. Every message a ShardCoordinator accepts is a claim about the sender’s own node: which shards its region hosts, that its region is gone, that a hand-off finished, where to send a shard home or a statistics reply. The coordinator read all of that out of the payload. Its only gate was isLeader() plus, with a lease, holding the lease — which answers “am I the authoritative coordinator?”, never “may this sender speak for that region?”.

So one well-formed sharding.Register naming somebody else’s address seized every shard of a type, and one sharding.RegionTerminated evicted its region. The eviction is the worse half: it sends no HandOff, so the victim keeps its shard actors and its entities running. The result is two owners for one live shard — the same entity id instantiated twice, and for a persistent entity two writers on one persistenceId.

The region-side origin check above does not blunt this, because the attacker never sends a ShardHome. It poisons the coordinator’s allocation map with one forged frame and the genuine coordinator then emits the redirect itself, from the leader’s own node, inside a wrapper every region rightly accepts. One forgery, every subsequent hop authentic.

The coordinator now claims its own well-known path on the envelope router and applies the same two conditions: the frame arrived inside the wrapper, and the address its payload names is the peer’s own. Both directions of the register loop are therefore attributed, which also fixed how a region addresses the coordinator — by the leader’s system name, not its own, since an actor path carries the system it belongs to.

hostedShards is bounded and capped on top of that. It was the only caller-sized input the coordinator had — one Register wrote an allocation entry per array element, with no range check and no length cap, into state that is broadcast to every region and persisted to the coordinator-state store, so the growth survived restarts. Entries outside 0 .. numShards - 1 are now dropped and the accepted set cannot exceed numShards.

A version cannot claim an address in advance

Section titled “A version cannot claim an address in advance”

Version is a logical clock seeded from Date.now(), so “highest version wins” also decides what happens the first time an address is mentioned at all. That made an address claimable before the node owning it exists. A stranger announces itself under the address the next pod is about to get — announcing your own record is the claim the rules above never refuse — dates it close to the 24 h skew cap, and attaches whatever roles it likes. The leader’s promotion loop lifts the record into the active set, and the node that really owns the address loses every merge afterwards, because it seeds its version from its own clock and that is lower. Roles are what routing, sharding placement, singleton hosting and downing quorums are computed from, so the phantom is not a cosmetic row in a member list.

A gossiped member version is therefore held to a tight clock-skew budget — 5 minutes by default:

const clusterOptions = ClusterOptions.create()
.withHost('10.0.0.5')
.withPort(2552)
.withMaxVersionSkewMs(30 * 60 * 1000);

The budget applies to every merge, not only to the record that introduces an address. It was originally the narrower rule — a tight cap on a first sighting, a generous 24 h one on every update — and that split could be stepped around by introducing the address first: two records for the same address in one frame, or — until #939 removed the door that filed a frame’s self-declared sender — a frame with no member records at all. Any rule that lets a record earn the wider budget fails the same way one step later, because without peer certificates every step of the earning is something the attacker can produce.

Raise it for a deployment whose clocks are known to run loose; 24 * 60 * 60 * 1000 restores the old single-cap behaviour. A refusal is exclusion, and it is durable: a node is introduced only by its own record, so a node whose clock runs further ahead than the budget is not filed at all. It stays out of the member list — with the WARN below naming the cap — until its clock comes back inside the budget. That was always this cap’s verdict on such a node; what changed is that the verdict now sticks instead of being softened by the sender door or reversed by the node’s second gossip frame.

Refused records are reported once per frame rather than once per record — a WARN naming the peer and the count, and the counter cluster_gossip_records_refused_total{reason}, whose reason label is closed at version-skew, map-cap, timestamp-skew, replayed-frame, self-claim, removed-run and foreign-cluster. Logging per record would hand a peer log amplification in place of the growth it just lost.

removed-run is a record for a run this node holds as removed — the same address and the same incarnation. A version is the sender’s to choose and could always be restamped past a tombstone; an incarnation names the process the cluster removed, and only a restart mints a new one. Since cluster wire v2 every address on a frame carries it, so the run the cluster downed cannot come back under any version, while a restart at the same address still joins at once (#823, #940).

self-claim is the rule above, seen from the counter: a peer asserting a status for the receiving node that is neither the one it already holds nor the leader promoting it to up. A peer echoing the status the node does hold is the ordinary content of every gossip round — its own record travels back to it in every frame — so that one is refused without being counted or logged, and the counter still reads zero on a healthy cluster.

A recorded frame cannot be played back to a receiver that has heard from its sender

Section titled “A recorded frame cannot be played back to a receiver that has heard from its sender”

A gossip frame is a snapshot of the member map, and a member’s version only moves when its status does — so a frame captured off the wire stays valid indefinitely. Against a converged receiver that costs nothing: every record in it loses the “higher version wins” comparison to the record already on file. What makes it an exploit is an entry the receiver has deleted, and two paths delete one. The failure detector’s own removal — the default with no strategy configured, under failure-detector.auto-down = on — deletes outright rather than leaving a tombstone. That was once justified as letting a healed partition re-discover the peer, and deleting buys no such thing: neither side heartbeats or gossips a peer it holds unreachable, and a deleted one is not in its map to address at all, so after a symmetric partition neither side hears from the other again (issue #930). Under the default, both sides of a real partition therefore remove each other and run on as two clusters, which is one more reason a production deployment configures lease-majority or another downing strategy. And an expired tombstone is pruned once the larger of its time-to-live and tombstone.min-retention has passed. Either way there is nothing left to compare against, and the branch that files a first sighting has no lower version bound at all. The other removals — an operator’s down(), a strategy’s decision, a peer’s leave — leave a tombstone that out-versions the recording until that prune; with a strategy configured or under auto-down = off, the detector itself removes no one. A restart that replaces a live record of its previous run deletes that record too, but the new run’s record takes the key at a higher version in the same step, so a recording of the old run still loses the comparison.

So replaying a downed member’s own pre-down record brought it back: same address, same version, up, carrying the roles it had. Shard placement and singleton hosting follow the roles, so the cluster starts routing work to a node it had evicted.

Every gossip frame therefore carries a sequence its author stamps — seeded from that node’s wall clock at startup and incremented once per frame — and a receiver remembers the highest one it has accepted from each connection peer. A frame that does not out-number that mark is dropped whole, before any record in it is looked at. The comparison itself needs no knob: it is between a peer and itself, so no clock-skew budget comes into it.

The mark is written in exactly one place — the gossip path, as it accepts a frame — so it exists for a peer this node has accepted a frame from, and for no other address. That is the bound, and the third residual below is what falls outside it.

A sequence must also be plausible — a finite number no further ahead of the receiver’s clock than maxVersionSkewMs, the same budget a gossiped version gets, because the number is stamped from the author’s wall clock. A frame outside that budget is refused, exactly like a repeat.

That started out as the weaker rule: the frame was merged, and only adopting its number as the mark was refused, on the argument that a frame numbered Number.MAX_SAFE_INTEGER cannot be a recording of a real frame. Only the sequence is fabricated in that attack. The members array is still the recording, and nothing on this wire binds the two together — so a captured frame with one field rewritten was merged, left the mark untouched, and was therefore merged again on every delivery, without limit, against a receiver that held a mark for a sender that was still alive. Refusing it loses nothing: the mark stays where the last plausible frame put it, so the real node’s next frame still out-numbers it and still lands. Silencing the real node forever — the exploit the version cap above exists to prevent — is closed from both sides rather than one.

An address additionally carries an incarnation: which process is answering at that system@host:port, minted once per Cluster.join. It is deliberately carried, and acted on only to decide what a restart announces — a record of a new incarnation ends the old run’s membership before the new run joins — never to refuse a record. The field is optional on the wire, so that a node running either version understands one running the other — and an optional field is bypassed by stripping it, which means a refusal keyed on a mismatch would be one an attacker opts out of while a legitimate peer of the previous version walks into it. Requiring the field breaks all eight address-bearing frame fields at once and waits on the wire-protocol versioning work. It stays out of toString, equals and compareTo, so nothing keyed on a node’s identity changes.

The one comparison that needs no agreement is made: a record a peer sends about this node keeps this node’s own incarnation. The leader’s promotion is the single claim about itself a node accepts, and it is merged wholesale, address included — so without that substitution the local record’s identifier would be whatever the last peer to promote it happened to say.

Three things this does not close. First, a peer that has earned standing and composes a fresh frame naming a deleted address at its old version: that is a forgery rather than a replay, and refusing it needs the incarnation to be required rather than carried, which is the wire break above.

Second, a missing mark admits everything, and there are three ways to be missing one — only one of which is the sender’s own eviction. An evicted member’s mark goes when its entry does, for the reason above: at once on the detector’s own auto-down route, which deletes, and on every other removal only when the tombstone is pruned, since a tombstone keeps the mark along with the entry. Under the default auto-down an ordinary partition takes out sender and subject together, so that route arrives ready to use; with a strategy configured or under auto-down = off a partition evicts nobody until something decides — both are parked, marks intact — and behind a tombstone the mark lasts exactly as long as the refusal does. A fresh or restarted process starts with none at all. And a member learned third-party never had one: gossip is epidemic, so a node files C as up on B’s word and has still never seen a frame from C. In that last case the sender is a full member the whole time and nothing anywhere was evicted — which is why this guard cannot be described as holding “while its sender is a member”. The earlier wording of this section, and of the changelog and roadmap entries, said exactly that, and it was wrong.

Refusing a frame from a peer with no mark is not the missing check: the first frame from every peer is one, so a receiver that refused them would never learn a cluster exists. What a recording gets from an empty mark is a two-frame bootstrap — the first frame buys standing and installs the mark, off its own recorded number, and the second out-numbers it and speaks for the subject. Against a member learned third-party it is shorter still, because the standing is already there and one frame does it.

Third, and this is why the two above stay open: nothing keyed on the sender’s own counter can separate a recording from a live frame, because both were stamped by the same counter. What separates them is which process emitted them, and the only receiver-checkable statement of that is the incarnation — which has to be required before a refusal can rest on it, and that waits on the wire-protocol versioning work.

Requiring it would close a recording of a previous incarnation outright, which is the ordinary restart case and the bulk of the exposure. Two edges would survive even then, and they are worth stating so nobody expects more: a node downed while still running answers under the same incarnation its recording carries; and a freshly started receiver holds no earlier incarnation of the subject to compare a first sighting against, so a record about a member it has never heard of remains admissible. On a correctly configured mTLS cluster neither edge is reachable by an outsider — hello is bound to the peer certificate — so the exposure is plaintext and server-only-TLS deployments, plus a compromised peer.

All three residuals are tracked separately.

production.ts
import { TcpTransport, Cluster, ClusterOptions } from 'actor-ts/cluster';
const tlsOptionsType = {
cert: fs.readFileSync(process.env.TLS_CERT_PATH!),
key: fs.readFileSync(process.env.TLS_KEY_PATH!),
ca: fs.readFileSync(process.env.TLS_CA_PATH!),
rejectUnauthorized: true,
};
const transport = new TcpTransport(self, log, { tls: tlsOptionsType });
const clusterOptions = ClusterOptions.create()
.withHost(host)
.withPort(port)
.withSeeds(seeds)
.withTransport(transport);
await Cluster.join(system, clusterOptions);

Env vars carry paths; the cert-manager / vault mounts the files. Code stays generic across environments.