Cluster security
此内容尚不支持你的语言。
The cluster transport defaults to plain TCP without authentication — fast, simple, fine for a private network. Not fine for any cluster that crosses an untrusted boundary (the public internet, multi-tenant kubernetes, cross-region links without VPN).
This page covers the production-security setup for the cluster transport.
Two concerns
Section titled “Two concerns”| Concern | How to address |
|---|---|
| Eavesdropping — peer-to-peer traffic readable by anyone on the wire. | TLS on the cluster transport. |
| Unauthorized joins — a malicious node connects + becomes a cluster member. | Mutual TLS (mTLS) — peer certificate verification. |
Enable TLS with mutual authentication (mTLS) for any external-facing cluster — it addresses both.
Enabling TLS
Section titled “Enabling TLS”import { TcpTransport, NodeAddress, Cluster, ClusterOptions } from 'actor-ts/cluster';import fs from 'node:fs';
const transport = new TcpTransport( NodeAddress.parse('my-app@10.0.0.5:2552'), system.log, { tls: { cert: fs.readFileSync('./tls/cluster.crt'), key: fs.readFileSync('./tls/cluster.key'), ca: fs.readFileSync('./tls/ca.crt'), rejectUnauthorized: true, // verify peer certs }, },);
const clusterOptions = ClusterOptions.create() .withHost('10.0.0.5') .withPort(2552) .withSeeds([...]) .withTransport(transport);await Cluster.join(system, clusterOptions);The TLS settings:
cert+key— this node’s certificate + private key. Both carry the material itself, not a path to it: nothing in the transport reads from disk, and none of the three runtimes accepts a filename in these fields either. Load it yourself, as the example above does. Both halves are required together on a listener — see the refusals below.ca— trusted CA bundle. Use to verify peers’ certificates.rejectUnauthorized: true— fail handshakes where the peer’s cert isn’t signed byca.requestClientCert— whether the listener demands a certificate from whoever connects to it. You will not normally set this: it defaults totruewhenevercais present, since a trust bundle on a cluster listener has no other purpose.
With a shared ca, the cluster is mutually authenticated —
every connection requires a peer cert signed by the trusted CA,
in both directions.
The certificate also decides which node
Section titled “The certificate also decides which node”mTLS answers “may this peer be in the cluster”. On its own it
never answered “is this peer the node it claims to be”: the
hello frame carries an address and no credential, so one
CA-signed node could announce itself under another member’s
address. That matters because the gossip-authority rules below
all key off the connection’s peer — they are exactly as strong
as the identity underneath them.
So when a peer presents a certificate, the address it claims in
hello must be one the certificate vouches for. A claim is
accepted when the CN or a SAN covers either:
- the address’s host — the ordinary case, where each node’s certificate carries its own hostname or IP, or
- the full
systemName@host— for deployments that mint a per-node identity and want the tighter binding.
Wildcards are honoured for the host, in the leftmost label only, exactly as TLS hostname verification does.
Nothing changes for a cluster with no peer certificate to read: plain TCP, one-way TLS, and a Deno listener (which cannot report one at all) behave exactly as before. The check strengthens mTLS deployments rather than adding a switch that can be left off.
Four configurations are refused at bind time rather than started in a weaker state than they read as:
- An incomplete server credential —
certwithoutkey,keywithoutcert, or atlsobject carrying neither (acaalone says which peers to trust when dialling; it gives the listener nothing to present). Empty counts as absent, which is what an unset environment variable or a mis-mounted secret looks like by the time it arrives. requestClientCert: truewith noca— there would be nothing to validate peer certificates against.- A
cathat holds no certificate — an empty or blank string, an empty byte array or array,null— whateverrequestClientCertsays. That is what a missing secret looks like by the time it arrives, not a way to ask for one-way TLS —requestClientCert: falseis that. The HTTP server holds itscato the same rule, stated once under Client certificates. - An mTLS listener on Deno —
Deno.listenTlstakes only a cert and a key, with no way to request or verify a client certificate, so the listener would authenticate nobody.
A Deno node can still join an mTLS cluster: it presents its
own cert / key when dialling, so the listener (on Node.js or
Bun) authenticates it like any other peer. Only hosting the
listener is unavailable — in a mixed deployment, keep the seed
nodes on Node.js or Bun.
One Deno difference worth knowing: rejectUnauthorized has no
equivalent there and is not mapped. Deno always validates the
chain, so reaching a self-signed peer means supplying its signing
CA in ca — which is the shape recommended here anyway.
Certificate management
Section titled “Certificate management”Three approaches:
Self-signed for development
Section titled “Self-signed for development”openssl req -x509 -newkey rsa:4096 -keyout cluster.key -out cluster.crt -days 365 -nodes -subj "/CN=actor-ts"Use the same cert + key on every node. Fine for dev / staging. Don’t use in production.
Private CA for production
Section titled “Private CA for production”# Create a CA once:openssl req -x509 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 3650 -nodes -subj "/CN=actor-ts-ca"
# Per-node certs signed by the CA:openssl req -newkey rsa:4096 -keyout node-1.key -out node-1.csr -nodes -subj "/CN=node-1"openssl x509 -req -in node-1.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out node-1.crt -days 365Each node gets its own cert; everyone trusts the CA. Rotating certs is per-node and doesn’t require touching the CA.
Cert-manager / vault for K8s
Section titled “Cert-manager / vault for K8s”For K8s deployments, use cert-manager with an internal CA or HashiCorp Vault. Certs are mounted as volume secrets; rotation handled by the cert manager.
# Example cert-manager Certificate spec:apiVersion: cert-manager.io/v1kind: Certificatemetadata: name: actor-ts-clusterspec: secretName: actor-ts-cluster-tls issuerRef: name: actor-ts-ca kind: ClusterIssuer commonName: actor-ts dnsNames: - actor-ts-cluster.svc duration: 8760h renewBefore: 720hThe pod mounts the secret as files; the actor reads them.
Firewall the cluster port
Section titled “Firewall the cluster port”Cluster port (2552) — internal-only:- pods can talk to pods on 2552- not exposed via Service / Ingress- LoadBalancer never sees itEven with TLS + auth, expose the cluster port narrowly. A NetworkPolicy in K8s:
apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata: name: actor-ts-cluster-internal-onlyspec: podSelector: matchLabels: app: actor-ts ingress: - from: - podSelector: matchLabels: app: actor-ts ports: - protocol: TCP port: 2552Only app=actor-ts pods can reach port 2552 on each other.
Threat model
Section titled “Threat model”| Threat | Mitigation |
|---|---|
| Network eavesdropping | TLS |
| Man-in-the-middle | TLS + cert verification |
| Unauthorized cluster join | mTLS (CA-signed peer certs) |
| Insider with stolen cert | Cert rotation + revocation |
| Malformed or hostile wire frame | Shape validation at the decode boundary |
| Frame split into thousands of chunks to multiply decode cost | Linear decode buffer — every byte copied once, on arrival |
| Socket that opens, sends half a frame and goes quiet | Stall deadline on an incomplete frame, plus a cap on inbound connections |
| Sockets opened in bulk that send nothing at all, to fill the inbound cap | Handshake deadline on every accepted socket, so no slot is held open-ended |
| Peer growing a node’s registries without bound | Per-registry caps on pub-sub and the receptionist |
| Address claimed before the node that owns it exists | Tight version-skew cap on every gossiped member version |
| Gossip frame captured off the wire and played back later | Per-sender frame sequence — a frame must out-number the last one accepted from that sender (partial: only once one has been) |
| Spoofed discovery answer steering the bootstrap | mTLS, plus pinnedAddresses on the seed provider |
| A ref in a message body naming an internal host, to make this node dial it | Frames leave only for members and nodes this node already holds a connection to (partial: without mTLS a peer still becomes a member under an address its own hello names) |
| Compromised pod inside the cluster | Application-level auth (out of scope here) |
The cluster transport handles transport-level security. Application-level concerns (auth between specific actors, per-tenant isolation) remain your job — the cluster transport is trusted within itself.
Where the seeds come from
Section titled “Where the seeds come from”A seed provider asks an outside party — DNS, the K8s API — which addresses to talk to, and that answer arrives before any certificate has been checked. It cannot let an attacker join the cluster: the address a peer claims must be vouched for by its certificate, so a spoofed seed produces a connection that dies at the handshake.
That guarantee is configuration-dependent, which is the part worth stating plainly. A node only demands a peer certificate if its listener was configured to request one; where TLS is off, or where a half-configured listener never asks, there is nothing to check the claimed address against and a poisoned discovery answer is followed as given.
So pin what the resolver is allowed to say:
const dnsSeedProviderOptions = DnsSeedProviderOptions.create() .withHostname('actor-ts.example.com') .withSystemName('my-app') .withPort(2552) .withPinnedAddresses(['10.0.0.0/8']) .withLog((message) => logger.warn(message));Addresses outside the list never become seeds. Details, including the SRV-mode caveat (targets are hostnames, so pins are suffixes there and the eventual A lookup stays unpinned), are on the DNS seed provider and Kubernetes API seed provider pages.
This is defence in depth: it lowers what a DNS or Endpoints compromise buys, and it is the only discovery-layer control left standing when mTLS is not in place.
Until #860 the snippet above was the only way to reach it:
Cluster.bootstrap({ discovery: 'dns' }) and the env-driven ladder built
their providers with no pin at all, so the documented mitigation was
unreachable on the path most deployments take. It is now a config key, and it
ships in the same block as the keys that decide which provider runs:
actor-ts.discovery { dns.pinned-addresses = ["10.0.0.0/8"] kubernetes.pinned-addresses = ["10.0.0.0/8"]}Set it wherever actor-ts.cluster.bootstrap.discovery.method or
service-name is set, and for the same reason both were added together: a
config file is a wider input surface than a code call, so making the
destination of discovery settable without making the pin settable would
have widened the attack surface and left the guard where it was. A list that
cannot match in the configured mode — CIDRs only in SRV mode, a host suffix
under Kubernetes — is refused at startup rather than silently discarding every
answer, because an empty seed set reads as “we are alone” and ends in a
one-node cluster.
What the wire edge rejects
Section titled “What the wire edge rejects”Frames arrive as JSON and used to be cast to the protocol type rather than checked against it, so a peer could hand a node a value the code then read as if the type were true. Every frame is now validated before any handler sees it, on every transport, the worker mesh included:
- The frame must be an object carrying a string
kind.null, a bare string and a number are refused — previouslynullalone was an eight-byte remote process kill, and it needed no completed handshake. - Node addresses must have a non-empty
systemNameandhostplus a positive integerport. A port that arrives as the string"2552"is the case worth knowing about: it renders identically in every log line and keys every map the same way, but never compares equal — a node that merged its own address in that shape stopped recognising itself. - A gossiped member’s
statusmust be one of the seven legal values. An unknown one used to reach an exhaustive match that threw after the member had been stored, so the node crashed and re-gossiped the poisoned entry to its peers. - A gossip batch is validated whole. One malformed member refuses the frame, so a bad entry cannot ride in behind a good one.
Two tiers, deliberately: a frame that fails validation is dropped and the connection stays up, because one bad frame should not cost a healthy peer its link. A handler that throws drops the connection — that is the case nobody understands, and it must not escape into the runtime’s socket callback.
The worker mesh takes the first tier and neither half of the
second. MessageChannelTransport — the transport a worker
thread and the main thread speak over a MessagePort — checks
the envelope’s own shape and then runs the payload through the
same validateWireFrame the socket transports use, so a
malformed frame from a worker never reaches a handler either.
WorkerBroker, which relays between threads, checks only the
envelope’s two address fields — the payload is the transport’s
job — and rewrites from to the port the frame arrived on, so
a worker cannot speak for a sibling. The broker also reports
what it refuses: a malformed envelope or a port that refuses a
frame is a folded WARN — one line per reason per 30 seconds,
carrying the count — that names the registered address of the
port the most recent frame arrived on and never the payload, and a frame to
an address nobody registered is the same line at DEBUG, since
that is what every healthy sibling sends for a moment after a
crash (#1276). The transport half reports too, on the same
pattern: a malformed envelope, a payload the wire checks refuse
and a handler that throws each bump
transport.dropped() and reach the log as one folded WARN per
reason per 30 seconds, carrying the count. Two of the three
carry the sender as well; the envelope one does not, and cannot:
when the envelope itself is what was refused, no address was read
far enough to name, so that line leaves most recently from off
entirely — which is what the mesh page says of the same line.
The payload one names the reason validateWireFrame gave — which
is what tells a version skew from a hostile peer — and none of
the three carries the frame (#1515). Before that the transport was
the last silent edge on this path: a worker running a skewed
version had its frames black-holed with no signal at all, which
is worse than before the guard existed, when the frame at least
arrived.
The sender is escaped on the way to the log, which the broker’s
line does not need to do. Through a broker from is rewritten
to the arriving port, so it is host-minted; wired port-to-port
there is no broker to do that, and a node address only has to
carry non-empty strings — so a host with a newline in it is a
well-formed address that would otherwise forge whole log lines.
Frame kinds registered by extensions (sharding, pub-sub, the receptionist, DistributedData, DevTools) pass this layer and validate their own payloads. So does replicated event sourcing, whose event envelopes ride on pub-sub rather than on a frame kind of their own — see what a replica trusts from its peers.
Validation is only half of it. A frame can be perfectly well-formed
and still speak for a node that did not send it, and that class of
defect is closed the same way everywhere in this codebase: the
identity is taken from the connection, not from the payload. A leave
may only retire the node that sent it; gossip replaces the sending
peer’s contribution and no one else’s; a sharding or singleton
directive is honoured only when it arrived wrapped by the per-path
envelope handler, which is the one place the authenticated address
exists; and a replicated event’s replica is held against the node
that published it. A frame that reaches an actor by generic path
resolution instead — which strips nothing but does carry no sender —
is treated as unauthenticated rather than as trusted.
A CRDT’s own state is the one place that rule does not reach.
DistributedData validates a decoded payload for shape and
plausibility: a counter slot must be a non-negative integer at
or under MAX_COUNTER_SLOT (≈ 2.2e12 — the largest per-slot
value for which a full counter’s value() is still an exact
integer), a register timestamp may be no more than five minutes
ahead of local time, a collection is capped at 4 096 entries, and
a __proto__ key is refused. It does not check that the
sending peer had any business writing the slot it wrote.
Grow-only counters merge by maximum, so any peer able to gossip
can raise any replica’s slot inside those bounds — the owning
node cannot lower it again, the value is written through to the
durable record, and delete(key) is best-effort because a peer
still holding the key re-gossips it. A DistributedData counter is
therefore trustworthy exactly as far as the set of peers you let
onto the wire, which makes the mTLS setup above a prerequisite
for a quota or billing counter rather than a hardening step.
On the socket transports every rejection is logged at WARN
naming the peer and the offending field, so a dropped frame is
diagnosable — a version mismatch and a hostile peer look
different in the log. On the worker path both halves of the
channel do the same, folded: the broker logs a WARN per reason
for what it will not route, and the transport logs one for what
it will not deliver; see above.
What decoding a frame costs
Section titled “What decoding a frame costs”Validation runs on a whole frame, and a whole frame has to be
reassembled from however many TCP chunks the sender chose to
split it into. The peer chooses that split, which used to
be the expensive half: the decoder rebuilt its accumulator on
every chunk, so assembling one frame cost work quadratic in
the chunk count. A frame just under the 16 MiB cap, delivered
in TCP-sized ~1400-byte writes, is about 12 000 chunks and
≈ 100 GB of memory copying — roughly 6000× amplification on
bytes the attacker never had to send, and reachable before the
hello gate, so no membership was needed. The decoder now
appends into a buffer it grows by doubling, so each byte is
copied once on arrival and the amplification is 1×.
Three bounds cover what an unauthenticated socket may still hold while it is connected:
- An incomplete frame may sit for 30 seconds without another byte arriving before the connection is closed. It is a stall bound, not a budget for the frame: the deadline is re-armed on every chunk, so a peer shipping a large frame over a congested link is never punished for being slow — only for going silent.
- The handshake itself has 5 seconds, from the moment the
socket is accepted, and an accepted socket that has not sent
its
helloby then is closed. This is the one an incomplete frame does not cover: a socket that sends no bytes at all is not stalled mid-frame, so nothing about it is being tracked. It is the same 5 seconds the dialling side already gives itself, and that clock starts earlier — before the TCP connect and the TLS handshake — so a peer that is still trying has always given up first. Once thehellolands the deadline is gone: an established peer between two gossip rounds is idle by design and is never dropped for it. - Inbound connections are capped at 1024. A fully-meshed cluster needs one per peer, so the cap is set far above any realistic topology; what it removes is the ability to multiply the per-connection cost by opening sockets in a loop. The newest connection is refused rather than an established one evicted — eviction would let an attacker push real peers off the node.
The second and third are a pair. A cap is only worth what its
slots’ turnover is worth, and without a handshake deadline the
cap becomes the attack: 1024 connections that send nothing
hold every slot for the life of the process, and every peer
and ClusterClient after them is refused. On Bun that is
reachable even under mTLS, because a socket’s open callback
fires before the TLS handshake completes — the slot would be
taken while there is still no certificate to check.
The first and third are what bound inbound decode memory at
all: the product of the two, with the per-connection term
already capped by actor-ts.remote.max-frame-bytes. Lowering
that key is also the lever for a deployment that never sends
large envelopes — the resident cost per connection follows it
directly.
Bytes and work are not the same bound, and this is where the
two part company. An accepted frame goes to JSON.parse and
only then to the tagged-tree walker, so the depth ceiling
(actor-ts.serialization.read-constraints.max-nesting-depth)
reports an over-deep payload after the parse has already built
it. Measured on Bun 1.4, one 16 MiB frame of balanced [ costs
about 1.0 s of CPU and 260 MiB of heap before the refusal —
roughly four times what a legitimate frame of the same size
costs, and repeatable by any peer that completed the handshake.
Two keys bound it and neither is on by default:
remote.max-frame-bytes, and
serialization.read-constraints.max-document-bytes, which is
checked against the same 4-byte length prefix and refuses the
frame before its payload is buffered at all. A pre-parse scan
of the raw buffer would close it with neither set; it was
measured at 18–28 % of every frame decode and not shipped, for
a fourfold amplification (#880).
What a well-formed frame still cannot grow
Section titled “What a well-formed frame still cannot grow”Shape validation says a frame is readable, not that acting on
it is free. Pub-sub gossip is the clean example: a peer naming
100 000 topics it claims subscribers for sends one perfectly
legal frame, and the receiver used to allocate 100 000 map
entries for it — no local Subscribe, no malformed field, no
log line. The receptionist had the same shape on its own gossip
path.
Membership itself had the same shape, and it is the one registry nobody has to opt into: every clustered node keeps a map of members, and gossip is what fills it. The rules below decide whether a claim is believable; none of them bounded how many believable claims one peer may make. A sender announcing its own address is waved through by design — refusing that would mean no node could ever join — so naming a fresh address per frame allocated an entry per name.
These registries are capped now, and the caps apply to the gossip path as much as to local calls:
| Registry | Capped by | Default |
|---|---|---|
| Live cluster members | cluster.max-members | 1000 |
removed tombstones | cluster.max-tombstones | 10000 |
| Pub-sub subscribers per topic | cluster.pub-sub.max-subscribers-per-topic | 10000 |
| Pub-sub topics | cluster.pub-sub.max-topics | 10000 |
| Pub-sub remote claimants per topic | cluster.pub-sub.max-remote-nodes-per-topic | 1000 |
| Receptionist subscribers per key | cluster.receptionist.max-subscribers-per-key | 1000 |
| Receptionist subscriptions total | cluster.receptionist.max-subscriptions-total | 10000 |
A claim over a cap is dropped and logged rather than refused on
the wire — the frame is well-formed, and dropping the connection
over it would let one noisy peer cost a healthy link. A local
Subscribe over a cap is answered with SubscribeRejected
instead, because the caller is in a position to do something
about it. Both pub-sub registries also watch their subscribers,
so the other half of the old growth — refs that stopped without
ever unsubscribing — is reclaimed rather than capped.
Lower the defaults for a deployment where the legitimate numbers are far below them: a cap is only a bound on the damage, and one set far above real usage bounds very little.
Why membership needs two caps
Section titled “Why membership needs two caps”Splitting members from tombstones is not tidiness: the two kinds
of phantom have different lifetimes. A phantom member in up /
joining / unreachable that arrives as a record in a gossip
frame is a member the failure detector watches from the moment it
is merged, and nothing ever heartbeats for it, so both detectors
declare it down — φ measuring the silence from that moment.
What that verdict does is the downing configuration’s call. Under
the default — no strategy, failure-detector.auto-down = on — the
detector removes it, down-after after the attacker stops feeding
it, and for good only once no other node still carries it, because
that route deletes without a tombstone and a node that has not
deleted the phantom yet gossips it back. With a strategy
configured, or under auto-down = off, the phantom is parked at
unreachable like any silent peer and holds its live slot until a
strategy or an operator removes it. A frame’s sender field is no
longer a route at all. A from naming an address this node held no
record of used to be filed as joining with no authority check and
promoted like any joining member — this node’s leader, if its address
sorted first, and an address the heartbeat and gossip ticks then
dialled. That door is closed
(issue #939): a
sender is introduced by its own record, merged like any other, and
every gossip frame carries one. max-members is therefore the one
bound on live phantoms, and the merge the one way in. A record gossiped as removed is watched by
nothing: only the tombstone prune reclaims it, once the larger of
cluster.tombstone.time-to-live and tombstone.min-retention has
passed — a day at the defaults. Two residues with two lifetimes, so
two caps. Refusing a tombstone costs nothing, because a tombstone for
an address this node holds no record of suppresses nothing that
exists — while a refused live record costs a legitimate member
one gossip round, and comes with a WARN naming the cap.
Both caps are charged on the bucket a record moves into, not
on whether it creates an entry. A gossiped record that keeps an
entry where it already is — up → unreachable, or a newer
tombstone over an older one — is a free in-place update. One that
crosses between the buckets is charged to the bucket it enters:
a tombstone re-incarnated as up needs room among the live
members, a live member gossiped as removed needs room among the
tombstones. Charging only entry creation let the two caps trade
headroom with each other for free — a re-incarnation vacated the
tombstone bucket without giving up a map slot, so the next flood
of tombstones was admitted too, and alternating the two grew the
map without bound while respecting both caps at every individual
step. A refused conversion leaves the member live, where it goes
the way any silent member does: removed by the detector at
down-after under the default, or — with a strategy configured or
under auto-down = off — parked at unreachable until a strategy
or an operator removes it.
Tombstones this node mints itself — a peer’s leave, a downing
decision, an operator down() — convert a record it already
holds and are never subject to the cap. Capping its own
bookkeeping would drop the suppression that stops stale gossip
resurrecting an evicted address, which is a liveness bug wearing
a security fix’s clothes.
The ceiling worth knowing is not the heap. Gossip carries the
whole member list, so at roughly 110 000 entries a node’s own
frame outgrows remote.max-frame-bytes and every peer terminates
the connection on the length prefix — the node evicts itself from
the cluster while still running, long before anything runs out of
memory.
The failure detector is held to the same bound. It keeps state per
peer, and it used to create that state for every connection identity
a frame arrived under — without mTLS, a name the peer picks in its
hello. A peer reconnecting under fresh names grew it by an entry per
name while the member map held at its cap: 500 names left 502 entries
beside 3 members, each carrying a sample window under the φ-accrual
detector. Liveness now reaches the detector only for a member that is
not a tombstone, or a configured seed that is not one either. Every
route that admits a member registers it, and every route that takes
it out of the live set forgets its entry — a tombstone this node
mints, a tombstone a gossiped record brings, and a deletion from the
member map alike. So the detector holds an entry per live member and
at most one per seed, bounded by max-members and the seed list
(#1103).
Who may say what
Section titled “Who may say what”Passing the shape check does not make a frame believable.
Gossip merges used to be decided purely by version magnitude,
and versions are seeded from Date.now() — so an attacker
could always pick a winning number and rewrite any member’s
status, including the receiving node’s own. Two rules now
sit in front of the merge:
- Nobody downgrades us. A claim about this node’s own
address is refused. The one exception is promotion out of
joining/weakly-upintoup, which has to come from outside because it is the leader’s decision — and which is harmless to accept, since a joining node is already trying to becomeup. What bounds it is how little it carries: the status, and a version limited only by themaxVersionSkewMscap (5 minutes by default, see A version cannot claim an address in advance below), so a stranger can date this node’s record up to that far ahead — clamping the version is tracked in #1797. Nothing else crosses over. The promotion asks nothing of its sender, which is why this node has to keep its own roles, incarnation, storage identities, configuration facts and app version over whatever the record says. Before #1104 the roles came from the record — and so did the storage identities of a node that had resolved none of its own — so one frame from a stranger during the joining window chose what this node hosts and places. Everything else about our own record we decide ourselves. - Third-party claims need a sender with standing. Saying something about another node requires the connection’s peer to be a member this node already considers active. A sender may always announce its own record — that is how joining works.
Both rules key on the connection’s peer, not on the
payload’s from field, since that field is the one thing an
attacker fully controls. The same reasoning covers three
neighbours. A leave is only accepted from the node that is
leaving. A heartbeat refreshes the failure detector for the
peer that sent it rather than the address it names — that one
previously let a peer keep a dead node looking healthy, and
made the receiver dial an attacker-chosen host. And a
ClusterClient ask is answered down the connection the
envelope arrived on: routing on the payload let any connected
party choose the address this node replied to, and opened a
connection to.
The client envelope no longer carries a sender field at all —
after the rule above it had exactly one correct value, which
is the one the receiver already holds. An envelope that still
names an address other than its own connection is counted on
cluster_envelope_from_mismatch_total{frame} and otherwise
ignored; see ClusterClient.
Unreachability is deliberately not covered by these rules: “I cannot reach C” is inherently a third-party observation, and every node must converge on the same view before a downing provider decides. Refusing those claims would leave each node with only its own reachability picture.
How a removed node finds out
Section titled “How a removed node finds out”The first rule means no peer can tell this node it was removed —
and a node the rest of the cluster downed while it was paused or
cut off used to go on running everything it hosted. So a peer that
holds this node’s run as removed answers its heartbeats with a
membership-refused frame instead of an acknowledgment, naming that
run with its incarnation (#940). The frame is taken as a statement
about the sender only: “I will not have you back”. The receiver
parks that peer at unreachable and stops letting its heartbeats
bring it back; its own record stays untouched, and the verdict stays
with its own downing strategy. Once every peer has refused it, the
view counts as settled and the strategy is asked at once rather than
after stable-after.
So a forged refusal buys what dropping that peer’s traffic already buys — the peer looks unreachable — and no more: the real peer’s gossip, which lists this node as a member, withdraws it, and a refusal naming another run at this address, or none, is ignored. Forging one for every peer makes the node’s strategy decide on a view in which it is alone; that is the reach of cutting all of its traffic, and the reason the caution below applies here too.
Who may be addressed by name
Section titled “Who may be addressed by name”The rules above bound what a frame may claim. One more bounds
what it may reach: an envelope carries the target path the
sender chose, and the node resolves it against its own actor
tree. That resolution used to walk from the root, so
/system/… was exactly as reachable as /user/… and any peer
that completed a hello could deliver straight to a shard
coordinator, a singleton manager or the pub-sub mediator —
bypassing the per-path handler each of them registers, and with
it the sender attribution the rules above are built on.
The /system subtree is now refused on that path, in every
configuration and at both seams — the cluster wire and the
ClusterClientReceptionist. Framework actors stay reachable
through their registered handlers, which is the door that knows
which connection a frame came in on. Nothing an operator can
configure opens the old route back up.
The /user half is a deployment’s own decision, because a peer
naming an application actor is how an ActorRef works across
nodes. Narrow it where the cluster network crosses a boundary
you do not control:
actor-ts.remote { untrusted-mode = true trusted-selection-paths = ["/user/orders/*", "/user/reporting/intake"]}An entry is an exact path unless it ends in /*, which makes it
that path and its subtree — segment-anchored, so
/user/orders/* does not admit /user/orders-archive. Refusals
are counted on cluster_envelope_refusals_total{reason, frame}
and folded into at most one WARN per reason every 30 seconds,
naming the connection’s peer and never the requested path; a
line per refusal would hand the peer log amplification instead.
Watch reason="not-allow-listed" on the day you turn the mode
on — it is where an incomplete allow-list shows up. See
configuration.
The same policy guards the one other frame in which a peer names
a path of its choosing: a watch. A refused path is answered like
a missing one — Terminated with existenceConfirmed = false —
so a watcher never hangs and the answer reveals nothing the
refusal did not already; the refusal is counted under
frame="watch".
A ref in a message cannot choose where this node dials
Section titled “A ref in a message cannot choose where this node dials”The other direction has the same shape. An ActorRef in a message
body crosses the wire as a marker naming its node — system, host,
port — and the receiver rebuilds a remote ref from it. That address
is the sender’s claim. The marker used to be rebuilt without even
the validation every other wire address gets, and whatever node a
ref named was dialled the moment it was used: an actor answering a
decoded replyTo, the idiom every handler follows, opened a
connection to an address of the sender’s choosing and wrote a frame
to it — a cloud metadata endpoint, an internal admin port. Watching
the ref did the same with a watch frame (#936).
A marker is now validated as a node address, and one that is not
valid decodes to Nobody. And a frame that carries a user payload or
a death-watch message — every send that is not the membership
protocol’s own — leaves only for:
- a member that is not a tombstone, or
- a node the transport already holds a connection to, acked or
still in its handshake — sending over an existing connection dials
nothing. It is how a reply reaches a node that has joined but that
this node has not heard of yet, and how a
tellto a seed straight afterCluster.joinresolves is queued on the connection the join opened instead of dropped. A custom transport that does not implement the optionalholdsConnectionTois asked throughpeers()instead, which knows acked connections only, so on it thattellis refused until the handshake lands.
Anything else is dropped, as a frame to an unreachable node is,
counted on cluster_outbound_refusals_total{frame} (envelope or
wire) and folded into at most one WARN every 30 seconds that
names the most recent destination, escaped. The commonest cause on a
healthy cluster is a node that has just left. A ref is rebuilt
lazily, so one to a node this node learns of a moment later still
works when it is used. Watching a ref whose node is refused is
answered at once with Terminated and existenceConfirmed false,
as a refused watch on the far side is, so the watcher does not wait
for an answer that cannot come.
The rule is only as strong as membership. Since #939 a gossip frame’s
payload from no longer files a member: a sender is introduced only
by its own record, merged as a self-announcement held to the identity
its connection presented. Without mTLS that identity is still the
peer’s own choice, so a peer that can connect can announce itself under
an address it picks — and from then on this gate lets frames through to
it, and the membership protocol’s own gossip and heartbeats dial it
every interval with no ref involved at all. So this closes the route
through a message body, not the minting of a member. mTLS is what binds
a member’s address to a certificate.
A sharding directive needs the coordinator behind it
Section titled “A sharding directive needs the coordinator behind it”The same reasoning reaches one layer up, into the extensions
whose frames pass the wire edge and validate their own
payloads. A ShardRegion used to treat any message whose
kind began with sharding. as a framework directive —
including the five only the coordinator may issue. HandOff
stops every entity under a shard; ShardHome moves ownership;
RememberedEntities pre-creates entities; ShardMapUpdate
publishes an allocation map to every local subscriber, DevTools
panel and application listener included. One small frame from
anyone who had completed hello did any of that, repeatably.
The region could not have checked: sharding registered no
per-path envelope handler, so an inbound frame reached the
actor through generic path resolution, which delivers with no
sender at all. It now claims its own path on the envelope
router, which hands the handler the connection’s peer, and
wraps the frame in a class instance before putting it in its
own mailbox — a shape a JSON wire body cannot mint, so the
directive arms can tell a routed frame from a forged one.
A tagged { kind } object would not have been enough: it is
reproducible verbatim from a payload.
Both halves are load-bearing. Without the wrapper check, a frame addressed non-canonically — a trailing slash, a doubled separator — misses the exact-string handler lookup, still resolves to the same region through the actor tree, and arrives unwrapped. And without the origin check, an authenticated peer is any cluster member, so every member could issue directives to every region.
Refused frames are dropped and logged at WARN. The
coordinator’s own local leg builds the same wrapper, so a
single-node cluster rebalances exactly as before.
A region can only speak for its own node
Section titled “A region can only speak for its own node”The same gap ran in the other direction, and it was the more
expensive one. Every message a ShardCoordinator accepts is a
claim about the sender’s own node: which shards its region
hosts, that its region is gone, that a hand-off finished, where
to send a shard home or a statistics reply. The coordinator read
all of that out of the payload. Its only gate was isLeader()
plus, with a lease, holding the lease — which answers “am I the
authoritative coordinator?”, never “may this sender speak for
that region?”.
So one well-formed sharding.Register naming somebody else’s
address seized every shard of a type, and one
sharding.RegionTerminated evicted its region. The eviction is
the worse half: it sends no HandOff, so the victim keeps its
shard actors and its entities running. The result is two owners
for one live shard — the same entity id instantiated twice, and
for a persistent entity two writers on one persistenceId.
The region-side origin check above does not blunt this, because
the attacker never sends a ShardHome. It poisons the
coordinator’s allocation map with one forged frame and the
genuine coordinator then emits the redirect itself, from the
leader’s own node, inside a wrapper every region rightly
accepts. One forgery, every subsequent hop authentic.
The coordinator now claims its own well-known path on the envelope router and applies the same two conditions: the frame arrived inside the wrapper, and the address its payload names is the peer’s own. Both directions of the register loop are therefore attributed, which also fixed how a region addresses the coordinator — by the leader’s system name, not its own, since an actor path carries the system it belongs to.
hostedShards is bounded and capped on top of that. It was the
only caller-sized input the coordinator had — one Register
wrote an allocation entry per array element, with no range check
and no length cap, into state that is broadcast to every region
and persisted to the coordinator-state store, so the growth
survived restarts. Entries outside 0 .. numShards - 1 are now
dropped and the accepted set cannot exceed numShards.
A version cannot claim an address in advance
Section titled “A version cannot claim an address in advance”Version is a logical clock seeded from Date.now(), so
“highest version wins” also decides what happens the first
time an address is mentioned at all. That made an address
claimable before the node owning it exists. A stranger
announces itself under the address the next pod is about to
get — announcing your own record is the claim the rules above
never refuse — dates it close to the 24 h skew cap, and
attaches whatever roles it likes. The leader’s promotion loop
lifts the record into the active set, and the node that
really owns the address loses every merge afterwards, because
it seeds its version from its own clock and that is lower.
Roles are what routing, sharding placement, singleton hosting
and downing quorums are computed from, so the phantom is not a
cosmetic row in a member list.
A gossiped member version is therefore held to a tight clock-skew budget — 5 minutes by default:
const clusterOptions = ClusterOptions.create() .withHost('10.0.0.5') .withPort(2552) .withMaxVersionSkewMs(30 * 60 * 1000);The budget applies to every merge, not only to the record that introduces an address. It was originally the narrower rule — a tight cap on a first sighting, a generous 24 h one on every update — and that split could be stepped around by introducing the address first: two records for the same address in one frame, or — until #939 removed the door that filed a frame’s self-declared sender — a frame with no member records at all. Any rule that lets a record earn the wider budget fails the same way one step later, because without peer certificates every step of the earning is something the attacker can produce.
Raise it for a deployment whose clocks are known to run loose;
24 * 60 * 60 * 1000 restores the old single-cap behaviour.
A refusal is exclusion, and it is durable: a node is introduced
only by its own record, so a node whose clock runs further ahead
than the budget is not filed at all. It stays out of the member
list — with the WARN below naming the cap — until its clock comes
back inside the budget. That was always this cap’s verdict on such
a node; what changed is that the verdict now sticks instead of
being softened by the sender door or reversed by the node’s second
gossip frame.
Refused records are reported once per frame rather than once per
record — a WARN naming the peer and the count, and the counter
cluster_gossip_records_refused_total{reason}, whose reason
label is closed at version-skew, map-cap, timestamp-skew,
replayed-frame, self-claim, removed-run and foreign-cluster. Logging per record
would hand a peer log amplification in place of the growth it just
lost.
removed-run is a record for a run this node holds as removed —
the same address and the same incarnation. A version is the
sender’s to choose and could always be restamped past a tombstone;
an incarnation names the process the cluster removed, and only a
restart mints a new one. Since cluster wire v2 every address on a
frame carries it, so the run the cluster downed cannot come back
under any version, while a restart at the same address still joins
at once (#823, #940).
self-claim is the rule above, seen from the counter: a peer
asserting a status for the receiving node that is neither the one
it already holds nor the leader promoting it to up. A peer
echoing the status the node does hold is the ordinary content of
every gossip round — its own record travels back to it in every
frame — so that one is refused without being counted or logged, and
the counter still reads zero on a healthy cluster.
A recorded frame cannot be played back to a receiver that has heard from its sender
Section titled “A recorded frame cannot be played back to a receiver that has heard from its sender”A gossip frame is a snapshot of the member map, and a member’s
version only moves when its status does — so a frame captured
off the wire stays valid indefinitely. Against a converged
receiver that costs nothing: every record in it loses the
“higher version wins” comparison to the record already on file.
What makes it an exploit is an entry the receiver has
deleted, and two paths delete one. The failure detector’s own
removal — the default with no strategy configured, under
failure-detector.auto-down = on — deletes outright rather than
leaving a tombstone. That was once justified as letting a healed
partition re-discover the peer, and deleting buys no such thing:
neither side heartbeats or gossips a peer it holds unreachable, and
a deleted one is not in its map to address at all, so after a
symmetric partition neither side hears from the other again
(issue #930).
Under the default, both sides of a real partition therefore remove
each other and run on as two clusters, which is one more reason a
production deployment configures lease-majority or another
downing strategy.
And an expired tombstone is pruned once the larger of its
time-to-live and tombstone.min-retention has passed. Either way
there is nothing left to compare against, and the branch that files
a first sighting has no lower version bound at all. The other
removals — an operator’s down(), a strategy’s decision, a peer’s
leave — leave a tombstone that out-versions the recording until
that prune; with a strategy configured or under auto-down = off,
the detector itself removes no one. A restart that replaces a live
record of its previous run deletes that record too, but the new
run’s record takes the key at a higher version in the same step, so
a recording of the old run still loses the comparison.
So replaying a downed member’s own pre-down record brought it
back: same address, same version, up, carrying the roles it
had. Shard placement and singleton hosting follow the roles, so
the cluster starts routing work to a node it had evicted.
Every gossip frame therefore carries a sequence its author stamps — seeded from that node’s wall clock at startup and incremented once per frame — and a receiver remembers the highest one it has accepted from each connection peer. A frame that does not out-number that mark is dropped whole, before any record in it is looked at. The comparison itself needs no knob: it is between a peer and itself, so no clock-skew budget comes into it.
The mark is written in exactly one place — the gossip path, as it accepts a frame — so it exists for a peer this node has accepted a frame from, and for no other address. That is the bound, and the third residual below is what falls outside it.
A sequence must also be plausible — a finite number no
further ahead of the receiver’s clock than maxVersionSkewMs,
the same budget a gossiped version gets, because the number is
stamped from the author’s wall clock. A frame outside that budget
is refused, exactly like a repeat.
That started out as the weaker rule: the frame was merged, and
only adopting its number as the mark was refused, on the
argument that a frame numbered Number.MAX_SAFE_INTEGER cannot
be a recording of a real frame. Only the sequence is fabricated
in that attack. The members array is still the recording, and
nothing on this wire binds the two together — so a captured frame
with one field rewritten was merged, left the mark untouched, and
was therefore merged again on every delivery, without limit,
against a receiver that held a mark for a sender that was still
alive. Refusing it loses nothing: the mark stays where the last
plausible frame put it, so the real node’s next frame still
out-numbers it and still lands. Silencing the real node forever —
the exploit the version cap above exists to prevent — is closed
from both sides rather than one.
An address additionally carries an incarnation: which
process is answering at that system@host:port, minted once
per Cluster.join. It is deliberately carried, and acted on only to
decide what a restart announces — a record of a new incarnation
ends the old run’s membership before the new run joins — never to
refuse a record. The field is optional on the wire, so that a node running
either version understands one running the other — and an
optional field is bypassed by stripping it, which means a
refusal keyed on a mismatch would be one an attacker opts out of
while a legitimate peer of the previous version walks into it.
Requiring the field breaks all eight address-bearing frame fields
at once and waits on the wire-protocol versioning work. It stays
out of toString, equals and compareTo, so nothing keyed on
a node’s identity changes.
The one comparison that needs no agreement is made: a record a peer sends about this node keeps this node’s own incarnation. The leader’s promotion is the single claim about itself a node accepts, and it is merged wholesale, address included — so without that substitution the local record’s identifier would be whatever the last peer to promote it happened to say.
Three things this does not close. First, a peer that has earned standing and composes a fresh frame naming a deleted address at its old version: that is a forgery rather than a replay, and refusing it needs the incarnation to be required rather than carried, which is the wire break above.
Second, a missing mark admits everything, and there are three ways
to be missing one — only one of which is the sender’s own
eviction. An evicted member’s mark goes when its entry does, for
the reason above: at once on the detector’s own auto-down route,
which deletes, and on every other removal only when the tombstone is
pruned, since a tombstone keeps the mark along with the entry.
Under the default auto-down an ordinary partition takes out
sender and subject together, so that route arrives ready to use;
with a strategy configured or under auto-down = off a partition
evicts nobody until something decides — both are parked, marks
intact — and behind a tombstone the mark lasts exactly as long as
the refusal does. A fresh or restarted process starts with none at
all. And a member learned
third-party never had one: gossip is epidemic, so a node files
C as up on B’s word and has still never seen a frame from C.
In that last case the sender is a full member the whole time and
nothing anywhere was evicted — which is why this guard cannot be
described as holding “while its sender is a member”. The earlier
wording of this section, and of the changelog and roadmap entries,
said exactly that, and it was wrong.
Refusing a frame from a peer with no mark is not the missing check: the first frame from every peer is one, so a receiver that refused them would never learn a cluster exists. What a recording gets from an empty mark is a two-frame bootstrap — the first frame buys standing and installs the mark, off its own recorded number, and the second out-numbers it and speaks for the subject. Against a member learned third-party it is shorter still, because the standing is already there and one frame does it.
Third, and this is why the two above stay open: nothing keyed on the sender’s own counter can separate a recording from a live frame, because both were stamped by the same counter. What separates them is which process emitted them, and the only receiver-checkable statement of that is the incarnation — which has to be required before a refusal can rest on it, and that waits on the wire-protocol versioning work.
Requiring it would close a recording of a previous incarnation
outright, which is the ordinary restart case and the bulk of the
exposure. Two edges would survive even then, and they are worth
stating so nobody expects more: a node downed while still running
answers under the same incarnation its recording carries; and a
freshly started receiver holds no earlier incarnation of the
subject to compare a first sighting against, so a record about a
member it has never heard of remains admissible. On a correctly
configured mTLS cluster neither edge is reachable by an outsider —
hello is bound to the peer certificate — so the exposure is
plaintext and server-only-TLS deployments, plus a compromised peer.
All three residuals are tracked separately.
Per-deployment recipe
Section titled “Per-deployment recipe”import { TcpTransport, Cluster, ClusterOptions } from 'actor-ts/cluster';
const tlsOptionsType = { cert: fs.readFileSync(process.env.TLS_CERT_PATH!), key: fs.readFileSync(process.env.TLS_KEY_PATH!), ca: fs.readFileSync(process.env.TLS_CA_PATH!), rejectUnauthorized: true,};
const transport = new TcpTransport(self, log, { tls: tlsOptionsType });
const clusterOptions = ClusterOptions.create() .withHost(host) .withPort(port) .withSeeds(seeds) .withTransport(transport);await Cluster.join(system, clusterOptions);Env vars carry paths; the cert-manager / vault mounts the files. Code stays generic across environments.
Where to next
Section titled “Where to next”- Operations overview — production-readiness checklist.
- TLS everywhere — TLS for HTTP + brokers + journals.
- Master key rotation — rotating data-at-rest encryption keys.
- Transports — the underlying transport interface.
- Configuration — the HOCON keys for TLS settings.
