Cluster security

The cluster transport defaults to plain TCP without authentication — fast, simple, fine for a private network. Not fine for any cluster that crosses an untrusted boundary (the public internet, multi-tenant kubernetes, cross-region links without VPN).

This page covers the production-security setup for the cluster transport.

Two concerns

Concern	How to address
Eavesdropping — peer-to-peer traffic readable by anyone on the wire.	TLS on the cluster transport.
Unauthorized joins — a malicious node connects + becomes a cluster member.	Mutual TLS (mTLS) — peer certificate verification.

Enable TLS with mutual authentication (mTLS) for any external-facing cluster — it addresses both.

Enabling TLS

import { TcpTransport, NodeAddress, Cluster, ClusterOptions } from 'actor-ts';
import fs from 'node:fs';

const transport = new TcpTransport(
  NodeAddress.parse('my-app@10.0.0.5:2552'),
  system.log,
  {
    cert: fs.readFileSync('./tls/cluster.crt'),
    key:  fs.readFileSync('./tls/cluster.key'),
    ca:   fs.readFileSync('./tls/ca.crt'),
    rejectUnauthorized: true,   // verify peer certs
  },
);

const clusterOptions = ClusterOptions.create()
  .withHost('10.0.0.5')
  .withPort(2552)
  .withSeeds([...])
  .withTransport(transport);
await Cluster.join(system, clusterOptions);

The TLS settings:

• cert + key — this node’s certificate + private key. Both carry the material itself, not a path to it: nothing in the transport reads from disk, and none of the three runtimes accepts a filename in these fields either. Load it yourself, as the example above does. Both halves are required together on a listener — see the refusals below.
• ca — trusted CA bundle. Use to verify peers’ certificates.
• rejectUnauthorized: true — fail handshakes where the peer’s cert isn’t signed by ca.
• requestClientCert — whether the listener demands a certificate from whoever connects to it. You will not normally set this: it defaults to true whenever ca is present, since a trust bundle on a cluster listener has no other purpose.

With a shared ca, the cluster is mutually authenticated — every connection requires a peer cert signed by the trusted CA, in both directions.

The certificate also decides which node

mTLS answers “may this peer be in the cluster”. On its own it never answered “is this peer the node it claims to be”: the hello frame carries an address and no credential, so one CA-signed node could announce itself under another member’s address. That matters because the gossip-authority rules below all key off the connection’s peer — they are exactly as strong as the identity underneath them.

So when a peer presents a certificate, the address it claims in hello must be one the certificate vouches for. A claim is accepted when the CN or a SAN covers either:

• the address’s host — the ordinary case, where each node’s certificate carries its own hostname or IP, or
• the full systemName@host — for deployments that mint a per-node identity and want the tighter binding.

Wildcards are honoured for the host, in the leftmost label only, exactly as TLS hostname verification does.

What a shared certificate cannot do

Two nodes that present the same host certificate are indistinguishable to this check, and no certificate-based rule could separate them. If you need node-level identity — so that a compromised node-a cannot claim to be node-b on the same host — issue per-node certificates naming systemName@host.

Nothing changes for a cluster with no peer certificate to read: plain TCP, one-way TLS, and a Deno listener (which cannot report one at all) behave exactly as before. The check strengthens mTLS deployments rather than adding a switch that can be left off.

One-way TLS authenticates nobody

The hello handshake carries no credential of its own, so on a TLS cluster the peer certificate is the admission control. A listener that does not request one will complete a handshake with any client — which may then claim any node identity it likes. Set requestClientCert: false only when something else (a service mesh, a private L2) is doing the authenticating.

Three configurations are refused at bind time rather than started in a weaker state than they read as:

• An incomplete server credential — cert without key, key without cert, or a tls object carrying neither (a ca alone says which peers to trust when dialling; it gives the listener nothing to present). Empty counts as absent, which is what an unset environment variable or a mis-mounted secret looks like by the time it arrives.
• requestClientCert: true with no ca — there would be nothing to validate peer certificates against.
• An mTLS listener on Deno — Deno.listenTls takes only a cert and a key, with no way to request or verify a client certificate, so the listener would authenticate nobody.

The incomplete credential used to bind plaintext

This one was a silent downgrade rather than an error. The listener decided whether to speak TLS by checking that it had both a cert and a key; with only one it concluded “no TLS” and bound in plaintext — while the same options object still dialled out over TLS, so the cluster formed and the node looked TLS-configured. A half-applied secret rotation was enough.

It now throws at bind. Which means a node that has been quietly serving plaintext will fail to start after this change rather than come up mis-secured — deliberately, but worth knowing before a rolling restart.

A Deno node can still join an mTLS cluster: it presents its own cert / key when dialling, so the listener (on Node.js or Bun) authenticates it like any other peer. Only hosting the listener is unavailable — in a mixed deployment, keep the seed nodes on Node.js or Bun.

One Deno difference worth knowing: rejectUnauthorized has no equivalent there and is not mapped. Deno always validates the chain, so reaching a self-signed peer means supplying its signing CA in ca — which is the shape recommended here anyway.

Certificate management

Three approaches:

Self-signed for development

openssl req -x509 -newkey rsa:4096 -keyout cluster.key -out cluster.crt -days 365 -nodes -subj "/CN=actor-ts"

Use the same cert + key on every node. Fine for dev / staging. Don’t use in production.

Private CA for production

# Create a CA once:
openssl req -x509 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 3650 -nodes -subj "/CN=actor-ts-ca"

# Per-node certs signed by the CA:
openssl req -newkey rsa:4096 -keyout node-1.key -out node-1.csr -nodes -subj "/CN=node-1"
openssl x509 -req -in node-1.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out node-1.crt -days 365

Each node gets its own cert; everyone trusts the CA. Rotating certs is per-node and doesn’t require touching the CA.

Cert-manager / vault for K8s

For K8s deployments, use cert-manager with an internal CA or HashiCorp Vault. Certs are mounted as volume secrets; rotation handled by the cert manager.

# Example cert-manager Certificate spec:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: actor-ts-cluster
spec:
  secretName: actor-ts-cluster-tls
  issuerRef:
    name: actor-ts-ca
    kind: ClusterIssuer
  commonName: actor-ts
  dnsNames:
    - actor-ts-cluster.svc
  duration: 8760h
  renewBefore: 720h

The pod mounts the secret as files; the actor reads them.

Firewall the cluster port

Cluster port (2552) — internal-only:
- pods can talk to pods on 2552
- not exposed via Service / Ingress
- LoadBalancer never sees it

Even with TLS + auth, expose the cluster port narrowly. A NetworkPolicy in K8s:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: actor-ts-cluster-internal-only
spec:
  podSelector:
    matchLabels:
      app: actor-ts
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: actor-ts
      ports:
        - protocol: TCP
          port: 2552

Only app=actor-ts pods can reach port 2552 on each other.

Threat model

Threat	Mitigation
Network eavesdropping	TLS
Man-in-the-middle	TLS + cert verification
Unauthorized cluster join	mTLS (CA-signed peer certs)
Insider with stolen cert	Cert rotation + revocation
Malformed or hostile wire frame	Shape validation at the decode boundary
Peer growing a node’s registries without bound	Per-registry caps on pub-sub and the receptionist
Address claimed before the node that owns it exists	Tight version-skew cap on every gossiped member version
Spoofed discovery answer steering the bootstrap	mTLS, plus pinnedAddresses on the seed provider
Compromised pod inside the cluster	Application-level auth (out of scope here)

The cluster transport handles transport-level security. Application-level concerns (auth between specific actors, per-tenant isolation) remain your job — the cluster transport is trusted within itself.

Where the seeds come from

A seed provider asks an outside party — DNS, the K8s API — which addresses to talk to, and that answer arrives before any certificate has been checked. It cannot let an attacker join the cluster: the address a peer claims must be vouched for by its certificate, so a spoofed seed produces a connection that dies at the handshake.

That guarantee is configuration-dependent, which is the part worth stating plainly. A node only demands a peer certificate if its listener was configured to request one; where TLS is off, or where a half-configured listener never asks, there is nothing to check the claimed address against and a poisoned discovery answer is followed as given.

So pin what the resolver is allowed to say:

const dnsSeedProviderOptions = DnsSeedProviderOptions.create()
  .withHostname('actor-ts.example.com')
  .withSystemName('my-app')
  .withPort(2552)
  .withPinnedAddresses(['10.0.0.0/8'])
  .withLog((message) => logger.warn(message));

Addresses outside the list never become seeds. Details, including the SRV-mode caveat (targets are hostnames, so pins are suffixes there and the eventual A lookup stays unpinned), are on the DNS seed provider and Kubernetes API seed provider pages.

This is defence in depth: it lowers what a DNS or Endpoints compromise buys, and it is the only discovery-layer control left standing when mTLS is not in place.

What the wire edge rejects

Frames arrive as JSON and used to be cast to the protocol type rather than checked against it, so a peer could hand a node a value the code then read as if the type were true. Every frame is now validated before any handler sees it:

• The frame must be an object carrying a string kind. null, a bare string and a number are refused — previously null alone was an eight-byte remote process kill, and it needed no completed handshake.
• Node addresses must have a non-empty systemName and host plus a positive integer port. A port that arrives as the string "2552" is the case worth knowing about: it renders identically in every log line and keys every map the same way, but never compares equal — a node that merged its own address in that shape stopped recognising itself.
• A gossiped member’s status must be one of the seven legal values. An unknown one used to reach an exhaustive match that threw after the member had been stored, so the node crashed and re-gossiped the poisoned entry to its peers.
• A gossip batch is validated whole. One malformed member refuses the frame, so a bad entry cannot ride in behind a good one.

Two tiers, deliberately: a frame that fails validation is dropped and the connection stays up, because one bad frame should not cost a healthy peer its link. A handler that throws drops the connection — that is the case nobody understands, and it must not escape into the runtime’s socket callback.

Frame kinds registered by extensions (sharding, pub-sub, the receptionist, DistributedData, DevTools) pass this layer and validate their own payloads.

Every rejection is logged at WARN naming the peer and the offending field, so a dropped frame is diagnosable — a version mismatch and a hostile peer look different in the log.

What a well-formed frame still cannot grow

Shape validation says a frame is readable, not that acting on it is free. Pub-sub gossip is the clean example: a peer naming 100 000 topics it claims subscribers for sends one perfectly legal frame, and the receiver used to allocate 100 000 map entries for it — no local Subscribe, no malformed field, no log line. The receptionist had the same shape on its own gossip path.

Membership itself had the same shape, and it is the one registry nobody has to opt into: every clustered node keeps a map of members, and gossip is what fills it. The rules below decide whether a claim is believable; none of them bounded how many believable claims one peer may make. A sender announcing its own address is waved through by design — refusing that would mean no node could ever join — so naming a fresh address per frame allocated an entry per name.

These registries are capped now, and the caps apply to the gossip path as much as to local calls:

Registry	Capped by	Default
Live cluster members	cluster.max-members	1000
removed tombstones	cluster.max-tombstones	10000
Pub-sub subscribers per topic	cluster.pub-sub.max-subscribers-per-topic	10000
Pub-sub topics	cluster.pub-sub.max-topics	10000
Pub-sub remote claimants per topic	cluster.pub-sub.max-remote-nodes-per-topic	1000
Receptionist subscribers per key	cluster.receptionist.max-subscribers-per-key	1000
Receptionist subscribers total	cluster.receptionist.max-subscribers-total	10000

A claim over a cap is dropped and logged rather than refused on the wire — the frame is well-formed, and dropping the connection over it would let one noisy peer cost a healthy link. A local Subscribe over a cap is answered with SubscribeRejected instead, because the caller is in a position to do something about it. Both pub-sub registries also watch their subscribers, so the other half of the old growth — refs that stopped without ever unsubscribing — is reclaimed rather than capped.

Lower the defaults for a deployment where the legitimate numbers are far below them: a cap is only a bound on the damage, and one set far above real usage bounds very little.

Why membership needs two caps, and why the second is the real one

Splitting members from tombstones is not tidiness. A phantom member in up / joining / unreachable is a member the failure detector is watching, so it is downed and dropped failure-detector.down-after after the attacker stops feeding it — five seconds, at the default. A record gossiped as removed is watched by nothing: only cluster.tombstone.time-to-live reclaims it, a day later. So the flood that persists is the tombstone flood, and max-tombstones is the cap doing the work. Refusing one costs nothing either, because a tombstone for an address this node holds no record of suppresses nothing that exists — while a refused live record costs a legitimate member one gossip round, and comes with a WARN naming the cap.

Both caps are charged on the bucket a record moves into, not on whether it creates an entry. A gossiped record that keeps an entry where it already is — up → unreachable, or a newer tombstone over an older one — is a free in-place update. One that crosses between the buckets is charged to the bucket it enters: a tombstone re-incarnated as up needs room among the live members, a live member gossiped as removed needs room among the tombstones. Charging only entry creation let the two caps trade headroom with each other for free — a re-incarnation vacated the tombstone bucket without giving up a map slot, so the next flood of tombstones was admitted too, and alternating the two grew the map without bound while respecting both caps at every individual step. A refused conversion leaves the member live, where the failure detector reclaims it the slower way.

Tombstones this node mints itself — a peer’s leave, a downing decision, an operator down() — convert a record it already holds and are never subject to the cap. Capping its own bookkeeping would drop the suppression that stops stale gossip resurrecting an evicted address, which is a liveness bug wearing a security fix’s clothes.

The ceiling worth knowing is not the heap. Gossip carries the whole member list, so at roughly 110 000 entries a node’s own frame outgrows remote.max-frame-bytes and every peer terminates the connection on the length prefix — the node evicts itself from the cluster while still running, long before anything runs out of memory.

The order these landed in matters

Today the failure detector’s down → delete cascade is what bounds phantom live members to seconds. A downing strategy that gates that deletion behind a configured DowningProvider would remove the implicit bound and make the 24-hour arithmetic apply to every status, not just to tombstones — so these caps are a precondition for that change rather than an independent hardening step.

Who may say what

Passing the shape check does not make a frame believable. Gossip merges used to be decided purely by version magnitude, and versions are seeded from Date.now() — so an attacker could always pick a winning number and rewrite any member’s status, including the receiving node’s own. Two rules now sit in front of the merge:

• Nobody downgrades us. A claim about this node’s own address is refused. The one exception is promotion out of joining/weakly-up into up, which has to come from outside because it is the leader’s decision — and which is harmless to accept, since a joining node is already trying to become up. Everything else about our own record we decide ourselves.
• Third-party claims need a sender with standing. Saying something about another node requires the connection’s peer to be a member this node already considers active. A sender may always announce its own record — that is how joining works.

Both rules key on the connection’s peer, not on the payload’s from field, since that field is the one thing an attacker fully controls. The same reasoning covers two neighbours: a leave is only accepted from the node that is leaving, and a heartbeat refreshes the failure detector for the peer that sent it rather than the address it names — the latter previously let a peer keep a dead node looking healthy and made the receiver dial an attacker-chosen host.

Unreachability is deliberately not covered by these rules: “I cannot reach C” is inherently a third-party observation, and every node must converge on the same view before a downing provider decides. Refusing those claims would leave each node with only its own reachability picture.

This is not authentication

These rules remove the free-for-all — a claim now needs standing this node granted, rather than a large number. They do not make an unauthenticated peer harmless: on a cluster with no peer certificates the hello handshake carries no credential, so an attacker can announce itself, wait to be promoted, and then satisfy the second rule. Run mTLS if the network is not fully trusted — under mTLS the claimed address must match the peer’s certificate, which is what gives these rules a real subject.

A version cannot claim an address in advance

Version is a logical clock seeded from Date.now(), so “highest version wins” also decides what happens the first time an address is mentioned at all. That made an address claimable before the node owning it exists. A stranger announces itself under the address the next pod is about to get — announcing your own record is the claim the rules above never refuse — dates it close to the 24 h skew cap, and attaches whatever roles it likes. The leader’s promotion loop lifts the record into the active set, and the node that really owns the address loses every merge afterwards, because it seeds its version from its own clock and that is lower. Roles are what routing, sharding placement, singleton hosting and downing quorums are computed from, so the phantom is not a cosmetic row in a member list.

A gossiped member version is therefore held to a tight clock-skew budget — 5 minutes by default:

const clusterOptions = ClusterOptions.create()
  .withHost('10.0.0.5')
  .withPort(2552)
  .withMaxVersionSkewMs(30 * 60 * 1000);

The budget applies to every merge, not only to the record that introduces an address. It was originally the narrower rule — a tight cap on a first sighting, a generous 24 h one on every update — and that split could be stepped around by introducing the address first: two records for the same address in one frame, or a frame with no member records at all, which still makes the receiver file the sender’s own address. Any rule that lets a record earn the wider budget fails the same way one step later, because without peer certificates every step of the earning is something the attacker can produce.

Raise it for a deployment whose clocks are known to run loose; 24 * 60 * 60 * 1000 restores the old single-cap behaviour. A refusal is not exclusion — a node announcing itself is still recorded, at version 1 and without roles — but it is durable: a node whose clock runs further ahead than the budget stays in the member list without roles until its clock comes back. That was always this cap’s verdict on such a node; what changed is that the verdict now sticks instead of being reversed by the node’s second gossip frame.

Refused records are reported once per frame rather than once per record — a WARN naming the peer and the count, and the counter cluster_gossip_records_refused_total{reason} with reason one of version-skew or map-cap. Logging per record would hand a peer log amplification in place of the growth it just lost.

Per-deployment recipe

production.ts

import { TcpTransport, Cluster, ClusterOptions } from 'actor-ts';

const tlsOptionsType = {
  cert: fs.readFileSync(process.env.TLS_CERT_PATH!),
  key:  fs.readFileSync(process.env.TLS_KEY_PATH!),
  ca:   fs.readFileSync(process.env.TLS_CA_PATH!),
  rejectUnauthorized: true,
};

const transport = new TcpTransport(self, log, tlsOptionsType);

const clusterOptions = ClusterOptions.create()
  .withHost(host)
  .withPort(port)
  .withSeeds(seeds)
  .withTransport(transport);
await Cluster.join(system, clusterOptions);

Env vars carry paths; the cert-manager / vault mounts the files. Code stays generic across environments.

Self-signed certs in production

// Same self-signed cert reused across years + many nodes

Self-signed is convenient but: no rotation story, no revocation, cert compromise = cluster compromise. Use a private CA from day one of production.

`rejectUnauthorized: false`

rejectUnauthorized: false,   // ✗ accepts any cert

Tempting during debug (“certs don’t match my hostnames”). Means any TLS connection is accepted — defeats the authentication. Fix the cert hostnames instead.

Cert expiration

// Cert expires; cluster suddenly can't form

TLS certs have expiry dates. Without rotation automation, certs eventually expire and the cluster goes offline. Use cert-manager, set up renewal alerts, or use very-long-lived internal CA certs (10+ years).

Where to next

• Operations overview — production-readiness checklist.
• TLS everywhere — TLS for HTTP + brokers + journals.
• Master key rotation — rotating data-at-rest encryption keys.
• Transports — the underlying transport interface.
• Configuration — the HOCON keys for TLS settings.