Troubleshooting
このコンテンツはまだ日本語訳がありません。
When something’s wrong in production, this page is the starting point. Each section is a symptom, the likely cause(s), and what to check / log / measure to confirm.
For the diagnostic tools themselves:
- Logging — structured logs.
- Stock metrics — built-in actor / cluster metrics.
- Tracing — per-request flow.
Cluster
Section titled “Cluster”Symptom: cluster won’t form
Section titled “Symptom: cluster won’t form”Pods start, but never reach Up.
Causes to check (in order):
- Seeds unreachable — wrong addresses / DNS / RBAC.
- Cluster port firewalled — pods can’t talk on 2552.
- Different system names —
ActorSystem.create('app-a')on one node,'app-b'on another. - TLS misconfiguration — handshake fails silently.
Diagnostics:
cluster.subscribe((evt) => { if (evt instanceof MemberJoined) { system.log.info(`saw join: ${evt.member.address}`); } else if (evt instanceof SelfUp) { system.log.info('self up'); }});Check whether SelfUp fires. If not, the local node hasn’t
even bootstrapped. Look at the seed-provider output:
kubectl logs pod-1 | grep -i seed# → "discovered N seeds: ..." ← should not be emptyIf empty: seed provider’s selector / DNS / config is wrong.
Symptom: cluster flaps
Section titled “Symptom: cluster flaps”Members oscillate between reachable and unreachable.
Causes:
- Tight failure-detector thresholds for the network’s jitter. See failure-detector tuning.
- Network partitions (real ones — diagnose at the
infrastructure level). With no downing strategy, one that lasts
past
down-afteris no longer a flap: each side removes the other and runs on as a cluster of its own. Configurelease-majorityor another downing strategy for production. - GC pauses longer than the unreachable threshold.
Diagnostics:
rate(cluster_unreachable_duration_ms_count[5m]) > 0.1# Frequent transitionsIn logs:
[INFO ] cluster — node-X marked unreachable[INFO ] cluster — node-X marked reachable[INFO ] cluster — node-X marked unreachablePersistent flapping signals threshold tuning.
Sharding
Section titled “Sharding”Symptom: sharded entities don’t spawn
Section titled “Symptom: sharded entities don’t spawn”Messages to a sharding region don’t reach entities.
Causes:
- Coordinator not yet started — sharding is async; messages sent before the coordinator is ready get buffered.
- No nodes match the
role— empty up-member set, so no shards are allocated. - No hosting member is
up— regions that could host the type are registered, but every one of them is on a member that isunreachable(or still joining), and a shard is only ever placed on a member that is up. New shards wait until one is. - extractEntityId returns undefined — the message routing has nothing to hash.
Diagnostics:
sharding_shards_hosted{type="entity-type"}# Should equal numShards (total) across the clusterLogs:
kubectl logs pod-1 | grep -i shard# → "coordinator: allocating shard X to node-Y"If you see “no candidates,” your role filter rejects every
node. Check role tags in Cluster.join.
If the coordinator warns that none of the registered regions that can
host the type “is on a member that is up”, the role is fine and the
hosting nodes are not. It says so once per episode, and the request
stays pending until a member comes back. Under the default a member
that stops answering is removed at down-after and its shards move to
the survivors; one parked at unreachable — by a split-brain strategy
that has not decided, or under failure-detector.auto-down = off —
stays there until it recovers or something downs it, so check the
member list before the sharding settings.
Symptom: rebalance storm
Section titled “Symptom: rebalance storm”Sharding metrics show continuous rebalances; entities flicker.
Causes:
- Aggressive allocation strategy —
LeastShardAllocationStrategywithrebalanceThreshold: 1and frequent membership changes. - Flapping cluster (see above) — a member that keeps going
unreachableand coming back postpones rebalancing rather than triggering it. The coordinator rebalances only once the set ofupmembers has held still for the settle window (down-after, orstable-afterwith a split-brain strategy), so a flap shorter than that moves no shard placed before it. Shards first asked for during the flap land on the members that are up, and those can be handed off once the member is back and the window has run out — see rebalance. A storm here comes from membership that changes more slowly than the window — real joins and removals, not blips.
Fix:
new LeastShardAllocationStrategy(/* threshold */ 5, /* max */ 3);Higher threshold = less sensitivity to small imbalances.
Persistence
Section titled “Persistence”Symptom: actor takes 30 seconds to start
Section titled “Symptom: actor takes 30 seconds to start”preStart takes a long time; the actor’s first
onReceive is delayed.
Cause: deep journal without snapshots. Recovery reads every event ever.
Diagnostics:
histogram_quantile(0.99, persistence_recovery_duration_ms_bucket)If P99 recovery is > 1 second, set a snapshot policy:
override snapshotPolicy() { return everyNEvents(100); }See Snapshots.
Symptom: an entity’s state depends on which node answers
Section titled “Symptom: an entity’s state depends on which node answers”State jumps back after a rebalance or deploy; remembered entities vanish after a coordinator failover; a projection processes events twice.
Cause: each node reads its own database. Per-node storage (a SQLite file each, separate in-memory stores, two nodes each pointing at their own Postgres) means a moved entity replays whatever its new node holds. Nothing errors — the optimistic append check runs against the local database, so it structurally cannot fire across nodes.
Diagnostics — two log needles, both once per node:
node-local storage # the store can never be shared (#1356)storage identity differs # shared-capable, but two instances (#1358)The second one also catches misconfiguration you can’t see in the backend name: stale connection strings, a restored backup, two “identical” database containers.
Fix: point every node at the same database instance — see
storage locality &
identity — or
use replicated event
sourcing where
per-node journals are the intended design. There is no
automatic repair for histories that already diverged: pick the
surviving database per persistence id and re-seed the others from
it (migrateBetweenJournals), then fix the wiring before
restarting.
Symptom: events recovered but state is wrong
Section titled “Symptom: events recovered but state is wrong”On restart, the actor’s state doesn’t match what it should be from the journal.
Causes:
onEventhas a side effect — runs during replay and somehow alters the state path.onEventusesDate.now()or random — non-deterministic; each replay produces different state.- Schema change without an adapter — old events have a
different shape than what
onEventexpects.
Diagnostics: compare the actor’s state after recovery to what’s in the journal. Replay manually in a test to isolate.
Symptom: the journal is unavailable and the actors just stop responding
Section titled “Symptom: the journal is unavailable and the actors just stop responding”No error, no restart, no dead letters. Persistent actors
accept commands and never answer; heap grows; the only log line
is mailbox high-water mark at 10 000 and again at every
doubling.
Cause: an append that neither succeeds nor fails. A
backend that refuses connections fails fast on its own. One that
accepts the connection and then stalls — a failed-over primary
still holding the socket, a saturated connection pool, a network
path that blackholes — leaves await this.persist(...) pending
forever. The actor’s turn never ends, so its commands queue in
the mailbox, which is unbounded by default; nothing throws and
supervision never runs.
Fix: give the append a deadline. The journal is called
through a circuit breaker, and the breaker’s per-call timeout is
what bounds a single append. That covers every journal call of a
PersistentActor, and a ReplicatedEventSourcedActor’s own
persist and recovery — but not yet its append of an event a peer
sent, which waits for #958. It ships unset, because omitting it
is what expresses “no deadline”:
actor-ts.circuit-breaker.persistence-journal.call-timeout = 10sWith it set, the stall surfaces as a CircuitBreakerTimeoutError
thrown out of your own onCommand — at the stall, naming it —
and after max-failures of them (10 by default) the breaker
opens and every later persist fails immediately with
CircuitBreakerOpenError until reset-timeout (15 s) lets a
probe through. Pick the timeout from the backend’s own P99 write
latency with headroom, not from how long you are willing to wait.
With metrics enabled that P99 is measured for you, at the breaker:
histogram_quantile(0.99, sum by (le) (rate(persistence_journal_latency_seconds_bucket{operation="append"}[5m])))Once the timeout is set, the same families tell the failures apart:
persistence_journal_failures_total{reason="timeout"} climbing is the
stall being cut, and persistence_breaker_state{breaker="journal"} == 2
is the breaker refusing every append. A call that never settles is
never timed, so before a call-timeout exists the stall itself does
not show up in the histogram — only as the appends that stop arriving
in it. See Persistence
metrics.
The timeout gives up rather than cancelling, so a write that blew
it may still land. That is safe — the throw skips the state fold
and leaves lastSequenceNr untouched, so the restart re-reads the
true head — but a non-replayable side effect in the handler has
already run. A ReplicatedEventSourcedActor is the exception: a
timed-out write that lands is never folded live and never reaches
its peers, because only a successful persist folds and broadcasts.
A later recovery folds it only if it replays it — a snapshot taken
after the landing counts the event as covered, so a recovery from
that snapshot never folds it, while a full replay does — see
replicated event
sourcing.
Related knobs, same block:
actor-ts.persistence.recovery-timeout(30s) does the same job for a recovery that never finishes: past it aPersistentActorfails with aRecoveryTimeoutErrorthroughonRecoveryFailure, and aReplicatedEventSourcedActor— which has no such hook — fails itspreStartwith it, instead of sitting inpreStart.actor-ts.persistence.max-concurrent-recoveries(50) keeps a restart storm from opening thousands of concurrent reads against a pool sized for tens.actor-ts.persistence.snapshot-is-optionallets an actor recover by full journal replay when only the snapshot store is down. It does not apply to an actor that setintegrity()orencryption()— see PersistentActor.
A JournalConcurrencyError never counts towards max-failures,
so a burst of losing conditional appends cannot open the breaker
for everyone. If a breaker is opening for something else you
consider benign, name it rather than raising the budget:
actor-ts.circuit-breaker.persistence-journal.ignored-error-names = ["MyTransientError"]Memory + performance
Section titled “Memory + performance”Symptom: memory grows unboundedly
Section titled “Symptom: memory grows unboundedly”Heap grows linearly with uptime; eventually OOM.
Causes (most common first):
- A mailbox with a slow consumer — mailboxes are unbounded by
default, so a producer that outruns its consumer grows the heap
until one of them stops.
actor_mailbox_sizeshows which actor, and the log carries a backlog warning from 10 000 messages up. - Subscriber set leaks — actors registering with the event stream or DistributedPubSub but never unsubscribing on stop.
- DistributedData keys accumulating —
LWWMapwith millions of keys. - Persistent buffers — stash buffers, ask reply-to refs.
Diagnostics:
actor_mailbox_size{class=...}# Find actors with persistently large queues
histogram_quantile(0.99, rate(actor_mailbox_wait_seconds_bucket[5m]))# …and how long messages are waiting before anyone gets to them.# A series only exists above 10 000 queued messages, so this is the# signal that moves first — a backlog shows up as wait long before# it shows up as size.
histogram_quantile(0.99, rate(actor_mailbox_depth_bucket[5m]))# How deep the queues actually got. Unlike the gauge this has no floor,# so a burst of a few thousand shows up here and nowhere else — and# unlike the gauge it is a distribution, so a spike between two 2-second# samples is still recorded.
histogram_quantile(0.99, rate(actor_dispatcher_queue_delay_seconds_bucket[5m]))# One level up: turns waiting for the dispatcher rather than messages# waiting for the actor. High here and low above means the actors are# fine and the scheduling is the queue.# Heap dump via runtime tools:node --inspect / Bun's profilerCheck mailbox sizing + the leak patterns in event stream.
Symptom: HTTP latency high under load
Section titled “Symptom: HTTP latency high under load”Actor work is fine; HTTP responses are slow.
Cause: an actor monopolizes the event loop, starving HTTP handlers.
Fix: per-actor ThroughputDispatcher on the heavy actor.
Tests + dev
Section titled “Tests + dev”Symptom: tests hang at the end
Section titled “Symptom: tests hang at the end”Bun / Vitest test process doesn’t exit.
Cause: await system.terminate() not called in a fixture
teardown. Leaked schedulers / actor cells keep the event
loop alive.
Fix:
afterEach(async () => { await tk.shutdown();});See TestKit.
Symptom: my message vanished
Section titled “Symptom: my message vanished”A tell returned normally, the recipient never handled it, and
nothing threw.
tell is fire-and-forget: it cannot report a recipient that has
stopped, a selection that matched nothing, or a behavior that answered
unhandled. Every one of those becomes a dead letter, and since
#1000 a default system says so in the log:
[…] INFO dead letter to actor-ts://app/user/worker: OrderPlaced from actor-ts://app/user/apiSearch the log for dead letter to and read the recipient path — that
is the actor the message failed to reach, not the dead-letter office.
Three things to know before concluding there was only one:
- The record is throttled. After ten letters a single
warnline announces that logging is suspending for five minutes; a second one reports the tally when it resumes. Raiseactor-ts.diagnostics.log-dead-letters, or setactor-ts.diagnostics.log-dead-letters-suspend-duration = 0to log every one, while you are looking. - The payload is not in the log, only the message’s class — that is
deliberate. To see the message itself, subscribe to
DeadLetteron the event stream, or turn on the queue withactor-ts.dead-letters.store = "memory"and read it back withsystem.deadLetterQueue.list(). - Shutdown is silent by default. If the message went missing during
terminate(), see the next symptom.
See Dead letters.
Symptom: shutdown dropped my messages
Section titled “Symptom: shutdown dropped my messages”Work that was queued when terminate() was called never ran, and
turns up in the dead-letter stream instead.
terminate() drains the actors under /user first, so a plain
backlog is handled. Four things it will not wait for:
- The drain budget ran out. It is
actor-ts.system.shutdown-drain-timeout, 2 s by default. An actor that keeps producing new work while draining — a self-tell loop, a rally between two actors — never goes quiet, so only the budget ends it. Raise the budget, and raiseactor-ts.coordinated-shutdown.default-phase-timeoutwith it if you shut down through the pipeline. - The mailbox was parked.
context.throttle(...)and a supervisor-suspended mailbox both count as quiet, because neither drains at a rate a shutdown can wait for. Cancel the throttle before shutting down if the backlog matters. - The work was not in a mailbox yet. A
context.timerstick that had not fired, or atellfrom a promise a handler started withoutawaiting it, arrives after the tree looks quiet. Await it, or fire it as a message instead. - It was addressed to
/system. Framework actors are never quiet by design, so they are not part of the drain.
To confirm which, subscribe to DeadLetter on the event stream before
shutting down — the dead letter names the message and the recipient.
Subscribing has to happen before, because nothing keeps a dead letter
by default, so there is nothing to go back and read afterwards. The log
will not help here either: this particular burst is the one case that is
not logged, because an orderly shutdown produces it every time (set
actor-ts.diagnostics.log-dead-letters-during-shutdown = on if you want
it anyway). Set actor-ts.dead-letters.store = "persistent" if you
would rather have the record without knowing in advance that you will
want it — see Dead letters.
Symptom: flaky timing-sensitive tests
Section titled “Symptom: flaky timing-sensitive tests”Tests pass locally, fail in CI.
The usual cause is real-clock timing: a fixed sleep long enough on an idle machine and too short on a loaded CI runner. Use ManualScheduler to control time deterministically, or wait on the observable state rather than on a duration.
That is not the only cause, and guessing between them wastes an afternoon. Diagnosing test flakes has the repeat-run harness that separates a flake from a broken test, plus the catalogued causes — a timer quantum that fires early, a dispatcher hop no poll interval can win, and the three multi-node suites CI does not run at all.
Where to start when nothing’s obvious
Section titled “Where to start when nothing’s obvious”1. Check logs for ERROR-level entries. Filter by time window around the issue.2. Check stock metrics — what changed? (rate of restarts, mailbox depth, member count).3. Check cluster events on the event stream.4. If a request is failing, follow the trace ID through logs / trace backend.5. Reproduce in a multi-node-spec test if you can.Making the framework talk
Section titled “Making the framework talk”The framework is quiet by default and stays that way unless you ask. Four
switches under actor-ts.diagnostics turn on what it can say about itself:
actor-ts { logger.level = debug # the second switch — see below diagnostics { log-config-on-start = on # one info record: every key, and which layer set it debug { unhandled = on # a message an actor was handed and declined lifecycle = on # actor start, stop, restart event-stream = on # subscribe / unsubscribe, never publish } }}log-config-on-start is the one to reach for first when a setting is not
doing what you wrote. It writes the merged tree — every key, sorted, each
with the layer that won (reference, project-reference, application,
profile, secrets or override), whether it displaced a lower one and the
environment variables its value came from — so “my application.conf is
being ignored” and “my application.conf says something else” stop looking
alike. reference is the reference.conf bundled with actor-ts,
project-reference your project’s
own one, its active
profile included, profile is the active profile’s block of the application
file, secrets is the secrets file,
its own active profile included, and override is what your code passed.
The header names the active profile and the four files that were looked
for — the project reference.conf, application.conf, the secrets file and
the .env file — and says when one was not there, which settles the first
of those on its own. A key
that holds a dot — a host name, say — prints in double quotes, as HOCON writes
it: webhooks."hooks.example.com" is one key, where
webhooks.hooks.example.com would be four. An excerpt, with production
active:
actor-ts.diagnostics.log-config-on-start = "on" [application, overrides a lower layer] actor-ts.diagnostics.log-dead-letters = 10 [reference] my-app.auth-header = <redacted> [application] my-app.db.dsn = <redacted> [application, from $DB_PASSWORD] my-app.db.host = "db.internal" [profile, overrides a lower layer] my-app.db.password = <redacted> [secrets] my-app.db.pool-size = 8 [project-reference] my-app.db.user = <redacted> [override, overrides a lower layer] my-app.payments.api-key = <redacted> [application] my-app.upstreams = [{"host":"db-1","password":"<redacted>"}] [application]Two things to know before you paste that record into a ticket. It is an
info record, so it needs no second switch — but it prints your
configuration, and four rules decide what it withholds. A value is
withheld when any whole word of its key name — of the key itself or of a
branch above it, wherever in the name the word stands — is password,
passphrase, passwd, pass, secret, token, key, credential, auth
or authorization, singular or plural, and the value is not a boolean: the
auth-header and the api-key above. A name is cut into words at every
character that is not a letter or a digit, at each camelCase hump and in
front of a digit after a letter, so apiKey holds key, password2 holds
the word it numbers and passivation-idle holds no pass. Those words
glued onto another (pgpassword, clientsecret, ghtoken) count too, all
but key, pass and auth, which end ordinary words; of the …key names
apikey, accesskey, secretkey and privatekey count, and pwd counts
beside another word of the key’s path, as in mysql-pwd or db.pwd — only
a variable called just PWD, a shell’s working directory, does not. A
value an environment variable supplied is withheld when the
variable’s name reads as a credential by the same rule, whatever its key is
called: the dsn above, built from ${DB_PASSWORD} out of a .env file,
and every value that refers to it. Every value the secrets file sets is
withheld whatever its key is called — even after a higher layer overrides
the key, like the user above. Both hold in a WorkerMesh worker’s dump
too. And a list is one key, so the objects inside it are read field by
field by the first rule: their password prints as "<redacted>", the rest
of the list as written. The one thing masked by value is the userinfo of a
URL: redis://default:s3cret@redis.example:6379 prints as
redis://***@redis.example:6379 under any key, dsn and connection-string
included. Every other secret outside the secrets file, under a key and out
of a variable that do not name one, is printed in full — a bare password
written out in a dsn, a ?password= or ?authToken= in a query string,
SQL Server’s Server=…;Password=… form, and memcached’s scheme-less
user:password@host server list. Ordinary tuning values are not withheld
just for spelling one
of those words inside a longer one — passivation-idle and keyspace print,
as they must for the dump to be worth reading. One that holds such a word
whole is withheld with it: a token-bucket-size of your own prints as
<redacted>, and only the keys reference.conf ships, such as
cache.redis.key-prefix, are exempted by name.
Configuration states the limits in full.
The three debug.* switches need two switches, and this is the usual
reason one of them appears not to work: the key decides whether a record is
produced, logger.level = debug decides whether it is written. Each is off
by default because each is one record per event on a path your application
drives — per declined message, per actor transition, per subscription — so
turn on the one that matches the symptom rather than all three.
Where to next
Section titled “Where to next”- Operations overview — the broader production checklist.
- FAQ — common questions and pitfalls.
- Stock metrics — what to read when symptoms emerge.
- Logging — how to make logs actually useful.
- Tracing — per-request flow when logs aren’t enough.
