Перейти к содержимому
Русский

Troubleshooting

Это содержимое пока не доступно на вашем языке.

When something’s wrong in production, this page is the starting point. Each section is a symptom, the likely cause(s), and what to check / log / measure to confirm.

For the diagnostic tools themselves:

Pods start, but never reach Up.

Causes to check (in order):

  1. Seeds unreachable — wrong addresses / DNS / RBAC.
  2. Cluster port firewalled — pods can’t talk on 2552.
  3. Different system names — ActorSystem.create('app-a') on one node, 'app-b' on another.
  4. TLS misconfiguration — handshake fails silently.

Diagnostics:

cluster.subscribe((evt) => {
if (evt instanceof MemberJoined) {
system.log.info(`saw join: ${evt.member.address}`);
} else if (evt instanceof SelfUp) {
system.log.info('self up');
}
});

Check whether SelfUp fires. If not, the local node hasn’t even bootstrapped. Look at the seed-provider output:

Terminal window
kubectl logs pod-1 | grep -i seed
# → "discovered N seeds: ..." ← should not be empty

If empty: seed provider’s selector / DNS / config is wrong.

Members oscillate between reachable and unreachable.

Causes:

  1. Tight failure-detector thresholds for the network’s jitter. See failure-detector tuning.
  2. Network partitions (real ones — diagnose at the infrastructure level). With no downing strategy, one that lasts past down-after is no longer a flap: each side removes the other and runs on as a cluster of its own. Configure lease-majority or another downing strategy for production.
  3. GC pauses longer than the unreachable threshold.

Diagnostics:

rate(cluster_unreachable_duration_ms_count[5m]) > 0.1
# Frequent transitions

In logs:

[INFO ] cluster — node-X marked unreachable
[INFO ] cluster — node-X marked reachable
[INFO ] cluster — node-X marked unreachable

Persistent flapping signals threshold tuning.

Messages to a sharding region don’t reach entities.

Causes:

  1. Coordinator not yet started — sharding is async; messages sent before the coordinator is ready get buffered.
  2. No nodes match the role — empty up-member set, so no shards are allocated.
  3. No hosting member is up — regions that could host the type are registered, but every one of them is on a member that is unreachable (or still joining), and a shard is only ever placed on a member that is up. New shards wait until one is.
  4. extractEntityId returns undefined — the message routing has nothing to hash.

Diagnostics:

sharding_shards_hosted{type="entity-type"}
# Should equal numShards (total) across the cluster

Logs:

Terminal window
kubectl logs pod-1 | grep -i shard
# → "coordinator: allocating shard X to node-Y"

If you see “no candidates,” your role filter rejects every node. Check role tags in Cluster.join.

If the coordinator warns that none of the registered regions that can host the type “is on a member that is up”, the role is fine and the hosting nodes are not. It says so once per episode, and the request stays pending until a member comes back. Under the default a member that stops answering is removed at down-after and its shards move to the survivors; one parked at unreachable — by a split-brain strategy that has not decided, or under failure-detector.auto-down = off — stays there until it recovers or something downs it, so check the member list before the sharding settings.

Sharding metrics show continuous rebalances; entities flicker.

Causes:

  1. Aggressive allocation strategy — LeastShardAllocationStrategy with rebalanceThreshold: 1 and frequent membership changes.
  2. Flapping cluster (see above) — a member that keeps going unreachable and coming back postpones rebalancing rather than triggering it. The coordinator rebalances only once the set of up members has held still for the settle window (down-after, or stable-after with a split-brain strategy), so a flap shorter than that moves no shard placed before it. Shards first asked for during the flap land on the members that are up, and those can be handed off once the member is back and the window has run out — see rebalance. A storm here comes from membership that changes more slowly than the window — real joins and removals, not blips.

Fix:

new LeastShardAllocationStrategy(/* threshold */ 5, /* max */ 3);

Higher threshold = less sensitivity to small imbalances.

preStart takes a long time; the actor’s first onReceive is delayed.

Cause: deep journal without snapshots. Recovery reads every event ever.

Diagnostics:

histogram_quantile(0.99, persistence_recovery_duration_ms_bucket)

If P99 recovery is > 1 second, set a snapshot policy:

override snapshotPolicy() { return everyNEvents(100); }

See Snapshots.

Symptom: an entity’s state depends on which node answers

Section titled “Symptom: an entity’s state depends on which node answers”

State jumps back after a rebalance or deploy; remembered entities vanish after a coordinator failover; a projection processes events twice.

Cause: each node reads its own database. Per-node storage (a SQLite file each, separate in-memory stores, two nodes each pointing at their own Postgres) means a moved entity replays whatever its new node holds. Nothing errors — the optimistic append check runs against the local database, so it structurally cannot fire across nodes.

Diagnostics — two log needles, both once per node:

node-local storage # the store can never be shared (#1356)
storage identity differs # shared-capable, but two instances (#1358)

The second one also catches misconfiguration you can’t see in the backend name: stale connection strings, a restored backup, two “identical” database containers.

Fix: point every node at the same database instance — see storage locality & identity — or use replicated event sourcing where per-node journals are the intended design. There is no automatic repair for histories that already diverged: pick the surviving database per persistence id and re-seed the others from it (migrateBetweenJournals), then fix the wiring before restarting.

Symptom: events recovered but state is wrong

Section titled “Symptom: events recovered but state is wrong”

On restart, the actor’s state doesn’t match what it should be from the journal.

Causes:

  1. onEvent has a side effect — runs during replay and somehow alters the state path.
  2. onEvent uses Date.now() or random — non-deterministic; each replay produces different state.
  3. Schema change without an adapter — old events have a different shape than what onEvent expects.

Diagnostics: compare the actor’s state after recovery to what’s in the journal. Replay manually in a test to isolate.

Symptom: the journal is unavailable and the actors just stop responding

Section titled “Symptom: the journal is unavailable and the actors just stop responding”

No error, no restart, no dead letters. Persistent actors accept commands and never answer; heap grows; the only log line is mailbox high-water mark at 10 000 and again at every doubling.

Cause: an append that neither succeeds nor fails. A backend that refuses connections fails fast on its own. One that accepts the connection and then stalls — a failed-over primary still holding the socket, a saturated connection pool, a network path that blackholes — leaves await this.persist(...) pending forever. The actor’s turn never ends, so its commands queue in the mailbox, which is unbounded by default; nothing throws and supervision never runs.

Fix: give the append a deadline. The journal is called through a circuit breaker, and the breaker’s per-call timeout is what bounds a single append. That covers every journal call of a PersistentActor, and a ReplicatedEventSourcedActor’s own persist and recovery — but not yet its append of an event a peer sent, which waits for #958. It ships unset, because omitting it is what expresses “no deadline”:

actor-ts.circuit-breaker.persistence-journal.call-timeout = 10s

With it set, the stall surfaces as a CircuitBreakerTimeoutError thrown out of your own onCommand — at the stall, naming it — and after max-failures of them (10 by default) the breaker opens and every later persist fails immediately with CircuitBreakerOpenError until reset-timeout (15 s) lets a probe through. Pick the timeout from the backend’s own P99 write latency with headroom, not from how long you are willing to wait. With metrics enabled that P99 is measured for you, at the breaker:

histogram_quantile(0.99, sum by (le) (rate(persistence_journal_latency_seconds_bucket{operation="append"}[5m])))

Once the timeout is set, the same families tell the failures apart: persistence_journal_failures_total{reason="timeout"} climbing is the stall being cut, and persistence_breaker_state{breaker="journal"} == 2 is the breaker refusing every append. A call that never settles is never timed, so before a call-timeout exists the stall itself does not show up in the histogram — only as the appends that stop arriving in it. See Persistence metrics.

The timeout gives up rather than cancelling, so a write that blew it may still land. That is safe — the throw skips the state fold and leaves lastSequenceNr untouched, so the restart re-reads the true head — but a non-replayable side effect in the handler has already run. A ReplicatedEventSourcedActor is the exception: a timed-out write that lands is never folded live and never reaches its peers, because only a successful persist folds and broadcasts. A later recovery folds it only if it replays it — a snapshot taken after the landing counts the event as covered, so a recovery from that snapshot never folds it, while a full replay does — see replicated event sourcing.

Related knobs, same block:

  • actor-ts.persistence.recovery-timeout (30s) does the same job for a recovery that never finishes: past it a PersistentActor fails with a RecoveryTimeoutError through onRecoveryFailure, and a ReplicatedEventSourcedActor — which has no such hook — fails its preStart with it, instead of sitting in preStart.
  • actor-ts.persistence.max-concurrent-recoveries (50) keeps a restart storm from opening thousands of concurrent reads against a pool sized for tens.
  • actor-ts.persistence.snapshot-is-optional lets an actor recover by full journal replay when only the snapshot store is down. It does not apply to an actor that set integrity() or encryption() — see PersistentActor.

A JournalConcurrencyError never counts towards max-failures, so a burst of losing conditional appends cannot open the breaker for everyone. If a breaker is opening for something else you consider benign, name it rather than raising the budget:

actor-ts.circuit-breaker.persistence-journal.ignored-error-names = ["MyTransientError"]

Heap grows linearly with uptime; eventually OOM.

Causes (most common first):

  1. A mailbox with a slow consumer — mailboxes are unbounded by default, so a producer that outruns its consumer grows the heap until one of them stops. actor_mailbox_size shows which actor, and the log carries a backlog warning from 10 000 messages up.
  2. Subscriber set leaks — actors registering with the event stream or DistributedPubSub but never unsubscribing on stop.
  3. DistributedData keys accumulating — LWWMap with millions of keys.
  4. Persistent buffers — stash buffers, ask reply-to refs.

Diagnostics:

actor_mailbox_size{class=...}
# Find actors with persistently large queues
histogram_quantile(0.99, rate(actor_mailbox_wait_seconds_bucket[5m]))
# …and how long messages are waiting before anyone gets to them.
# A series only exists above 10 000 queued messages, so this is the
# signal that moves first — a backlog shows up as wait long before
# it shows up as size.
histogram_quantile(0.99, rate(actor_mailbox_depth_bucket[5m]))
# How deep the queues actually got. Unlike the gauge this has no floor,
# so a burst of a few thousand shows up here and nowhere else — and
# unlike the gauge it is a distribution, so a spike between two 2-second
# samples is still recorded.
histogram_quantile(0.99,
rate(actor_dispatcher_queue_delay_seconds_bucket[5m]))
# One level up: turns waiting for the dispatcher rather than messages
# waiting for the actor. High here and low above means the actors are
# fine and the scheduling is the queue.
Terminal window
# Heap dump via runtime tools:
node --inspect / Bun's profiler

Check mailbox sizing + the leak patterns in event stream.

Actor work is fine; HTTP responses are slow.

Cause: an actor monopolizes the event loop, starving HTTP handlers.

Fix: per-actor ThroughputDispatcher on the heavy actor.

Bun / Vitest test process doesn’t exit.

Cause: await system.terminate() not called in a fixture teardown. Leaked schedulers / actor cells keep the event loop alive.

Fix:

afterEach(async () => {
await tk.shutdown();
});

See TestKit.

A tell returned normally, the recipient never handled it, and nothing threw.

tell is fire-and-forget: it cannot report a recipient that has stopped, a selection that matched nothing, or a behavior that answered unhandled. Every one of those becomes a dead letter, and since #1000 a default system says so in the log:

[…] INFO dead letter to actor-ts://app/user/worker: OrderPlaced from actor-ts://app/user/api

Search the log for dead letter to and read the recipient path — that is the actor the message failed to reach, not the dead-letter office. Three things to know before concluding there was only one:

  • The record is throttled. After ten letters a single warn line announces that logging is suspending for five minutes; a second one reports the tally when it resumes. Raise actor-ts.diagnostics.log-dead-letters, or set actor-ts.diagnostics.log-dead-letters-suspend-duration = 0 to log every one, while you are looking.
  • The payload is not in the log, only the message’s class — that is deliberate. To see the message itself, subscribe to DeadLetter on the event stream, or turn on the queue with actor-ts.dead-letters.store = "memory" and read it back with system.deadLetterQueue.list().
  • Shutdown is silent by default. If the message went missing during terminate(), see the next symptom.

See Dead letters.

Work that was queued when terminate() was called never ran, and turns up in the dead-letter stream instead.

terminate() drains the actors under /user first, so a plain backlog is handled. Four things it will not wait for:

  • The drain budget ran out. It is actor-ts.system.shutdown-drain-timeout, 2 s by default. An actor that keeps producing new work while draining — a self-tell loop, a rally between two actors — never goes quiet, so only the budget ends it. Raise the budget, and raise actor-ts.coordinated-shutdown.default-phase-timeout with it if you shut down through the pipeline.
  • The mailbox was parked. context.throttle(...) and a supervisor-suspended mailbox both count as quiet, because neither drains at a rate a shutdown can wait for. Cancel the throttle before shutting down if the backlog matters.
  • The work was not in a mailbox yet. A context.timers tick that had not fired, or a tell from a promise a handler started without awaiting it, arrives after the tree looks quiet. Await it, or fire it as a message instead.
  • It was addressed to /system. Framework actors are never quiet by design, so they are not part of the drain.

To confirm which, subscribe to DeadLetter on the event stream before shutting down — the dead letter names the message and the recipient. Subscribing has to happen before, because nothing keeps a dead letter by default, so there is nothing to go back and read afterwards. The log will not help here either: this particular burst is the one case that is not logged, because an orderly shutdown produces it every time (set actor-ts.diagnostics.log-dead-letters-during-shutdown = on if you want it anyway). Set actor-ts.dead-letters.store = "persistent" if you would rather have the record without knowing in advance that you will want it — see Dead letters.

Tests pass locally, fail in CI.

The usual cause is real-clock timing: a fixed sleep long enough on an idle machine and too short on a loaded CI runner. Use ManualScheduler to control time deterministically, or wait on the observable state rather than on a duration.

That is not the only cause, and guessing between them wastes an afternoon. Diagnosing test flakes has the repeat-run harness that separates a flake from a broken test, plus the catalogued causes — a timer quantum that fires early, a dispatcher hop no poll interval can win, and the three multi-node suites CI does not run at all.

1. Check logs for ERROR-level entries. Filter by time window
around the issue.
2. Check stock metrics — what changed? (rate of restarts,
mailbox depth, member count).
3. Check cluster events on the event stream.
4. If a request is failing, follow the trace ID through logs /
trace backend.
5. Reproduce in a multi-node-spec test if you can.

The framework is quiet by default and stays that way unless you ask. Four switches under actor-ts.diagnostics turn on what it can say about itself:

actor-ts {
logger.level = debug # the second switch — see below
diagnostics {
log-config-on-start = on # one info record: every key, and which layer set it
debug {
unhandled = on # a message an actor was handed and declined
lifecycle = on # actor start, stop, restart
event-stream = on # subscribe / unsubscribe, never publish
}
}
}

log-config-on-start is the one to reach for first when a setting is not doing what you wrote. It writes the merged tree — every key, sorted, each with the layer that won (reference, project-reference, application, profile, secrets or override), whether it displaced a lower one and the environment variables its value came from — so “my application.conf is being ignored” and “my application.conf says something else” stop looking alike. reference is the reference.conf bundled with actor-ts, project-reference your project’s own one, its active profile included, profile is the active profile’s block of the application file, secrets is the secrets file, its own active profile included, and override is what your code passed. The header names the active profile and the four files that were looked for — the project reference.conf, application.conf, the secrets file and the .env file — and says when one was not there, which settles the first of those on its own. A key that holds a dot — a host name, say — prints in double quotes, as HOCON writes it: webhooks."hooks.example.com" is one key, where webhooks.hooks.example.com would be four. An excerpt, with production active:

actor-ts.diagnostics.log-config-on-start = "on" [application, overrides a lower layer]
actor-ts.diagnostics.log-dead-letters = 10 [reference]
my-app.auth-header = <redacted> [application]
my-app.db.dsn = <redacted> [application, from $DB_PASSWORD]
my-app.db.host = "db.internal" [profile, overrides a lower layer]
my-app.db.password = <redacted> [secrets]
my-app.db.pool-size = 8 [project-reference]
my-app.db.user = <redacted> [override, overrides a lower layer]
my-app.payments.api-key = <redacted> [application]
my-app.upstreams = [{"host":"db-1","password":"<redacted>"}] [application]

Two things to know before you paste that record into a ticket. It is an info record, so it needs no second switch — but it prints your configuration, and four rules decide what it withholds. A value is withheld when any whole word of its key name — of the key itself or of a branch above it, wherever in the name the word stands — is password, passphrase, passwd, pass, secret, token, key, credential, auth or authorization, singular or plural, and the value is not a boolean: the auth-header and the api-key above. A name is cut into words at every character that is not a letter or a digit, at each camelCase hump and in front of a digit after a letter, so apiKey holds key, password2 holds the word it numbers and passivation-idle holds no pass. Those words glued onto another (pgpassword, clientsecret, ghtoken) count too, all but key, pass and auth, which end ordinary words; of the …key names apikey, accesskey, secretkey and privatekey count, and pwd counts beside another word of the key’s path, as in mysql-pwd or db.pwd — only a variable called just PWD, a shell’s working directory, does not. A value an environment variable supplied is withheld when the variable’s name reads as a credential by the same rule, whatever its key is called: the dsn above, built from ${DB_PASSWORD} out of a .env file, and every value that refers to it. Every value the secrets file sets is withheld whatever its key is called — even after a higher layer overrides the key, like the user above. Both hold in a WorkerMesh worker’s dump too. And a list is one key, so the objects inside it are read field by field by the first rule: their password prints as "<redacted>", the rest of the list as written. The one thing masked by value is the userinfo of a URL: redis://default:s3cret@redis.example:6379 prints as redis://***@redis.example:6379 under any key, dsn and connection-string included. Every other secret outside the secrets file, under a key and out of a variable that do not name one, is printed in full — a bare password written out in a dsn, a ?password= or ?authToken= in a query string, SQL Server’s Server=…;Password=… form, and memcached’s scheme-less user:password@host server list. Ordinary tuning values are not withheld just for spelling one of those words inside a longer one — passivation-idle and keyspace print, as they must for the dump to be worth reading. One that holds such a word whole is withheld with it: a token-bucket-size of your own prints as <redacted>, and only the keys reference.conf ships, such as cache.redis.key-prefix, are exempted by name. Configuration states the limits in full.

The three debug.* switches need two switches, and this is the usual reason one of them appears not to work: the key decides whether a record is produced, logger.level = debug decides whether it is written. Each is off by default because each is one record per event on a path your application drives — per declined message, per actor transition, per subscription — so turn on the one that matches the symptom rather than all three.

  • Operations overview — the broader production checklist.
  • FAQ — common questions and pitfalls.
  • Stock metrics — what to read when symptoms emerge.
  • Logging — how to make logs actually useful.
  • Tracing — per-request flow when logs aren’t enough.