Dead letters
이 콘텐츠는 아직 번역되지 않았습니다.
A dead letter is a message the system could not deliver: a tell to a
ref whose actor has stopped, a selection that resolved to nothing, a
behavior that answered unhandled, an actor that declined one with
this.unhandled(m),
a cluster singleton with no host. Nothing was lost by accident — the
framework noticed, and this is how it says so.
The two unhandled entries are a different animal from the rest: the
message was delivered, and a live actor turned it down. A counter tells
them apart — actor_unhandled_total{class} rises only for a declined
message, and unlike actor_dead_letters_total it does not need the queue
below to be switched on first.
Three framework base classes decline this way by default. The
onSelfMessage hook of WebsocketServerActor, WebsocketClientActor and
MqttActor receives whatever none of the actor’s own kinds matched — an
application message your subclass never handles, or a body a remote node
addressed to the actor’s path — and unless you override it, it turns that
message into a dead letter naming the actor, after one warn that names
the message’s class and never its payload.
What happens by default
Section titled “What happens by default”Every dead letter is wrapped in a DeadLetter, logged, and published
on the event stream. That is the whole of
the default behaviour.
The record is one info line naming the recipient path, the sender when
there is one, and the message’s class — never the payload, for the same
data-protection reason store below defaults to off. It is throttled: the
first ten dead letters are logged in full, then a single warn line says
logging is suspending, and the next five minutes are silent. When it
resumes, one more warn line says how many letters went unreported. Both
numbers are settings — see
actor-ts.diagnostics — and 0 means “never
log” for the count and “never suspend” for the window.
The burst terminate() produces while draining mailboxes is not logged
unless you ask for it with log-dead-letters-during-shutdown = on: an
orderly shutdown drains every queued and stashed message to dead letters,
and reporting that makes a clean stop read as an incident. An individual
ref.stop() still logs, because the system is not terminating.
What the log line is not is a record. It carries no payload and nothing keeps it in the process; if nothing subscribes to the stream either, the letter itself is gone. That is fine for a development run and a poor answer to “what did we drop during last night’s incident?”, which is what the queue below is for.
import { Actor, DeadLetter } from 'actor-ts';import { classNameOf } from 'actor-ts/util';
class Watcher extends Actor<DeadLetter> { override preStart(): void { this.system.eventStream.subscribe(this.self, DeadLetter); } override onReceive(letter: DeadLetter): void { // The class, never the payload: interpolating the message prints a // string payload verbatim and throws on a decoded body with no usable // toString. this.system.log.warn(`undelivered: ${classNameOf(letter.message)} -> ${letter.recipient.path}`); }}Every dead letter names its recipient — the actor the message failed to reach, not the dead-letter office. That is what makes one letter distinguishable from another, and it is what the filters and the replay below are built on.
Turning the queue on
Section titled “Turning the queue on”system.deadLetterQueue always exists and, by default, keeps nothing. Four
stores, one axis — how much of the letter is kept:
store | Keeps | Survives a restart |
|---|---|---|
off (default) | nothing | — |
metrics | a counter, no payload | — |
memory | a bounded ring | no |
persistent | the same ring, plus an append-only journal log | yes |
metrics is there because retaining a payload is a different decision
from observing a rate. A ring holds a strong reference to every
undeliverable message, which is a data-protection question the moment a
payload says anything about a person; a counter is not. list() stays
empty and replay answers unknown-entry for every id — nothing was kept,
so there is nothing to hand back. You cannot approximate it with a tiny
ring: max-entries must be positive, so even the smallest one keeps a live
payload for a whole retention window.
actor-ts.dead-letters { store = "memory" max-entries = 1000 retention = 1h max-replays = 3}The same knobs are available as an options family, handed to the system at creation time:
import { ActorSystem, ActorSystemOptions, DeadLetterQueueOptions } from 'actor-ts';
const deadLetterOptions = DeadLetterQueueOptions.create() .withStore('persistent') .withMaxEntries(5_000) .withRetentionMs(6 * 60 * 60 * 1_000);
const systemOptions = ActorSystemOptions.create() .withDeadLetters(deadLetterOptions);
const system = ActorSystem.create('orders', systemOptions);Precedence is the usual one — explicit options beat HOCON beat the
built-in defaults — and it applies field by field, so naming one knob in
code leaves the rest of the actor-ts.dead-letters block in effect.
Every key in this block decides what is retained, which is why they live
under actor-ts.dead-letters and not beside the logging knobs in
actor-ts.diagnostics: those decide how loudly
a letter is announced, are read on the publish path rather than by the
queue, and sit downstream of capture by design — so a suppression setting
can never quietly make this record incomplete.
Inspecting
Section titled “Inspecting”The queue also has a panel: once it is switched on, the dead-letter inspector in DevTools shows the same entries in a browser, with their payloads. In code:
list() returns entries newest first, narrowed by an optional filter.
Everything is awaited, because a persistent queue may still be reading
back a previous run’s log.
const recent = await system.deadLetterQueue.list({ recipient: 'actor-ts://orders/user/checkout', sinceMs: Date.now() - 15 * 60 * 1_000, limit: 20,});
for (const entry of recent) { console.log(entry.id, entry.timestampMs, entry.recipientPath, entry.payload);}recipient matches a path or its subtree, so
actor-ts://orders/user selects everything the application spawned.
An entry’s payload is one of two shapes:
{ kind: 'captured', message }— the original message, untouched.{ kind: 'degraded', className, reason }— the payload could not be written to the journal. The tagged-JSON encoder refuses functions, symbols,Promise, weak collections and cycles rather than corrupting them silently, so apersistentqueue keeps the provenance instead of losing the letter. Such an entry cannot be replayed — there is nothing left to send.
Replaying
Section titled “Replaying”const result = await system.deadLetterQueue.replay(entry.id);if (result.kind !== 'replayed') { console.warn('not replayed:', result.kind);}Replay resolves the recipient path again rather than reusing a ref captured at failure time — the whole point is that the actor has come back since, at the same address as a new instance.
Every outcome is named, so nothing fails quietly:
kind | Meaning |
|---|---|
replayed | Handed back; the entry left the queue. recipientPath is where it actually went. |
unknown-entry | No such id — already replayed, or aged out. |
unresolved-recipient | Nothing at the destination path; the entry stays. |
degraded-payload | Provenance only; there is nothing to redeliver. |
quarantined | Already replayed max-replays times. |
Replaying to a different recipient
Section titled “Replaying to a different recipient”A second argument sends the letter somewhere other than the address it was originally sent to — the actor was renamed, the shard moved, the path was a typo:
const result = await system.deadLetterQueue.replay( entry.id, 'actor-ts://orders/user/checkout-v2',);Three things about a redirect are worth knowing:
- Only the destination changes. The sender stays the recorded one, so
the recipient’s
senderstill answers the actor that sent the message and a reply goes where a reply always would. - The recorded path is not resolved at all. Redirecting is most useful exactly when the original address is gone for good, so requiring it to still exist would refuse the cases that need it most.
- The replay cap still applies.
max-replaysbounds how often this letter is redelivered, not how often one recipient is asked — a quarantined letter stays refused however it is addressed. Otherwise alternating between two paths would hand back the unbounded retry loop the cap exists to close.
If the letter dead-letters again at the alternate it comes back as the same
entry, now recording the alternate as its recipientPath — that is where
the message failed this time.
Poison messages cannot grow the queue
Section titled “Poison messages cannot grow the queue”A replayed message that dead-letters again comes back as the same
entry with a higher replayCount — not as a new one. Without that, an
operator (or a script) retrying a message the recipient simply cannot
handle would add an entry per attempt, while every individual attempt still
looked like a first. Past max-replays the letter is quarantined and
replay refuses it.
Durability
Section titled “Durability”With store = "persistent" the queue writes an append-only log to the
configured journal and reads it back on the next
start, so a redeploy does not lose what the previous process captured.
What ships is graceful-shutdown durability, not crash durability, and the difference is worth being precise about:
- The stream is derived from the system name, so two systems sharing
one journal keep separate queues. Override it with
actor-ts.dead-letters.persistence-idif you want something else. - Writes are issued as letters arrive and settled at shutdown — once in
the
before-actor-system-terminatephase of CoordinatedShutdown, and again after the actor tree is down, because the teardown itself produces dead letters after the last phase has run. Anything captured before aterminate()is in the journal after it, however large the burst. - A hard kill loses the backlog, which is not the same as losing one
write.
tellis synchronous and cannot wait for a journal, so appends are issued fire-and-forget onto a serialized chain. A burst arriving faster than the journal accepts it therefore leaves many un-settled appends outstanding, and aSIGKILL, a power loss or an OOM kill loses all of them — not just the one in flight.
The queue is a diagnostic record, not a transactional outbox. If you need a letter to survive an uncontrolled stop, the journal write has to be on the sending path, which is a different design and not what this is.
Metrics
Section titled “Metrics”actor_dead_letters_total{outcome} counts what the queue saw — outcome
is captured, replayed or replay-failed. See
stock metrics.
It carries no recipient label. That would be one permanent time series
per distinct undeliverable path, and under sharding the path is
entity-<entityId> — a value chosen by whoever addresses the shard region,
at a price of one lost message per series. Which actor a letter was
addressed to is answered where it costs nothing per-series: list({ recipient }) above, and the DeadLetter on the event stream.
What the queue sees only when asked
Section titled “What the queue sees only when asked”A message discarded by a bounded or priority mailbox becomes a dead
letter when — and only when — that mailbox was built with
deadLetterDrops:
import { ActorOptions, BoundedMailbox, BoundedMailboxOptions } from 'actor-ts';
const sheddingMailbox = BoundedMailboxOptions.create() .withCapacity(1_000) .withOverflow('drop-head') .withDeadLetterDrops(true);
const workerOptions = ActorOptions.create() .withMailbox(() => new BoundedMailbox(sheddingMailbox));Left off — which is the default — an overflow shows up in
actor_mailbox_dropped_total and nowhere here.
The switch exists because a drop happens on the sender’s stack, and a dead letter is a durable capture followed by a synchronous publish: routing every shed envelope turns load shedding into per-message work under exactly the pressure the bound was drawn to absorb. Turn it on for the actors whose losses you would have to explain afterwards — a command stream, a delivery confirmation — and leave it off for the telemetry firehose the bound exists for. See mailbox sizing.
The letter carries the message, its sender and the actor it never reached —
the same three fields every other loss path records — and, in attribution,
the envelope’s MDC context and its tracing span:
class Auditor extends Actor<DeadLetter> { override preStart(): void { this.system.eventStream.subscribe(this.self, DeadLetter); } override onReceive(letter: DeadLetter): void { const { context, trace } = letter.attribution; this.system.log .withFields({ ...context, traceId: trace?.traceId ?? 'none' }) .warn(`undelivered -> ${letter.recipient.path}`); }}That pair is what a counter can never give you: not that a message was lost,
but which request lost it. Both are absent when the send carried neither —
a tell outside any LogContext.run, or a system with tracing disabled —
and attribution itself is always present, so reading through it needs no
guard.
Only a loss path that still knows where the message came from can fill it.
An overflow drop copies both fields straight off the discarded envelope; a
BackoffSupervisor stash eviction carries the MDC its entry captured when
the message was buffered, and no span — nothing exposes an arriving
envelope’s span to actor code, so the supervisor has none to copy. A letter
built from a bare message — a projection event, a singleton hand-off —
carries an empty attribution, and that is the truth rather than an omission.
The attribution stays on the event. A captured DeadLetterEntry — what
list() returns — does not carry it, deliberately. An
entry is retained for retentionMs and, under store: 'persistent', written
to a journal that outlives the process — while an MDC routinely carries
tenant, user and request identifiers. That is the same decision that keeps
the store off by default and the payload out of the log line, taken one
layer down. Subscribe to DeadLetter if you want the attribution durably,
under a retention you chose.
