Skip to content
English

Dead letters

A dead letter is a message the system could not deliver: a tell to a ref whose actor has stopped, a selection that resolved to nothing, a behavior that answered unhandled, an actor that declined one with this.unhandled(m), a cluster singleton with no host. Nothing was lost by accident — the framework noticed, and this is how it says so.

The two unhandled entries are a different animal from the rest: the message was delivered, and a live actor turned it down. A counter tells them apart — actor_unhandled_total{class} rises only for a declined message, and unlike actor_dead_letters_total it does not need the queue below to be switched on first.

Three framework base classes decline this way by default. The onSelfMessage hook of WebsocketServerActor, WebsocketClientActor and MqttActor receives whatever none of the actor’s own kinds matched — an application message your subclass never handles, or a body a remote node addressed to the actor’s path — and unless you override it, it turns that message into a dead letter naming the actor, after one warn that names the message’s class and never its payload.

Every dead letter is wrapped in a DeadLetter, logged, and published on the event stream. That is the whole of the default behaviour.

The record is one info line naming the recipient path, the sender when there is one, and the message’s class — never the payload, for the same data-protection reason store below defaults to off. It is throttled: the first ten dead letters are logged in full, then a single warn line says logging is suspending, and the next five minutes are silent. When it resumes, one more warn line says how many letters went unreported. Both numbers are settings — see actor-ts.diagnostics — and 0 means “never log” for the count and “never suspend” for the window.

The burst terminate() produces while draining mailboxes is not logged unless you ask for it with log-dead-letters-during-shutdown = on: an orderly shutdown drains every queued and stashed message to dead letters, and reporting that makes a clean stop read as an incident. An individual ref.stop() still logs, because the system is not terminating.

What the log line is not is a record. It carries no payload and nothing keeps it in the process; if nothing subscribes to the stream either, the letter itself is gone. That is fine for a development run and a poor answer to “what did we drop during last night’s incident?”, which is what the queue below is for.

import { Actor, DeadLetter } from 'actor-ts';
import { classNameOf } from 'actor-ts/util';
class Watcher extends Actor<DeadLetter> {
override preStart(): void {
this.system.eventStream.subscribe(this.self, DeadLetter);
}
override onReceive(letter: DeadLetter): void {
// The class, never the payload: interpolating the message prints a
// string payload verbatim and throws on a decoded body with no usable
// toString.
this.system.log.warn(`undelivered: ${classNameOf(letter.message)} -> ${letter.recipient.path}`);
}
}

Every dead letter names its recipient — the actor the message failed to reach, not the dead-letter office. That is what makes one letter distinguishable from another, and it is what the filters and the replay below are built on.

system.deadLetterQueue always exists and, by default, keeps nothing. Four stores, one axis — how much of the letter is kept:

storeKeepsSurvives a restart
off (default)nothing—
metricsa counter, no payload—
memorya bounded ringno
persistentthe same ring, plus an append-only journal logyes

metrics is there because retaining a payload is a different decision from observing a rate. A ring holds a strong reference to every undeliverable message, which is a data-protection question the moment a payload says anything about a person; a counter is not. list() stays empty and replay answers unknown-entry for every id — nothing was kept, so there is nothing to hand back. You cannot approximate it with a tiny ring: max-entries must be positive, so even the smallest one keeps a live payload for a whole retention window.

actor-ts.dead-letters {
store = "memory"
max-entries = 1000
retention = 1h
max-replays = 3
}

The same knobs are available as an options family, handed to the system at creation time:

import { ActorSystem, ActorSystemOptions, DeadLetterQueueOptions } from 'actor-ts';
const deadLetterOptions = DeadLetterQueueOptions.create()
.withStore('persistent')
.withMaxEntries(5_000)
.withRetentionMs(6 * 60 * 60 * 1_000);
const systemOptions = ActorSystemOptions.create()
.withDeadLetters(deadLetterOptions);
const system = ActorSystem.create('orders', systemOptions);

Precedence is the usual one — explicit options beat HOCON beat the built-in defaults — and it applies field by field, so naming one knob in code leaves the rest of the actor-ts.dead-letters block in effect.

Every key in this block decides what is retained, which is why they live under actor-ts.dead-letters and not beside the logging knobs in actor-ts.diagnostics: those decide how loudly a letter is announced, are read on the publish path rather than by the queue, and sit downstream of capture by design — so a suppression setting can never quietly make this record incomplete.

The queue also has a panel: once it is switched on, the dead-letter inspector in DevTools shows the same entries in a browser, with their payloads. In code:

list() returns entries newest first, narrowed by an optional filter. Everything is awaited, because a persistent queue may still be reading back a previous run’s log.

const recent = await system.deadLetterQueue.list({
recipient: 'actor-ts://orders/user/checkout',
sinceMs: Date.now() - 15 * 60 * 1_000,
limit: 20,
});
for (const entry of recent) {
console.log(entry.id, entry.timestampMs, entry.recipientPath, entry.payload);
}

recipient matches a path or its subtree, so actor-ts://orders/user selects everything the application spawned.

An entry’s payload is one of two shapes:

  • { kind: 'captured', message } — the original message, untouched.
  • { kind: 'degraded', className, reason } — the payload could not be written to the journal. The tagged-JSON encoder refuses functions, symbols, Promise, weak collections and cycles rather than corrupting them silently, so a persistent queue keeps the provenance instead of losing the letter. Such an entry cannot be replayed — there is nothing left to send.
const result = await system.deadLetterQueue.replay(entry.id);
if (result.kind !== 'replayed') {
console.warn('not replayed:', result.kind);
}

Replay resolves the recipient path again rather than reusing a ref captured at failure time — the whole point is that the actor has come back since, at the same address as a new instance.

Every outcome is named, so nothing fails quietly:

kindMeaning
replayedHanded back; the entry left the queue. recipientPath is where it actually went.
unknown-entryNo such id — already replayed, or aged out.
unresolved-recipientNothing at the destination path; the entry stays.
degraded-payloadProvenance only; there is nothing to redeliver.
quarantinedAlready replayed max-replays times.

A second argument sends the letter somewhere other than the address it was originally sent to — the actor was renamed, the shard moved, the path was a typo:

const result = await system.deadLetterQueue.replay(
entry.id,
'actor-ts://orders/user/checkout-v2',
);

Three things about a redirect are worth knowing:

  • Only the destination changes. The sender stays the recorded one, so the recipient’s sender still answers the actor that sent the message and a reply goes where a reply always would.
  • The recorded path is not resolved at all. Redirecting is most useful exactly when the original address is gone for good, so requiring it to still exist would refuse the cases that need it most.
  • The replay cap still applies. max-replays bounds how often this letter is redelivered, not how often one recipient is asked — a quarantined letter stays refused however it is addressed. Otherwise alternating between two paths would hand back the unbounded retry loop the cap exists to close.

If the letter dead-letters again at the alternate it comes back as the same entry, now recording the alternate as its recipientPath — that is where the message failed this time.

A replayed message that dead-letters again comes back as the same entry with a higher replayCount — not as a new one. Without that, an operator (or a script) retrying a message the recipient simply cannot handle would add an entry per attempt, while every individual attempt still looked like a first. Past max-replays the letter is quarantined and replay refuses it.

With store = "persistent" the queue writes an append-only log to the configured journal and reads it back on the next start, so a redeploy does not lose what the previous process captured.

What ships is graceful-shutdown durability, not crash durability, and the difference is worth being precise about:

  • The stream is derived from the system name, so two systems sharing one journal keep separate queues. Override it with actor-ts.dead-letters.persistence-id if you want something else.
  • Writes are issued as letters arrive and settled at shutdown — once in the before-actor-system-terminate phase of CoordinatedShutdown, and again after the actor tree is down, because the teardown itself produces dead letters after the last phase has run. Anything captured before a terminate() is in the journal after it, however large the burst.
  • A hard kill loses the backlog, which is not the same as losing one write. tell is synchronous and cannot wait for a journal, so appends are issued fire-and-forget onto a serialized chain. A burst arriving faster than the journal accepts it therefore leaves many un-settled appends outstanding, and a SIGKILL, a power loss or an OOM kill loses all of them — not just the one in flight.

The queue is a diagnostic record, not a transactional outbox. If you need a letter to survive an uncontrolled stop, the journal write has to be on the sending path, which is a different design and not what this is.

actor_dead_letters_total{outcome} counts what the queue saw — outcome is captured, replayed or replay-failed. See stock metrics.

It carries no recipient label. That would be one permanent time series per distinct undeliverable path, and under sharding the path is entity-<entityId> — a value chosen by whoever addresses the shard region, at a price of one lost message per series. Which actor a letter was addressed to is answered where it costs nothing per-series: list({ recipient }) above, and the DeadLetter on the event stream.

A message discarded by a bounded or priority mailbox becomes a dead letter when — and only when — that mailbox was built with deadLetterDrops:

import { ActorOptions, BoundedMailbox, BoundedMailboxOptions } from 'actor-ts';
const sheddingMailbox = BoundedMailboxOptions.create()
.withCapacity(1_000)
.withOverflow('drop-head')
.withDeadLetterDrops(true);
const workerOptions = ActorOptions.create()
.withMailbox(() => new BoundedMailbox(sheddingMailbox));

Left off — which is the default — an overflow shows up in actor_mailbox_dropped_total and nowhere here.

The switch exists because a drop happens on the sender’s stack, and a dead letter is a durable capture followed by a synchronous publish: routing every shed envelope turns load shedding into per-message work under exactly the pressure the bound was drawn to absorb. Turn it on for the actors whose losses you would have to explain afterwards — a command stream, a delivery confirmation — and leave it off for the telemetry firehose the bound exists for. See mailbox sizing.

The letter carries the message, its sender and the actor it never reached — the same three fields every other loss path records — and, in attribution, the envelope’s MDC context and its tracing span:

class Auditor extends Actor<DeadLetter> {
override preStart(): void {
this.system.eventStream.subscribe(this.self, DeadLetter);
}
override onReceive(letter: DeadLetter): void {
const { context, trace } = letter.attribution;
this.system.log
.withFields({ ...context, traceId: trace?.traceId ?? 'none' })
.warn(`undelivered -> ${letter.recipient.path}`);
}
}

That pair is what a counter can never give you: not that a message was lost, but which request lost it. Both are absent when the send carried neither — a tell outside any LogContext.run, or a system with tracing disabled — and attribution itself is always present, so reading through it needs no guard.

Only a loss path that still knows where the message came from can fill it. An overflow drop copies both fields straight off the discarded envelope; a BackoffSupervisor stash eviction carries the MDC its entry captured when the message was buffered, and no span — nothing exposes an arriving envelope’s span to actor code, so the supervisor has none to copy. A letter built from a bare message — a projection event, a singleton hand-off — carries an empty attribution, and that is the truth rather than an omission.

The attribution stays on the event. A captured DeadLetterEntry — what list() returns — does not carry it, deliberately. An entry is retained for retentionMs and, under store: 'persistent', written to a journal that outlives the process — while an MDC routinely carries tenant, user and request identifiers. That is the same decision that keeps the store off by default and the payload out of the log line, taken one layer down. Subscribe to DeadLetter if you want the attribution durably, under a retention you chose.