Dead letters

A dead letter is a message the system could not deliver: a tell to a ref whose actor has stopped, a selection that resolved to nothing, a behavior that answered unhandled, a cluster singleton with no host. Nothing was lost by accident — the framework noticed, and this is how it says so.

What happens by default

Every dead letter is wrapped in a DeadLetter and published on the event stream. That is the whole of the default behaviour.

In particular the framework does not log dead letters. DeadLetterRef holds no logger; if nothing subscribes to the stream, the letter is gone. That is fine for a development run and a poor answer to “what did we drop during last night’s incident?”, which is what the queue below is for.

import { DeadLetter } from 'actor-ts';

class Watcher extends Actor<DeadLetter> {
  override preStart(): void {
    this.system.eventStream.subscribe(this.self, DeadLetter);
  }
  override onReceive(letter: DeadLetter): void {
    this.system.log.warn(`undelivered: ${letter.message} -> ${letter.recipient.path}`);
  }
}

Every dead letter names its recipient — the actor the message failed to reach, not the dead-letter office. That is what makes one letter distinguishable from another, and it is what the filters and the replay below are built on.

Turning the queue on

system.deadLetterQueue always exists and, by default, keeps nothing. Four stores, one axis — how much of the letter is kept:

store	Keeps	Survives a restart
off (default)	nothing	—
metrics	a counter, no payload	—
memory	a bounded ring	no
persistent	the same ring, plus an append-only journal log	yes

metrics is there because retaining a payload is a different decision from observing a rate. A ring holds a strong reference to every undeliverable message, which is a data-protection question the moment a payload says anything about a person; a counter is not. list() stays empty and replay answers unknown-entry for every id — nothing was kept, so there is nothing to hand back. You cannot approximate it with a tiny ring: max-entries must be positive, so even the smallest one keeps a live payload for a whole retention window.

actor-ts.dead-letters {
  store       = "memory"
  max-entries = 1000
  retention   = 1h
  max-replays = 3
}

The same knobs are available as an options family, handed to the system at creation time:

import { ActorSystem, ActorSystemOptions, DeadLetterQueueOptions } from 'actor-ts';

const deadLetterOptions = DeadLetterQueueOptions.create()
  .withStore('persistent')
  .withMaxEntries(5_000)
  .withRetentionMs(6 * 60 * 60 * 1_000);

const systemOptions = ActorSystemOptions.create()
  .withDeadLetters(deadLetterOptions);

const system = ActorSystem.create('orders', systemOptions);

Precedence is the usual one — explicit options beat HOCON beat the built-in defaults — and it applies field by field, so naming one knob in code leaves the rest of the actor-ts.dead-letters block in effect.

Caution

withDeadLetters is the only way to configure the queue the system uses. new DeadLetterQueue(system, options) builds a valid object, but nothing installs it as the capture sink, so it never receives a letter — there is one sink slot and the system owns it.

Note

store is a storage decision, not a failure strategy. What a projection should do when a handler throws is recoveryStrategy — retry-and-skip, skip and friends. Once a projection has skipped an event it publishes a dead letter, and this setting decides whether anything keeps it. The two compose; neither replaces the other.

Every key in this block decides what is retained, which is why they live under actor-ts.dead-letters and not beside the eventual logging and rate-limiting knobs: those decide how loudly a letter is announced, are read on the publish path rather than by the queue, and sit downstream of capture by design — so a suppression setting can never quietly make this record incomplete.

Inspecting

The queue also has a panel: once it is switched on, the dead-letter inspector in DevTools shows the same entries in a browser, with their payloads. In code:

list() returns entries newest first, narrowed by an optional filter. Everything is awaited, because a persistent queue may still be reading back a previous run’s log.

const recent = await system.deadLetterQueue.list({
  recipient: 'actor-ts://orders/user/checkout',
  sinceMs: Date.now() - 15 * 60 * 1_000,
  limit: 20,
});

for (const entry of recent) {
  console.log(entry.id, entry.timestampMs, entry.recipientPath, entry.payload);
}

recipient matches a path or its subtree, so actor-ts://orders/user selects everything the application spawned.

An entry’s payload is one of two shapes:

• { kind: 'captured', message } — the original message, untouched.
• { kind: 'degraded', className, reason } — the payload could not be written to the journal. The tagged-JSON encoder refuses functions, symbols, Promise, weak collections and cycles rather than corrupting them silently, so a persistent queue keeps the provenance instead of losing the letter. Such an entry cannot be replayed — there is nothing left to send.

Replaying

const result = await system.deadLetterQueue.replay(entry.id);
if (result.kind !== 'replayed') {
  console.warn('not replayed:', result.kind);
}

Replay resolves the recipient path again rather than reusing a ref captured at failure time — the whole point is that the actor has come back since, at the same address as a new instance.

Every outcome is named, so nothing fails quietly:

kind	Meaning
replayed	Handed back; the entry left the queue. recipientPath is where it actually went.
unknown-entry	No such id — already replayed, or aged out.
unresolved-recipient	Nothing at the destination path; the entry stays.
degraded-payload	Provenance only; there is nothing to redeliver.
quarantined	Already replayed max-replays times.

Replaying to a different recipient

A second argument sends the letter somewhere other than the address it was originally sent to — the actor was renamed, the shard moved, the path was a typo:

const result = await system.deadLetterQueue.replay(
  entry.id,
  'actor-ts://orders/user/checkout-v2',
);

Three things about a redirect are worth knowing:

• Only the destination changes. The sender stays the recorded one, so the recipient’s sender still answers the actor that sent the message and a reply goes where a reply always would.
• The recorded path is not resolved at all. Redirecting is most useful exactly when the original address is gone for good, so requiring it to still exist would refuse the cases that need it most.
• The replay cap still applies. max-replays bounds how often this letter is redelivered, not how often one recipient is asked — a quarantined letter stays refused however it is addressed. Otherwise alternating between two paths would hand back the unbounded retry loop the cap exists to close.

If the letter dead-letters again at the alternate it comes back as the same entry, now recording the alternate as its recipientPath — that is where the message failed this time.

Poison messages cannot grow the queue

A replayed message that dead-letters again comes back as the same entry with a higher replayCount — not as a new one. Without that, an operator (or a script) retrying a message the recipient simply cannot handle would add an entry per attempt, while every individual attempt still looked like a first. Past max-replays the letter is quarantined and replay refuses it.

Durability

With store = "persistent" the queue writes an append-only log to the configured journal and reads it back on the next start, so a redeploy does not lose what the previous process captured.

What ships is graceful-shutdown durability, not crash durability, and the difference is worth being precise about:

• The stream is derived from the system name, so two systems sharing one journal keep separate queues. Override it with actor-ts.dead-letters.persistence-id if you want something else.
• Writes are issued as letters arrive and settled at shutdown — once in the before-actor-system-terminate phase of CoordinatedShutdown, and again after the actor tree is down, because the teardown itself produces dead letters after the last phase has run. Anything captured before a terminate() is in the journal after it, however large the burst.
• A hard kill loses the backlog, which is not the same as losing one write. tell is synchronous and cannot wait for a journal, so appends are issued fire-and-forget onto a serialized chain. A burst arriving faster than the journal accepts it therefore leaves many un-settled appends outstanding, and a SIGKILL, a power loss or an OOM kill loses all of them — not just the one in flight.

The queue is a diagnostic record, not a transactional outbox. If you need a letter to survive an uncontrolled stop, the journal write has to be on the sending path, which is a different design and not what this is.

Metrics

actor_dead_letters_total{outcome} counts what the queue saw — outcome is captured, replayed or replay-failed. See stock metrics.

It carries no recipient label. That would be one permanent time series per distinct undeliverable path, and under sharding the path is entity-<entityId> — a value chosen by whoever addresses the shard region, at a price of one lost message per series. Which actor a letter was addressed to is answered where it costs nothing per-series: list({ recipient }) above, and the DeadLetter on the event stream.

What the queue sees only when asked

A message discarded by a bounded or priority mailbox becomes a dead letter when — and only when — that mailbox was built with deadLetterDrops:

import { ActorOptions, BoundedMailbox, BoundedMailboxOptions } from 'actor-ts';

const sheddingMailbox = BoundedMailboxOptions.create()
  .withCapacity(1_000)
  .withOverflow('drop-head')
  .withDeadLetterDrops(true);

const workerOptions = ActorOptions.create()
  .withMailbox(() => new BoundedMailbox(sheddingMailbox));

Left off — which is the default — an overflow shows up in actor_mailbox_dropped_total and nowhere here.

The switch exists because a drop happens on the sender’s stack, and a dead letter is a durable capture followed by a synchronous publish: routing every shed envelope turns load shedding into per-message work under exactly the pressure the bound was drawn to absorb. Turn it on for the actors whose losses you would have to explain afterwards — a command stream, a delivery confirmation — and leave it off for the telemetry firehose the bound exists for. See mailbox sizing.

The letter carries the message, its sender and the actor it never reached — the same three fields every other loss path records. The envelope’s MDC context and its tracing span do not survive it; DeadLetter has no slot for either.