Coordinated shutdown
system.terminate() stops the actor system, but a production app
usually has work to do before that: drain in-flight HTTP requests,
tell the cluster you’re leaving, flush a journal, close broker
connections. And those steps have an order — hand the sharding
region’s shards off before you leave the cluster; stop the HTTP
server before the actors that handle requests.
Coordinated shutdown is the DSL for that. You register tasks
against named phases; the framework runs them in dependency
order, one phase after the next, each with a timeout cap. Calling
run() from any trigger (SIGTERM, K8s PreStop hook, an admin
endpoint) executes the whole pipeline once.
The minimal example
Section titled “The minimal example”import { ActorSystem, CoordinatedShutdownId, Phases, type Reason,} from 'actor-ts';
const system = ActorSystem.create('my-app');const coordinatedShutdown = system.extension(CoordinatedShutdownId);
coordinatedShutdown.addTask(Phases.ServiceUnbind, 'close-http', async (reason) => { await httpServer.close();});
coordinatedShutdown.addTask(Phases.ServiceRequestsDone, 'drain-in-flight', async () => { await waitForInFlightRequests(/* up to 10s */);});
await system.runUntilTerminated(); // SIGTERM/SIGINT → the pipeline → resolves when it is doneThree things happen when SIGTERM lands:
- The runtime calls
coordinatedShutdown.run(new ProcessTerminateReason('SIGTERM')). - The phases run in canonical order. Inside each phase, all registered tasks run in parallel; the phase waits for them all (or for their timeouts).
- The pipeline finishes with the built-in
actor-system-terminatetask, which callssystem.terminate()for you.
Some of what happens in between is already wired for you; the rest is whatever you added.
The 12 canonical phases
Section titled “The 12 canonical phases”Listed in execution order. Wired marks the phases the framework populates by itself — everything else is empty until you register something.
| # | Phase name | Wired | Typical tasks |
|---|---|---|---|
| 1 | before-service-unbind | — | Last-chance announcements before the server stops accepting connections. |
| 2 | service-unbind | system.http(...).bind(...), DevTools | Stop the HTTP server / gRPC server / WebSocket listener from accepting new connections. |
| 3 | service-requests-done | — | Wait for in-flight requests to finish; abort the rest. |
| 4 | service-stop | broker actors (MQTT, Kafka, NATS, AMQP, …) | Close client connections, release sockets. |
| 5 | before-cluster-shutdown | — | Optional pre-cluster-leave hooks. |
| 6 | cluster-sharding-shutdown-region | Cluster.join(...) | Tell the sharding region to hand off entities. |
| 7 | cluster-leave | Cluster.join(...) | Issue a Cluster.leave() — gossip leaving status. |
| 8 | cluster-exiting | — | Wait for the cluster to acknowledge the leave. |
| 9 | cluster-exiting-done | — | Confirm cluster transition is complete. |
| 10 | cluster-shutdown | — | Tear down cluster transports. |
| 11 | before-actor-system-terminate | — | Last-chance app-level cleanup (flush journals, close brokers). |
| 12 | actor-system-terminate | the built-in terminator | The built-in system.terminate() task. |
You don’t have to use every phase. Empty phases are no-ops; only phases with registered tasks do anything.
Opting out of the wired tasks
Section titled “Opting out of the wired tasks”Set actor-ts.coordinated-shutdown.auto-register-tasks = false and the
framework registers none of them. The phases stay, the built-in
terminator stays, and every resource above is yours to register — for
an embedder that owns the lifecycle of what it handed the system. It
is one switch rather than one per subsystem, because the reason to
reach for it is “I own the lifecycle”, never “unbind the HTTP server
but leave the brokers to me”.
The Phases constant exports the canonical names — prefer it over
string literals for autocomplete:
import { Phases } from 'actor-ts';
coordinatedShutdown.addTask(Phases.ServiceUnbind, ...); // ✓ typedcoordinatedShutdown.addTask('service-unbind', ...); // ✗ stringly-typed, no auto-completeAdding custom phases
Section titled “Adding custom phases”For app-specific work that doesn’t fit a canonical phase, declare your own:
coordinatedShutdown.addPhase({ name: 'flush-metrics', timeoutMs: 3_000, dependsOn: [Phases.BeforeActorSystemTerminate], recover: true,});
coordinatedShutdown.addTask('flush-metrics', 'push-prometheus', async () => { await metricsRegistry.flush();});The dependsOn field is what makes the order DAG-shaped rather
than linear — your phase runs after before-actor-system-terminate.
It does not by itself put the phase before
actor-system-terminate. Both become ready in the same round of the
topological sort — actor-system-terminate has the same single
dependency — and the sort’s alphabetical tie-breaker then decides, so
flush-metrics above actually runs after the system has terminated.
Adding the reverse edge is what places it, and that is depends-on
from config (below), which is added to a canonical phase’s existing
edges. From code, name a phase whose successor is not the one you
need to precede.
Because addPhase requires every dependsOn entry to name an
already-registered phase, a cycle can’t be formed in the first
place. The order itself is computed by a topological sort when
run() executes — that sort would throw Error: cycle in phase dependencies if a cycle ever did arise.
From config
Section titled “From config”The same graph is expressible in HOCON, under
actor-ts.coordinated-shutdown.phases — one child per phase name,
each taking timeout, recover and depends-on:
actor-ts.coordinated-shutdown.phases { flush-metrics { timeout = 3s recover = true depends-on = ["before-actor-system-terminate"] } actor-system-terminate { depends-on = ["flush-metrics"] }}A name that is one of the twelve canonical phases merges into
that phase; anything else declares a new one, and there
depends-on is required — a phase with no edges sorts into the
first ready batch and would run before before-service-unbind.
Note what the second child buys. depends-on on a canonical
phase is added to the edge that phase already has, never a
replacement, so naming flush-metrics there is what actually puts
your phase before actor-system-terminate rather than merely
after before-actor-system-terminate — the two would otherwise
become ready in the same round and the sort’s alphabetical
tie-breaker would decide. The additive rule is also what stops a
config file re-parenting cluster-leave ahead of service-unbind:
the sort catches cycles, not a pipeline quietly running in the
wrong order.
Declaration order in the file is irrelevant — two phases may name
each other and are registered dependency-first. An unresolvable
name, a cycle, an unknown setting under a phase, or a declared
phase with no depends-on is a ConfigError when the extension is
built, not when the shutdown runs.
Ask what the merge produced with phaseDefinition:
coordinatedShutdown.phaseDefinition('flush-metrics');// { name: 'flush-metrics', timeoutMs: 3000, dependsOn: [...], recover: true }Task semantics
Section titled “Task semantics”Every task is a function from a Reason to void | Promise<void>:
type ShutdownTask = (reason: Reason) => Promise<void> | void;The reason lets a task branch on why shutdown was triggered:
coordinatedShutdown.addTask(Phases.BeforeClusterShutdown, 'deregister', async (reason) => { if (reason instanceof ClusterDowningReason) { // We were downed — the registry has already written us off. return; } await serviceRegistry.deregister(cluster.selfAddress.toString());});Built-in Reason classes:
| Class | When |
|---|---|
ProcessTerminateReason(signal) | SIGTERM/SIGINT, from runUntilTerminated() or installProcessHooks(). |
ActorSystemTerminateReason | What system.terminate() passes when it runs the pipeline — asked for with runCoordinatedShutdown: true, or by default with actor-ts.coordinated-shutdown.run-by-actor-system-terminate on. An ordinary shutdown your program decides on is that call, rather than a run() you hand this reason: the call also tears the system down when the run does not. |
ClusterLeavingReason | What Cluster.bootstrap’s shutdown() passes — a member leaving on purpose. |
ClusterDowningReason | What a node passes when its own downing decision removes it and actor-ts.cluster.run-coordinated-shutdown-when-down is on — see when the cluster downs this node. |
ClusterJoinUnsuccessfulReason | What a node passes when it gives its join up — actor-ts.cluster.shutdown-after-unsuccessful-join passed with nothing able to promote it — and what Cluster.bootstrap passes when its readiness wait fails, or when it refuses to go on after building its system; see when no seed ever answers. A run with it never exits 0. |
StartupFailedReason(phase, task, cause) | What the system passes when a startup task failed or outlived its phase’s budget — it names the phase and the task and carries the cause. |
UnknownReason | The default when run() is called with no argument. |
A failed startup runs this pipeline too: the system starts it with a
StartupFailedReason and tears down afterwards, whatever the pipeline
did — so a task here can tell a node that never finished starting from
one that is being stopped. Nor does it end the process with a status
that reads as a clean stop: with exit-process = on it exits with
exit-code when that is not 0, and with 1 when it is — the rule a
run that halts exits by too. Left at the default
exit-process = off, runUntilTerminated() rejects with the startup’s
error instead, and a main that awaits it exits non-zero by itself.
A node that could not join runs it as well, and its reason says so.
ClusterJoinUnsuccessfulReason arrives when the join gives up, when a
Cluster.bootstrap readiness wait fails — which used to pass
ClusterLeavingReason, so a task that told a failed start by that reason
now sees this one — and when the bootstrap refuses to go on after it has
built its system, where run-by-actor-system-terminate makes that
teardown a run. None of them ends the process on its own: the built-in
terminator ends the actor system, and the process exits only with
actor-ts.coordinated-shutdown.exit-process = on, and then by the rule
above: a node that could not join did not stop cleanly either, and a
failure never exits 0. The bootstrap says why at ERROR before the run
starts, because the exit comes before its error could reach your main.
You can subclass Reason for app-specific triggers
(AdminEndpointReason, HotReloadReason, etc.).
Parallelism within a phase
Section titled “Parallelism within a phase”All tasks in a phase run concurrently — they’re started together
and the phase waits for the last one (or its timeout). If you have
ordering requirements within a phase (task B must wait for task
A), put them in different phases with a dependsOn.
Timeouts
Section titled “Timeouts”Each phase has a timeoutMs (default 5 s); each task is wrapped in
a timeout race. A task that doesn’t finish in time is logged and
either:
-
Recovered from (the phase continues,
recover: true— the default). The next phase starts. -
Halts the pipeline (
recover: false). Subsequent phases are not run; shutdown stops mid-flight. A run thatterminate({ runCoordinatedShutdown: true })started still ends in the teardown, because that call asked for a terminated system; so does a run that a signal, this node’s own downing decision or a join that could not complete started — nobody awaits those, so the run answers its own halt — unlessterminate-actor-system = falsesays the pipeline never takes the system down. Either way the halt is logged as an ERROR. Arun()you call yourself, with any other reason, rejects with the task’s error and leaves the system as the halt found it.With
exit-process = onthe process exits at the halt instead, whichever trigger started the run — a signal,terminate(), your ownrun()or the cluster’s — and with a status that does not read as a clean stop:exit-codewhere that is not0,1where it is. Nothing after the exit runs, so it takes the place of those teardowns — but not of what they flush on the way down: the dead-letter queue’s writes and then the log sinks are flushed before the exit, each withinactor-ts.logger.close-timeout, so the ERROR line naming the phase, logged just before, reaches a buffering sink too. A completed run that left the system up (terminate-actor-system = false) is flushed the same way before its exit; one whose terminator tore the system down was flushed by that teardown already. The run is in flight until its exit, so aterminate()that arrives while those flushes run starts nothing.
Override per phase:
coordinatedShutdown.setPhaseTimeout(Phases.ServiceRequestsDone, 30_000); // 30s drain budgetOr define your own with the wanted timeoutMs / recover:
coordinatedShutdown.addPhase({ name: 'aggressive-cleanup', timeoutMs: 1_000, // strict cap dependsOn: [Phases.BeforeActorSystemTerminate], recover: false, // failure → halt});Both are also config, per phase and without a rebuild — the same
phases block as above, and it retimes only the phase it names:
actor-ts.coordinated-shutdown { default-phase-timeout = 5s # every phase that says nothing phases { service-requests-done { timeout = 30s } # …except this one service-stop { recover = false } # a failure here halts the run }}The drain budget sits inside the last phase
Section titled “The drain budget sits inside the last phase”actor-system-terminate is a task that awaits the system’s teardown —
what a bare system.terminate() runs — and the teardown begins by
draining the actors under /user — see
Terminating. So two budgets
are nested, and the inner one has to be the smaller:
| Budget | Key | Default |
|---|---|---|
The whole actor-system-terminate phase | actor-ts.coordinated-shutdown.phases.actor-system-terminate.timeout, falling back to …default-phase-timeout | 5 s |
| The drain inside it | actor-ts.system.shutdown-drain-timeout | 2 s |
Raise the drain budget above the phase timeout and the phase is abandoned while the drain is still running — before a single actor has been told to stop. Raise both together, or raise the phase timeout first.
Reach for the per-phase key when only the terminate step needs more
room. Raising default-phase-timeout instead hands the same longer
budget to eleven other phases, and each of those is a phase a stuck
task can now hold the shutdown in for that much longer.
SIGTERM / SIGINT hooks
Section titled “SIGTERM / SIGINT hooks”Almost every service wants the same thing — install the handlers, wait, shut down, exit — so that whole shape is one call:
await system.runUntilTerminated();// Or, to listen for something else as well:await system.runUntilTerminated(['SIGTERM', 'SIGINT', 'SIGUSR2']);It installs the handlers, resolves once the system is down and
the pipeline has finished, and detaches the handlers on the way out.
That last step is not housekeeping: on Deno a signal listener holds
the event loop open and has no unref, so a program that shuts down
for any other reason — a terminate() from inside, an admin endpoint
— would never exit.
A signal the platform cannot deliver is skipped rather than registered. Windows has no SIGTERM under any runtime, and Deno throws on a signal it cannot deliver instead of ignoring it, so asking for one is never a startup failure.
The process stays alive for as long as the promise is pending, and
that is a guarantee the call makes rather than one it inherits from the
handlers it installed. A signal handler is not by itself a reason for a
runtime to keep running: Node unrefs its signal handles, so a service
with nothing else on the event loop — no bound port, no open socket, no
referenced timer — would otherwise drain its loop the moment it started
waiting, and exit before the SIGTERM it had just armed itself for
could arrive. So the call holds the loop open itself and lets go in the
same step that detaches the handlers. A headless node needs no
setInterval of your own to stay up.
The two halves are still available separately when you are embedding the system in a host that owns the process lifetime:
coordinatedShutdown.installProcessHooks(); // → run(new ProcessTerminateReason(signal))coordinatedShutdown.removeProcessHooks(); // detaches exactly what it installedCalling either twice is harmless. removeProcessHooks() never
touches a listener it did not install, so a second ActorSystem in
the same process — or the application’s own SIGTERM handling — is
left alone.
The default is config, so a host that owns the signals can take the handlers off every node it runs without touching the code:
actor-ts.coordinated-shutdown.run-by-process-signals = falseThat switches what runUntilTerminated() and the cluster bootstrap do
when the caller named no signals. Naming them —
runUntilTerminated(['SIGTERM']), shutdownOnSignals in the bootstrap
options, or installProcessHooks() itself — still installs, because
explicit options outrank config. With it off, runUntilTerminated()
keeps both of its other promises: it waits, and it holds the event loop
open.
K8s PreStop integration
Section titled “K8s PreStop integration”In Kubernetes, the pod-shutdown sequence is:
1. K8s sends SIGTERM and ends the pod's grace period clock.2. K8s also calls the PreStop hook (if configured), running concurrently.3. After max(graceful-shutdown, grace-period), K8s sends SIGKILL.The standard recipe:
// On SIGTERM, run coordinated shutdown:await system.runUntilTerminated();
// PreStop hook script (in your container image):// #!/bin/sh// sleep 10 # give upstream LBs time to drain this pod// exit 0The sleep in PreStop gives the load balancer time to drop this
pod from rotation before the actor system starts shutting down
— so in-flight HTTP requests don’t see “I’m draining, go away.”
See Operations — Kubernetes for the full deployment manifest.
Multi-trigger safety
Section titled “Multi-trigger safety”coordinatedShutdown.run() is idempotent — calling it multiple times returns the
same in-flight promise. Three independent triggers (SIGTERM, an
admin endpoint, and a cluster downing) all calling run doesn’t
re-run the pipeline. The first call starts it; subsequent calls
await the same completion.
This matters because in production you often have multiple shutdown paths:
// SIGTERM path:await system.runUntilTerminated();
// Admin-endpoint path:app.post('/shutdown', async (req, res) => { await coordinatedShutdown.run(new AdminEndpointReason()); res.send('ok');});
// Cluster-downing path: run-coordinated-shutdown-when-down = on — see below.Each of them ends up running the same shutdown sequence, once.
When the cluster downs this node
Section titled “When the cluster downs this node”A node whose own downing decision removes it — a
strategy’s verdict that includes this node,
or down-all-when-unstable — always leaves the cluster. Whether it also
stands down is run-coordinated-shutdown-when-down:
const clusterOptions = ClusterOptions.create() .withDowning(new KeepMajority()) .withRunCoordinatedShutdownWhenDown(true);actor-ts.cluster.run-coordinated-shutdown-when-down = onWith it on, the node leaves first and then runs this pipeline with
ClusterDowningReason, even when the leave fails: its services unbind, and
the built-in terminator ends the actor system and with it every entity and
singleton the node hosted. terminate-actor-system and exit-process decide
what the run ends in, as they do for a signal — a run that a phase
halts still ends in the teardown, because a downed node must not
keep its actors running beside the side that won — and
auto-register-tasks = off does not suppress it: that switch withholds the
framework’s own tasks and decides nothing about when the pipeline runs. The
node says so at warn before the run starts. A decision that downs only
peers changes nothing on this node: the switch answers its own removal, not a
downing it decides.
It ships on:
- On, a wrong self-down terminates the node. The two known ones —
static-quorumdowning a cluster still smaller thanquorum-size(issue #933), andkeep-oldestdowning every survivor once the lowest-addressed node crashed (issue #932) — are fixed, which is when the default moved. - Off, the node leaves and keeps running its actors — the entities and singletons it hosted, any HTTP binding — beside the side that won. That is a second writer, the very thing a downing decision exists to prevent.
Only this node’s own verdict reaches the switch. A peer’s claim about this
node’s own record is refused, so the winning side cannot down it directly.
It does not have to: a peer that removed this node answers its heartbeats
with a refusal, the node parks every such peer as unreachable, and once every
peer has refused it its own strategy is asked at once — a node paused past
the downing window stands down within a heartbeat round of resuming
(issue #940). A self-down
ends in removed and announces SelfRemoved for this node, on or off, so an
application that wants to stand down on its own terms can subscribe to it.
A node the switch stands down ends its process the way a clean stop does.
With exit-process = on, the run exits with exit-code, which ships as 0:
a downing is none of the failures that never exit 0 — a failed startup, a
node that could not join, a run that a phase halts, which exits 1 here as
well. Left at exit-process = off, the process ends the way your main
ends once runUntilTerminated() resolves: with 0, for the production main
below. So a supervisor that restarts a process only after a failure —
systemd’s Restart=on-failure, as the
systemd unit sets it —
does not bring the node back. Two ways do:
- Restart whatever the status:
Restart=always, or the same policy of another supervisor. - A non-zero
exit-code, withexit-process = on. That status is not the downing’s alone: every runexit-processends exits with it, whatever its reason — a signal’s run and the run of asystem.terminate()that runs the pipeline as much as this one — and the runs for a failure exit with it instead of1. A supervisor then reads every stop as a failure, and the status no longer tells a failed start, a failed join or a halted run from an orderly stop.
What runs after coordinatedShutdown.run() completes
Section titled “What runs after coordinatedShutdown.run() completes”By the time the promise resolves:
- Every task in every phase has either succeeded or timed out.
- The built-in
actor-system-terminatetask has calledsystem.terminate(), which has stopped every actor and closed the dispatcher and scheduler. - The log sinks have been flushed and closed, bounded by
actor-ts.logger.close-timeout(3 s by default) — so a batching sink’s last records are on disk or on the wire before the process goes. That flush lives insideterminate()rather than in a phase, which means it also covers a program that callsterminate()directly; see multi-sink logging. - The process is free to exit (
process.exit(0)). Nothing left to do.
A whole production main:
async function main() { const system = ActorSystem.create('my-app');
await system.http(8080).bind(routes); // registers its own unbind // Register anything the framework does not own...
await system.runUntilTerminated();}
main().catch((err) => { console.error(err); process.exit(1);});When SIGTERM arrives the pipeline runs, the system terminates,
runUntilTerminated() resolves, its handlers come off, and the
process exits because nothing is keeping the loop alive.
Common pitfalls
Section titled “Common pitfalls”Where to next
Section titled “Where to next”- Actor system —
the
terminate()that runs at the end of the pipeline. - Cluster overview —
Cluster.join()registers thecluster-sharding-shutdown-regionandcluster-leavetasks; the other cluster phases are yours to fill. - Kubernetes deployment — the full PreStop + SIGTERM + grace-period recipe.
- Persistence — Migration — rolling shutdown for journal migrations.
The CoordinatedShutdown
API reference covers addTask, addPhase, run, and the full
phase constant set.
