コンテンツにスキップ
日本語

Coordinated shutdown

このコンテンツはまだ日本語訳がありません。

system.terminate() stops the actor system, but a production app usually has work to do before that: drain in-flight HTTP requests, tell the cluster you’re leaving, flush a journal, close broker connections. And those steps have an order — hand the sharding region’s shards off before you leave the cluster; stop the HTTP server before the actors that handle requests.

Coordinated shutdown is the DSL for that. You register tasks against named phases; the framework runs them in dependency order, one phase after the next, each with a timeout cap. Calling run() from any trigger (SIGTERM, K8s PreStop hook, an admin endpoint) executes the whole pipeline once.

import {
ActorSystem,
CoordinatedShutdownId,
Phases,
type Reason,
} from 'actor-ts';
const system = ActorSystem.create('my-app');
const coordinatedShutdown = system.extension(CoordinatedShutdownId);
coordinatedShutdown.addTask(Phases.ServiceUnbind, 'close-http', async (reason) => {
await httpServer.close();
});
coordinatedShutdown.addTask(Phases.ServiceRequestsDone, 'drain-in-flight', async () => {
await waitForInFlightRequests(/* up to 10s */);
});
await system.runUntilTerminated(); // SIGTERM/SIGINT → the pipeline → resolves when it is done

Three things happen when SIGTERM lands:

  1. The runtime calls coordinatedShutdown.run(new ProcessTerminateReason('SIGTERM')).
  2. The phases run in canonical order. Inside each phase, all registered tasks run in parallel; the phase waits for them all (or for their timeouts).
  3. The pipeline finishes with the built-in actor-system-terminate task, which calls system.terminate() for you.

Some of what happens in between is already wired for you; the rest is whatever you added.

Listed in execution order. Wired marks the phases the framework populates by itself — everything else is empty until you register something.

#Phase nameWiredTypical tasks
1before-service-unbind—Last-chance announcements before the server stops accepting connections.
2service-unbindsystem.http(...).bind(...), DevToolsStop the HTTP server / gRPC server / WebSocket listener from accepting new connections.
3service-requests-done—Wait for in-flight requests to finish; abort the rest.
4service-stopbroker actors (MQTT, Kafka, NATS, AMQP, …)Close client connections, release sockets.
5before-cluster-shutdown—Optional pre-cluster-leave hooks.
6cluster-sharding-shutdown-regionCluster.join(...)Tell the sharding region to hand off entities.
7cluster-leaveCluster.join(...)Issue a Cluster.leave() — gossip leaving status.
8cluster-exiting—Wait for the cluster to acknowledge the leave.
9cluster-exiting-done—Confirm cluster transition is complete.
10cluster-shutdown—Tear down cluster transports.
11before-actor-system-terminate—Last-chance app-level cleanup (flush journals, close brokers).
12actor-system-terminatethe built-in terminatorThe built-in system.terminate() task.

You don’t have to use every phase. Empty phases are no-ops; only phases with registered tasks do anything.

Set actor-ts.coordinated-shutdown.auto-register-tasks = false and the framework registers none of them. The phases stay, the built-in terminator stays, and every resource above is yours to register — for an embedder that owns the lifecycle of what it handed the system. It is one switch rather than one per subsystem, because the reason to reach for it is “I own the lifecycle”, never “unbind the HTTP server but leave the brokers to me”.

The Phases constant exports the canonical names — prefer it over string literals for autocomplete:

import { Phases } from 'actor-ts';
coordinatedShutdown.addTask(Phases.ServiceUnbind, ...); // ✓ typed
coordinatedShutdown.addTask('service-unbind', ...); // ✗ stringly-typed, no auto-complete

For app-specific work that doesn’t fit a canonical phase, declare your own:

coordinatedShutdown.addPhase({
name: 'flush-metrics',
timeoutMs: 3_000,
dependsOn: [Phases.BeforeActorSystemTerminate],
recover: true,
});
coordinatedShutdown.addTask('flush-metrics', 'push-prometheus', async () => {
await metricsRegistry.flush();
});

The dependsOn field is what makes the order DAG-shaped rather than linear — your phase runs after before-actor-system-terminate.

It does not by itself put the phase before actor-system-terminate. Both become ready in the same round of the topological sort — actor-system-terminate has the same single dependency — and the sort’s alphabetical tie-breaker then decides, so flush-metrics above actually runs after the system has terminated. Adding the reverse edge is what places it, and that is depends-on from config (below), which is added to a canonical phase’s existing edges. From code, name a phase whose successor is not the one you need to precede.

Because addPhase requires every dependsOn entry to name an already-registered phase, a cycle can’t be formed in the first place. The order itself is computed by a topological sort when run() executes — that sort would throw Error: cycle in phase dependencies if a cycle ever did arise.

The same graph is expressible in HOCON, under actor-ts.coordinated-shutdown.phases — one child per phase name, each taking timeout, recover and depends-on:

actor-ts.coordinated-shutdown.phases {
flush-metrics {
timeout = 3s
recover = true
depends-on = ["before-actor-system-terminate"]
}
actor-system-terminate { depends-on = ["flush-metrics"] }
}

A name that is one of the twelve canonical phases merges into that phase; anything else declares a new one, and there depends-on is required — a phase with no edges sorts into the first ready batch and would run before before-service-unbind.

Note what the second child buys. depends-on on a canonical phase is added to the edge that phase already has, never a replacement, so naming flush-metrics there is what actually puts your phase before actor-system-terminate rather than merely after before-actor-system-terminate — the two would otherwise become ready in the same round and the sort’s alphabetical tie-breaker would decide. The additive rule is also what stops a config file re-parenting cluster-leave ahead of service-unbind: the sort catches cycles, not a pipeline quietly running in the wrong order.

Declaration order in the file is irrelevant — two phases may name each other and are registered dependency-first. An unresolvable name, a cycle, an unknown setting under a phase, or a declared phase with no depends-on is a ConfigError when the extension is built, not when the shutdown runs.

Ask what the merge produced with phaseDefinition:

coordinatedShutdown.phaseDefinition('flush-metrics');
// { name: 'flush-metrics', timeoutMs: 3000, dependsOn: [...], recover: true }

Every task is a function from a Reason to void | Promise<void>:

type ShutdownTask = (reason: Reason) => Promise<void> | void;

The reason lets a task branch on why shutdown was triggered:

coordinatedShutdown.addTask(Phases.BeforeClusterShutdown, 'deregister', async (reason) => {
if (reason instanceof ClusterDowningReason) {
// We were downed — the registry has already written us off.
return;
}
await serviceRegistry.deregister(cluster.selfAddress.toString());
});

Built-in Reason classes:

ClassWhen
ProcessTerminateReason(signal)SIGTERM/SIGINT, from runUntilTerminated() or installProcessHooks().
ActorSystemTerminateReasonWhat system.terminate() passes when it runs the pipeline — asked for with runCoordinatedShutdown: true, or by default with actor-ts.coordinated-shutdown.run-by-actor-system-terminate on. An ordinary shutdown your program decides on is that call, rather than a run() you hand this reason: the call also tears the system down when the run does not.
ClusterLeavingReasonWhat Cluster.bootstrap’s shutdown() passes — a member leaving on purpose.
ClusterDowningReasonWhat a node passes when its own downing decision removes it and actor-ts.cluster.run-coordinated-shutdown-when-down is on — see when the cluster downs this node.
ClusterJoinUnsuccessfulReasonWhat a node passes when it gives its join up — actor-ts.cluster.shutdown-after-unsuccessful-join passed with nothing able to promote it — and what Cluster.bootstrap passes when its readiness wait fails, or when it refuses to go on after building its system; see when no seed ever answers. A run with it never exits 0.
StartupFailedReason(phase, task, cause)What the system passes when a startup task failed or outlived its phase’s budget — it names the phase and the task and carries the cause.
UnknownReasonThe default when run() is called with no argument.

A failed startup runs this pipeline too: the system starts it with a StartupFailedReason and tears down afterwards, whatever the pipeline did — so a task here can tell a node that never finished starting from one that is being stopped. Nor does it end the process with a status that reads as a clean stop: with exit-process = on it exits with exit-code when that is not 0, and with 1 when it is — the rule a run that halts exits by too. Left at the default exit-process = off, runUntilTerminated() rejects with the startup’s error instead, and a main that awaits it exits non-zero by itself.

A node that could not join runs it as well, and its reason says so. ClusterJoinUnsuccessfulReason arrives when the join gives up, when a Cluster.bootstrap readiness wait fails — which used to pass ClusterLeavingReason, so a task that told a failed start by that reason now sees this one — and when the bootstrap refuses to go on after it has built its system, where run-by-actor-system-terminate makes that teardown a run. None of them ends the process on its own: the built-in terminator ends the actor system, and the process exits only with actor-ts.coordinated-shutdown.exit-process = on, and then by the rule above: a node that could not join did not stop cleanly either, and a failure never exits 0. The bootstrap says why at ERROR before the run starts, because the exit comes before its error could reach your main.

You can subclass Reason for app-specific triggers (AdminEndpointReason, HotReloadReason, etc.).

All tasks in a phase run concurrently — they’re started together and the phase waits for the last one (or its timeout). If you have ordering requirements within a phase (task B must wait for task A), put them in different phases with a dependsOn.

Each phase has a timeoutMs (default 5 s); each task is wrapped in a timeout race. A task that doesn’t finish in time is logged and either:

  • Recovered from (the phase continues, recover: true — the default). The next phase starts.

  • Halts the pipeline (recover: false). Subsequent phases are not run; shutdown stops mid-flight. A run that terminate({ runCoordinatedShutdown: true }) started still ends in the teardown, because that call asked for a terminated system; so does a run that a signal, this node’s own downing decision or a join that could not complete started — nobody awaits those, so the run answers its own halt — unless terminate-actor-system = false says the pipeline never takes the system down. Either way the halt is logged as an ERROR. A run() you call yourself, with any other reason, rejects with the task’s error and leaves the system as the halt found it.

    With exit-process = on the process exits at the halt instead, whichever trigger started the run — a signal, terminate(), your own run() or the cluster’s — and with a status that does not read as a clean stop: exit-code where that is not 0, 1 where it is. Nothing after the exit runs, so it takes the place of those teardowns — but not of what they flush on the way down: the dead-letter queue’s writes and then the log sinks are flushed before the exit, each within actor-ts.logger.close-timeout, so the ERROR line naming the phase, logged just before, reaches a buffering sink too. A completed run that left the system up (terminate-actor-system = false) is flushed the same way before its exit; one whose terminator tore the system down was flushed by that teardown already. The run is in flight until its exit, so a terminate() that arrives while those flushes run starts nothing.

Override per phase:

coordinatedShutdown.setPhaseTimeout(Phases.ServiceRequestsDone, 30_000); // 30s drain budget

Or define your own with the wanted timeoutMs / recover:

coordinatedShutdown.addPhase({
name: 'aggressive-cleanup',
timeoutMs: 1_000, // strict cap
dependsOn: [Phases.BeforeActorSystemTerminate],
recover: false, // failure → halt
});

Both are also config, per phase and without a rebuild — the same phases block as above, and it retimes only the phase it names:

actor-ts.coordinated-shutdown {
default-phase-timeout = 5s # every phase that says nothing
phases {
service-requests-done { timeout = 30s } # …except this one
service-stop { recover = false } # a failure here halts the run
}
}

The drain budget sits inside the last phase

Section titled “The drain budget sits inside the last phase”

actor-system-terminate is a task that awaits the system’s teardown — what a bare system.terminate() runs — and the teardown begins by draining the actors under /user — see Terminating. So two budgets are nested, and the inner one has to be the smaller:

BudgetKeyDefault
The whole actor-system-terminate phaseactor-ts.coordinated-shutdown.phases.actor-system-terminate.timeout, falling back to …default-phase-timeout5 s
The drain inside itactor-ts.system.shutdown-drain-timeout2 s

Raise the drain budget above the phase timeout and the phase is abandoned while the drain is still running — before a single actor has been told to stop. Raise both together, or raise the phase timeout first.

Reach for the per-phase key when only the terminate step needs more room. Raising default-phase-timeout instead hands the same longer budget to eleven other phases, and each of those is a phase a stuck task can now hold the shutdown in for that much longer.

Almost every service wants the same thing — install the handlers, wait, shut down, exit — so that whole shape is one call:

await system.runUntilTerminated();
// Or, to listen for something else as well:
await system.runUntilTerminated(['SIGTERM', 'SIGINT', 'SIGUSR2']);

It installs the handlers, resolves once the system is down and the pipeline has finished, and detaches the handlers on the way out. That last step is not housekeeping: on Deno a signal listener holds the event loop open and has no unref, so a program that shuts down for any other reason — a terminate() from inside, an admin endpoint — would never exit.

A signal the platform cannot deliver is skipped rather than registered. Windows has no SIGTERM under any runtime, and Deno throws on a signal it cannot deliver instead of ignoring it, so asking for one is never a startup failure.

The process stays alive for as long as the promise is pending, and that is a guarantee the call makes rather than one it inherits from the handlers it installed. A signal handler is not by itself a reason for a runtime to keep running: Node unrefs its signal handles, so a service with nothing else on the event loop — no bound port, no open socket, no referenced timer — would otherwise drain its loop the moment it started waiting, and exit before the SIGTERM it had just armed itself for could arrive. So the call holds the loop open itself and lets go in the same step that detaches the handlers. A headless node needs no setInterval of your own to stay up.

The two halves are still available separately when you are embedding the system in a host that owns the process lifetime:

coordinatedShutdown.installProcessHooks(); // → run(new ProcessTerminateReason(signal))
coordinatedShutdown.removeProcessHooks(); // detaches exactly what it installed

Calling either twice is harmless. removeProcessHooks() never touches a listener it did not install, so a second ActorSystem in the same process — or the application’s own SIGTERM handling — is left alone.

The default is config, so a host that owns the signals can take the handlers off every node it runs without touching the code:

actor-ts.coordinated-shutdown.run-by-process-signals = false

That switches what runUntilTerminated() and the cluster bootstrap do when the caller named no signals. Naming them — runUntilTerminated(['SIGTERM']), shutdownOnSignals in the bootstrap options, or installProcessHooks() itself — still installs, because explicit options outrank config. With it off, runUntilTerminated() keeps both of its other promises: it waits, and it holds the event loop open.

In Kubernetes, the pod-shutdown sequence is:

1. K8s sends SIGTERM and ends the pod's grace period clock.
2. K8s also calls the PreStop hook (if configured), running concurrently.
3. After max(graceful-shutdown, grace-period), K8s sends SIGKILL.

The standard recipe:

// On SIGTERM, run coordinated shutdown:
await system.runUntilTerminated();
// PreStop hook script (in your container image):
// #!/bin/sh
// sleep 10 # give upstream LBs time to drain this pod
// exit 0

The sleep in PreStop gives the load balancer time to drop this pod from rotation before the actor system starts shutting down — so in-flight HTTP requests don’t see “I’m draining, go away.”

See Operations — Kubernetes for the full deployment manifest.

coordinatedShutdown.run() is idempotent — calling it multiple times returns the same in-flight promise. Three independent triggers (SIGTERM, an admin endpoint, and a cluster downing) all calling run doesn’t re-run the pipeline. The first call starts it; subsequent calls await the same completion.

This matters because in production you often have multiple shutdown paths:

// SIGTERM path:
await system.runUntilTerminated();
// Admin-endpoint path:
app.post('/shutdown', async (req, res) => {
await coordinatedShutdown.run(new AdminEndpointReason());
res.send('ok');
});
// Cluster-downing path: run-coordinated-shutdown-when-down = on — see below.

Each of them ends up running the same shutdown sequence, once.

A node whose own downing decision removes it — a strategy’s verdict that includes this node, or down-all-when-unstable — always leaves the cluster. Whether it also stands down is run-coordinated-shutdown-when-down:

const clusterOptions = ClusterOptions.create()
.withDowning(new KeepMajority())
.withRunCoordinatedShutdownWhenDown(true);
actor-ts.cluster.run-coordinated-shutdown-when-down = on

With it on, the node leaves first and then runs this pipeline with ClusterDowningReason, even when the leave fails: its services unbind, and the built-in terminator ends the actor system and with it every entity and singleton the node hosted. terminate-actor-system and exit-process decide what the run ends in, as they do for a signal — a run that a phase halts still ends in the teardown, because a downed node must not keep its actors running beside the side that won — and auto-register-tasks = off does not suppress it: that switch withholds the framework’s own tasks and decides nothing about when the pipeline runs. The node says so at warn before the run starts. A decision that downs only peers changes nothing on this node: the switch answers its own removal, not a downing it decides.

It ships on:

  • On, a wrong self-down terminates the node. The two known ones — static-quorum downing a cluster still smaller than quorum-size (issue #933), and keep-oldest downing every survivor once the lowest-addressed node crashed (issue #932) — are fixed, which is when the default moved.
  • Off, the node leaves and keeps running its actors — the entities and singletons it hosted, any HTTP binding — beside the side that won. That is a second writer, the very thing a downing decision exists to prevent.

Only this node’s own verdict reaches the switch. A peer’s claim about this node’s own record is refused, so the winning side cannot down it directly. It does not have to: a peer that removed this node answers its heartbeats with a refusal, the node parks every such peer as unreachable, and once every peer has refused it its own strategy is asked at once — a node paused past the downing window stands down within a heartbeat round of resuming (issue #940). A self-down ends in removed and announces SelfRemoved for this node, on or off, so an application that wants to stand down on its own terms can subscribe to it.

A node the switch stands down ends its process the way a clean stop does. With exit-process = on, the run exits with exit-code, which ships as 0: a downing is none of the failures that never exit 0 — a failed startup, a node that could not join, a run that a phase halts, which exits 1 here as well. Left at exit-process = off, the process ends the way your main ends once runUntilTerminated() resolves: with 0, for the production main below. So a supervisor that restarts a process only after a failure — systemd’s Restart=on-failure, as the systemd unit sets it — does not bring the node back. Two ways do:

  • Restart whatever the status: Restart=always, or the same policy of another supervisor.
  • A non-zero exit-code, with exit-process = on. That status is not the downing’s alone: every run exit-process ends exits with it, whatever its reason — a signal’s run and the run of a system.terminate() that runs the pipeline as much as this one — and the runs for a failure exit with it instead of 1. A supervisor then reads every stop as a failure, and the status no longer tells a failed start, a failed join or a halted run from an orderly stop.

What runs after coordinatedShutdown.run() completes

Section titled “What runs after coordinatedShutdown.run() completes”

By the time the promise resolves:

  • Every task in every phase has either succeeded or timed out.
  • The built-in actor-system-terminate task has called system.terminate(), which has stopped every actor and closed the dispatcher and scheduler.
  • The log sinks have been flushed and closed, bounded by actor-ts.logger.close-timeout (3 s by default) — so a batching sink’s last records are on disk or on the wire before the process goes. That flush lives inside terminate() rather than in a phase, which means it also covers a program that calls terminate() directly; see multi-sink logging.
  • The process is free to exit (process.exit(0)). Nothing left to do.

A whole production main:

async function main() {
const system = ActorSystem.create('my-app');
await system.http(8080).bind(routes); // registers its own unbind
// Register anything the framework does not own...
await system.runUntilTerminated();
}
main().catch((err) => {
console.error(err);
process.exit(1);
});

When SIGTERM arrives the pipeline runs, the system terminates, runUntilTerminated() resolves, its handlers come off, and the process exits because nothing is keeping the loop alive.

  • Actor system — the terminate() that runs at the end of the pipeline.
  • Cluster overview — Cluster.join() registers the cluster-sharding-shutdown-region and cluster-leave tasks; the other cluster phases are yours to fill.
  • Kubernetes deployment — the full PreStop + SIGTERM + grace-period recipe.
  • Persistence — Migration — rolling shutdown for journal migrations.

The CoordinatedShutdown API reference covers addTask, addPhase, run, and the full phase constant set.