Actor & cluster panels

Three views land together because they answer the same question from different distances: is this system healthy, and if not, where?

Overview

The page DevTools opens on. Each tile carries a sparkline, and the charts below plot the same series over the last few minutes — because a single number cannot tell you whether 40 queued messages is the steady state or the start of a pile-up.

Common answers what am I looking at: actor system, uptime, runtime, cluster status. Uptime is the system’s own, measured from ActorSystem.startedAtMs and sent by the server, so reloading the page or losing the socket does not make the system look younger than it is.

Numbers is the figures themselves:

Tile	Reads
Actors	Live actors, including the guardians
Messages / s, Processed messages	Throughput, from actor_messages_delivered_total
Spawns / s, Stops / s	Actor churn
Restarts	Supervisor-driven restarts since attach
Mailbox backlog	Every pending message in the system, summed
Stashed	Every stashed message, summed
Suspended actors	Actors halted by supervision — nonzero means trouble
Dead letters	Undeliverable messages since attach
Mailbox drops	Messages a bounded mailbox threw away on overflow
Handler p99	Time in onReceive, interpolated from histogram buckets

Counters are cumulative on the wire; the UI derives the per-second rates itself. That means a browser that reconnects, throttles in a background tab, or misses a tick still shows correct figures rather than a phantom spike.

In a cluster the figures are the whole cluster: every node runs a DevTools agent that answers with its own numbers, and the totals above are the sum of them. A Per node section breaks the same figures down one row per node. A node that stops answering keeps its last reading, marked not answering, rather than disappearing — and is forgotten an hour later, the same window the cluster panel uses.

Charts plots those series over a timespan you choose — one minute to twenty-four hours, five minutes by default. The series is recorded on the server from the moment DevTools attaches, so a panel opened an hour in shows that hour rather than starting an empty graph; reloading the page does not lose it either.

Longer spans are summarised: a day arrives in two-minute buckets, an hour in fifteen-second ones, and the panel says which. Levels keep each interval’s peak so a spike survives being summarised, and cumulative counters keep the interval’s last reading so the per-second maths stays correct however coarse the buckets are.

The three plots are split so that a level (queued messages) and a rate (messages per second) never share a y-axis — on one scale a backlog of 400 flattens a 2/s line to nothing. Below them, Busiest mailboxes lists the deepest queues right now: the fastest way from “something is backing up” to “this actor is backing up”.

Throughput needs metrics

Message counts, mailbox drops and handler latency come from the framework’s own counters, which are inert until the metrics registry is enabled. DevTools enables it at attach if you have not, and restores the noop on detach — see Cost.

Actors

The live supervision tree, from the root guardian down.

• Filter by path, class name or display name. Matching keeps ancestors visible, so a hit deep in a collapsed branch is still reachable — and a filter temporarily overrides collapsing, because a match you cannot see reads as a search that failed.
• The row label is the actor’s own name when it has one — whatever displayName() resolves to — and the last path segment otherwise. Worth setting for Behaviors actors above all, whose class column reads TypedActor for every row. The full path stays in the row’s tooltip either way, and sibling order stays on the path segment, so a rename never makes rows jump.
• Badges show what is unusual: messages waiting, stashed messages, a suspended mailbox. An idle actor carries none.
• The state dot is live. Green running, amber suspended or terminating, red terminated. Lifecycle events only announce births, deaths and restarts — nothing announces a suspension — so the tap re-inspects the tree each second and sends the cells that moved.
• A restart flashes the row. The path survives a restart, so nothing moves in the tree — without the flash it would be invisible.
• A stopped actor stays for 30 seconds, greyed out with its name struck through and a “stopped 12s ago” badge, then disappears. The actor you want to look at is usually the one that just died; removing its row the instant it stopped meant you never saw it. Uncheck Keep stopped if you would rather see only the living.
• Hide DevTools actors is on by default. DevTools runs its own hub and taps as ordinary actors; they are real, so they are listed, but they are not what you came to look at. They are marked at spawn with ActorOptions.withInternal() rather than matched by name, so their children — a DevTools websocket connection is a child of its hub — are hidden too.

In a cluster the panel lists every node, each under its own heading — which is how you see that one node is hosting a singleton and the others are not, or that only one of them is piling up work. Paths repeat across nodes (they all run the same system name), so the trees are kept apart by address rather than by path.

The local tree arrives as one snapshot followed by deltas, so a system that spawns constantly does not re-send its whole tree every time. Remote nodes report their whole tree each round — they have no channel to push a spawn as it happens — and an actor that stops appearing in one is tombstoned for the same thirty seconds a local one gets. A node that stops answering keeps its last tree, dimmed and with its state dots neutral: those actors are not running, nobody knows what they are. Mailbox depths come on their own stream and refresh once a second.

Why the tree can look busier than your code

Every WebSocket connection to DevTools spawns a connection actor, and the framework’s own machinery (sharding, singletons, pub-sub) spawns actors too. Nothing is hidden except DevTools’ own, and that is a checkbox.

Reading the two branches

/user holds what your code spawned; /system holds what the framework spawned for itself, grouped by subsystem — /system/cluster/sharding, /system/cluster/pubsub, /system/devtools, and so on. If you are looking for your own actors, /user is the whole story.

The group levels are actors too (empty supervisors that exist to hold the grouping and a supervision policy), so they show up as nodes with children and no messages. Groups appear the first time something needs them, so a system that never starts DevTools or clustering has a /system with nothing under it.

Cluster

Available when you pass a Cluster — a system cannot hand out its own, the same reason managementRoutes takes one:

import { DevTools, DevToolsOptions } from 'actor-ts/devtools';

const devtoolsOptions = DevToolsOptions.create().withCluster(cluster);
await DevTools.attach(system, devtoolsOptions);

Without it the panel stays visible but explains why it is unavailable, rather than disappearing.

• Topology — members on a ring, coloured by status. The leader draws larger, this node draws with a thick outline. A ring rather than a force-directed layout: every node gossips with every other, so the graph is complete and a simulation would only produce a slowly settling circle — while a ring draws the same membership the same way every time.
• Members — address, status, roles, and who leads. A node that leaves the membership is kept for an hour, struck through and red, with how long ago it was last seen. The node that drops out is the one you want to look at, and it used to disappear at exactly that moment. The record lives on the server, so reloading the page — which is often the first thing you do — does not erase it.
• Shard distribution — shards per region, per sharded type. Types appear as their coordinator republishes them (the framework keeps no central registry of sharded types), so a freshly opened panel fills in within a coordinator tick.
• Membership history — transitions since you attached, newest first. Unreachable and down are coloured as problems; joins and leaves are not.

Cost

Both sampling streams idle until a panel subscribes, so a browser tab left open in the background costs the system nothing. The tree tap listens to actor lifecycle events, which the runtime publishes whether or not anyone is watching — the same events that already drive the actor_created_total / actor_terminated_total metrics.

Trying it

bun run examples/cluster/singleton-hello.ts --devtools

Every example is wired for DevTools; a three-node cluster demo gives you three ports, 9333 upwards. See the overview for the details.