Stock metrics
이 콘텐츠는 아직 번역되지 않았습니다.
When the metrics extension is enabled, the framework automatically records a baseline of metrics covering the actor lifecycle, message handling, bounded mailboxes, and the cluster.
import { ActorSystem } from 'actor-ts';import { MetricsExtensionId } from 'actor-ts/metrics';
const metrics = system.extension(MetricsExtensionId).enable();// Stock metrics now record into the live registry — no further setupThese are the metrics you’d write yourself anyway. Shipping them out of the box lets you wire a dashboard immediately.
Actor metrics
Section titled “Actor metrics”Actor-lifecycle and message-handling metrics, recorded across the whole system — each is a single unlabeled series:
| Metric | Type | Labels | Meaning |
|---|---|---|---|
actor_created_total | counter | — | Actors successfully started. |
actor_terminated_total | counter | — | Actors stopped (clean stop or post-failure). |
actor_restarted_total | counter | — | Supervisor-driven restarts. |
actor_messages_delivered_total | counter | — | User messages delivered to onReceive. |
actor_message_handler_seconds | histogram | — | Time spent inside onReceive handlers, in seconds. |
actor_message_handler_seconds uses the default seconds-scale
buckets; its p99 is your “how slow is a handler” signal.
Mailbox metrics
Section titled “Mailbox metrics”| Metric | Type | Labels | Meaning |
|---|---|---|---|
actor_mailbox_size | gauge | class, path | Queued user messages, sampled. Only actors at or above 10 000 are represented. |
actor_mailbox_depth | histogram | — | Queued user messages at the moment one was delivered, including itself. |
actor_mailbox_wait_seconds | histogram | — | Time a user message spent queued before delivery, in seconds. |
actor_mailbox_dropped_total | counter | class, reason | Messages dropped by a bounded mailbox’s overflow policy. |
actor_mailbox_size is the backlog signal. Mailboxes are
unbounded by default, so an actor that falls behind accumulates rather
than sheds — and a series only exists once one crosses 10 000 queued
messages. Its presence is therefore the alert: on a healthy system
the metric is empty. A mailbox that drains back below the floor — or
whose actor stops — has its series removed, not set to 0, so
“healthy system, no series” holds after an incident and not just before
one. The same threshold produces a log warning (repeated at each
doubling) whether or not metrics are on.
Removal is why the family’s width counts concurrent incidents rather
than incidents ever had: an actor that recovers gives its slot under the
cardinality cap
back. Alert on the series existing (count(actor_mailbox_size) > 0) or
on its value, not on a transition to 0 — there is no 0 to transition to.
actor_mailbox_depth is the distribution the gauge cannot be, and
the two are designed to meet at one number. The gauge tells you which
actor is behind, and pays for that with a path label it can only
afford above 10 000 — below that it reports nothing at all, which is the
entire 1–9 999 range, and a spike between two of its 2 second samples is
recorded nowhere. The histogram covers exactly that range: it is
observed once per delivery, so nothing is missed between samples, and it
carries no labels, so the whole family costs one series per bucket
however many actors or entities exist. Its last bucket boundary is
the gauge’s floor, so anything in the +Inf overflow is by construction
an actor the gauge is already reporting by path.
Read the histogram to learn that a backlog exists and how deep its tail goes; read the gauge to learn whose it is.
Buckets run 1 → 10 000 messages on a 1-2-5 ladder. The floor is 1
rather than 0 because the observation counts the message being
delivered: a quiet actor reads exactly 1, so there is no bucket that
nothing can fall into. Its count matches
actor_messages_delivered_total exactly — one observation per delivery,
with none of the exclusions the wait histogram has.
actor_mailbox_wait_seconds is the latency signal, and the half
actor_message_handler_seconds cannot give you: that histogram starts
measuring once a message is already being handled, so an actor that is
slow and an actor that is merely behind look identical in it. Read the
two together — handler p99 high and wait p99 low means the handler
itself is the problem; wait p99 high and handler p99 low means the
actor is fine and there is simply more work arriving than it can take.
Its buckets are not the default seconds-scale ladder. They run 1 ms → 10 s, because a mailbox that is keeping up drains in well under the 5 ms the defaults start at, and a histogram whose first bucket holds everything answers nothing. 1 ms is also the finest the metric could be: the stamp is wall-clock, so anything faster is reported as a zero-millisecond wait and lands in the first bucket, which reads as “delivered within a millisecond” and is exactly true.
Two kinds of message are deliberately left out, which matters when you
compare its count against actor_messages_delivered_total:
- Replayed stashed messages. A message that comes back out of the
stash kept the stamp from its original arrival, so counting it
would report however long your actor chose to hold it as queueing
delay. One actor stashing for thirty seconds while it waits on a
resource would drown out every other actor’s signal in a metric that
has no labels to separate them. The
explain plan makes the
opposite choice — it shows the whole arrival-to-handling span,
because there you can see the
stashedentry that accounts for it. - Messages queued before metrics were switched on. They carry no stamp, so their wait is omitted rather than invented. This corrects itself within one drain.
Throttled messages are counted. A message parked by an actor’s throttle really is waiting in the queue for an actor that cannot keep up, which is the question this metric asks.
actor_mailbox_dropped_total is the shedding signal, and only appears
for actors whose mailbox discards something — which, since the default is
unbounded, means actors someone deliberately bounded. The reason label
records the policy that fired (drop-head / drop-new). Both ways of
bounding are covered: withMailboxCapacity, and a mailbox you build
yourself and pass to withMailbox.
It counts per class, not per actor. There is no path label: a
bounded mailbox sheds as its designed steady state rather than as an
anomaly, so a path-labelled counter minted one permanent series for
every actor that was working as intended — and under sharding the value
came from whoever addressed the shard region. If you need to know which
instance is shedding, pass an onDrop of your own; it fires alongside
the stock counter rather than replacing it, so the series is yours to
label and to size:
import { ActorOptions, BoundedMailbox } from 'actor-ts';
// `onDrop` runs alongside the stock counter, never instead of it.const workerOptions = ActorOptions.create() .withMailbox(() => new BoundedMailbox({ capacity: 1_000, overflow: 'drop-head', onDrop: (reason) => metrics.counter('jobs_shed_total', { entityId, reason }).inc(), }));
system.spawn(Worker, entityId, workerOptions);The cardinality of jobs_shed_total is then yours to budget for — which
is the point. The framework will not spend it on your behalf.
Dispatcher metrics
Section titled “Dispatcher metrics”| Metric | Type | Labels | Meaning |
|---|---|---|---|
actor_dispatcher_queue_delay_seconds | histogram | dispatcher | Time an actor turn waited between being handed to a dispatcher and starting, in seconds. |
This is the saturation signal: whether the dispatcher is keeping up
with the turns being handed to it. One observation per turn, not per
message, so a batch of up to
throughput messages counts
once. The dispatcher label is Dispatcher.id, so every dispatcher in
the system reports separately — including a per-actor
ActorOptions.withDispatcher(…), which nothing else in the framework can
even enumerate.
At rest the delay is a single hand-off: around 1 µs through
MicrotaskDispatcher and 3 µs through ImmediateDispatcher. The default,
HybridDispatcher, hands most turns over through the microtask queue and
every 64th through setImmediate; no at-rest figure has been measured for
it on its own.
Under saturation the delay grows without bound, because the turns queue up
behind each other. So the alert to write is on the quantile against your
own latency budget:
histogram_quantile(0.99, rate(actor_dispatcher_queue_delay_seconds_bucket[5m])) > 0.05# A turn is waiting 50 ms for a slot. Something on this dispatcher is# not yielding, or its throughput budget is too high for the load.Buckets run 10 µs → 10 s on a 1-5 ladder. The floor is two decades below the mailbox families’ 1 ms because a healthy hand-off is microseconds, so the first bucket means “scheduled immediately” and the second already means “something was queued ahead of it”.
One reading needs care. The delay measures the queue the dispatcher
itself uses, and MicrotaskDispatcher’s queue is the microtask queue,
which the runtime drains before any timer or I/O. Its delay therefore
stays low even while actors are starving the event loop — which is that
dispatcher’s documented hazard, not a contradiction. A low delay there
is evidence that microtask scheduling is not the bottleneck, not
evidence of headroom; use ThroughputDispatcher or the default if you
want the number to reflect loop pressure.
Why there is no saturation ratio
Section titled “Why there is no saturation ratio”A 0–1 “how busy is the dispatcher” fraction would be a more familiar
number, and this framework does not publish one, because it cannot
compute one honestly on every runtime it supports. The only primitive
that could is performance.eventLoopUtilization, and it behaves
differently on each of the three:
| Runtime | performance.eventLoopUtilization |
|---|---|
| Bun 1.3 / 1.4 | absent until something imports node:perf_hooks; from then on present, and permanently { idle: 0, active: 0, utilization: 0 } (measured on 1.4.2) |
| Node 26 | present, and a real reading |
| Deno 2.6 | present, and permanently { idle: 0, active: 0, utilization: 0 } |
A feature check therefore passes on Node and Deno, and on Bun as soon
as anything in the process has imported node:perf_hooks — and only
Node’s reading is real. A ratio built on it would read a flat 0 % on
Deno for ever, and on Bun wherever the check passed, which is worse
than publishing nothing at all: an alert on a metric that never fires
looks identical to a system that is never saturated. And even where
the reading is real it covers the whole event loop, so it can never be
attributed to one of several dispatchers.
Scheduling delay needs nothing but a clock, so it is the same measurement on all three, and it answers the same question in a form an alert rule can state: instead of “utilization is at 100 %”, “turns are waiting longer than my budget”.
If you want a true per-node loop-occupancy figure, that is a different signal from a different source — timer lag, sampled from the runtime. The framework does not ship one yet: an event-loop starvation detector built on it is planned under #886.
Dead-letter metrics
Section titled “Dead-letter metrics”| Metric | Type | Labels | Meaning |
|---|---|---|---|
actor_dead_letters_total | counter | outcome | Undeliverable messages the dead-letter queue captured, by outcome. |
actor_unhandled_total | counter | class | Messages a live actor was handed and declined, by actor class. |
Only present once
actor-ts.dead-letters.store is something
other than off — the counter is minted by the queue, so a system that
captures nothing counts nothing. store = "metrics" is the setting for
this counter and nothing else: it counts every letter and retains no
payload. outcome is one of:
captured— a new letter entered the queue.replayed— a letter was handed back.replay-failed— a replayed letter came back, i.e. the recipient still cannot take it. A risingreplay-failedagainst a flatcapturedis a poison message being retried, not a new outage.
There is deliberately no recipient label. It used to carry the
full path of the actor the message failed to reach, which under sharding
is entity-<entityId> — chosen by whoever addresses the shard region —
and for an anonymous actor is a fresh path per spawn. Against the
stock-label rule at the end of this page that is a per-instance value
with nothing paying for it: the sibling actor_mailbox_size keeps
its path because minting one costs a sustained 10 000-message backlog,
while minting one here cost a single undeliverable message.
Which path a letter was addressed to has not gone anywhere — it was never
only on the counter. Every letter is published on the
event stream as a DeadLetter carrying its
recipient ref, and under store = "memory" or "persistent" the queue
records recipientPath on the entry itself. Both are per-event rather
than per-series, so they cost nothing permanent; deadLetterQueue.list({ recipient }) is the query that answers “which actor”, and the counter
answers “how much, and how is it trending”.
actor_dead_letters_total is a loss signal, not a backlog one. Any
sustained non-zero rate means messages are being sent to actors that are
not there — a stale ref held past a restart, a selection built from a
path that changed, a singleton with no host.
actor_unhandled_total is the neighbouring signal, and the one that is
present by default. It counts a message that was delivered and then
declined — an actor calling
this.unhandled(m),
a behavior answering Behaviors.unhandled, or a cluster wire frame no
handler claimed — and unlike the counter above it does not need the queue
switched on, because it is incremented at the sink rather than by the
queue. class is the declining actor’s constructor name, or Cluster
for the wire case, which has no actor to name; it is the only label, for
exactly the reason the paragraph above gives about recipient.
A sustained rate here is a protocol problem rather than a routing one: something is sending this node messages it will not handle, which after a rolling upgrade usually means a peer speaking a newer wire protocol than this build knows. Nothing detects a declined message on your behalf — an actor that silently ignores one still counts nothing, which is what makes a non-zero rate a statement rather than noise.
Cluster sharding declines through the same sink: ShardRegion,
ShardCoordinator and Shard count a message none of their arms takes —
a sharding. kind from a newer member, or one sent to the wrong one of the
three — and since none of them declines anything under healthy traffic, a
rate on those classes is drift rather than load.
Cluster metrics
Section titled “Cluster metrics”| Metric | Type | Labels | Meaning |
|---|---|---|---|
cluster_members_up | gauge | — | Members currently in the up state (this node’s view). |
cluster_gossip_rounds_total | counter | — | Gossip-push rounds initiated by this node. |
cluster_gossip_records_refused_total | counter | reason | Gossiped member records a merge-path guard refused. |
cluster_envelope_from_mismatch_total | counter | frame | Envelopes whose payload named a sender other than the connection they arrived on. |
cluster_envelope_refusals_total | counter | reason, frame | Inbound envelopes refused because their target path is outside what this node accepts by name. |
cluster_outbound_refusals_total | counter | frame | Frames this node refused to send because the destination was neither a member nor a node it already holds a connection to. |
For monitoring cluster health:
cluster_members_upshould equal your configured replica count; a persistent shortfall means members are down or unreachable.cluster_gossip_rounds_totalrate confirms gossip is flowing — a flat line means this node has stopped gossiping.cluster_gossip_records_refused_totalshould sit at zero. Movement onreason="removed-run"means a run this node removed is still gossiping about itself — a process that was paused or cut off and resumed, which stands down on its own (#940). Movement onreason="version-skew"means gossiped versions are further ahead of this node’s clock thanmaxVersionSkewMs— either a peer’s clock has drifted or someone is trying to pre-claim an address. Movement onreason="map-cap"meansmax-members/max-tombstonesis full.reason="timestamp-skew"is a tombstone whoseremovedAtis not a plausible instant.reason="replayed-frame"is a whole gossip frame whosesequencedid not out-number the last one accepted from that peer, or was not a finite number withinmaxVersionSkewMsof this node’s clock — a duplicate, someone re-sending a frame captured off the wire, or someone re-sending one with itssequencerewritten to clear the mark. The counter is incremented once per frame with that frame’s count, and the label set is closed at those four values.cluster_envelope_from_mismatch_totalshould sit at zero too. It moves when an envelope’s payload names a sender other than the connection it arrived on — a client old enough to still send that field and wrong about its own address, or someone testing whether the node routes on payload. It never changes what the node does: the reply goes down the connection either way. Theframelabel is the wire kind, drawn from code and never from the payload, so the series count is bounded by how many wire handlers make the check — one today,cluster-client-envelope. The claimed address is deliberately not a label; putting it there would mint one series per address a sender cares to invent. Turn the node’s log todebugto see which connection is carrying them.cluster_envelope_refusals_totalcounts a target path this node would not resolve by name.reason="system-path"should sit at zero on a healthy cluster: it means a peer, or an outsideClusterClient, addressed a/systemframework actor directly rather than through the handler that actor registered. A rolling upgrade past the release that closed that route is the one benign cause; anything else is worth looking at.reason="not-allow-listed"only ever moves withactor-ts.remote.untrusted-modeon, and on the day you turn it on it is the metric to watch — every entry yourtrusted-selection-pathsis missing shows up here as refused traffic. Both labels are closed sets drawn from code:reasonhas those two values,frameisenvelopeorcluster-client-envelope, so the family is four series whatever arrives. The requested path is deliberately not a label, for the same reason the claimed address above is not one — and it is not in the log line either, since it is an unvalidated string from the payload.cluster_outbound_refusals_totalcounts a frame this node would not send because its destination was neither a member nor a node it already holds a connection to — a ref in a message body naming some other host, or, far more often, a node that has just left. A short burst after a node leaves is normal; a steady rate on a stable membership means something keeps handing this node refs to addresses outside the cluster.frameisenvelopeorwire, so the family is two series; the destination is not a label, and the log line that accompanies it at most every 30 seconds names it escaped.
DistributedData metrics
Section titled “DistributedData metrics”The CRDT replicator’s quorum path — updateAsync / getAsync — plus
its wire decoder and its gossip packer. Every label set here is fixed
and tiny, so the five families contribute eight series in total:
| Metric | Type | Labels | Meaning |
|---|---|---|---|
distributed_data_quorum_pending | gauge | — | Quorum reads and writes currently awaiting peer replies on this replica. |
distributed_data_quorum_timeouts_total | counter | operation | Quorum requests that hit their deadline before enough replicas replied. |
distributed_data_quorum_rejected_total | counter | operation | Quorum requests refused because max-pending-quorum-requests was reached. |
distributed_data_dropped_values_total | counter | — | CRDT values this replica dropped: peer-supplied ones it could not decode, and any of another CRDT kind than the value it holds for the key. |
distributed_data_gossip_skipped_keys_total | counter | reason | Local keys a gossip frame could not carry — over the frame budget, or unserialisable. |
operation is write or read — those two values and no others.
reason is oversize or unserialisable, likewise.
distributed_data_quorum_pendingriding near your configuredmax-pending-quorum-requestsmeans the next caller is about to be refused; either peers have stopped acking or the cap is too low for the workload.distributed_data_quorum_rejected_totaland..._timeouts_totalseparate the two failure shapes: refused means the request never started, timed out means it started and nobody answered in time. See Quorum reads and writes.distributed_data_dropped_values_totalis not a tuning signal at all — it counts values this replica could not use: payloads it cannot decode, which means a broken or hostile node, and values of another CRDT kind than the one it holds for the key, which means two replicas disagree on what the key is.distributed_data_gossip_skipped_keys_total{reason="oversize"}above zero means a key is not converging: its own encoding exceedsmax-gossip-bytes(or the wire cap it is clamped to), and one key’s state is the smallest unit gossip can send, so no amount of slicing helps. Raise both knobs or split the value; the accompanying warning names the key and its size.reason="unserialisable"means a value the tagged-JSON codec refuses — a function or aPromiseinside anLWWRegister, which is an application bug rather than a tuning one.
Persistence metrics
Section titled “Persistence metrics”The calls a PersistentActor makes
to its stores — recovery, persist, snapshots and deleteHistory —
go through the circuit breaker in front of each store, and that is
where they are measured:
| Metric | Type | Labels | Meaning |
|---|---|---|---|
persistence_journal_latency_seconds | histogram | operation | Duration of a journal call, in seconds. |
persistence_journal_failures_total | counter | operation, reason | Journal calls that failed. |
persistence_snapshot_latency_seconds | histogram | operation | Duration of a snapshot-store call, in seconds. |
persistence_snapshot_failures_total | counter | operation, reason | Snapshot-store calls that failed. |
persistence_breaker_state | gauge | breaker | State of the breaker in front of a store: 0 closed, 1 half-open, 2 open. |
operation is append, replay or delete for the journal, and
save, load or delete for the snapshot store. replay is the
whole journal half of a recovery rather than one read — the snapshot’s
claimed sequence checked against the journal head, the read, the check
on what came back, and the head probe; the fold that follows runs your
onEvent and is not timed. load is the snapshot read a recovery
starts with. Both histograms use the default seconds-scale buckets.
reason takes five values and no others:
outage— the store threw something that is none of the below. The breaker counts it towardsmax-failureson an append, a replay or a load, and never on a journal delete or a snapshot save or delete.timeout— the breaker’s owncall-timeoutcut the call. The breaker counts it on an append, a replay or a load, and not on a delete or a save.breaker-open— the breaker was open and refused the call, so the store was never asked.ownership— a conditional append lost to a second writer (JournalConcurrencyError).integrity— the store answered with data that failed a trust check (JournalIntegrityError,SnapshotIntegrityError). A snapshot whose claimed sequence the journal contradicts is caught duringreplay, so it is counted on the journal family.
breaker is journal or snapshot — the store a breaker guards,
never its configured id — so the gauge has at most two series, and
none for a store whose breaker id is "".
- The latency histogram is the early signal, and the only one
for slowness. No
call-timeoutships, so a slow call that succeeds is a success to the breaker: a journal degrading from 2 ms to 200 ms movespersistence_journal_latency_secondsand nothing else. Its p99 foroperation="append"is also the number to pick acall-timeoutfrom. - Every call that reached the store is timed, failed ones included —
a store failing slowly is what you want to see. A call the open
breaker refused is counted as
breaker-openand not timed: a stream of instant refusals would drag every quantile towards zero exactly while the store is down. A call that never settles is not timed either, until acall-timeoutturns it into atimeout. ownershipandintegrityare the verdicts the breaker deliberately does not count, and they page different people: a lost race means two live instances of one entity, a failed trust check is a data incident. Neither opens the breaker, so neither shows up anywhere else; both are worth an alert on any movement.- An error the breaker’s
ignored-error-namesexcuses is still counted, under the reason its class gives. The list decides what trips the breaker; the counter reports what the store did. persistence_breaker_state{breaker="journal"} == 2is the outage alert: every append on the node is being refused. A breaker publishes its state when it is first used and on every transition after that; half-open is entered by the first call after the reset window.
None of these is labelled by persistence id or by store:
operation, reason and breaker are closed sets, so the series
count is fixed however many entities you run. Not every store call
goes through these breakers — see
When the stores are down
for the ones that do not.
Projection metrics
Section titled “Projection metrics”A projection is a background loop nobody watches, which is exactly the kind of thing that fails quietly. These three are what tell you a read model has stopped tracking its journal:
| Metric | Type | Labels | Meaning |
|---|---|---|---|
persistence_projection_stalled | gauge | projection | 1 while the projection is blocked on an event whose handler failed. |
persistence_projection_failures_total | counter | projection, reason | Projection ticks that failed. |
persistence_projection_events_skipped_total | counter | projection | Events a skip strategy stepped past and published as dead letters. |
projection is the name you gave it; reason is handler (your
handle or handleGroup callback threw) or poll (the query layer or
the offset store did) — those two values and no others.
persistence_projection_stalledis the alert. It is published as 0 when a projection starts, so the series exists for every running projection and== 1is a working alert expression. A stall clears on its own if the handler recovers or the strategy skips past the event; one that stays at 1 has either exhausted its retries and stopped, or is configured to retry forever.- Separating
reason="handler"fromreason="poll"separates the two fixes: a handler failure is a bad event or a broken read model, a poll failure is the journal underneath. Only handler failures spend themaxRetriesbudget. persistence_projection_events_skipped_totalis the data-loss signal. Any movement means the read model is missing something askipstrategy chose not to block on — cross-reference the dead letters for which events.
projection is bounded by how many projections you declare, which is
normally a handful. The one shape that breaks that is the per-pid
fan-out,
where the name is derived from the entity id — there the label
inherits the entity count and the same
cardinality cap
applies as to actor_mailbox_size{path}. That is a cardinality you
opted into by declaring a projection per entity, which is why the label
stays; see the stock-label rule at the end of this page.
See Projections for the strategies these describe.
Worker-mesh metrics
Section titled “Worker-mesh metrics”A worker mesh relays every worker’s registry into the main thread’s exposition, and one family belongs to the relay itself:
| Metric | Type | Labels | Meaning |
|---|---|---|---|
worker_mesh_snapshot_age_seconds | gauge | worker | Seconds since the main thread last received a metrics snapshot from this worker. |
worker is the slot — worker-0, worker-1, … — the same value the
relayed samples carry in their thread label, so the two join by eye.
It is bounded by worker-mesh.workers, minted by the main thread from
its own slot table and never off the wire, and the series is removed
when the slot leaves the mesh.
- Read at scrape time, not at the relay tick, so the value is the age at
the moment you asked. Under the default
10sinterval a healthy worker reads under ten; a worker that is alive but has stopped answering the relay keeps its last snapshot exported and this gauge climbing, which is the alert —worker_mesh_snapshot_age_seconds > 30is a working expression. A snapshot the relay refused (a name outside the exposition grammar, too many samples) leaves the gauge climbing too, with awarnline naming the reason — provided an earlier snapshot from that worker was accepted. The series is minted by the first accepted snapshot, so a worker refused from its very first answer has no age series at all, and thewarnline is the only signal. - The relayed families themselves are the stock families on this page,
minted on the worker and exported here with
thread="worker-N"added at export. Every row above applies to them unchanged, one series per thread; see the scope of a scrape for thethreadlabel and how to aggregate across it.
Overhead
Section titled “Overhead”There is no opt-out flag — stock metrics are always wired into
the framework. Until you call .enable(), every instrumentation
call resolves against a noop registry (a single object lookup
that records nothing), so the cost is negligible. Enabling the
extension swaps in the live registry and the same calls start
capturing.
Where to next
Section titled “Where to next”- Observability overview — the bigger picture.
- Core metrics — for your own custom metrics.
- Prometheus exporter — how to scrape these.
- Management overview —
for the
/metricsendpoint.
