Circuit breaker
이 콘텐츠는 아직 번역되지 않았습니다.
A circuit breaker wraps calls that might fail (a flaky HTTP endpoint, a slow DB query, an ask to a struggling actor) and short-circuits when failure is sustained. Three states form a simple state machine:
- Closed (start): calls pass through; failures are counted.
- Open: calls fail immediately with
CircuitBreakerOpenError; no traffic reaches the downstream. - Half-open: the first call after the reset timeout is allowed through. Success → closed. Failure → open again.
A minimal example
Section titled “A minimal example”import { CircuitBreaker, CircuitBreakerOpenError } from 'actor-ts';
const breaker = new CircuitBreaker({ maxFailures: 5, // open after 5 consecutive failures resetTimeoutMs: 30_000, // try a probe after 30s callTimeoutMs: 2_000, // any call > 2s counts as a failure});
try { const data = await breaker.call(() => fetch('https://flaky.example/items')); // closed → call passed through} catch (e) { if (e instanceof CircuitBreakerOpenError) { // breaker is open — don't even try the upstream } else { // either the call itself failed, or it timed out }}The breaker doesn’t care what the call is — anywhere you have a
function returning a Promise<T>, you can wrap it. For
actor-to-actor calls, wrap an ask:
const breaker = new CircuitBreaker({ maxFailures: 3, resetTimeoutMs: 10_000 });
async function askWithBreaker(): Promise<Reply> { return breaker.call(() => target.ask({ kind: 'q' }, 5_000));}The state machine in detail
Section titled “The state machine in detail”Closed → open
Section titled “Closed → open”Each call’s outcome updates a counter:
- Success resets the failure counter to 0.
- Failure increments it. When the counter reaches
maxFailuresconsecutively, the breaker transitions to open and records when it next allows a probe (now + resetTimeoutMs).
What counts as a failure:
- The promise rejects.
- The call exceeds
callTimeoutMs(if configured) — the breaker rejects withCircuitBreakerTimeoutErrorand counts it as a failure. - Optional: if
isFailure(err)returnsfalse, the error bypasses counting (the promise still rejects to the caller, but the breaker doesn’t increment). Use this for “this isn’t really a service failure” errors — 404s, validation failures, etc.
new CircuitBreaker({ maxFailures: 5, resetTimeoutMs: 30_000, isFailure: (err) => !(err instanceof ValidationError),});The options are validated at construction: a missing or non-positive
maxFailures, a negative resetTimeoutMs, or a non-positive
callTimeoutMs throws OptionsError (omit callTimeoutMs entirely to
disable the per-call timeout).
Ignoring errors by name
Section titled “Ignoring errors by name”isFailure is a predicate, so it lives in code. ignoredErrorNames
is the same idea expressed as data — a list of Error.name values
that never count — which is what lets an operator change it in
application.conf without a deploy:
new CircuitBreaker({ maxFailures: 5, resetTimeoutMs: 30_000, ignoredErrorNames: ['ValidationError', 'NotFoundError'],});The list is consulted before isFailure, and a listed name is
never a failure whatever the predicate says. That order is deliberate:
the list is the operator’s half of the classifier and the predicate is
the developer’s, and a compiled predicate that could overrule the list
would make the config key go inert for exactly the deployments that
set it. For an unlisted name, isFailure still decides.
'CircuitBreakerTimeoutError' is listable, and it means what it says:
a call that blew its own callTimeoutMs does not count against the
breaker. Coherent — “slow is not down” — and surprising enough to be
worth writing on the deployment that chooses it.
Open → half-open
Section titled “Open → half-open”Once open, every call rejects with CircuitBreakerOpenError
immediately. The breaker stays open until its clock — the wall clock,
or the scheduler it was given — reaches nextProbeAt. At that point,
the next call transitions to half-open and is allowed through.
This isn’t a scheduled wake-up — the breaker checks lazily on the next
call().
The refusal says how long is left: error.remainingMs is the wait, from
the moment the call was refused, until a call would be let through. A
caller that wants to come back reads that rather than nextProbeAt,
since only the breaker knows which clock the timestamp is on.
Half-open → closed (or back to open)
Section titled “Half-open → closed (or back to open)”The probe call either:
- Succeeds → breaker closes. Counter resets. Normal operation.
- Fails → breaker re-opens with a fresh
nextProbeAt.
While in half-open, only the one probe is in flight. Concurrent calls during half-open also go through (the breaker doesn’t serialize), but the breaker’s state transitions are driven by the first one to complete.
The reopen schedule
Section titled “The reopen schedule”By default the window is flat: every open waits resetTimeoutMs.
Two optional knobs change that, and both ship neutral so the default
behaviour is the one every earlier release had.
const breaker = new CircuitBreaker({ maxFailures: 5, resetTimeoutMs: 15_000, // the first window backoffFactor: 2, // each further open doubles it maxResetTimeoutMs: 60_000, // …up to here randomFactor: 0.2, // then spread by ±20%});The delay for the n-th consecutive open is
resetTimeoutMs × backoffFactor^(n − 1), clamped to
maxResetTimeoutMs, then multiplied by 1 ± randomFactor. Above:
15 s, 30 s, 60 s, 60 s…, each ±20 %. The counter resets when a probe
succeeds and the breaker closes, so a dependency that recovers starts
over at the first window rather than staying in a long one it earned
while it was down.
A resetTimeoutMs of 0 — “probe on the very next call” — stays 0
at every open count, whatever the factor: the growth is multiplicative
and there is nothing to multiply. It is a legal pairing and it stays
one; the window is a finite number of milliseconds for every legal
combination of these four settings, however long the outage runs.
Jitter is the same contract retry and exponentialBackoff use, and
it buys the same thing here: N replicas that opened on one upstream
event stop probing it in lockstep, so the recovering dependency meets
a spread of probes instead of a synchronised wave.
maxResetTimeoutMs bounds the whole window, not only the growth, so
a resetTimeoutMs above the ceiling is refused at construction rather
than quietly shortened — clamping silently would replace a window an
operator chose with one nobody wrote down. It defaults to 60 s, which
is what a resetTimeoutMs longer than that has to raise.
breaker.nextProbeAt reads the scheduled timestamp and
breaker.consecutiveOpens the exponent behind it, which is how a
metric sees the schedule without waiting it out.
Breakers from configuration
Section titled “Breakers from configuration”new CircuitBreaker({ … }) is the door for a breaker whose settings
live in code. The CircuitBreakerExtension is the other one: it
resolves a breaker by id, so the numbers come from HOCON and the
instance is owned by the system rather than by whatever created it.
import { CircuitBreakerExtensionId } from 'actor-ts';
const breakers = system.extension(CircuitBreakerExtensionId);const payments = breakers.breaker('payment-api');const shipping = breakers.breaker('shipping-api');actor-ts.circuit-breaker { default { max-failures = 5 reset-timeout = 15s } payment-api { max-failures = 3 reset-timeout = 30s backoff-factor = 2.0 random-factor = 0.2 ignored-error-names = ["ValidationError"] }}payment-api takes its own four leaves and inherits the rest from
default; shipping-api has no block at all and is entirely the
default one. Each id is a separate instance with separate settings,
which is the “one breaker per logical dependency” advice below made
configurable instead of only recommended. The same id always returns
the same instance, and it reads the system’s clock — under a
ManualScheduler its window moves with the advance.
The framework’s own breakers are resolved this way too, so they are
configured in the same block: the persistence stores name theirs with
actor-ts.persistence.journal-breaker / snapshot-breaker, and a broker
actor names one with withCircuitBreaker('kafka') or
circuit-breaker = "kafka" under its root — off until it does. Every
actor that names the same id shares the instance: one breaker per broker
cluster, opened by any of them, with at most one attempt per window from each
— and a single probe for all of them only while a failing connect is short
against the spread of their wake-ups
(see BrokerActor base).
Options a config file cannot express are passed as the second argument, above both blocks:
const payments = breakers.breaker('payment-api', { isFailure: (err) => err.name !== 'ValidationError',});Read them once, when the instance is created — a later
breaker('payment-api') returns the breaker that already exists
rather than reconfiguring it. A bad value in either block throws
OptionsError at that first call, so a typo’d override fails loudly
instead of leaving one breaker quietly running the framework defaults.
A classifier that belongs to the dependency, not to one caller
Section titled “A classifier that belongs to the dependency, not to one caller”That “read once” rule has a sharp edge as soon as more than one place
resolves the same id: whoever gets there first decides, and every later
caller’s isFailure is silently dropped. Passing the predicate to
call instead sidesteps it — it classifies that call’s errors and
takes precedence over whatever the instance was built with:
const journal = breakers.breaker('persistence-journal');
await journal.call( () => store.append(id, entries, expectedSeq), (err) => !(err instanceof ConcurrencyError),);Use the construction option when the breaker’s settings are yours
alone, and the per-call argument when the id is shared or published —
which is what the framework does for
actor-ts.persistence.journal-breaker, whose id is a documented config
path anyone may resolve.
The persistence breakers also choose the classifier by call, because
not every call through them is one they guard. An append, a recovery
and a snapshot load are classified as above; a journal delete and a
snapshot save or delete still run through the breaker — so
call-timeout bounds them and an open breaker refuses them — with a
classifier that counts nothing. A store that refuses deletes or
snapshot writes but serves everything else therefore cannot open the
breaker in front of every append and recovery on the node; see When
the stores are down.
ignoredErrorNames still comes first either way, and an excused error
is still neither a failure nor a success: the consecutive-failure count
survives it untouched, so one excused error between every two real ones
cannot keep a dead dependency from opening the breaker.
The full key list is in the configuration reference.
Observability
Section titled “Observability”import { CircuitBreaker, type CircuitState } from 'actor-ts';import { MetricsExtensionId } from 'actor-ts/metrics';
const breaker = new CircuitBreaker({ maxFailures: 5, resetTimeoutMs: 30_000 });
// A gauge holds a number, so the state is encoded — ordered by how much// traffic is refused, the encoding persistence_breaker_state uses too.const stateValue: Record<CircuitState, number> = { closed: 0, 'half-open': 1, open: 2 };const metrics = system.extension(MetricsExtensionId).enable();const stateGauge = metrics.gauge( 'payments_breaker_state', {}, { help: 'Payments API circuit breaker: 0 closed, 1 half-open, 2 open.' },);stateGauge.set(stateValue[breaker.state]);
const unsubscribe = breaker.onStateChange((state) => { stateGauge.set(stateValue[state]); log.info(`circuit breaker → ${state}`);});
// Later: `unsubscribe()` to remove.Use this to wire state transitions into your logging or metrics
pipeline. The listener fires on every transition, including
forced ones via breaker.setState(...).
metrics.gauge(name, labels, options) returns the gauge; it is
not a setter, so the value goes through .set(). The name must be
a Prometheus metric name — underscores, not dots — and a gauge holds
a number, which is why the state is mapped rather than passed.
Setting it once before the first transition makes a closed breaker
read 0 instead of being absent from the scrape.
breaker.state reads the current state synchronously.
The two breakers in front of the persistence stores need none of
this: with metrics enabled they are exported as
persistence_breaker_state{breaker="journal"|"snapshot"} — see
Persistence metrics.
A broker actor’s breaker needs none of it either: each actor naming one
publishes its transitions on the event stream as
BrokerCircuitBreakerStateChanged.
Manual overrides
Section titled “Manual overrides”breaker.setState('open'); // force open — useful for admin "drain this dep"breaker.setState('closed'); // force close — manual recoverysetState is mostly for tests and admin endpoints. Production
code should rely on the natural transition path; reaching for
setState from regular code usually means the breaker isn’t
configured right.
Picking the numbers
Section titled “Picking the numbers”Three parameters decide when the breaker trips; here’s how to think about them:
maxFailures— high enough that a single transient blip doesn’t open the breaker, low enough that a real outage opens it before too much traffic piles up. 3-10 is typical. Lower for critical paths (open fast); higher for paths where false trips are expensive.resetTimeoutMs— long enough for the downstream to recover but short enough that you notice when it has. 10-60 seconds for typical HTTP / DB calls; sub-second only if the breaker fronts a truly local resource.callTimeoutMs— the longest the individual call should take. If the upstream’s p99 is 800 ms, set this to 2-3 s. Setting it lower than the upstream’s normal latency means you’re declaring everything a timeout.
Three more shape the reopen schedule, and all three are off by default:
backoffFactor— leave it at1for a dependency whose outages are short and self-clearing; reach for2when a repeatedly-failing upstream is one you should stop asking so often. Growth only compounds while the breaker never closes, so it costs nothing in the common case.maxResetTimeoutMs— how long you are willing to not notice a recovery. It has to be at leastresetTimeoutMs, and the default 60 s is the ceiling the rest of the project converged on for a backoff someone is waiting behind.randomFactor—0.2if more than one process runs this breaker against the same dependency,0if exactly one does. Jitter is about de-synchronising a fleet, and a single instance has nothing to de-synchronise from.
Compared to retry
Section titled “Compared to retry”A breaker and a retry helper are complementary, not substitutes:
- Retry says “try again after a delay if this call fails.” It treats each call independently and burns its budget on the current operation.
- Breaker says “after enough recent failures, stop trying for a while.” It carries state across calls.
The right combination is retry inside a breaker call:
breaker.call(() => retry( () => fetch('https://flaky.example'), { attempts: 3, delayMs: 100, factor: 2 },));The retry handles the per-call resilience; the breaker handles the
“stop hammering the broken dep” coordination. Don’t put the retry
outside the breaker — you’d be retrying through the
CircuitBreakerOpenErrors, which is pointless.
Where to next
Section titled “Where to next”- Retry — the complementary per-call retry helper.
- Backoff supervisor — exponential-backoff restarts for an actor child.
- Ask pattern — what you’d typically wrap when the call is actor-to-actor.
The CircuitBreaker API
reference covers the full surface.
