Ir al contenido
Español

Circuit breaker

Esta página aún no está disponible en tu idioma.

A circuit breaker wraps calls that might fail (a flaky HTTP endpoint, a slow DB query, an ask to a struggling actor) and short-circuits when failure is sustained. Three states form a simple state machine:

failures ≥ maxFailures

reset timeout elapsed

probe succeeds

probe fails

Closed\ncalls pass through

Open\ncalls fail fast

Half-open\nprobe call

  • Closed (start): calls pass through; failures are counted.
  • Open: calls fail immediately with CircuitBreakerOpenError; no traffic reaches the downstream.
  • Half-open: the first call after the reset timeout is allowed through. Success → closed. Failure → open again.
import { CircuitBreaker, CircuitBreakerOpenError } from 'actor-ts';
const breaker = new CircuitBreaker({
maxFailures: 5, // open after 5 consecutive failures
resetTimeoutMs: 30_000, // try a probe after 30s
callTimeoutMs: 2_000, // any call > 2s counts as a failure
});
try {
const data = await breaker.call(() => fetch('https://flaky.example/items'));
// closed → call passed through
} catch (e) {
if (e instanceof CircuitBreakerOpenError) {
// breaker is open — don't even try the upstream
} else {
// either the call itself failed, or it timed out
}
}

The breaker doesn’t care what the call is — anywhere you have a function returning a Promise<T>, you can wrap it. For actor-to-actor calls, wrap an ask:

const breaker = new CircuitBreaker({ maxFailures: 3, resetTimeoutMs: 10_000 });
async function askWithBreaker(): Promise<Reply> {
return breaker.call(() => target.ask({ kind: 'q' }, 5_000));
}

Each call’s outcome updates a counter:

  • Success resets the failure counter to 0.
  • Failure increments it. When the counter reaches maxFailures consecutively, the breaker transitions to open and records when it next allows a probe (now + resetTimeoutMs).

What counts as a failure:

  • The promise rejects.
  • The call exceeds callTimeoutMs (if configured) — the breaker rejects with CircuitBreakerTimeoutError and counts it as a failure.
  • Optional: if isFailure(err) returns false, the error bypasses counting (the promise still rejects to the caller, but the breaker doesn’t increment). Use this for “this isn’t really a service failure” errors — 404s, validation failures, etc.
new CircuitBreaker({
maxFailures: 5,
resetTimeoutMs: 30_000,
isFailure: (err) => !(err instanceof ValidationError),
});

The options are validated at construction: a missing or non-positive maxFailures, a negative resetTimeoutMs, or a non-positive callTimeoutMs throws OptionsError (omit callTimeoutMs entirely to disable the per-call timeout).

isFailure is a predicate, so it lives in code. ignoredErrorNames is the same idea expressed as data — a list of Error.name values that never count — which is what lets an operator change it in application.conf without a deploy:

new CircuitBreaker({
maxFailures: 5,
resetTimeoutMs: 30_000,
ignoredErrorNames: ['ValidationError', 'NotFoundError'],
});

The list is consulted before isFailure, and a listed name is never a failure whatever the predicate says. That order is deliberate: the list is the operator’s half of the classifier and the predicate is the developer’s, and a compiled predicate that could overrule the list would make the config key go inert for exactly the deployments that set it. For an unlisted name, isFailure still decides.

'CircuitBreakerTimeoutError' is listable, and it means what it says: a call that blew its own callTimeoutMs does not count against the breaker. Coherent — “slow is not down” — and surprising enough to be worth writing on the deployment that chooses it.

Once open, every call rejects with CircuitBreakerOpenError immediately. The breaker stays open until its clock — the wall clock, or the scheduler it was given — reaches nextProbeAt. At that point, the next call transitions to half-open and is allowed through. This isn’t a scheduled wake-up — the breaker checks lazily on the next call().

The refusal says how long is left: error.remainingMs is the wait, from the moment the call was refused, until a call would be let through. A caller that wants to come back reads that rather than nextProbeAt, since only the breaker knows which clock the timestamp is on.

The probe call either:

  • Succeeds → breaker closes. Counter resets. Normal operation.
  • Fails → breaker re-opens with a fresh nextProbeAt.

While in half-open, only the one probe is in flight. Concurrent calls during half-open also go through (the breaker doesn’t serialize), but the breaker’s state transitions are driven by the first one to complete.

By default the window is flat: every open waits resetTimeoutMs. Two optional knobs change that, and both ship neutral so the default behaviour is the one every earlier release had.

const breaker = new CircuitBreaker({
maxFailures: 5,
resetTimeoutMs: 15_000, // the first window
backoffFactor: 2, // each further open doubles it
maxResetTimeoutMs: 60_000, // …up to here
randomFactor: 0.2, // then spread by ±20%
});

The delay for the n-th consecutive open is resetTimeoutMs × backoffFactor^(n − 1), clamped to maxResetTimeoutMs, then multiplied by 1 ± randomFactor. Above: 15 s, 30 s, 60 s, 60 s…, each ±20 %. The counter resets when a probe succeeds and the breaker closes, so a dependency that recovers starts over at the first window rather than staying in a long one it earned while it was down.

A resetTimeoutMs of 0 — “probe on the very next call” — stays 0 at every open count, whatever the factor: the growth is multiplicative and there is nothing to multiply. It is a legal pairing and it stays one; the window is a finite number of milliseconds for every legal combination of these four settings, however long the outage runs.

Jitter is the same contract retry and exponentialBackoff use, and it buys the same thing here: N replicas that opened on one upstream event stop probing it in lockstep, so the recovering dependency meets a spread of probes instead of a synchronised wave.

maxResetTimeoutMs bounds the whole window, not only the growth, so a resetTimeoutMs above the ceiling is refused at construction rather than quietly shortened — clamping silently would replace a window an operator chose with one nobody wrote down. It defaults to 60 s, which is what a resetTimeoutMs longer than that has to raise.

breaker.nextProbeAt reads the scheduled timestamp and breaker.consecutiveOpens the exponent behind it, which is how a metric sees the schedule without waiting it out.

new CircuitBreaker({ … }) is the door for a breaker whose settings live in code. The CircuitBreakerExtension is the other one: it resolves a breaker by id, so the numbers come from HOCON and the instance is owned by the system rather than by whatever created it.

import { CircuitBreakerExtensionId } from 'actor-ts';
const breakers = system.extension(CircuitBreakerExtensionId);
const payments = breakers.breaker('payment-api');
const shipping = breakers.breaker('shipping-api');
actor-ts.circuit-breaker {
default {
max-failures = 5
reset-timeout = 15s
}
payment-api {
max-failures = 3
reset-timeout = 30s
backoff-factor = 2.0
random-factor = 0.2
ignored-error-names = ["ValidationError"]
}
}

payment-api takes its own four leaves and inherits the rest from default; shipping-api has no block at all and is entirely the default one. Each id is a separate instance with separate settings, which is the “one breaker per logical dependency” advice below made configurable instead of only recommended. The same id always returns the same instance, and it reads the system’s clock — under a ManualScheduler its window moves with the advance.

The framework’s own breakers are resolved this way too, so they are configured in the same block: the persistence stores name theirs with actor-ts.persistence.journal-breaker / snapshot-breaker, and a broker actor names one with withCircuitBreaker('kafka') or circuit-breaker = "kafka" under its root — off until it does. Every actor that names the same id shares the instance: one breaker per broker cluster, opened by any of them, with at most one attempt per window from each — and a single probe for all of them only while a failing connect is short against the spread of their wake-ups (see BrokerActor base).

Options a config file cannot express are passed as the second argument, above both blocks:

const payments = breakers.breaker('payment-api', {
isFailure: (err) => err.name !== 'ValidationError',
});

Read them once, when the instance is created — a later breaker('payment-api') returns the breaker that already exists rather than reconfiguring it. A bad value in either block throws OptionsError at that first call, so a typo’d override fails loudly instead of leaving one breaker quietly running the framework defaults.

A classifier that belongs to the dependency, not to one caller

Section titled “A classifier that belongs to the dependency, not to one caller”

That “read once” rule has a sharp edge as soon as more than one place resolves the same id: whoever gets there first decides, and every later caller’s isFailure is silently dropped. Passing the predicate to call instead sidesteps it — it classifies that call’s errors and takes precedence over whatever the instance was built with:

const journal = breakers.breaker('persistence-journal');
await journal.call(
() => store.append(id, entries, expectedSeq),
(err) => !(err instanceof ConcurrencyError),
);

Use the construction option when the breaker’s settings are yours alone, and the per-call argument when the id is shared or published — which is what the framework does for actor-ts.persistence.journal-breaker, whose id is a documented config path anyone may resolve.

The persistence breakers also choose the classifier by call, because not every call through them is one they guard. An append, a recovery and a snapshot load are classified as above; a journal delete and a snapshot save or delete still run through the breaker — so call-timeout bounds them and an open breaker refuses them — with a classifier that counts nothing. A store that refuses deletes or snapshot writes but serves everything else therefore cannot open the breaker in front of every append and recovery on the node; see When the stores are down.

ignoredErrorNames still comes first either way, and an excused error is still neither a failure nor a success: the consecutive-failure count survives it untouched, so one excused error between every two real ones cannot keep a dead dependency from opening the breaker.

The full key list is in the configuration reference.

import { CircuitBreaker, type CircuitState } from 'actor-ts';
import { MetricsExtensionId } from 'actor-ts/metrics';
const breaker = new CircuitBreaker({ maxFailures: 5, resetTimeoutMs: 30_000 });
// A gauge holds a number, so the state is encoded — ordered by how much
// traffic is refused, the encoding persistence_breaker_state uses too.
const stateValue: Record<CircuitState, number> = { closed: 0, 'half-open': 1, open: 2 };
const metrics = system.extension(MetricsExtensionId).enable();
const stateGauge = metrics.gauge(
'payments_breaker_state', {},
{ help: 'Payments API circuit breaker: 0 closed, 1 half-open, 2 open.' },
);
stateGauge.set(stateValue[breaker.state]);
const unsubscribe = breaker.onStateChange((state) => {
stateGauge.set(stateValue[state]);
log.info(`circuit breaker → ${state}`);
});
// Later: `unsubscribe()` to remove.

Use this to wire state transitions into your logging or metrics pipeline. The listener fires on every transition, including forced ones via breaker.setState(...).

metrics.gauge(name, labels, options) returns the gauge; it is not a setter, so the value goes through .set(). The name must be a Prometheus metric name — underscores, not dots — and a gauge holds a number, which is why the state is mapped rather than passed. Setting it once before the first transition makes a closed breaker read 0 instead of being absent from the scrape.

breaker.state reads the current state synchronously.

The two breakers in front of the persistence stores need none of this: with metrics enabled they are exported as persistence_breaker_state{breaker="journal"|"snapshot"} — see Persistence metrics. A broker actor’s breaker needs none of it either: each actor naming one publishes its transitions on the event stream as BrokerCircuitBreakerStateChanged.

breaker.setState('open'); // force open — useful for admin "drain this dep"
breaker.setState('closed'); // force close — manual recovery

setState is mostly for tests and admin endpoints. Production code should rely on the natural transition path; reaching for setState from regular code usually means the breaker isn’t configured right.

Three parameters decide when the breaker trips; here’s how to think about them:

  • maxFailures — high enough that a single transient blip doesn’t open the breaker, low enough that a real outage opens it before too much traffic piles up. 3-10 is typical. Lower for critical paths (open fast); higher for paths where false trips are expensive.
  • resetTimeoutMs — long enough for the downstream to recover but short enough that you notice when it has. 10-60 seconds for typical HTTP / DB calls; sub-second only if the breaker fronts a truly local resource.
  • callTimeoutMs — the longest the individual call should take. If the upstream’s p99 is 800 ms, set this to 2-3 s. Setting it lower than the upstream’s normal latency means you’re declaring everything a timeout.

Three more shape the reopen schedule, and all three are off by default:

  • backoffFactor — leave it at 1 for a dependency whose outages are short and self-clearing; reach for 2 when a repeatedly-failing upstream is one you should stop asking so often. Growth only compounds while the breaker never closes, so it costs nothing in the common case.
  • maxResetTimeoutMs — how long you are willing to not notice a recovery. It has to be at least resetTimeoutMs, and the default 60 s is the ceiling the rest of the project converged on for a backoff someone is waiting behind.
  • randomFactor — 0.2 if more than one process runs this breaker against the same dependency, 0 if exactly one does. Jitter is about de-synchronising a fleet, and a single instance has nothing to de-synchronise from.

A breaker and a retry helper are complementary, not substitutes:

  • Retry says “try again after a delay if this call fails.” It treats each call independently and burns its budget on the current operation.
  • Breaker says “after enough recent failures, stop trying for a while.” It carries state across calls.

The right combination is retry inside a breaker call:

breaker.call(() => retry(
() => fetch('https://flaky.example'),
{ attempts: 3, delayMs: 100, factor: 2 },
));

The retry handles the per-call resilience; the breaker handles the “stop hammering the broken dep” coordination. Don’t put the retry outside the breaker — you’d be retrying through the CircuitBreakerOpenErrors, which is pointless.

  • Retry — the complementary per-call retry helper.
  • Backoff supervisor — exponential-backoff restarts for an actor child.
  • Ask pattern — what you’d typically wrap when the call is actor-to-actor.

The CircuitBreaker API reference covers the full surface.