콘텐츠로 이동
한국어

Diagnosing test flakes

이 콘텐츠는 아직 번역되지 않았습니다.

Testing overview tells you how to write a test that does not flake. This page is the other half: you have a test that already does, and you need to know which kind it is.

A test that fails once and passes on a re-run is not a category — it is an observation with a sample size of one. Run it again, deliberately:

Terminal window
# 20 repeats of one file, naming anything that failed at least once
bun run test:stress -- --runs=20 tests/unit/http/websocket/WebsocketServerActor.test.ts

bun run test:stress (scripts/stress-test.mjs) runs the suite N times, keeps every run’s JUnit report and log, and aggregates failures by test identity. Its output splits the two cases that matter:

VerdictMeaningWhat to do
flaky — failed in some runsThe outcome depends on timing, ordering or the machine.Match it against the catalog below.
consistently failing — failed in every runBroken. Repetition tells you nothing more.Fix it, or open an issue. It is not a flake.

A test that fails more than once inside a single run used to fall out of both categories, and the harness then printed “No test failed in any run that reported” and exited 0 — over sixteen runs of which none was green, two lines under a 100 % failure rate (#1359). failedRuns counts runs now, once per run however many testcases carried the identity, and the occurrence count is reported beside it (4 failing testcases in total) rather than folded into the classification. A hook timeout is still the easy way to produce the shape: a describe whose beforeAll and afterAll both blow their budget emits two (unnamed) cases under one classname.

The verdict now also fails on runs, not only on tests. A run that was not green and that no tolerated offender explains is red however the tables read — which is the field the quarantine’s exit criterion is written in (greenRuns == runs), and the one the old verdict computed, printed, and then did not act on.

Other things it names rather than swallows: a run that never exited (see below), a run that produced no JUnit report at all (Bun died before the reporter flushed, so those failures are missing from the totals rather than absent), and a run that exited non-zero with no failing test (a crash in teardown, an unreleased handle). All three read as “green” to a naive loop.

Terminal window
bun run test:stress # 10 repeats of the whole suite
bun run test:stress -- --runs=3 # 3 repeats
bun run test:stress -- --concurrency=4 # 4 at once, for real CPU contention
bun run test:stress -- --run-timeout=300000 # call a run hung after 5 minutes

A hang is data, not an abort. The failure the quarantined suites show on hosted runners is not a red test — workers spawn, handshake and then never run, so bun test does not exit at all. --run-timeout (20 minutes by default) kills such a run, records it as HUNG — kept apart from a failure and from a truncated report, because each points at a different cause — and carries on with the next repeat. Without it the loop stalled on the first hang and the job was killed by its own timeout-minutes with nothing to show: the one measurement the harness exists to make was the one it could not survive. Hung runs appear in summary.json as runsTimedOut, and a night with any of them is red.

Reports, logs and a machine-readable summary.json land in .stress/.

Everything above is a computed verdict, and a classifier that is wrong looks exactly like a measurement that is surprising. So the harness is checked against inputs whose answer is known before the run starts:

  • tests/unit/ci/StressHarnessAggregation.test.ts drives the JUnit parser and the aggregation over fixtures — a pass, a <failure>, an <error>, a <skipped/>, an escaped test name, and the same test spelled with a POSIX path, a Windows path and an absolute path, which must collapse to one identity or “failed 3 of 5 runs” becomes three unrelated reds.
  • tests/unit/ci/StressHarnessClassification.test.ts runs the real script over a synthetic suite whose behaviour is decided by a counter file: one test fails in run 3 of 5, one fails in all five, one never fails, one is skipped. The expected verdict is a fact rather than a judgement, which is the only way to tell “reported as flaky” from “misclassified”.
  • tests/unit/ci/StressHarnessWatchdog.test.ts drives it against a suite that genuinely never finishes.

The distinction those three protect is the one this page is built on. A skip counted as a pass, or a hang counted as green, produces a clean-looking number for a night in which nothing was measured — and that number would then be quoted here.

A loop drives up the probability of a load-sensitive flake: a fixed sleep that is long enough on an idle machine and short under contention. That is the dominant family here, and the loop finds it.

It says nothing about a deterministic ordering bug. The clearest example in this repository is a test that awaited one actor’s recorded state and then asserted on another actor’s — two states separated by a dispatcher hop. A throwaway probe ran the broken condition 200 times: 0 failures at a 5 ms poll interval, and still 0 at 1 ms. The probe ran while the default dispatcher scheduled every turn with setImmediate; today’s hybrid default hands most turns over as microtasks, sooner still. Either way the second actor’s turn is queued ahead of the poller’s next timer and wins every time work runs promptly. The race is real by construction and not locally reproducible.

So a green stress run is evidence about one family, not a clean bill of health. The second family is found by reading:

Between the state you wait on and the state you assert on, is there a message send?

If yes, the wait is a proxy that a half-finished step already satisfies — wait on the asserted state instead.

Three places, each asking a different question.

  • Every commit, test.yml and multi-runtime.yml run the whole suite once, shuffled — “does anything depend on running after anything else”.
  • Every pull request, a probe re-runs the branch’s changed test files five times — “does a test this branch adds depend on the run it was written in”. That is the moment a new flake is cheapest to fix, and the only moment it has an obvious owner.
  • Every night, nightly-flakes.yml loops the whole suite and aggregates by test identity — the load-sensitive family above, over a population no branch touches. A night that is not green opens an issue, and a green one closes it again.

That last clause is the part that took longest to get right, and it is worth saying why. The nightly measured this suite for months and the measurement reached nobody: both jobs were continue-on-error, and the verdict went to a step summary and a run annotation. The workflow predicted its own failure in a comment — “Nothing accumulates the streak. It is counted by a human reading these annotations, which is the same failure that made the quarantine permanent in the first place.” Nobody counted. The fourteen-night exit criterion stood at twenty-one before anyone looked.

An issue has an owner, a history and a notification, which an annotation never had. And with the finding actionable, a red job is a fact rather than noise, so the suppression came off: nothing is gated on a scheduled workflow, and red there blocks no pull request.

The probe follows the changed test files rather than everything a change reaches, and the numbers are the reason. bun test --changed follows imports transitively: measured on this tree, one edit to a persistence module selected 346 files and 5 505 tests in 209 s, and a base whose diff reached a tests/util/ helper selected 468 files and 8 777 tests in 252 s. Five reruns of either is a half-hour job, and what it would add over the nightly loop is the same question asked on a smaller population. Repetition pays off where the code is new, so the probe follows the new code — and a change under src/ is covered once by the shuffled suite, exactly as before.

Generative tests run against a pinned seed

Section titled “Generative tests run against a pinned seed”

A property test with a random seed can fail on one run and pass on the next. That is a flake by definition, and the worst kind available: the counterexample exists only in the run log, so when that rotates the failing case is gone and nobody can reproduce what happened. Before #1372 this repository had ~52 fc.assert call sites, no seed, no endOnFailure, and no regression corpus.

tests/setup/property-seed.ts pins the seed for the whole process through fc.configureGlobal, loaded by bunfig.toml’s preload — one place rather than fifty-two, so a new property file gets the policy without its author having to know it exists.

The trade is real and worth stating: a fixed seed explores the same cases every run, so the suite stops finding new counterexamples on its own. That is acceptable because it was never really doing that. A case found by an unlucky nightly and then lost to log rotation was never turned into a regression test, so the exploration produced red builds rather than knowledge. New cases now come from raising PROPERTY_RUNS or changing PROPERTY_SEED — a decision somebody makes, in a diff, and expects to be the commit that finds something.

When a property does fail, copy the shrunk counterexample into a plain example test beside it before fixing anything. The property proves the law; the example is what stops that specific case coming back, and unlike a log line it survives.

tests/unit/ci/PropertySeedPolicy.test.ts keeps the two halves wired together, and its last assertion is the one that matters: it asks fast-check what seed it is actually running with, so a preload that was registered and never loaded fails rather than quietly restoring the random default.

Causes this suite has actually had, with how each was settled.

FamilyStatusSignature
Fixed sleep before an assertionConverting, ratchetedawait sleep(N) then expect(...) on state a background step produces
A budget the per-test timeout cannot reachGatedthis test timed out after 5000ms where an awaitCondition label was expected
Real work in a hook, against a cap nobody setSettled(unnamed) and “a beforeEach/afterEach hook timed out”, naming no test
Real work in a test body, against a cap nobody setSettledthis test timed out after 5000ms naming a test that does bounded work and passes standalone
Bun’s timer quantumSettledAn elapsed-time assertion fails by ~15 ms, on an idle machine
A dispatcher hop between wait and assertSettledNever reproduces in isolation; fails only in a big parallel run
Port collisionsSettledEADDRINUSE, only when suites overlap
Filesystem racesSettledTwo suites reading each other’s fixtures
An assertion over process-global stateSettledPasses in isolation, fails in a whole-suite run, and no timing is involved
A test that depends on running after another oneSettled, gatedGreen in declaration order, red under --seed=N; a repeat run repeats the same order and never finds it
Hosted-runner worker respawnRetired, guarded nightlyWorkers spawn, handshake, then never run — never once in 63 measured executions

The dominant shape, by a wide margin. N was picked from a run that passed, so it encodes the latency of one machine on one day; under load the step takes longer and the assertion reads a value that was never written.

// ✗ the sleep is a bet on how fast the machine is
await sleep(50);
expect(stopped).toBe(true);
// ✓ returns as soon as it holds; the timeout only bounds the broken case
await awaitCondition(() => stopped, { label: 'the failing child was stopped' });
expect(stopped).toBe(true);

tests/util/AwaitCondition.ts is the shared helper, and Wait on state, not on elapsed time covers when a sleep is still the right call — an absence you can only give a window to, a duration that is the assertion, an interleaving a test needs.

The conversion has largely landed, and the shape of what is left is the useful part. 486 fixed-delay waits remain under tests/ — 440 written await sleep(N) and 46 as an inline Bun.sleep(20) — but 483 of them now state a reason for the delay. Three do not, and all three are in helper files rather than in a test: tests/util/AwaitCondition.ts (the shared sleep itself), and two in the integration-scenario tree.

The two duplications the page used to report have essentially gone. One file re-declares its own sleep — tests/integration/scenarios/Types.ts, which exports it for that tree — against 129 that import the shared one, and 11 modules still hand-roll a while (Date.now() < deadline) loop rather than calling awaitCondition. Treat a sleep-then-expect pair in a failing test as a suspect before looking for anything cleverer, and read its reason comment first: it will usually tell you whether the wait is the bet or the assertion.

Those numbers are not only documentation. tests/unit/ci/SleepRatchet.test.ts re-measures them on every bun test and fails when one goes up, per module: the debt can shrink and cannot grow back. It does not forbid waiting — an absence cannot be polled for, and many of the waits are followed by an assertion that something did not happen — it forbids an unexplained wait, a re-declared sleep and a re-invented poll loop, each with a one-line remedy the failure message spells out.

Measured 2026-09-08 at 5c7064f5, over source with comments and string literals blanked, which is what the gate’s own scanner does. A raw grep -ro '<pattern>' tests/ --include=*.ts | wc -l re-measures them roughly and reads high, because it also counts the prose and the quoted examples — this page included.

The previous figures on this page were measured at 95db877c on 2026-08-18, before the #418 conversion sweep merged, so they described the tree before the work the page is about. Two of them had not merely drifted but inverted: the page told a contributor that 93 files re-declare sleep “rather than importing the shared one (8 import it)”, when today one file declares it and 129 import it (#1313). A measured claim carries the revision it was measured at for exactly this reason.

A budget the per-test timeout cannot reach

Section titled “A budget the per-test timeout cannot reach”

Bun kills a test after 5 000 ms unless the test declares otherwise, and nothing in this repository raises that globally. So an awaitCondition budget at or above 5 s is not generous — it is unreachable, and the run reports this test timed out after 5000ms instead of the label, which is the one thing the helper exists to give you. Worse, the budget’s own rejection still lands later, as an unhandled error attributed to no test at all.

Give the test the room instead:

test('shards rebalance when a node leaves', async () => {
await awaitCondition(/* … */, { timeoutMs: 10_000, label: '…' });
}, 30_000); // the cap is a backstop; the budget is what reports

tests/unit/ci/AwaitConditionBudgets.test.ts gates this across the whole tree — largest reachable budget plus 1 s must fit the cap — and follows budgets reached through a module-level helper too. It re-measures bun’s behaviour in a child process rather than assuming it, so the day bun changes, the gate says so instead of quietly meaning nothing.

Real work in a hook, against a cap nobody set

Section titled “Real work in a hook, against a cap nobody set”

Bun caps a hook at 5 000 ms exactly as it caps a test — and the previous section’s remedy has a counterpart that is easy not to know exists: beforeAll(fn, timeoutMs) takes a second argument, just as test() takes a third.

tests/unit/docs/DocSampleHarnessEndToEnd.test.ts is the case this suite had (#1282). Its beforeAll writes a fixture documentation tree and then runs the doc-sample harness twice — and each of those runs spawns bunx tsc twice, because the fixture deliberately contains an unparseable fence and re-checking the rest without it is the property the file exists to prove. So the hook drives four compilers in series: 3.1 s idle, 4.3 s inside a full bun test, 9.0 s with copies of the file contending. Against a 5 000 ms cap that is not a slow test, it is a coin flip — the file passed every time it was run alone and failed roughly three whole-suite runs in four.

Two things make this family harder to read than the test-timeout one above:

  • The failure names nothing. It reports as (fail) <describe> > (unnamed) [5001.31ms] with “a beforeEach/afterEach hook timed out for this test” — the wrong hook kind, and no test name, because no test is at fault. Grepping the log for the file name finds it; grepping for a test name does not.
  • It under-reports how much did not run. Every test in the block is skipped and one failure is recorded. The same file green executes 13 tests per run; red it executed 1.

The remedy is layered budgets rather than one larger number:

const RUN_BUDGET_MS = 30_000; // what one spawned run may take
const HOOK_BUDGET_MS = 3 * RUN_BUDGET_MS; // a backstop, not the budget
function run(): Run {
const startedAt = performance.now();
const result = spawnSync(command, args, { encoding: 'utf8', timeout: RUN_BUDGET_MS });
if (result.error !== undefined || result.signal !== null) {
const elapsed = Math.round(performance.now() - startedAt);
throw new Error(`${command} did not complete after ${elapsed} ms`);
}
return { status: result.status ?? -1, stdout: result.stdout ?? '', stderr: result.stderr ?? '' };
}
beforeAll(() => {
/* … */
}, HOOK_BUDGET_MS); // reached only if the stall was not in a spawn

The hook’s cap sits clear of the work’s own budget on purpose. A thrown error carries the command, the elapsed time and the budget it blew; bun’s hook timeout carries none of the three. Same principle as the section above — the cap is a backstop; the budget is what reports — applied one level out, to a hook.

Sizing it is a measurement, not a guess. Time the hook under the load that actually breaks it (running several copies of the file at once is the cheapest way to get there), then leave a multiple of the worst figure, and write the figures next to the constant so the next person can tell a budget that was measured from one that was doubled until the red went away.

Real work in a test body, against a cap nobody set

Section titled “Real work in a test body, against a cap nobody set”

The same cap, one level in. tests/unit/CoreStaticImports.test.ts guards the #1005 invariant — import { ActorSystem } must never statically reach Fastify — by walking the static import graph of the sources. That walk ran inside the test body: 148 files, 1.28 MiB of source, 29 ms on an idle machine. A full bun test --coverage run stretched it to 6 723.89 ms (#1392):

(fail) core static import closure (#1005) > ActorSystem never statically
reaches fastify [6723.89ms]
^ this test timed out after 5000ms.

Roughly 230x, against a file that takes 276 ms standalone. Nothing about the invariant was in question — the machine was busy.

Neither remedy above applies, and reaching for one is the trap. A third argument sizes a failure budget: a wait that is meant to expire and print a label. A closure walk has no budget — it is bounded work that always completes, so there is nothing to report, and no honest number to pick against a 230x spread. Layered budgets need a spawn boundary for the inner operation to carry its own timeout, and there is none here.

The remedy is to take the work off the clock entirely. Module scope carries no per-test timeout at all:

// Walked once, here, and not inside the tests that assert on it.
const coreClosure = staticClosure('src/ActorSystem.ts');
describe('core static import closure', () => {
test('ActorSystem never statically reaches fastify', () => {
const reached = [...coreClosure.files].filter((f) => f.endsWith('FastifyBackend.ts'));
expect(reached).toEqual([]); // the body is an assertion, nothing more
});
});

That is measured rather than assumed: a fixture with 6 s of synchronous work at module scope passes, while the identical 6 s inside a test body is killed at 5 000 ms. It is also what this repository’s other repo-file guards already do — AwaitConditionBudgets and WorkflowHygiene scan the tree at module scope and keep their test() bodies to assertions, and NoDeadConfigKeys computes its whole perturbation matrix, every reference.conf leaf against every registered reader, at module scope too. The two CoreStaticImports bodies went from 29.15 ms and 1.24 ms to 0.41 ms and 0.04 ms, which puts the same contention at ~95 ms against the cap.

The trade is that a throw at module scope fails the whole file rather than one test. For a guard that reads the tree and asserts on what it found, that is the right side of the trade, and the three siblings above already take it.

One warning about reproducing this family. The previous section’s advice — run several copies of the file at once — does not transfer here. Measured: the pre-fix file under 12 concurrent CPU hogs still completed in 31.45 ms and passed. External load competes with spawned work; work done inside the bun process is slowed by that process’s own heap, garbage collection and coverage counters, which only a real whole-suite run recreates. Establish the mechanism by measurement instead — the module-scope fixture above settles it in a single run, without ever reproducing the flake.

Bun’s timer quantum — the early half is retired

Section titled “Bun’s timer quantum — the early half is retired”

Windows’ default timer resolution is 15.625 ms, and Bun 1.3’s event loop decided a timer was due on that tick boundary — so a setTimeout whose deadline sat just below a tick multiple fired a full tick early. A setTimeout(30) was measured firing at 18.67 ms on an idle machine, 30 ms being pathological because it sits just under 2 × 15.625.

Bun 1.4 fixed it. Re-measured 2026-09-08, 60 samples at each of 5, 20, 30 and 60 ms: not one fire below nominal, and the 30 ms row came back at 30.04 ms minimum. So minimumElapsedMs() no longer gives a quantum away — its slack is lateness-only, and tests/util/TimerTolerance.ts carries both measurements, because a tolerance that records only what is true today explains nothing about why it exists.

A consequence worth stating, since it removes work rather than adding it: a 30 ms sleep is no longer a value to avoid. It was on the list of things to sweep out of the suite; the sweep is off, because the reason for it is gone and churning 44 sites to change a number that now behaves would buy nothing.

The late half has not gone anywhere, and it is the half that matters. There is still no safe upper bound — the same 30 ms timer was measured at 201 ms under CPU load, lateness is whatever the OS scheduler decides, and a bound loose enough to survive it can no longer tell the delay it is checking apart from a longer one. Assert virtual time with ManualScheduler instead, which now reaches nearly everywhere the framework waits.

Described above. The tell is that it never reproduces in isolation — not at any poll interval, not under CPU load — and fails only inside a large parallel run, where the macrotask queue is congested enough for the ordering to invert. Do not spend an afternoon trying to reproduce it; read the two states and ask whether a message crosses between them.

Both settled, and worth knowing because a new test can reintroduce either.

  • Ports. Bind 0 and read the assigned port back off the listener, then hand that to every client in the test. A hard-coded port is a collision waiting for a second suite.
  • Temp directories. mkdtemp or a randomUUID suffix per test, never a shared fixture path. Roughly two dozen suites do this today; copy one of the object-storage suites if you need a pattern.

The one family with no timing in it at all, and therefore the one a repeat run diagnoses least well: the test is correct about its own object and wrong about its scope.

bun test runs the whole tree in one process, so a static collection is shared by every suite in it. An assertion on the contents of one is an assertion about everything that ran first:

// ✗ InMemoryTransport.registry is a `private static Map`, and peers() returns
// every entry except self — so this says "no other transport is registered
// anywhere in this process". 75 test files construct one.
await transportA.shutdown();
expect(transportA.peers()).toEqual([]);
// ✓ the transition this test caused, read through a live peer
await transportA.shutdown();
expect(transportB.peers().some((peer) => peer.port === 40501)).toBe(false);

The signature is unmistakable once you know it: passes in isolation, fails only in a full run, and no amount of waiting changes anything — because nothing is in flight. Ask what the assertion’s denominator is. If it is a static field, a module-level registry or a process-wide counter, narrow it to the entry this test owns.

A test that depends on running after another one

Section titled “A test that depends on running after another one”

The sibling of the entry above, and the one a repeat run cannot find at all: a repeat run repeats the same order. Only a shuffle asks the question, and bun test --seed=N implies --randomize and reproduces one exact order:

Terminal window
bun test --seed=413044451

Four suites failed the first time that was asked of them, in three shapes.

A value carried from one test to the next. The block meant “the fixture boots once, not once per test” and said it by writing in one test and reading in the sibling that followed. Shuffled, the reader went first and compared against an unwritten null.

// ✗ the premise is the order of its own block
let observed: ActorSystem | null = null;
test('returns a live system', () => { observed = sys(); });
test('returns the SAME instance', () => { expect(sys()).toBe(observed); });
// ✓ compare against what the block booted, which every test can see
let booted!: ReturnType<typeof sys>;
beforeAll(() => { booted = sys(); });
test('returns the SAME instance the block booted', () => { expect(sys()).toBe(booted); });

A module mock that outlived its file. mock.restore() does not undo mock.module() — a module mock is process-wide and permanent, so a file that replaces node:zlib to make a capability look absent hands that to whichever file runs next. Two suites in tests/unit/persistence/object-storage/ were failing each other in whichever direction the shuffle put them. Snapshot the real module before mocking, put it back in afterEach, and reset whatever the production code memoised from it:

const REAL_NODE_ZLIB = { ...nodeZlibModule };
afterEach(() => {
mock.module('node:zlib', () => REAL_NODE_ZLIB);
resetCompressionCache();
});

Note the beforeEach half as well: a memo inherited from the previous file is just as wrong as one leaked to the next, and only one of the two is your file’s fault. Reset on both edges.

That fixes the file that leaked. It does nothing for the file that was moved, which is where the damage landed: three cases pinning the #580 allocation-time bound went on running on a rung that has no such bound, asserting its security property there. They failed rather than passing quietly, and that was luck — the error message’s tail was the only thing separating “refused before allocating” from “allocated, then refused”, and a differently-worded assertion would have been satisfied by the wrong path.

So a suite that consumes a resolved capability should be able to state which one it got, and assert it once up front:

beforeAll(async () => {
expect(await zstdDecompressorRung()).toBe('node-zlib');
});

A wrong path is then one failure that names the cause, instead of five message mismatches that do not. The general form: if a block’s subject is a property of one implementation behind a resolver, the resolver has to be able to say which implementation answered — otherwise the block is asserting the thing it is trying to establish.

A fixed number of microtask turns. A test fired an operation without awaiting it, let two Promise.resolve()s go by, and asserted that a lazily imported SDK client had been constructed. That is a bet on how many turns a dynamic import() takes, which is one number when the specifier is already in the module registry and another when this is the run’s first resolution of it.

// ✗ two turns is a guess about the module registry's state
void backend.list({ prefix: '' }).catch(() => {});
await Promise.resolve(); await Promise.resolve();
expect(clientsConstructed.length).toBe(1);
// ✓ a completed operation has constructed the client just as well
expect(await backend.list({ prefix: '' })).toEqual([]);
expect(clientsConstructed.length).toBe(1);

Both per-commit suites now shuffle, with the seed drawn by the workflow and printed. It is drawn there rather than left to bun deliberately: bun reports its own seed in the trailing summary block, and under GitHub Actions it stops printing that block and reports the totals out of band — the same trap that once turned an unreadable pass count into a green tests-0 of 0 badge. A seed nobody can read is not a reproduction.

Tests observed failing intermittently whose cause is not yet established. Ruling causes out is not the same as establishing one, so these are listed without a verdict rather than guessed at. One of the five is no longer open — its cause is established below — and stays in the table because it is the entry a whole-suite run is most likely to name again:

TestFileSleep ruled out?
stoppingStrategy stops a failing childtests/Actor.test.tsYes — wait converted
shards rebalance when a node leavestests/Cluster.test.tsYes — wait converted
explicit seeds bypass discoverytests/ClusterBootstrap.test.tsYes — deadline loop converted
peers list is empty after shutdowntests/unit/InMemoryTransport.test.tsYes — the test is fully synchronous
partition + heal flips reachability without dropping the workerstests/unit/testkit/ParallelMultiNodeSpec.test.tsYes — the file has no wait to convert

Ports and filesystem paths have been ruled out for all five, and so has “the sleep was too short” — but not by the same argument in each case, and the difference is what tells you where to look next:

  • Three had a wait, and it was converted. Actor, Cluster and ClusterBootstrap each waited on a fixed delay — ClusterBootstrap through a hand-rolled deadline loop that fell through silently — and now wait on the state the assertion reads.

  • Two never had one. peers list is empty after shutdown is synchronous end to end: every step is awaited, there is no poll and no delay. Its cause is not timing at all — it is a process-global assertion, and that one is settled rather than open. InMemoryTransport.registry is a private static Map and peers() returns every entry in it except self, so expect(transportA.peers()).toEqual([]) asserts that no other transport anywhere in the process is registered. 75 test files construct one. A single suite that leaves one registered — or whose shutdown() has not run yet — fails this test, and only in a whole-suite run, which is exactly the observed signature.

    That is a fifth family, and worth recognising by shape: an assertion over a static collection is a test of the whole process, not of the unit. The test has since been narrowed to the transition it causes — that this address left the registry, read through another live transport — and renamed to shutdown removes the transport from a live peer's view; the old name is kept in the table above because that is the identity the measurement recorded.

    Worth knowing why the narrowing is not merely tidier. Deleting the registry.delete(…) line from InMemoryTransport.shutdown() leaves the old assertion green when it runs alone: peers() filters self out, so A’s own leftover registration is invisible to it. It only went red inside its own file, and only because the tests above it leave registrations behind — so it detected its siblings rather than its subject. The narrowed form goes red on that mutation either way.

    ParallelMultiNodeSpec.test.ts contains no fixed-delay wait at all; it waits through spec.awaitMembers / spec.awaitMemberStatus budgets.

    For that second one there is now a measurement rather than a suspicion. A 15-repeat local run of the three quarantined suites (2026-08-18) caught partition + heal once, and the failure is not an assertion at all:

    InvalidStateError: Worker has been terminated
    at postMessage (src/runtime/worker/WebWorkerBackend.ts:88)
    at postMessage (src/testkit/ParallelMultiNodeSpec.ts:409)
    at onMessage (src/testkit/internal/MultiNodeBroker.ts:90)

    The broker forwarded a message to a worker that had already been terminated, and the rejection was attributed to whichever test was running. That is a teardown race in the harness, not a timing bet in the test — so no wait, of any shape, could have been the cause.

    That trace can no longer escape. MultiNodeBroker.onMessage now routes inside the try/catch the production broker has carried since #701, so a destination port that throws — a worker crash() already terminated is the case that reaches it — counts as the unroutable destination it is and the frame is dropped. The race itself is unchanged; what is gone is the part that made it a flake, an unrelated test failing for it. Read the entry above as a record of the symptom, not as something still to reproduce.

That is the value of writing an open entry down instead of guessing: an entry whose sleep was converted and an entry that never had one lead to different next steps, and only the count in summary.json distinguishes “seen once” from “seen often”. Add the count here rather than starting over.

For months, three suites ran in no CI job at all. ACTOR_TS_SKIP_FLAKY_MNS=1 was set in test.yml, multi-runtime.yml and publish.yml, and three files skipped themselves when it was:

  • tests/multi-node/LeaseMajority.test.ts
  • tests/multi-node/ParallelPubSub.test.ts
  • tests/unit/testkit/ParallelMultiNodeSpec.test.ts

The stated reason was that Bun on GitHub’s hosted runners cannot respawn functional worker threads after the first worker test — they spawn, handshake, and then never run — and that the same resource starvation delayed LeaseMajority’s renewal timer past the lease TTL, so both sides of a partition acquired and the test saw a false split-brain.

All three run in CI again, and how that happened is worth more than the fact.

  • The two worker-thread suites earned it by measurement. The exit criterion was fourteen consecutive green nights of nightly-flakes.yml running exactly those suites with the flag off. They reached twenty-one — 63 executions on ubuntu-latest, not one hang. The criterion had been met and nobody had looked, which is the failure the workflow predicted about itself: “Nothing accumulates the streak. It is counted by a human reading these annotations, which is the same failure that made the quarantine permanent in the first place.”
  • LeaseMajority was never a runner problem. Its cause was a product defect in split-brain resolution: a partition detected one peer at a time was resolved as a run of majority decisions instead of one equal-split decision, so both halves survived and the lease was never consulted at all. Every hypothesis about renewal timers and TTLs was looking at machinery that had not run. Fixed by the stability window in Downing strategies; the suite goes from 8 failures in 15 runs to 0 in 30.

Two mechanisms came out of it, and they are the part to remember.

  • An environment variable may not decide whether a test runs. tests/unit/ci/NoEnvironmentGatedSkips.test.ts refuses one, because that is exactly the thing a workflow can set — and setting it is how a red suite becomes an absent one. A capability probe is fine and is the shape to reach for instead: available ? describe : describe.skip asks the machine what it can do, and no workflow can answer for it. An entry in that guard’s allow-list needs a reason it cannot hide a failure.
  • Every job carries timeout-minutes. A suite that stops making progress now fails inside the hour rather than burning GitHub’s six-hour default — which is the failure mode the quarantine was justified by in the first place.

nightly-flakes.yml keeps running those three suites on their own, three repeats a night. It is the regression guard now rather than the parole board: a claim retired by measurement stays retired only while somebody keeps measuring.

  • Testing overview — how to write a test that does not flake in the first place.
  • ManualScheduler — assert on virtual time instead of on the wall clock.
  • TestProbe — wait for a message rather than for a duration.
  • ParallelMultiNodeSpec — the worker-thread harness two of the quarantined suites exercise.