No Known Bugs

TL;DR: “No known bugs” tells you where someone looked, not how the system behaves.

On 17 November 2025 Cloudflare’s bot management had no known bugs; on 18 November it took a large share of the web down for the better part of an afternoon, and no step in the chain was a typo. What you can honestly write instead is one row per critical component: what’s in, how it’s verified, what escaped. This post builds that row out of Cloudflare’s outage.

Two Hundred #

On 18 November 2025, a configuration file grew past a number, and from 11:28 to 17:06 UTC a large share of the web behind Cloudflare answered with errors.

The chain, from their own post-mortem, has five links. A permissions change on a ClickHouse cluster made the underlying tables of a second database visible to a query that used to see one. The query that generates the bot-management feature file started returning every column twice. The rows in the file more than doubled. The proxy that loads the file preallocates memory for a fixed number of features - “currently that limit is set to 200, well above our current use of ~60 features” - and the file crossed it. The Rust code that reads the file had not been written for that case:

thread fl2_worker_thread panicked: called Result::unwrap() on an Err value

Every worker, every data centre. “Today was Cloudflare’s worst outage since 2019.”

Read the chain again. There is no typo in it. Each step is locally correct: the permissions change was intended, the query was right for the schema it knew, the limit was generous, and the unwrap() was reasonable for a file that had never been too big. Memory-safe language, some of the best infrastructure engineers in the industry, and a component that had never once failed this way.

On 17 November, Cloudflare’s bot management had no known bugs.

You have this chain too. Not the ClickHouse part, the shape. A file that some job regenerates, a query written against the schema it knew, a limit somebody picked with headroom years ago, and one line that assumes the input is fine because it always has been. The scale was Cloudflare’s. The shape is in every codebase I have worked on, and the rest of this post is about what to do with it.

“No Known Bugs” #

The sentence shows up everywhere. Release notes, status updates, the handover doc before a holiday. It’s true and it’s empty. “Known” means found, and found means your tests and your users have looked in some places and not others. Dijkstra put it in one line in 1969: “Testing shows the presence, not the absence of bugs.” Fifty-seven years later we still write the sentence as if it said something about the system.

Three words I’ll keep apart from here on:

• A bug is a fault in code or configuration. It exists whether or not anyone has seen it.
• A defect is a bug with a consequence someone cares about.
• An escaped defect is a defect that reached production and was noticed - by a user, a customer, or an auditor. It’s the only count a non-engineer can verify, and the only one worth promising anything about.

“No known bugs” is a claim about the first kind, and nobody can check it. “Zero escaped defects, here” is a different claim. It’s a standard, not a guarantee, and it can be held - if here is chosen carefully and every escape closes more than the one bug.

The Core #

You don’t owe your users zero bugs. You owe them zero escapes in the places where an escape costs more than an apology.

Call that set the critical core, and pick it by consequence, not by which service scares you most. A component is in the core if a defect in it can:

1. Move money wrongly - charges, refunds, invoices, payouts, balances.
2. Lose or corrupt data the customer cannot recreate.
3. Give the wrong person access - auth, tenant isolation, permissions.
4. Breach a rule you’re bound by - tax, retention, reporting.
5. Stop the product for every customer at once.

The fifth is where Cloudflare’s feature loader sits. It reads one file, and a defect in it stops every request through every data centre. Their analytics dashboard is out of the core. Not because bugs there don’t matter, but because a bug there costs an apology and a bug in the loader costs the web.

Write the list down and review it when the product changes. And notice what the list is not: it is not “the components where we’ve never had an incident”. Never having had an incident is a statement about luck and traffic, not about verification. The feature file had never crossed 200 either.

Inside the core, every escape is a process failure to fix, not a ticket to close. Outside it, move fast and fix forward. A team that applies core-grade rigour everywhere ships nothing; a team that applies dashboard-grade rigour to the core ships an outage.

This assumes you can name the consequence. In a product nobody understands yet, you can’t, and the right move is fast feedback, not a core.

Kill Classes, Not Bugs #

Nobody at Cloudflare had a bug called “feature file exceeds preallocated limit”. What they had was an unmapped defect class: a config value crossing a hard limit in a component that panics on surprise. A class is the shape of a failure, not one instance of it. One class produces any number of bugs. Fixing a bug removes one instance; removing the class removes all of them, including the ones nobody has hit yet. The habit comes from safety engineering, where a catastrophe is never one mistake but several harmless-looking ones lining up - Richard Cook’s How Complex Systems Fail says it in eighteen points on two pages.

That’s the whole method. You don’t defend a core by hunting bugs. You ask which classes it can still admit and make each one impossible. Three practices I’ve run on real projects. Each is tested against one link in Cloudflare’s chain, then shown on a project with none of Cloudflare’s scale, because a class does not care about scale.

Make the Illegal State Impossible to Write #

The fix is structural, not a bigger number. The loader is a parser, and a parser returns an error the caller has to handle, so an oversize file degrades one feature instead of killing the process. Cloudflare’s first commitment in the post-mortem says the same thing in their words: ingest internally generated config “in the same way we would for user-generated input”. The guard lives at the boundary and the panic becomes unreachable.

Your version of this link is every place your code reads a file, a message, or a row and assumes its shape: the settings loader, the webhook handler, the job that pops from the queue.

When I built a saga orchestrator in Python, the whole thing rested on this idea: one enum of booking states and a guard on every transition, so a stale or duplicate event falls through instead of advancing the booking. That’s not a bug fix. It’s a class of bugs that cannot be expressed.

Test Against the Real Database, Fast Enough That Nobody Skips It #

No unit test saw the duplicate rows, because no unit test holds a ClickHouse cluster with two databases and a fresh permission grant. Only a test against the real thing, under the real schema, sees them.

Your version of this link is the query that passes on SQLite and fails on Postgres, the migration that drifted from the model, the unique constraint that only fires on real rows.

The fix is well known and routinely abandoned, because the suite gets slow and slow suites get skipped. On one project, our suite took 40 to 90 minutes, each of 1,154 tests building a fresh database through 60 migrations, so people ran the tests they’d touched and let CI tell them the rest. A verification you can’t afford to run every time isn’t part of your core’s defence, whatever the README says. How we made that suite 10 to 15 times faster is a post of its own, coming next.

Every Escape Ends Upstream, Not in a Restart #

Cloudflare could have raised the limit to 400 and gone home. The post-mortem lists four commitments instead, and none of them is “raise the limit”: harden config ingestion, add global kill switches for features, stop error reports from overwhelming system resources, review failure modes across every core proxy module. That’s what closing a class looks like. The fix lands where the class lives, and the symptom is beside the point.

Your version of this link is the nightly restart, the memory bump, the retry that hides the error, each one a class left open with a snooze button on it.

A Cloud Run function of ours once started dying with out-of-memory errors. The cheap fix was more memory and more restarts. The leak turned out to be in the OpenTelemetry Python SDK, and the fix went into the library, which means every other serverless user got the class removed without knowing it existed. That’s what the same move looks like on a function that handles a few thousand requests a day instead of a large share of the web.

Review belongs on this list too, and I’ve written a whole post on making it scale with the code.

The Ledger #

Now write Cloudflare’s row. Three columns, owned by the team, looked at in every retro.

The verification column holds only what the post-mortem commits to. The row makes three things visible: the boundary is real, the instruments are named, and the escape closed a class rather than a ticket. A ledger with a one, a named class, and a structural fix is more credible than a column of zeros. A column of zeros says you haven’t looked.

Yours has the same three columns and one row per component in your core. The first column is the list you wrote down earlier. The second holds only what runs on every change, not what the README promises. The third is a count, and next to every count above zero, the class it closed. It takes an afternoon to write, and it’s the thing to send the next time you would have written “no known bugs”.

“No known bugs” tells you where someone looked. The row tells you how the system behaves.