The Line of Rust That „Broke the Internet“

TL;DR

On November 18, 2025, a single unwrap() call in Cloudflare’s Rust code triggered a global outage. But blaming the language feature misses the point entirely. The real story is about a database permission change that rolled straight into production without ever being tested in a staging environment. This article breaks down what actually happened, why „config is code,“ and how to build a development lifecycle where misconfigurations die in dev instead of taking down half the Internet.

---

🔥 Introduction

On November 18, 2025, Cloudflare had its worst outage since 2019.¹ A large chunk of the Internet effectively faceplanted with a wave of 5xx errors. CDNs went dark. APIs stopped responding. Dashboards froze. Even services that check whether things are down were themselves down.

The root cause turned into a perfect headline: „A single Rust .unwrap() took down Cloudflare.“¹

The Internet, predictably, exploded with hot takes. „Never use unwrap() in production.“ „Rust isn’t as safe as advertised.“ „This is why you should stick to X language.“

But that narrative is too shallow. The interesting lesson isn’t „unwrap() bad.“ The interesting lesson is this: if a configuration change in production can trigger a new code path for the first time, your problem isn’t just error handling. Your problem is your entire software development lifecycle.

This post is about that. Why configuration changes deserve the same rigor as code. Why staging environments that actually mirror production are critical. And how to design a pipeline where a misconfigured database doesn’t get to audition for „global outage of the year.“

---

🧠 Main Content

What Actually Happened (Technically, Not Clickbait-ly)

Let’s briefly break down the incident in engineering terms.

Cloudflare’s request path can be simplified as: Client connects via HTTP/TLS, hits the core proxy (FL / FL2), passes through modules like Bot Management, and finally reaches origins or edge services.

One of these modules, Bot Management, uses a machine learning model to assign bot scores to each request. It consumes a „feature configuration file“ listing all features the model can use, and it refreshes this file every few minutes across Cloudflare’s global edge network.

The file is generated by a ClickHouse query over metadata tables. Cloudflare rolled out a database permissions change so that users could see metadata for underlying tables (r0 schema) in addition to the default database. That changed the behavior of a simple query that selected column names and types from a system table. Previously, it only returned rows for the default schema. After the permission change, it also returned rows for r0, effectively more than doubling the result set.¹

Now here’s where it gets interesting. In the Bot Management module, there was a limit of 200 features. They were using around 60. The code preallocated memory up to 200 as a performance optimization and enforced that limit. When the oversized feature file rolled out with more than 200 features, the module hit that limit and treated it as an impossible condition. The relevant Rust code enforced an invariant and then called unwrap() on a Result, assuming „this can’t fail.“ It did. Rust did exactly what it promises: it panicked.² In FL2, that panic surfaced as 5xx errors.¹

So the chain looked like this: ClickHouse permission change leads to query returning more rows than expected. Generated feature file size grows beyond the hard-coded limit. Bot Management module hits a „this should never happen“ invariant. Result::unwrap() sees an Err and panics. Core proxy crashes for affected traffic paths. Global 5xx storm ensues.

Here’s how that cascade of failures unfolded:

sequenceDiagram
    participant Admin as DB Admin
    participant CH as ClickHouse
    participant Gen as Config Generator
    participant BM as Bot Management Module
    participant Proxy as FL2 Proxy
    participant Client as Clients

    Admin->>CH: Apply permission change<br/>(expose r0 schema)
    Note over CH: Query now returns<br/>default + r0 rows

    Gen->>CH: SELECT features FROM system.columns
    CH-->>Gen: Returns 120+ features<br/>(was 60)

    Gen->>Gen: Generate feature file
    Note over Gen: File now has >200 features<br/>(exceeds hard limit)

    Gen->>BM: Deploy new feature file
    BM->>BM: Load config
    Note over BM: Feature count > 200<br/>Invariant violated!

    BM->>BM: unwrap() on Err
    Note over BM: PANIC!

    BM--xProxy: Module crashes
    Proxy--xClient: HTTP 5xx errors
    Note over Client: Global outage

Yes, there is a Rust unwrap() at the end of this domino chain. But the first domino is a configuration change.

The Real Villain: Untested Configuration Changes

Most teams treat code with respect and configuration with casual optimism.

Code change? Pull request, review, CI, unit tests, integration tests, maybe performance tests.

Config change? „It’s just a permission tweak. A limit bump. A YAML entry. Ship it.“

Here’s what that looks like in practice:

flowchart TB
    subgraph code["Code Change Process"]
        c1[Write Code] --> c2[Create PR]
        c2 --> c3[Code Review]
        c3 --> c4[Run CI]
        c4 --> c5[Unit Tests]
        c5 --> c6[Integration Tests]
        c6 --> c7[Performance Tests]
        c7 --> c8[Deploy to Prod]
    end

    subgraph config["Config Change Process"]
        cf1[Edit Config] --> cf2["Ship it! 🚀"]
        cf2 -.->|Hope| cf3[Pray nothing breaks]
    end

    style code fill:#90EE90
    style config fill:#FFB6C6

Cloudflare’s own postmortem explicitly calls out that a database access-control change and an assumption in their query behavior led to a configuration file that violated a module’s expectations.¹

That’s the first core lesson. Config is code.³ It can change behavior, trigger new branches, and violate invariants just as violently as a commit to main.

If you believe an invariant like „this file will never have more than 200 entries,“ you must test the things that could violate it in environments that resemble production.

Let’s connect that to a proper development lifecycle. In your development or test environment, you apply the same ClickHouse permission change first. You regenerate the feature file. You see that the file suddenly doubles in size and crosses the 200-feature limit. You watch your dev proxy crash miserably where only you notice it.

In your staging environment, after fixing the behavior or adjusting limits, you repeat the process against a staging cluster with realistic data. You validate that the feature generation pipeline and proxy modules behave correctly with realistic traffic patterns.

Only then do you promote the same configuration change to production, ideally with progressive rollout through canaries, partial regions, and gradual percentage increases.

If each environment had the same query, same schema assumptions, and same feature-file generation path, this bug had multiple chances to die long before it reached all 330+ data centers. The absence of that multi-stage, config-aware testing pipeline is the real failure.

Here’s what a proper config rollout pipeline should look like:

flowchart LR
    subgraph dev["Dev/Test Environment"]
        d1[Apply DB Change] --> d2[Regenerate Config]
        d2 --> d3[Run Tests]
        d3 --> d4{Tests Pass?}
        d4 -->|No| d5[Fix Issues]
        d5 --> d1
        d4 -->|Yes| d6[Validate]
    end

    subgraph staging["Staging Environment"]
        s1[Apply DB Change] --> s2[Regenerate Config]
        s2 --> s3[Synthetic Traffic Test]
        s3 --> s4{Metrics OK?}
        s4 -->|No| s5[Rollback & Debug]
        s5 --> d1
        s4 -->|Yes| s6[Validate]
    end

    subgraph prod["Production"]
        p1[Canary Deployment<br/>1 Region] --> p2{Metrics OK?}
        p2 -->|No| p3[Rollback]
        p3 --> s1
        p2 -->|Yes| p4[Gradual Rollout<br/>25% → 50% → 100%]
        p4 --> p5{Metrics OK?}
        p5 -->|No| p6[Rollback]
        p6 --> s1
        p5 -->|Yes| p7[Full Deployment]
    end

    d6 --> s1
    s6 --> p1

    style dev fill:#E3F2FD
    style staging fill:#FFF3E0
    style prod fill:#E8F5E9

Would Staging Really Have Saved Them?

Here’s where I want to gently challenge the conventional wisdom.

Saying „we should test config changes in staging“ is correct but incomplete. Plenty of companies think they do this already. The devil is in the details.

For staging to catch this kind of bug, at least three things must be true.⁴

First, staging must run the same codepaths. If feature-file generation only runs in production because „that’s where the real data is,“ staging won’t see the failure.

Second, staging must have representative data and permissions. This specific issue was triggered by a subtle change in DB permissions and query behavior. If your staging cluster doesn’t mirror those grants, you won’t see it.

Third, the config rollout pipeline must be environment-aware. If the feature file is generated once in prod and then pushed everywhere, staging will never see the broken version.

So „just stage it“ only works if you design your system so staging actually experiences the same shape of reality.⁴

Concretely, that means your ClickHouse migration or permission change should roll out like any other change: dev/test, then staging, then prod. The config generator should run in each environment, using that environment’s DB. You should have invariants and assertions around config artifacts before they hit critical systems, things like „Feature file size is less than or equal to 200“ or „No duplicate column definitions“ or „Schema version matches supported range.“

In other words, we don’t just need more environments. We need better designed environments and pipelines.

Here’s what production and staging should look like to actually catch these issues:

graph TB
    subgraph prod["Production Environment"]
        pdb[(ClickHouse<br/>Production)]
        pgen[Config Generator]
        pproxy[FL2 Proxy]
        pbm[Bot Management]

        pdb -->|"Permissions: default + r0"| pgen
        pgen -->|Feature File<br/>120+ features| pbm
        pbm --> pproxy

        style pdb fill:#FFCDD2
        style pgen fill:#FFCDD2
    end

    subgraph staging["Staging Environment"]
        sdb[(ClickHouse<br/>Staging)]
        sgen[Config Generator]
        sproxy[FL2 Proxy]
        sbm[Bot Management]

        sdb -->|"MUST MATCH:<br/>Permissions: default + r0"| sgen
        sgen -->|"MUST MATCH:<br/>Feature File 120+ features"| sbm
        sbm --> sproxy

        style sdb fill:#C8E6C9
        style sgen fill:#C8E6C9
    end

    note1[✓ Same DB permissions]
    note2[✓ Same query behavior]
    note3[✓ Same data volume/shape]
    note4[✓ Same codepaths]

    staging -.->|Must mirror| prod

    style note1 fill:#FFF9C4
    style note2 fill:#FFF9C4
    style note3 fill:#FFF9C4
    style note4 fill:#FFF9C4

Defense in Depth: It’s Not Just SDLC vs unwrap()

Now let’s talk about the unwrap() in the room.

I agree that blaming unwrap() alone is lazy. But completely absolving it is also a miss.

In Rust, unwrap() is basically an assertion. It says „I’m so confident this is Ok that if it isn’t, I’d rather crash than continue.“² Sometimes that is the right choice. In deeply internal invariants where the only honest response to violation is to fail fast, a panic can be the safest behavior.

But there are three levels of responsibility here.⁵

The configuration pipeline should validate that generated configs adhere to limits before they touch the hot path. It should fail fast on bad config and roll back or quarantine it.³

The service or module behavior should have safe failure modes. If bot features look broken, disable Bot Management and keep routing traffic. Fall back to last-known-good feature file.⁶ Treat internally generated config with the same suspicion as user input. Cloudflare’s own remediation plan explicitly calls this out.¹

The local code decision between unwrap() and proper error handling matters too. At that exact call site, you have a choice. You can panic the whole worker thread, which was the current behavior. Or you can bubble up a typed error that the module-level error handler can turn into something like „module unhealthy, disable bot scoring, keep proxy alive.“

Think of it as concentric layers of defense:

graph TB
    subgraph outer["Layer 1: Config Pipeline"]
        v1[Schema Validation]
        v2[Size Limit Checks]
        v3[Invariant Assertions]
        v4[Rollback on Failure]
    end

    subgraph middle["Layer 2: Service/Module Behavior"]
        m1[Safe Failure Modes]
        m2[Fallback to Last Known Good]
        m3[Graceful Degradation]
        m4[Health Check Exposure]
    end

    subgraph inner["Layer 3: Code-Level Error Handling"]
        c1[Result Types vs unwrap]
        c2[Error Propagation]
        c3[Panic vs Degrade Decision]
        c4[Lint Rules]
    end

    outer -.->|If this fails| middle
    middle -.->|If this fails| inner

    style outer fill:#E8F5E9
    style middle fill:#FFF3E0
    style inner fill:#FFEBEE

The real improvement isn’t „rewrite unwrap() to match everywhere.“ It’s designing error propagation such that you can cleanly enter a degraded but safe state rather than catastrophic failure.

Here’s how error handling should flow through those layers:

flowchart TD
    start[Config Change Deployed] --> load[Load New Config]

    load --> validate{Layer 1:<br/>Pipeline Validation}
    validate -->|Invalid| reject[Reject & Rollback]
    reject --> alert1[Alert: Config Rejected]

    validate -->|Valid| apply{Layer 2:<br/>Module Load}
    apply -->|Error Detected| degrade[Enter Degraded Mode]
    degrade --> fallback[Use Last Known Good Config]
    fallback --> alert2[Alert: Module Degraded]
    fallback --> serve1[Continue Serving Traffic]

    apply -->|Unexpected Error| panic{Layer 3:<br/>Code Decision}
    panic -->|unwrap/panic| crash[Module Crashes]
    crash --> down[Proxy Down]
    down --> outage[OUTAGE]

    panic -->|Result/match| handle[Handle Error Gracefully]
    handle --> disable[Disable Module]
    disable --> serve2[Continue Serving Traffic]
    disable --> alert3[Alert: Module Disabled]

    apply -->|Success| healthy[Healthy State]
    healthy --> serve3[Serve Traffic Normally]

    style reject fill:#C8E6C9
    style degrade fill:#FFF9C4
    style fallback fill:#FFF9C4
    style serve1 fill:#C8E6C9
    style serve2 fill:#C8E6C9
    style serve3 fill:#C8E6C9
    style crash fill:#FFCDD2
    style down fill:#FFCDD2
    style outage fill:#D32F2F,color:#fff

I’d phrase it like this: unwrap() was the visibility of the failure, not the root cause. But better error propagation could have reduced the blast radius dramatically.

That’s the nuance worth understanding. SDLC and staging are your outer fortress walls. Error handling and panic policies are the internal fire doors. You need both.

Practical Guidance: How to Avoid Your Own „Feature-File“ Moment

Let’s turn this into actionable guidance for anyone building systems that are config-heavy and latency-sensitive.

Treat Config as First-Class Code

Store configs in version control with human-readable history.³ Enforce schema and invariants via JSON Schema, protobufs, or custom validators. Add CI checks that parse config, enforce limits like max counts and sizes, and run snapshot tests against representative inputs.⁷

Here’s what a proper CI/CD pipeline for config should look like:

flowchart LR
    commit[Config Commit] --> parse[Parse & Validate]
    parse --> schema{Schema Valid?}
    schema -->|No| fail1[❌ Fail CI]

    schema -->|Yes| limits{Size/Count<br/>Limits OK?}
    limits -->|No| fail2[❌ Fail CI]

    limits -->|Yes| snapshot[Snapshot Tests]
    snapshot --> compare{Matches<br/>Expected?}
    compare -->|No| fail3[❌ Fail CI]

    compare -->|Yes| deploy[Deploy to Dev]
    deploy --> integration[Integration Tests]
    integration --> metrics{Tests Pass?}
    metrics -->|No| fail4[❌ Fail CI]

    metrics -->|Yes| promote[✅ Promote to Staging]

    style fail1 fill:#FFCDD2
    style fail2 fill:#FFCDD2
    style fail3 fill:#FFCDD2
    style fail4 fill:#FFCDD2
    style promote fill:#C8E6C9

Design a Multi-Stage Config Rollout

Instead of „DB change leads to new config leads to instantly shipped to all prod nodes,“ prefer something more deliberate.

In dev/test, apply the DB or permission change. Regenerate config. Run unit and integration tests that load the config into the module and assert invariants.

In staging, run the same change, same generator, but hitting a staging DB with realistic data volumes and permissions. Run synthetic traffic through the staging proxy and watch for panics or high error rates.

In production, canary the change.⁸ Start with one region, one cluster, or a low percentage of traffic. Only roll out globally if metrics are clean.

If your config is generated centrally like Cloudflare’s feature file, you can still simulate this by having the generator run as if it were targeting each environment, with the same query and permissions.

Make Config Errors Degrade Gracefully

Build guardrails directly into the code that consumes config. On load, validate counts, sizes, and schema. If something looks off, mark the module unhealthy rather than panicking the entire proxy.⁶ Keep a last-known-good config snapshot. If the new config fails validation, log loudly, alert, and keep serving with the old config. Expose health signals so „Bot module unhealthy but proxy OK“ is vastly better than „proxy dead.“

Here’s how a service should transition between health states:

stateDiagram-v2
    [*] --> Healthy: Service starts with valid config

    Healthy --> Validating: New config received

    Validating --> Healthy: Config valid,<br/>load successful
    Validating --> Degraded: Config invalid,<br/>using last known good
    Validating --> Failed: No valid config available

    Degraded --> Validating: Retry with new config
    Degraded --> Failed: Last known good config expired

    Failed --> [*]: Service shutdown

    Healthy --> Failed: Critical error<br/>(unwrap panic)

    note right of Healthy
        ✓ Serving traffic normally
        ✓ All features enabled
    end note

    note right of Degraded
        ⚠ Serving traffic with fallback
        ⚠ Some features disabled
        ⚠ Alerts firing
    end note

    note right of Failed
        ❌ Not serving traffic
        ❌ Requires intervention
    end note

Use Panics Intentionally, Not Casually

In Rust, use unwrap() for truly impossible states, things like static invariants that cannot be influenced by configuration or runtime data.² Use it for very early prototype code, but treat those as TODOs before shipping.

Prefer Result or custom error types at module boundaries. Let the module decide whether to panic, degrade, or bypass. And absolutely lint aggressively in critical services. Many teams enable Clippy’s unwrap_used and expect_used lints for production code and only allow carefully documented exceptions.⁹

---

🎯 Conclusion

The Cloudflare outage is going to live in slides and talks for years as „the Rust unwrap() that took down the Internet.“ It’s catchy. It’s also misleading.

What really happened was this: A database permissions change altered query results. A config generator produced a file that violated an implicit invariant. That config rolled out globally without failing in lower environments first. A panic in a critical module was allowed to take down the core proxy.

Rust did exactly what it promised. It refused to silently proceed in a state the developers had declared „impossible.“ The real question is why that state was allowed to first appear in global production, instead of dying noisily in dev or staging.

So if you’re a staff or principal engineer reading about this incident, don’t just paste #![deny(clippy::unwrap_used)] into your repo and call it a day. Ask harder questions. Do your staging environments actually behave like production in terms of config, data shape, and permissions? Do your configuration changes have a rollout pipeline, or do they silently leapfrog straight into prod? When invariants are violated, do you fail fast in the right place and degrade gracefully elsewhere?

Because in the end, it’s not about demonizing unwrap().

It’s about building a development lifecycle where a misconfigured database never gets the chance to break the Internet in the first place.

---

📚 References

---

Notes

1. Cloudflare. (2025). „Cloudflare outage on November 18, 2025.“ Cloudflare Blog ↩
2. Klabnik, S., & Nichols, C. „To panic! or Not to panic!“ The Rust Programming Language. Rust Documentation ↩
3. HashiCorp. „What is Infrastructure as Code with Terraform?“ HashiCorp Developer ↩
4. Wiggins, A. „Dev/prod parity – The Twelve-Factor App.“ 12factor.net ↩
5. Cloudflare. „What is defense in depth?“ Cloudflare Learning Center. Cloudflare Learning ↩
6. Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. „Addressing Cascading Failures.“ Site Reliability Engineering: How Google Runs Production Systems. Google SRE Book ↩
7. AWS. „Best practices for using the Terraform AWS Provider.“ AWS Prescriptive Guidance. AWS Documentation ↩
8. Netflix Technology Blog. (2018). „Automated Canary Analysis at Netflix with Kayenta.“ Netflix TechBlog ↩
9. Rust Team. „Clippy Documentation – Usage.“ Rust Clippy Docs ↩