Anatomy of a Cascade: Lessons Learned (and Paid For)

July 23, 2026 • by rob

Anatomy of a Cascade: How One Dead Host Took Down Our Monitoring, Our Mechanics, and Almost Our Sanity

The setup

We run a fleet of RPC nodes across five regions, listed on dRPC. Over the past months we built central monitoring around it: a traffic-insights pipeline (gateway access logs shipped to a central listener), an oracle that scores every upstream, an incident babysitter that opens tickets, sentinels that restart things, and a pool of "mechanic" agents that wake every few minutes, claim open issues, and fix them.

Each piece worked. Together they nearly killed each other.

The timeline (2026-07-22, all times UTC)

16:00 — Berachain's bepolia testnet hardforks. Our two bepolia nodes run a client one minor version too old; both freeze at the same second. First lesson of the day, learned within hours: two "independent" backends sharing one client stack are not independent. A same-second freeze looks exactly like a chain halt until you compare against an external endpoint.

17:28 — Conduit, the rollup-as-a-service provider behind several of our chains, has a batcher outage. Mode mainnet and the sepolia-family chains stop advancing. The sepolia side recovers by 18:30. Mode does not: when the feed comes back, our op-geth nodes need p2p peers to backfill the gap — and mode's peering has been broken for weeks (a known regression we'd been living with, because with fresh blocks arriving over the sequencer feed you never notice you have zero peers). The outage ends; our nodes stay stuck. A dormant defect activated by an unrelated provider incident.

19:45 — The real domino: us-32, our sole US-East host carrying 16 chains, starts dying. sshd stops answering, then every chain falls behind, then the RPC data plane goes dark. Load/OOM death spiral; no management path left; only the provider console can help. Nothing to do with bepolia, nothing to do with Conduit. Just entropy.

20:30 — Here is where architecture bites. The traffic oracle runs its update sweep, tries to poll a host that no longer answers, and cannot finish the run. Its output — the derived series every downstream sensor reads — comes up empty. Within one poll tick, the incident system declares every chain on every host dark: 97 incidents in two minutes, across hosts that were serving traffic perfectly.

20:30 onward — The mechanics do what they were built to do: wake every five minutes, claim an issue, investigate. Except there are now ~97 phantom issues, and — due to an unrelated MCP cold-start race — many mechanics die on step one with a tool error, burn their session, and leave the claim for the next wake. The pool grinds through its API quota claiming and re-claiming phantom incidents caused by a monitor artifact caused by a dead host. Automated workflows halt. The claim/retry churn and the failed sessions fill the logs; CouchDB on the insights host grows; that host's disk climbs to 95%; disk pressure starts suffocating the same mechanics that were already dying of quota exhaustion. Meanwhile the controller's own disk quietly passes 86% from job logs and image pulls.

Everything is spiralling, and not one of the spiral's arms knew about the others.

What actually went wrong

Not the host crash. Hosts crash. What went wrong is that we built a distributed system while reasoning about a monolith:
  • The oracle assumed "my sweep finishes" — a monolith assumption. One unreachable
host turned partial data into no data, and no data was rendered as "everything is down" instead of "I don't know".
  • The babysitter assumed "an incident in the queue is real" — reasonable inside a
monolith where the sensor and the actor share state, false in a pipeline where a sensor artifact fans out into 97 tickets.
  • The mechanics assumed "my tools are loaded when I wake" — true on a warm
monolith, a race on cold-started per-session processes.
  • Every cron, on every host, made its own local decision with no shared circuit
breaker. Monolithic architecture, composed of cronjobs and decentralized decisions. Each host and each process introduced another uncorrelated failure case — and the failures correlate anyway, downstream, where they compound.

And the ultimate correlation, the one that makes all of this one system whether we like it or not: everything hangs off the same accounts. The same API quota. The same disks we forgot to cap. The same credit card. Given enough time, everything that can go wrong, will — and it will all be "correlated by billing".

What saved us

The boring stuff. The ultimate backup turned out not to be any database: it's a pile of scripts, task files, and ansible roles in git. Point them at hosts — or just at a Hetzner cloud API key — and they rebuild the system. That property held. de-33 died in the morning and was fully restored by evening from exactly those pieces. Nodes are cattle; the herd definition is the asset.

Also: hash-compare before believing "forked". Check an external endpoint before believing "chain halt". Check tslb across backends before letting a sentinel restart a gateway. Verify a "recovered" notice before celebrating. Every one of those cheap verification habits stopped at least one wrong action this week.

Lessons (paid for)

  • A monitor must distinguish "down" from "I can't see". An unfinished sweep
    • should degrade to unknown, never to offline. Absence of data is not data of absence.
  • Alert pipelines need circuit breakers. 97 simultaneous incidents from one
    • source should trip a fuse ("this is a sensor event"), not fan out to the actor pool.
  • Actors need budgets per cause, not just per action. The mechanics had a
    • fleet-wide restart budget but no cap on claims-per-root-cause. Quota is an availability resource; phantom work is a denial-of-service on yourself.
  • Cold-start races are architecture, not noise. If a tool takes 1.6s to
    • initialize and your agent is told tools are ready on turn one, you built a race. Persistent services beat per-session spawns for anything called on wake.
  • Same client stack = same failure domain. Diversity of hosts is not
    • diversity of software. A fork you can't follow freezes every replica at the same block.
  • Dormant defects activate under stress. Zero peers didn't matter until the
    • day the feed stopped. Audit the "works anyway" warnings; they are your future outage's accomplices.
  • Disk is a shared fate. Logs, job archives, CouchDB, container images —
    • every subsystem writes, none of them owns cleanup. Retention must be a first-class, monitored budget on every host, including (especially) the controller and the monitoring host itself.
  • Keep the rebuild path sacred. Scripts + inventory + roles + an API key
    • restored a host in hours. That is the disaster recovery plan; everything else is optimization. Test it like it matters, because it is the only thing that does.

    Epilogue

    The referee kept flapping on healthy nodes all night — twenty-two false fork alarms on one chain, each one hash-verified, each one wrong. The damping rule we approved for one testnet ("hold one sweep before acting") went five-for-five with zero wasted actions. The next architectural investment is obvious: make the sensors trustworthy enough that the actors can be brave again. Until then, the on-call agent verifies everything twice — because the system we built to watch the fleet turned out to need a watcher of its own.