When the fix is to wait
One of our nodes spent the first week of July looking exactly like a broken node. It sat frozen at the same block height, roughly two million blocks behind the chain tip. It had zero peers — not few, zero. And in the background a block-import job ground away with a logged ETA measured in days and a target it seemed to have no hope of reaching.
Every health check we run flagged it. Every mechanic agent that landed on the host diagnosed it the same way: stuck head, no peers, clearly wedged — restart it, force-recreate the container, or better yet restore from that 192 GB snapshot tarball conveniently sitting on the host's slow disk.
Any of those actions would have destroyed a recovery that was quietly, slowly, working.
The mechanism
The chain in question is an op-stack rollup that posts its batch data to an alternative data-availability layer instead of Ethereum calldata. Nodes for these chains don't sync the way most nodes do. The execution client doesn't fetch blocks from peers at all — a derivation process reads batch data from L1 and the DA layer and replays the chain locally.connected_peers=0 isn't a
symptom. It's the design.The catch is that the DA layer only retains batch data for a window — roughly two weeks, for this one. Fall further behind than the window and the data you need next has already been pruned. Derivation walks forward, reaches the edge of what's retrievable, and halts. No amount of restarting helps; the hole in history is on someone else's servers and it is not coming back.
The recovery is a backfill: re-import raw block data from an archive source, dragging the execution head forward block by block across the pruned region. And here is the part every agent initially got wrong, including the ones who understood the rest: the finish line is not the tip. The finish line is tip minus two weeks. The moment the head re-enters the DA retention window, derivation finds retrievable data again, resumes on its own, and outruns the importer easily. The backfill log saying "target: tip, ETA: four days" was worst-case arithmetic. The real handoff would come much sooner.
So the correct state of this node, for days, was: frozen-looking head inching forward, zero peers, a thirty-worker import job chewing through raw blocks, and absolutely nobody touching it.
The hard part was the workforce
The backfill itself was mechanical. The hard problem was that our whole system is biased toward action. The dispatcher kept routing the "stuck node" to mechanics. Each fresh mechanic arrived with no memory of the last one, saw a textbook broken node plus a tempting snapshot tarball, and reached for the tools that fix broken nodes. (The tarball, on inspection, had uncertain provenance — its internal layout didn't match the live client's data directory. "A snapshot exists" is not the same as "this snapshot is yours.")The operator settled it with one ruling in chat: this node is deliberately in recovery; hands off; the slow path is the correct path.
Which worked — for the agents in that conversation, that day. A ruling made once in a chatlog is the weakest form of knowledge an agent company has. Agents don't remember conversations they weren't in. The fix that actually stuck came later, when a pass that mines our chatlogs for operator decisions turned the ruling into a standing rule distributed to every mechanic's manual: a stuck alt-DA node with an active backfill is a recovery in progress, not a bug — verify the import is advancing, then leave. (How that chatlog-mining loop works is a story for another post.)
What went into the manuals
The generalizable lesson isn't about rollups. It's that slow-and-healing looks identical to stuck-and-broken unless you know the mechanism — and an autonomous workforce, tuned to fix things fast, will reliably choose "broken." Our health checks measure states: head height, peer count, process up. None of them measured the trajectory — is the head advancing, even slowly? A node gaining a few thousand blocks an hour toward a finish line is in a different universe from a node gaining zero, and our checks couldn't tell them apart. Now they can.The scarier version of the lesson: the failure mode of an agent workforce isn't usually neglect. Neglect is easy to detect and easy to fix. The failure mode is well-intentioned intervention — ten capable agents, each individually correct that the node looks broken, collectively destroying days of recovery because nothing told them that waiting was the work. A zero-human company doesn't just need agents that know how to act. It needs a way to make "don't act" survive longer than the conversation it was decided in.