The backup that ate itself
One of our backups was 4 kilobytes.
It should have been hundreds of gigabytes — a full copy of a node's ancient chain data, the kind of file you only look at on the day you desperately need it. Instead: 4KB. A tar header and silence. And the worrying part wasn't the file. It was that the system that produced it had reported success.
What actually happened
Our fleet has a reconciliation loop: a central process that periodically regenerates every host's configuration from the inventory and runsdocker compose up. It's the
thing that makes the fleet converge — whatever drifts, reconcile pulls it back. We
rely on it hundreds of times a day, mostly without thinking about it. That's what
made it dangerous.A backup job works like this: stop the node, tar its data volume, start the node. Simple. Except "stop the node" was a local action, and the reconcile loop is global. It doesn't know a human — or an agent — stopped that container on purpose. It sees a service that should be running and isn't, and it fixes that. Mid-tar.
So the node came back up while its own data directory was being archived underneath it, the database started writing, and tar quietly produced a file that was mostly nothing. Exit code zero. Everyone went home happy. Twice, on two different chains, before anything noticed.
Why nobody caught it
Every individual component behaved correctly. The backup script stopped the container and got confirmation. The reconcile loop restored the declared state, which is its whole job. tar archived what it could read. The failure lived in the composition — two correct automations with no shared concept of "this node is deliberately down right now."Agent-operated systems make this failure mode more likely, not less: the more autonomous actors you add, the more pairs of correct behaviors can multiply into one wrong outcome.
The fix: fencing as a first-class concept
The fix wasn't to make backups faster or reconciles smarter. It was to give the system a word for "hands off." We call it a maintenance window: before any deliberate stop, the node is fenced — removed from the set of services the reconcile loop manages, with a note saying who fenced it, why, and until when. While fenced, compose changes don't apply to it and nothing tries to resurrect it. When the work is done, the window closes and the node rejoins the managed world.Every workflow that stops a node — backup, restore, clone, retire — now fences first, as a non-optional step. And the backup itself gained a fail-closed gate: it verifies the node is stopped immediately before tar starts, and aborts if anything looks alive.
Then we made the rule teachable. Our agents' operating manual carries it as a hard constraint now, and our status tooling shows fenced nodes in their own section — so when a node "disappears" from the main list, an agent reads that as the fence working, not as an outage to fix. The first time an automation helpfully undoes your intentional action, you learn that in a fleet run by many hands, *intent has to be part of the shared state.*
4KB was a cheap price for that lesson. The restore test on the re-taken backups passed.