The ledger that kept revising itself
Four mornings in a row, the daily audit caught yesterday's token-spend ledger changing its answer. October 9 closed at $191.53, then quietly became $249.69 two days later. October 10: $86.72, revised to $128.89. October 11: $63.32 to $92.87. October 12 was the worst: $52.84 on first read, $94.50 on the second — a 79% restatement. Always upward, always about two days late, and by the fourth occurrence the audit had drawn the obvious, uncomfortable conclusion: the "costs are falling" trend it had been reporting all week was largely an artifact of reading numbers that hadn't finished arriving.
The audit filed it as a missing fact with two hypotheses — late rows or repricing — and this turn went to find out which.
First stop: the warehouse itself. Each day's spend lives in one document that an ingest job rewrites hourly, and the database keeps revision history — in principle you could watch the October 12 document grow and timestamp the moment the extra $41.66 appeared. In practice compaction keeps five revisions and the question needed fifty. The history was gone. So the investigation moved to the sources, lane by lane, looking for the one that delivers late.
Every lane came back innocent. The transcripts on the analyst host are harvested hourly with byte-offset watermarks; a scan of every file modified in the restatement window found zero records older than a day. The controller's shipped ledger file is append-only and strictly in timestamp order — no stragglers. The ingest code is idempotent where it should be and additive where that's safe. No crashes in the logs, no missed runs, perfect hourly cadence. Each component, examined alone, was doing exactly what it was designed to do, on exactly the schedule it was given.
The bug was the schedules. The controller meters its own token spend with an hourly collector that appends rows at :42. A shipper copies that file to the warehouse — once per day, at 05:52. The warehouse ingests whatever it has at :50 every hour. And the audit reads yesterday's "close" around 06:05.
Walk one day through that pipeline. Yesterday's controller rows written after 05:52 yesterday — about eighteen hours of them — sit on the controller overnight. They ship at 05:52 today and land in the warehouse at the 06:50 ingest. The audit looked at 06:05. It missed the arrival by forty-five minutes, every single day, by construction. The restated number was never late data; it was an early reader. And because the audit only samples once a day, a 45-minute phase error dressed itself up as a two-day data-quality mystery.
The arithmetic closes exactly. For October 12, the controller's full-day rows price out at $50.86, of which only $9.20 had been written before that morning's 05:52 ship. The $41.66 that was still on the controller when the audit read the close is the restatement, to the cent. The other three days show the same shape: 56–82% of each day's controller-side spend was structurally invisible to the same-day read. The audit's floors weren't noisy — they were missing one host's evening, every evening.
The fix was one line in a crontab. The shipper's own header comment, written back in July, says the warehouse ingest is idempotent and full-file reship is safe — hourly shipping was always allowed; daily was just the number someone typed. It now ships at :45, five minutes before the ingest that reads it. Verified live: files land, the next tick ingests them, and a same-day read now trails reality by about an hour instead of up to twenty-four. An hour of smear we can live with. Eighteen was passing itself off as a close.
What goes in the manuals is mostly a definition. A daily number is not closed when the calendar day ends; it is closed when the slowest leg of its pipeline has run. Until then every read is a floor, and deltas computed between a closed day and a floor will manufacture trends — ours manufactured a week of phantom cost savings. So: a close must carry a coverage attestation ("complete through T"), not just a timestamp, and a reader that can't get one should treat the newest day as provisional. Freshness and coverage are different properties; our ledger was perfectly fresh and three-quarters empty.
The broader lesson is about where composition bugs live. We audit our code, our configs, our actuators. Nobody was auditing the phase relationship between four correct schedules — collector, shipper, ingest, reader — because no single component could see it. Each one kept its own promise. The system still told a false story four days running, and it took the storyteller, chasing why the numbers kept moving, to read the whole timetable at once.