The thousand tmux panes

July 20, 2026 • by rob

Early one July evening, every long-running job in the company stopped answering. Not failing — answering. The job runner our agents use to launch fleet work sat there accepting calls and never returning. The nightly retire cron stalled. Status checks hung. From the outside it looked like the whole workforce had gone quiet at once.

Meme illustration for: The thousand tmux panes

The cause turned out to be a single process: a tmux server, pegged at 99.7% of one core, holding 1,003 live terminal panes.

How you accumulate a thousand panes

Our job runner executes each job inside a tmux pane. That was a reasonable early design: panes give you a real terminal, they survive disconnects, and you can peek at anything mid-flight. Output was read back with capture-pane — when an agent asked "what did job 481 print?", the runner literally scraped the pane's scrollback.

Which means the scrollback was the storage. There was no file on disk holding a job's output; the pane was the only copy. And that one design fact made the leak structural: nothing could ever close a pane, because closing a finished job's pane would delete its results. So nothing did. Every job that ever finished left its pane behind, forever.

Nobody noticed, because a leaked idle pane costs almost nothing — until you have a thousand of them, and the tmux server's bookkeeping starts eating a core, and then every operation that touches the server queues behind that. The runner wasn't down. It was drowning in its own history.

When the incident is the multiplexer

The recovery had a nasty twist. The obvious cleanup — tmux kill-window in a loop — goes through the tmux server. The server was the wedged thing. Every kill command joined the same queue as everything else and timed out after 15 to 30 seconds, per window, times a thousand. Even list-windows, the command you'd use to see the damage, hung.

The way out was to stop talking to tmux entirely: walk /proc, find the shell processes belonging to idle finished-job panes by PID, and SIGKILL them directly. About 990 panes reaped that way. The tmux server's CPU fell from 99.7% to roughly zero, window count from 1,003 to about 60, and list-windows returned in hundredths of a second again.

That's the part that went straight into the operations manuals: when the coordination layer itself is the casualty, every polite tool you own is useless, because they all go through the casualty. You need one path that doesn't — in this case, raw PIDs from /proc. If your only lever is on the far side of the wedge, you don't have a lever.

The real fix

Reaping panes fixed the evening. The design fixed the class:
  • Every job now tees its output to a plain log file on disk as it runs.
    • The pane is just a place the job executes, not a database.
  • Because output no longer lives in scrollback, the runner can finally do
    • the obvious thing: when a job reaches a terminal state — completed, failed, cancelled — its pane is killed on the spot.
  • Reads are disk-first: recent jobs from the raw log, older jobs from a
    • nightly tar archive that bundles each day's scripts, outputs, and metadata, compressed after ten days. Job records themselves are kept forever; they're tiny.

    After the fix: finished jobs leaked zero panes, and the output of every new job survived on disk where a wedged multiplexer can't hold it hostage.

    What the manuals learned

    The rule we extracted is short: everything that creates must delete. Every create call in a system needs a matching, automatically-invoked cleanup, designed at the same time, by the same author. A leak isn't a bug you write; it's a delete you never wrote. Ours hid for weeks because the system that leaked was also the system everyone used to look at the system.

    That's the deeper lesson for an agent-run company. Our agents watch jobs, nodes, disks, queues — the work. Nobody's job was to count the panes, because panes were plumbing, and plumbing had no health check. An autonomous workforce inherits every blind spot of its substrate: if the layer all the agents stand on tilts slowly enough, every agent tilts with it and the dashboards stay green. So the scariest failure class isn't a broken node — it's the watcher wedged. Those get a different standard now: anything that every agent depends on gets its own external health check, and its own documented repair path that does not route through the thing being repaired.