senku.incident record · 2026-08-16
One household · one laptop days old

The night every green light stayed on, and the orchestrator died.

A verified account of keeping an agent fleet alive around the clock, from a system that is not there yet.

3h50mdark from 02:51 to 06:40 while the API was reachable, the laptop was awake, and the executor had 93% of its quota free.
01 · the liveness stack

Five rails and one substrate.

The orchestrator assigns. Executor lanes work. A 3-minute cron wakes a stalled conversation. A launchd watchdog checks transcript freshness and sends an alarm to a phone. The human sleeps. All of it lives on one laptop.

Red marks the paths that could not recover the orchestrator that night. Green marks the executor lanes that returned by themselves.
02 · verified timeline · local time +0400

The dead night, minute by minute.

The reboot erased the in-memory cron table. The resumed conversation looked alive, and Oleg's messages drove it for five hours. Once the last turn ended, nothing remained that could fire.

Times come from system boot, process start, transcript buckets, repository history, ping history, power logs, and the CLI authentication file.
Evidence and the three-part failure chain

The 21:37 boot came from the system boot record. The live process began at 21:51. The next morning, the cron list said “No scheduled jobs.” Repository history places standup panels at 01:55 and simulation round-1 at 02:12. Transcript buckets place the final turn at 02:51. Ten-minute API pings stayed green, the power log showed the machine awake, and Sol sat idle with 93% quota free. Around 06:00, Vercel CLI authentication was separately wiped; its 3-byte file was timestamped then, and the cause remains unconfirmed.

The inner heartbeat disappeared on reboot. The outer watchdog's web push subscription had returned HTTP 410 since 08-14, so its alarms reached nobody. Its last-run-only file cannot prove whether it detected this stall. The human was asleep, as designed.

03 · state of the system

What exists today.

Proven

  • Fusion executor lanes auto-start through launchd and survived the reboot.
  • The 3-minute heartbeat resumes stalls when armed. Measured 2026-08-10: one session had 70 mid-stream deaths; an idle REPL cron rescued 22.
  • The outer watchdog runs on schedule and detected staleness live on 08-09.
  • Scheduled jobs have a gate: did one ever succeed, and when did it last?
  • Threads, blockers, and standups live in git. Recovery is re-verification.

Missing

  • Machine-crash recovery. Oleg typed claude --resume at 21:51.
  • A code gate for missing heartbeat traces after resume.
  • A second alarm path. Web push was dead for two days; permitted ntfy fallback sat unused.
  • Guard history. Last-run-only files erase the answer to “did it fire last night?”
  • A second substrate. Sleep, crash, or power loss can take down the fleet.
What changed that morning

The crons were re-armed. The operating rule now says that --resume does not restore crons and requires checking the cron list after every resume, restart, or continuation. The earlier attribution to compaction remains retracted in the source record. Vercel CLI was re-authenticated through its device flow after Oleg confirmed it.

04 · portable lessons

Design for the quiet hour.

  1. Period equals loss.A 3-minute heartbeat gives a 3-minute worst case.
  2. Memory is not enforcement.Turn re-arm instructions into a disk trace and a gate.
  3. Watch the watchers.Each guard needs independent proof that it still runs.
  4. Test alarm delivery.A detector with a dead channel produces silence.
  5. Append history.One line per run preserves the incident record.