docs/deploy/dev-plane-restart-hygiene.md
Operational runbook for restarting a dev/shared Paperclip control plane without killing in-flight agent work. Written after the 2026-07-06/07 failure spike, where a restart-heavy deploy window was the single largest source of failed tasks.
Every control-plane restart hard-kills any heartbeat run in flight at that moment. The run is finalized as failed with error_code = 'process_lost'. There is an automatic single retry (process_lost_retry), but it is best-effort: in the 07-06/07 incident window only 12 of 36 lost runs recovered. On the evening of 07-06, 9 restarts in 90 minutes killed 16 in-flight runs.
Batch your deploys. Stack up changes and restart once, instead of restart-per-change. A restart storm (several restarts within an hour) multiplies run loss for near-zero benefit.
Restart during low fleet activity. Check for active runs before restarting; prefer windows where the fleet is quiet. A quick check against the control-plane DB:
SELECT count(*) FROM heartbeat_runs WHERE status = 'running';
Drain before restart (upcoming). Graceful SIGTERM drain — stop accepting new runs, let in-flight runs finish or checkpoint, then exit — is being added in PAP-12930. Once it lands, send SIGTERM and wait for drain instead of hard-restarting. Until then, rule 2 is your drain.
After any restart, glance at the damage. See the detection queries below; confirm lost runs either retried successfully or get manual follow-up.
Two signals, cross-referenced:
1. Server start markers in the instance log. Each boot logs Server listening on <host>:<port>. Logs live at ~/.paperclip/instances/<instance>/server.log, rotated daily to server.log-YYYYMMDD.gz.
grep -h "Server listening" ~/.paperclip/instances/default/server.log
zgrep -h "Server listening" ~/.paperclip/instances/default/server.log-20260706.gz
Many markers minutes apart = restart burst.
2. Same-minute process_lost clusters in heartbeat_runs. Runs killed by a restart are finalized together, so they cluster on the same minute:
SELECT date_trunc('minute', finished_at) AS minute, count(*)
FROM heartbeat_runs
WHERE error_code = 'process_lost'
GROUP BY 1 HAVING count(*) > 1
ORDER BY 1 DESC;
If the cluster minutes line up with the Server listening timestamps, the failures are restart-inflicted, not a product regression.
Note: query the DB directly for this — the /heartbeat-runs list API ignores the status= filter and omits error fields.
Each process_lost run should have triggered one retry wake (reason = 'process_lost_retry'). To find lost runs that never recovered, look for process_lost failures with no subsequent successful run for the same issue, and re-wake or reassign those issues manually.
Seen in the same incident window; do not confuse them with restart damage:
workspace_validation_failed — deterministic workspace validation retry loops (self-heal: PAP-12931).claude_transient_upstream failures (quota-aware handling: PAP-12932).