RCA — Woodpecker CI crash loop via systemd Requires= dependency

Incident ID: 2026-07-11-woodpecker-restart-loop Date: 2026-07-11 Severity: SEV-2 Duration: 5h 47m (08:00:02 UTC — first loop iteration; 13:47:03 UTC — service stable) Blast radius: All CI pipelines blocked. No webhook-triggered builds ran from 08:00 to 13:47 UTC. ALB returned 502 for all ci.moosequest.net requests during the window. No data loss. No Woodpecker database writes during loop (container never reached serving state). Author: sre-agent

Summary

At 08:00:02 UTC the woodpecker-restart.timer fired its daily restart of Woodpecker CI. The woodpecker-restart.service unit contained Requires=woodpecker.service, which caused systemd to SIGTERM the restart process the moment it internally stopped woodpecker.service (the first half of systemctl restart). Systemd then re-queued the start of woodpecker.service and attempted to restore woodpecker-restart.service to active, restarting the loop every ~2 seconds. The WP container never remained up long enough for the ALB health check to pass. The healthcheck edge-triggered one alert email at ~08:15 UTC. The operator acted on the alert at ~13:36 UTC. The fix — removing Requires=woodpecker.service from the restart.service unit — broke the loop; the service was stable within 10 seconds.

Timeline (all times UTC)

• 08:00:02 — woodpecker-restart.timer fires; woodpecker-restart.service starts
• 08:00:02 — woodpecker-restart.service killed (SIGTERM) due to Requires=woodpecker.service stop-propagation
• 08:00:03 — woodpecker.service starts (queued restart from partial systemctl restart command); loop begins
• ~08:15 — GHA healthcheck (woodpecker-healthcheck.yml) detects unhealthy; sends edge-triggered email to kris@moosequest.net
• 13:36 — Operator acts on alert; reports 502 from ALB
• 13:37 — sre-agent engaged; SSM diagnostics begin; pre-fix state captured
• 13:37 — Root cause identified: Requires=woodpecker.service crash loop; loop confirmed running since 08:00
• 13:46 — systemctl stop woodpecker-restart.timer woodpecker-restart.service woodpecker issued; loop halted
• 13:46 — sed -i '/^Requires=woodpecker.service/d' applied to unit file; daemon-reload run
• 13:47:03 — systemctl start woodpecker completed; container up; /healthz returns 200
• 13:47:08 — woodpecker-restart.timer re-enabled; next trigger: 2026-07-12 08:00:00 UTC
• 13:47 — External curl https://ci.moosequest.net/api/version returns 200; ALB recovery confirmed

Impact

• Users affected: 0 (internal CI only)
• User-visible symptoms: none (CI is internal)
• CI pipelines blocked: all push/PR webhook triggers from 08:00–13:47 UTC (~5h 47m)
• Data integrity: ok
• Revenue / billing: ok
• Agent cost (EC2): t4g.small running during the loop — billed normally, no spike

What went well

• SSM Session Manager access worked immediately; no SSH key dependency
• Root cause was identifiable from journalctl within one SSM round-trip
• The fix was a one-line sed -i with zero restart downtime beyond the ~10s clean restart
• The woodpecker-healthcheck.yml edge-trigger fired correctly (one email, no spam)
• wp-config-svc was confirmed not involved: 34 MiB / 1.8 GiB limit, 0% CPU, healthy throughout
• No OOM — the t4g.small has adequate headroom (1.2 GiB available after fix)

What didn't go well

• The woodpecker-restart.service unit was deployed with Requires=woodpecker.service, an incorrect dependency that makes the restart self-defeating. This was not caught by review.
• The unit files were deployed as a live-patch directly to the instance (via SSM on 2026-07-10), without a corresponding IaC change. A future server rebuild would have reproduced the bug.
• The loop ran for 5h 47m before action was taken. The alert email fired at ~08:15 but was not acted on until 13:36 — a 5h 21m response gap for a SEV-2 CI outage.
• The healthcheck only sends one email at unhealthy→healthy transition. There is no escalation or repeat alert path for sustained outages.

Root cause analysis

• Contributing factor 1: Requires=woodpecker.service in woodpecker-restart.service — The direct cause. systemctl restart X = stop X + start X. During the stop phase, Requires=woodpecker.service propagates the deactivation to woodpecker-restart.service, killing its ExecStart process with SIGTERM before the restart can complete. After=woodpecker.service provides equivalent boot-ordering semantics without the stop-propagation side effect.

• Contributing factor 2: Unit files deployed as live-patches without IaC — The woodpecker-restart.service and .timer units were installed directly on the running EC2 instance on 2026-07-10 and never added to infra/ci/templates/user_data_server.sh.tpl. The IaC and the instance were out of sync. A rebuild would have silently lost both the timer AND the fix applied today.

• Contributing factor 3: No smoke test for the daily restart before the next scheduled fire — A test run of systemctl start woodpecker-restart.service immediately after deploying the unit would have revealed the loop on 2026-07-10, not at 08:00 on 2026-07-11 during production.

• Contributing factor 4: Single-shot healthcheck alert with no escalation path — The GHA healthcheck sends one email per degraded event. There is no paging, no re-alert after N minutes, and no Slack notification. The alert at 08:15 was unactioned for 5+ hours.

Detection

• What alerted us: GHA woodpecker-healthcheck.yml — edge-triggered email to kris@moosequest.net
• Time between cause and detection: ~13 minutes (08:00 → ~08:13 healthcheck run → email)
• Time between detection and action: ~5h 21m
• How to detect faster next time: add a 30-minute re-alert if the woodpecker-health anchor issue remains open (or add Slack notification alongside email for ops-channel visibility)

Resolution

• What was changed: Removed Requires=woodpecker.service from /etc/systemd/system/woodpecker-restart.service via sed -i on i-082ee835595d90ae0. systemctl daemon-reload run. systemctl start woodpecker completed cleanly. Timer re-enabled; next fire is 2026-07-12 08:00:00 UTC.
• IaC fix: Added section 9 to infra/ci/templates/user_data_server.sh.tpl with the correct unit files (no Requires=). Main checkout requires the same patch on the next merge.
• Validation: curl https://ci.moosequest.net/api/version → 200. systemctl status woodpecker → active (exited) with no cycling. Journalctl shows no new woodpecker-restart.service starts after the fix was applied. Timer is active (waiting) for next trigger.

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md (Failure mode J)
• Prior incident: docs/incidents/2026-07-04-waf-webhook-block-batch1-parse-error.md
• Instance: i-082ee835595d90ae0, us-east-2, account 521228113048
• IaC fix: infra/ci/templates/user_data_server.sh.tpl section 9