RCA — CI spot-fleet wedge: 5h silent outage, scale_arbiter blind guard

Incident ID: 2026-07-13-ci-spot-wedge-silent-outage Date: 2026-07-13 Severity: SEV-2 Duration: ~289m total (00:41 UTC detection → 05:30 UTC resolution) Blast radius: All Woodpecker CI pipelines; no Raptor production impact Author: sre-agent

Summary

A spot interruption killed the sole CI agent at ~00:41 UTC. The ASG kept desired=1 but could not place a replacement spot in the AZ. The scale_arbiter Lambda had a three-way scale-up guard (pending > 0 and workers == 0 and current_desired == 0); with current_desired stuck at 1, the condition was permanently false and the arbiter did nothing for ~5 hours. No alert existed for "queued work with zero workers," so the outage was silent until manually discovered. Fixed by removing the current_desired == 0 guard, adding stateless wedge detection and escalation, and adding the WoodpeckerNoWorkersWithBacklog Prometheus alert with a 10-minute window.

Timeline (all times UTC)

• 00:41 — Spot instance interrupted; sole WP agent terminated. ASG desired stayed at 1. ASG began attempting to replace spot in same AZ — requests not fulfilled (capacity unavailable).
• 00:43 — Arbiter tick: pending > 0, workers == 0, current_desired == 1 — guard current_desired == 0 is false. No action taken. (Repeated every 2 min for ~5h.)
• ~05:25 — Outage discovered manually (no alert fired).
• 05:30 — Operator manually set ASG desired=0, then desired=1. A spot was placed in a different AZ. Agent registered. Pipelines drained.
• 05:30 — Resolved.

Impact

• Users affected: 0 (CI is internal)
• User-visible symptoms: none directly; PRs accumulated unprocessed CI checks
• Data integrity: ok
• Revenue / billing: ok

What went well

• Scale-down and cold-start paths were unaffected; no runaway costs.
• The ASG did not go into an error loop; it silently kept trying spot requests without consuming quota.
• Resolution was fast once noticed (manual desired toggle < 5 min).

What didn't go well

• No alert for "work queued but zero workers for > N minutes" — the canonical signal for this failure class.
• The scale_arbiter's guard current_desired == 0 was semantically incorrect: it conflates "ASG is idle" with "no action needed." These are different states.
• The arbiter had no escalation path for the post-interruption recovery scenario; it only handled the clean cold-start case.
• Detection gap was ~5 hours. The system allowed this because the arbiter's logging ("NO ACTION") was indistinguishable from normal work-in-progress ticks.

Root cause analysis

• Contributing factor 1: current_desired == 0 guard in scale_arbiter — The scale-up branch required all three conditions: pending > 0, workers == 0, AND current_desired == 0. After a spot interruption, current_desired remains at its last set value (1). The guard was permanently false even though the real state (no instances, no workers, work queued) warranted immediate action. The system allowed a misleadingly-named "scale up" condition to also encode "ASG was fully idle," conflating two independent states.

• Contributing factor 2: No "queued work, zero workers" alert — The existing WoodpeckerPipelineQueueStuck alert fires on woodpecker_pending_pipelines > 0 for 30m. This fires correctly when agents ARE connected but pipelines don't drain. It does NOT capture the case where there are no agents at all — which is the more urgent failure mode. The alert gap left a 5h window with no automated notification.

• Contributing factor 3: No ASG instance-health inspection in the arbiter — The arbiter queried the WP queue (/api/queue/info) and the ASG desired capacity, but never inspected the ASG Instances list. Checking whether any instance is InService or launching would have made the wedge state immediately observable (desired=1, Instances=[]).

Detection

• What alerted us: manual discovery (~05:25 UTC)
• How long between cause and detection: ~5h (289 min)
• How to detect faster next time: WoodpeckerNoWorkersWithBacklog alert (now deployed) fires at 10 min. Would have fired at 00:51 UTC.

Resolution

• Operator toggled ASG desired 0→1; spot placed in different AZ; agent registered; pipelines drained.
• Permanent fix deployed via PR #4210 (merged to develop):
• scale_arbiter.py: removed current_desired == 0 guard; added wedge detection (inspect Instances list) + escalation ladder (bump desired → instance refresh).
• lambda.tf: added autoscaling:StartInstanceRefresh IAM permission.
• cloud-init.sh.tpl: added WoodpeckerNoWorkersWithBacklog alert, severity: critical, for: 10m.
• test_scale_arbiter.py: 13 unit tests including regression test for this exact wedge state.
• Hot-applied alert to running monitoring box (i-0a656c6631ddc45b3) via SSM Run Command; Prometheus reloaded, rule confirmed loaded via /api/v1/rules.

Action items

References

• PR: https://github.com/raxx-app/TradeMasterAPI/pull/4210
• Runbook: docs/ops/runbooks/ci-monitoring.md
• Lambda logs: /aws/lambda/ci-scale-arbiter (CloudWatch, us-east-2)
• Monitoring instance: i-0a656c6631ddc45b3 (us-east-2)