RCA — Woodpecker CI server replace (pip-cache activation) ran ~22min vs 3-5min accepted downtime

Incident ID: 2026-08-07-woodpecker-server-replace-pip-cache-activation Date: 2026-08-07 Severity: SEV-2 (deploy pipeline blocked — CI fully down for part of the window) Duration: ~23m total (destroy of old instance ~06:20 UTC → first pipeline resumed executing 06:42:46 UTC) Blast radius: Woodpecker CI (ci.moosequest.net) — no pipelines could run or complete during the window. No customer-facing production impact (CI/build system only). Author: sre-agent

Summary

Issue #4416 authorized a standalone terraform apply to replace the Woodpecker CI server instance and activate the PyPI cache wiring (docs/ops/runbooks/ci-build-cache.md), accepting an estimated ~3-5 minute downtime and the known risk that the out-of-band wp-config-svc container would be re-provisioned from the Terraform template. The replace itself (terraform apply -target=aws_instance.woodpecker_server) completed cleanly in under 2 minutes, but three post-replace validation checks each surfaced a gap that the plan/apply step could not have shown, extending real downtime to ~23 minutes: (1) the boot template lacked an ECR login step needed to pull the private wp-config-svc image, so the server itself crash-looped; (2) the -target-scoped apply correctly avoided ~23 unrelated pending resources but also skipped the ALB target-group re-attachment, leaving the public URL unreachable after the server itself came up; (3) it also skipped the CI agent launch template, leaving the running agent instance permanently pointed at the destroyed instance's dead private IP, so queued pipelines sat pending indefinitely. All three were fixed forward during validation. Final state: server healthy, wp-config-svc healthy and functionally equivalent to its pre-replace out-of-band config, agent fleet reconnected, and the pip cache is confirmed active — live devpi-server access logs show real pip install traffic from a pipeline run hitting the cache (/root/pypi/+simple/... and /root/pypi/+f/... sub-second responses).

Timeline (all times UTC)

• 05:50–06:06 — Prior authorization exchange on issue #4416 (see docs/incidents/ cross-reference below); operator authorization posted under the operator's own GitHub identity (comment 5213132551), satisfying the AC.
• 06:19:39 — Pre-flight check: zero pending/running Woodpecker pipelines. Cleared to proceed.
• 06:19:39–06:21:15 — terraform apply -target=aws_instance.woodpecker_server (plan confirmed as a single-resource replace, forced by the user_data hash change that adds wp-config-svc + pip-cache env vars). Old instance i-082ee835595d90ae0 destroyed, new instance i-0563f92028b656333 created (LaunchTime 06:20:54).
• 06:22:48 — First boot failure: woodpecker.service crash-loops. docker compose up fails with no basic auth credentials pulling raxx-wp-config-svc:latest from the private ECR repo — the boot template never performs aws ecr get-login-password | docker login; the prior out-of-band container had been logged in manually/separately, outside the templated boot flow.
• 06:26:04 — Fix-forward #1: docker login to ECR via aws ecr get-login-password (short-lived IAM token), systemctl restart woodpecker. Both woodpecker-server and wp-config-svc containers come up healthy.
• ~06:27–06:39 — Discovered fix-forward #2: aws elbv2 describe-target-health on ci-woodpecker-server-tg showed zero registered targets — the -target=aws_instance.woodpecker_server apply correctly avoided the ~23 unrelated pending resources but also excluded aws_lb_target_group_attachment.woodpecker_server, a direct dependent whose target_id still pointed at the destroyed instance's ID. Applied terraform apply -target=aws_lb_target_group_attachment.woodpecker_server (1 to add — Terraform recreates the attachment resource since target_id is not updatable in place). Target health reported healthy shortly after; https://ci.moosequest.net/api/user returned 200 with valid admin session.
• 06:36:59 — Triggered a manual pipeline on develop (POST /api/repos/1/pipelines) to validate the golden path end-to-end. Pipeline queued as pending — no agent claimed it.
• ~06:39 — Discovered fix-forward #3: GET /api/agents showed no recent last_contact; infra/ci/agents.tf bakes aws_instance.woodpecker_server.private_ip into the agent launch template's user_data at apply time (static-mode server discovery, ADR- 0136). The running agent instance (i-05d83ba5fa5f0734b, launch template version 5) was still configured with the destroyed instance's private IP (10.42.10.143) instead of the new one (10.42.10.157) — this LT was also outside the -target scope. Applied terraform apply -target=aws_launch_template.woodpecker_agent (updates WOODPECKER_SERVER_HOST in the template, bumps to LT version 6). The ASG's instance_refresh block does not auto-trigger from a bare LT content update when version = "$Latest" is a literal string with no diff on the ASG resource itself, so manually started aws autoscaling start-instance-refresh (0% healthy tolerated — min_size=1 on-demand floor, single instance).
• 06:42:43–06:42:46 — New agent instance (i-05863a9e4985b7d31, LT version 6) boots, connects to the new server IP, woodpecker-agent.service healthy.
• 06:42:46 — Pipeline #4832 transitions from pending to running — agent immediately claims the queued work. Golden path restored.
• 06:49:01–06:52:05 — Confirmed via devpi-server access logs (journalctl -u devpi-server) on ci-monitoring: real pip install traffic from the running pipeline hitting /root/pypi/+simple/... and /root/pypi/+f/... with sub-second response times — matches the runbook's documented cache-hit signature.
• 06:55 — Spot-checked one pipeline failure (deploy-console-shim) to rule out replace-caused regressions: confirmed pre-existing/unrelated (Node 20.20.2 runtime vs wrangler@4.87.0 requiring Node ≥22 — a pipeline definition issue, not infra).
• lifecycle { ignore_changes = [user_data] } guard re-added to infra/ci/server.tf immediately after the instance apply, per the runbook's documented pattern. Final untargeted terraform plan confirms zero drift on all three resources touched (aws_instance.woodpecker_server, aws_lb_target_group_attachment.woodpecker_server, aws_launch_template.woodpecker_agent); remaining plan diff (16 to add, 10 to change) is exactly the pre-existing, explicitly out-of-scope backlog (#4140 config-svc SSM/ECR, alarms, ASG/IAM/Lambda) — untouched, as directed.

Impact

• Users affected: none (internal CI/build system, no customer-facing surface).
• User-visible symptoms: none.
• Data integrity: ok — stateless server (RDS-backed), no data loss; old instance's EBS root volume was delete_on_termination = true by design (immutable infrastructure pattern), nothing needed preserving beyond the config already captured pre-replace.
• Revenue / billing: ok — no billing-affecting systems touched.
• CI impact: ~23 minutes where no Woodpecker pipeline could execute (queued but not claimed) or where the server/UI was unreachable. One in-flight manual trigger (#4832, a full monorepo multi-workflow run including several deploy-* workflows) was queued during this window and resumed automatically once the agent reconnected — no pipeline was lost or corrupted, only delayed.

What went well

• The pre-flight check (zero pipelines running before the apply) held — no in-flight work was interrupted mid-run by the instance destroy.
• The -target-scoped apply behaved exactly as planned for the one resource it was scoped to: single clean replace, zero collateral changes to the ~23 unrelated pending resources in the same root, as instructed.
• Pre-replace state capture (compose file, docker ps, docker inspect wp-config-svc) meant the post-replace comparison was fast and precise — every field of the re-provisioned wp-config-svc container could be checked against a known-good baseline rather than guessed at.
• All three fix-forwards were low-risk, additive, and reversible (ECR login token is a 12h IAM-issued credential, not a stored secret; the two Terraform applies were themselves scoped, single-resource, and immediately verified).
• The ignore_changes lifecycle guard was correctly re-added immediately after the apply, preventing future unrelated terraform apply runs from proposing an unintended second replace.
• Final validation used independent, hard-to-fake evidence (devpi server's own access log showing real request traffic) rather than trusting the pipeline step's own --quiet pip output.

What didn't go well

• The accepted-risk estimate (~3-5 min downtime) assumed the template-driven boot would succeed on the first attempt. It didn't: the out-of-band wp-config-svc container that had been running for 3 weeks was never fully represented in the template's boot sequence — specifically, whatever process started it originally included an ECR login step that the Terraform-managed user_data_server.sh.tpl does not perform. This gap was invisible to terraform plan (plan only diffs declared resources, not boot-script runtime success).
• -target scoping a single-resource replace correctly avoided unrelated drift but also silently excluded two resources that are direct, functional dependents of the replaced instance (aws_lb_target_group_ attachment.woodpecker_server, whose target_id references the instance ID directly, and aws_launch_template.woodpecker_agent, whose user_data bakes in the instance's private IP). Neither gap surfaced in the -target plan output because -target plans only show the targeted resource by design — there was no automated check flagging "this instance has N direct dependents outside the target set."
• The agent ASG's instance_refresh block does not auto-trigger from a launch-template content update alone (when version = "$Latest" is a literal, unchanged string, Terraform sees no diff on the aws_autoscaling_group resource itself) — this required a manual aws autoscaling start-instance-refresh that isn't documented anywhere as part of the standalone-replace procedure.
• Validating the golden path via POST /api/repos/1/pipelines triggered the entire monorepo pipeline definition (60+ workflows, including several real deploy-* workflows), not a narrow smoke test — a higher blast radius than intended for a cache-verification step. It happened to be low-risk here (all triggered deploys targeted the already-deployed tip of develop, making them idempotent re-deploys of currently-live code), but that was incidental, not by design.

Root cause analysis

• Contributing factor 1 — undocumented boot dependency: the live wp-config-svc container was provisioned out-of-band at some point outside the Terraform-managed boot flow, and whatever process started it performed an ECR login that was never captured back into user_data_server.sh.tpl. The system had no mechanism to detect that the template's declared behavior ("pull raxx-wp-config-svc:latest from ECR") required a precondition ("be authenticated to ECR") that the template itself never established.
• Contributing factor 2 — -target scoping has no dependent-resource awareness for this pattern: the runbook's standalone-replace procedure (remove ignore_changes → plan → apply → re-add guard) was written before this resource graph grew a target-group attachment and an agent launch template that both depend on the instance's identity/IP rather than on a stable reference (e.g., an Elastic IP, a DNS-based discovery mechanism, or the ASG-of-1 + SSM-discovery mode that already exists in this same file for the server_use_asg=true path but is not the active mode). The procedure doesn't enumerate "resources that must be re-applied alongside the instance replace," so each dependent had to be discovered live, one at a time, during validation.
• Contributing factor 3 — instance refresh trigger semantics are a known Terraform/AWS provider gotcha (refresh triggers on a change to the aws_autoscaling_group resource's own arguments, not on the referenced launch template's content when the version pointer is the literal string "$Latest") that wasn't previously exercised by this specific replace path and so wasn't caught in the runbook.

Detection

• What alerted us: no automated alert — this was caught entirely through the post-replace validation steps mandated by the runbook and the task's explicit validation checklist (WP healthy, agents reconnect, pip-cache active via a real pipeline run), not through any monitoring signal.
• How long between cause and detection: near-zero for the boot failure (systemd's own restart-loop was visible within ~2 minutes of the apply); near-zero for the ALB/agent gaps once the validation checklist was followed step by step — but all three gaps would have been invisible to a "the instance shows running in AWS" check alone.
• How to detect faster next time: none of this needs a new dashboard so much as a documented, mandatory post-replace checklist that includes ALB target health and agent reconnection as explicit gates before declaring the replace successful — see action items below.

Resolution

• What was changed: 1. docker login to the private ECR repo on the new server instance (aws ecr get-login-password | docker login), then systemctl restart woodpecker. 2. terraform apply -target=aws_lb_target_group_attachment.woodpecker_server to re-register the new instance ID with the ALB target group. 3. terraform apply -target=aws_launch_template.woodpecker_agent to bake the new server private IP into the agent launch template (LT version 5 → 6), followed by a manual aws autoscaling start-instance-refresh on ci-woodpecker-agents-asg to roll the single on-demand-floor agent onto the new version.
• Validation:
• GET /api/user on https://ci.moosequest.net returns 200 with a valid admin session (WP server + ALB routing confirmed).
• docker inspect wp-config-svc post-replace vs. pre-replace capture: same image, same WP_CSVC_SHADOW_MODE=true, same bind host/port, same WP_CSVC_WP_ADMIN_TOKEN value; WP_CSVC_GITHUB_PAT changed from PLACEHOLDER to a real value now that /ci/config-svc/github-pat is populated in SSM (a pre-existing SSM state difference, not caused by this replace — the old container just predated that SSM param being set).
• docker inspect on a live pipeline step container confirms PIP_INDEX_URL=http://10.42.10.76:3141/root/pypi/+simple/ and PIP_TRUSTED_HOST=10.42.10.76 are injected via WOODPECKER_ENVIRONMENT into every step, as designed.
• journalctl -u devpi-server on ci-monitoring shows real GET /root/pypi/+simple/<pkg>/ and GET /root/pypi/+f/<hash>/<wheel> requests during the pipeline's execution window (06:49:01–06:52:05), with sub-second response times consistent with warm-cache serving — the same cache-hit signature documented in the runbook's registry-mirror verification section, now demonstrated for the PyPI cache.
• Agent ASG instance refresh completed successfully (100%); new agent instance i-05863a9e4985b7d31 InService/Healthy on LT version 6.
• Final terraform plan (untargeted) confirms zero drift on all three resources touched by this incident; remaining plan diff is exactly the pre-existing, explicitly out-of-scope backlog (config-svc SSM/ECR from #4140, alarms, ASG/IAM/Lambda — untouched per operator instruction).

Action items

References

• Runbook: docs/ops/runbooks/ci-build-cache.md
• Authorization: issue #4416, comment https://github.com/raxx-app/TradeMasterAPI/issues/4416#issuecomment-5213132551
• Prior authorization-chain near-miss on the same card (separate concern, already resolved before this incident): issue #4424
• Terraform config: infra/ci/server.tf, infra/ci/agents.tf, infra/ci/alb.tf