RCA — queue-docker-smoke WP pipeline permanently broken, blocking Phase 1b branch-protection swap

Incident ID: 2026-09-28-queue-docker-smoke-wp-broken Date: 2026-09-28 Severity: SEV-3 (non-blocking today only because a redundant GHA required check masked it; would have become SEV-2 the moment that GHA context stopped reporting) Duration: ~2h5m (diagnosis start to verified-green merge + branch-protection swap) Blast radius: CI/CD reliability only — ci/woodpecker/pr/queue-docker-smoke (the Woodpecker replacement for the retired GHA Queue Docker build + run smoke check). No user-facing impact, no data integrity or revenue impact. Author: sre-agent

Summary

The Woodpecker replacement for the Queue service's Docker build+smoke gate (.woodpecker/queue-docker-smoke.yaml) had been failing on every real run since it was first ported (f526c3e3cf, #4112) — confirmed on the last several release→main promotion PRs (#4586, #4582, #4580). Tracking issue #4495 could not close its Phase 1b branch-protection swap (retired-GHA-context → WP-context) because the WP context itself had never actually gone green. Root-caused via a live failing pipeline (9252) to two independent bugs in the same file: a nonexistent Woodpecker env var reference, and a cross-step state-passing pattern that silently doesn't work in this WP version. Both fixed, verified green on a real PR (#4614), merged to main, and the branch-protection swap completed. The now-redundant GHA workflow was retired on main (PR #4630), matching the retirement already applied on develop.

Timeline (all times UTC)

• 20:15 — Task received: diagnose real WP failure on pipeline 9252 (PR #4586), fix, verify green, then perform the operator-pre-authorized branch-protection swap.
• 20:20 — Read #4495 history: GHA retirement prep already done on develop (PR #4602); main/release still carried the active GHA file; WP check confirmed failing on 3 consecutive real promotion PRs.
• 20:25 — Pulled pipeline 9252 via WP API (AWS SSM /ci/woodpecker/admin-api-token). Isolated failure to workflow queue-docker-smoke, step docker-build-smoke, exit code 2.
• 20:30 — Decoded base64 step log: /bin/sh: CI_BUILD_NUMBER: parameter not set, immediately following set -eu, at the SMOKE_TAG=... $CI_BUILD_NUMBER line. Confirmed via grep that every other .woodpecker/*.yaml file in the repo correctly uses $CI_PIPELINE_NUMBER — this file was the sole outlier, present identically on main, release, and develop.
• 20:45 — Branched off main (fix/queue-docker-smoke-ci-pipeline-number), fixed CI_BUILD_NUMBER → CI_PIPELINE_NUMBER (4 occurrences), opened PR #4614.
• 20:50–22:10 — CI queue backlog: single-agent WP fleet (ci-woodpecker-agents-asg, desired=1) had ~19 pending steps ahead of PR #4614's pipeline. Confirmed each "stuck-looking" step (console-tests, backend smoke-suite) was CPU-bound and genuinely progressing (verified via SSM top showing 90-100% CPU on the actual test/compile process), not wedged — no intervention taken on other PRs' pipelines. Temporarily burst-scaled the WP fleet (spot capacity unfulfillable for r6a.xlarge; switched to on-demand base capacity 1→2) to speed up verification; reverted to baseline (desired=1, on-demand-base=1) once no longer needed.
• 22:31 — Pipeline 9252-equivalent for PR #4614 reached docker-build-smoke: CI_BUILD_NUMBER bug confirmed fixed (script executed past the previously-fatal line), but hit a second, independent bug: ERROR: /tmp/ecr-password is missing or empty. ecr-auth step may have failed. — despite the separate ecr-auth step reporting success and "ECR auth token written to /tmp/ecr-password" moments earlier in the same run.
• 22:35 — Root-caused bug #2: each WP step runs in a fresh container; only the git-checked-out workspace is a shared volume across steps in the same workflow, not /tmp. The ecr-auth/docker-build-smoke file-handoff pattern (introduced to work around WOODPECKER_ENV_FILE being unavailable in WP v3.16, per this file's own comments) never actually worked for values that cross a different-image step boundary — it only appeared to work for the SKIP flag because that variable's safe-default fallback ("false") is indistinguishable from a real read. Confirmed the correct, already-proven pattern for this exact case exists elsewhere in the repo: deploy-queue.yaml's build-container step installs aws-cli via apk and performs the ECR login inline, in the same container that runs the build.
• 22:45 — Folded ECR auth into docker-build-smoke (apk-install aws-cli, login via the agent's IMDS-backed IAM role), removed the now-dead ecr-auth step. Pushed as a second commit to PR #4614.
• 23:09 — docker-build-smoke (cold build, no prior cache — the build had never succeeded far enough to populate the ECR cache) completed successfully. ci/woodpecker/pr/queue-docker-smoke = SUCCESS on PR #4614, confirmed via GitHub's status-check rollup, not just the WP API.
• 22:52 (session time, see note) — Merged PR #4614 to main (admin merge, per established precedent — PR #4602 was merged the same way; enforce_admins: false on main).
• 22:57 — Swapped main branch protection's required_status_checks.contexts: removed "Queue Docker build + run smoke", added "ci/woodpecker/pr/queue-docker-smoke". Verified via GET .../protection.
• 22:58 — Opened and merged PR #4630: retired the now-redundant GHA queue-docker-smoke.yml on main (git mv to workflows-retired/, trigger neutered to workflow_dispatch), matching the develop-side retirement already in place from PR #4602.
• 22:59 — Reverted the temporary CI fleet burst-capacity change back to baseline.

(Note: WP pipeline internal clocks and this session's wall-clock references are UTC-consistent; the exact minute-level timestamps above are reconstructed from WP pipeline created/started/finished epochs and GitHub PR timestamps, not live-typed.)

Impact

• Users affected: none (internal CI/CD only).
• User-visible symptoms: none.
• Data integrity: ok.
• Revenue / billing: ok.
• CI/CD impact: ci/woodpecker/pr/queue-docker-smoke had been silently broken since its introduction (#4112). It never blocked merges because the retired-but-still-required GHA context (Queue Docker build + run smoke) continued to satisfy branch protection. The real risk (flagged by a prior sre-agent pass on 2026-09-28, see #4495 comments) was that once any PR reached main from a state where the GHA file no longer existed (e.g. via develop, where it was already retired in PR #4602), the required GHA context could never report, hard-blocking that merge with no safe fallback. That risk is now closed.

What went well

• The prior sre-agent pass on #4495 (2026-09-28, earlier in the day) correctly identified the live risk and correctly declined to swap branch protection while the WP check was still red — that discipline prevented trading one unblockable check for another broken one.
• deploy-queue.yaml already contained a proven, working pattern (in-step ECR auth via IMDS) that could be directly reused rather than inventing a new one.
• The WP admin API token (AWS SSM /ci/woodpecker/admin-api-token) and AWS SSM access to agent instances made it possible to pull real step-level logs and directly observe CPU activity, ruling out "wedged pipeline" as a false lead before it triggered an unnecessary intervention on someone else's in-flight PR.
• deploy-queue.yaml provided a second, independent confirmation source once the first fix (CI_BUILD_NUMBER) surfaced the deeper /tmp-handoff bug — the fix pattern didn't need to be invented from scratch.

What didn't go well

• The pipeline had been broken since its original port (#4112) and was never caught, because its own failures were masked by the redundant GHA check satisfying branch protection. A check that is "required" in name but never actually gates anything is worse than no check — it hides a real gap.
• The /tmp-file cross-step handoff pattern was introduced as a workaround for WOODPECKER_ENV_FILE unavailability (documented in the file's own comments, referencing ADR-0135) but was never actually validated against a step using a different container image. It happened to look like it worked for the SKIP flag purely because the safe-default fallback masked the untested assumption.
• No CI linter currently catches "shell-defined value written in step A, read in step B with a different image:" as a suspicious cross-step dependency — this is the second time this class of WP gotcha has appeared (see docs/ops/runbooks/ci-woodpecker.md Failure mode E-4 for the sibling ${VAR}-at-parse-time gotcha).

Root cause analysis

• Contributing factor 1 — wrong env var name: .woodpecker/queue-docker-smoke.yaml referenced $CI_BUILD_NUMBER (a Drone-era variable name) instead of Woodpecker's actual built-in $CI_PIPELINE_NUMBER. Under set -eu, referencing an unset variable is a hard failure (exit 2). Every other .woodpecker/*.yaml file in the repo used the correct name — this was a one-off transcription error at the file's original creation, never exercised end-to-end before merge (no dry-run against a real WP pipeline, since the required GHA check made the WP result inconsequential at review time).
• Contributing factor 2 — false shared-state assumption: The file's own comments correctly identified that WOODPECKER_ENV_FILE doesn't exist in WP v3.16, but the chosen workaround (write to /tmp/<name>, read it in a later step) implicitly assumed /tmp is a shared, persistent volume across all steps in a workflow. It is not — each step is a fresh container; only the checked-out workspace is a shared volume. This assumption was never caught in review or CI because the one variable that used this pattern successfully by observation (SKIP) has a safe-default fallback that produces the same value whether the file is actually shared or not, making the bug invisible until a different value (the ECR password, with no safe default) needed to cross the same boundary.
• Systemic factor — required check that never actually gated: Branch protection on main retained the retired GHA context ("Queue Docker build + run smoke") as required specifically because the WP replacement wasn't verified green yet — a reasonable interim safeguard, but one with no expiry or escalation trigger. The WP check silently failed on every real run for weeks without anyone being paged, because its failure had zero effect on mergeability.

Detection

• What alerted us: none automatically — this was found via a dispatched diagnostic task (issue #4495, operator-action label) referencing three consecutive real-PR failures (#4586, #4582, #4580) already observed and documented by a prior sre-agent pass.
• How long between cause and detection: the CI_BUILD_NUMBER bug existed from the file's original merge; unknown total duration since it was always masked by the redundant GHA gate.
• How to detect faster next time: issue #4495 itself already tracked a fast-follow (PR #4602, prior session) — scripts/ci/check_wp_non_required_reliability.py, a weekly sweep that flags WP checks with a sustained (>80% over ≥5 runs) failure rate even when they aren't in required_status_checks.contexts. This exact failure class (a required-in-intent-but-not-in-branch-protection check silently red) is precisely what that sweep is designed to catch going forward.

Resolution

• What was changed: 1. .woodpecker/queue-docker-smoke.yaml — $CI_BUILD_NUMBER → $CI_PIPELINE_NUMBER (4 occurrences: 2 comments, 2 script lines). 2. .woodpecker/queue-docker-smoke.yaml — removed the separate ecr-auth step; folded apk add aws-cli + aws ecr get-login-password | docker login inline into docker-build-smoke, mirroring deploy-queue.yaml's build-container step. 3. main branch protection required_status_checks.contexts: "Queue Docker build + run smoke" → "ci/woodpecker/pr/queue-docker-smoke". 4. .github/workflows/queue-docker-smoke.yml retired on main (moved to workflows-retired/, trigger neutered to workflow_dispatch), matching the existing develop-side retirement.
• Validation:
• ci/woodpecker/pr/queue-docker-smoke = SUCCESS on a real PR (#4614) against main, confirmed via GitHub's status-check rollup (not just the WP API) — full cold Docker build (no prior cache, since the build had never previously succeeded far enough to populate one), binary-exists check, and startup-banner smoke check all passed.
• Post-swap main protection confirmed via GET repos/.../branches/main/protection: contexts are exactly ["ci/woodpecker/pr/ci-pr", "ci/woodpecker/pr/pii-scan", "ci/woodpecker/pr/queue-docker-smoke"].
• Follow-up PR #4630 (retiring the GHA file on main) itself required and passed the newly-swapped ci/woodpecker/pr/queue-docker-smoke check (fast-skip path, since it touched no queue/** files), giving a second, independent green confirmation of the swap actually gating real PRs.

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md (new Failure mode added for this incident)
• Related incidents: docs/incidents/2026-08-10-queue-docker-smoke-ci-commit-before-unset.md (prior bug in the same file's detect-changes step — third latent bug found in this file across two incidents)
• Tracking issue: #4495 ("Retire duplicate GHA queue-docker-smoke.yml in favor of WP equivalent")
• Fix PRs: #4614 (both bug fixes), #4630 (GHA retirement on main)
• Precedent pattern: .woodpecker/deploy-queue.yaml's build-container step (in-step ECR auth via IMDS)