RCA — stale-branch-guard hardcoded to develop, false-failing every main-based hotfix PR

Incident ID: 2026-07-25-ci-pr-stale-branch-guard-hardcoded-base Date: 2026-07-25 Severity: SEV-4 (CI/CD pipeline failure blocking a deploy path; no user-facing impact by itself, but blocked a P0 prod hotfix's ci-pr gate) Blast radius: Any PR whose base branch is main or release (e.g. every ADR-0115 emergency-hotfix PR). Confirmed impact: PR #4346 (console Billing Summary prod hotfix). Author: sre-agent

Summary

The stale-branch-guard step in .woodpecker/ci-pr.yaml hardcoded GITHUB_BASE_REF: develop instead of deriving the PR's actual base branch, so it always compared the branch's freshness against origin/develop regardless of what the PR was actually targeting. Any PR based on main (the documented ADR-0115 emergency-hotfix path) is structurally guaranteed to fail this gate, because develop is always far ahead of main by design (that's the entire point of the develop→release→main promotion model). This was discovered while diagnosing why ci/woodpecker/pr/ci-pr was red on hotfix PR #4346, whose diff was a 3-line middleware exemption plus a test and had nothing to do with branch staleness.

Timeline (all times UTC)

• 14:32 — ci/woodpecker/pr/ci-pr reports FAILURE on PR #4346 (pipeline 3032, workflow 1, step stale-branch-guard, exit code 1).
• ~14:45 — sre-agent pulls WP pipeline detail via /api/repos/1/pipelines/3032; isolates failing step to stale-branch-guard.
• ~14:48 — sre-agent reads .woodpecker/ci-pr.yaml step config: GITHUB_BASE_REF: develop is a static, hardcoded environment value (not derived from the PR's real base).
• ~14:50 — sre-agent computes fork-point diff manually: PR #4346's head is 0 commits / 0h behind its real base (origin/main), but 27 commits / 89h behind origin/develop — both over the guard's thresholds (10 commits / 48h).
• ~14:52 — sre-agent confirms PR #4346's diff does not touch .woodpecker/ci-pr.yaml (gh pr diff --name-only), and confirms the hardcoded line is unchanged on main since commit a1a16b8be (2026-07-11) — pre-existing, not introduced by this PR.
• ~14:53 — sre-agent confirms ADR-0115's own migration plan (docs/ops/runbooks/ci-woodpecker.md §CI migration) explicitly specifies stale-branch-guard should compare against the PR's actual GITHUB_BASE_REF, not a hardcoded develop — this is an implementation gap relative to the ADR's own design, not an intentional policy.
• ~14:55 — Root cause confirmed unrelated to PR #4346's content; proceeding to merge per FIX-mode authorization, but see companion escalation for a separate, larger blocker (main-protection ruleset).
• ~15:05 — sre-agent applies the fix on a develop-targeted branch (fix/ci-pr-stale-branch-guard-base-ref), mirroring the pattern already used by the b1-migration-path-lint / flag-promotion-check steps in the same file (${CI_COMMIT_TARGET_BRANCH:-develop}).

Impact

• Users affected: none directly.
• User-visible symptoms: none directly — but this bug independently delayed the console Billing Summary prod hotfix (#4346) by forcing a manual root-cause investigation instead of a green gate.
• Data integrity: ok.
• Revenue / billing: ok.

What went well

• The WP pipeline API exposed per-step exit codes cleanly, making root-cause isolation fast (~15 min from "ci-pr red" to confirmed root cause).
• The fix pattern already existed elsewhere in the same file (b1-migration-path-lint, flag-promotion-check), so the correct implementation was a known-good copy, not a new design.

What didn't go well

• stale-branch-guard's implementation diverged from its own ADR (ADR-0115's CI migration plan explicitly called for base-ref-aware comparison) without anyone catching the gap — the step was left comparing against a hardcoded literal.
• This bug is invisible until someone opens a PR against main or release — which by definition only happens during incidents (hotfixes) or scheduled promotions, i.e., exactly when speed matters most and false gates are most costly.
• No test/lint validates that .woodpecker/*.yaml steps which read CI_COMMIT_TARGET_BRANCH-equivalent context actually do so consistently across steps in the same file.

Root cause analysis

• Contributing factor 1 — hardcoded literal instead of dynamic base-ref: the stale-branch-guard step's environment.GITHUB_BASE_REF was set to the literal string develop at write time (commit a1a16b8be, 2026-07-11) and never revisited when ADR-0115 introduced the develop→release→main model and its emergency-hotfix-to-main exception. The step's own script (scripts/ci/check_stale_branch.sh) was correctly parameterized on $GITHUB_BASE_REF — the bug was entirely in the workflow YAML failing to pass the real value through.
• Contributing factor 2 — no drift check between sibling steps in the same file: three other steps in .woodpecker/ci-pr.yaml (b1-migration-path-lint, flag-promotion-check, and a third) correctly derive their base branch from ${CI_COMMIT_TARGET_BRANCH:-develop} inline in commands:. stale-branch-guard predates that pattern and was never brought into alignment — nothing enforces consistency across steps that all need "what's my real base branch" logic.

Detection

• What alerted us: ci/woodpecker/pr/ci-pr required-status-check failure on PR #4346, investigated as part of an SRE FIX-mode dispatch (not a monitoring alert).
• How long between cause and detection: cause is 15 days old (introduced 2026-07-11); detection triggered by the first main-based PR to actually exercise the path (2026-07-25).
• How to detect faster next time: a CI lint step that asserts every .woodpecker/*.yaml job reading "the PR's base branch" derives it from CI_COMMIT_TARGET_BRANCH (or equivalent) rather than a literal — see action items.

Resolution

• What was changed: .woodpecker/ci-pr.yaml — stale-branch-guard no longer hardcodes GITHUB_BASE_REF: develop as a static environment value. It now computes export GITHUB_BASE_REF="${CI_COMMIT_TARGET_BRANCH:-develop}" inside commands:, and the subsequent git fetch origin ... line fetches that resolved branch instead of a hardcoded develop. Fix targets develop (per branching model — this is a routine CI fix, not itself a P0 hotfix) via PR (see References).
• Validation: manual fork-point computation for PR #4346 confirms 0 commits / 0h behind the true base (main) — the fixed guard would pass. The develop-targeted fix PR will additionally self-validate: since its own base IS develop, the resolved GITHUB_BASE_REF for that PR is develop itself, exercising the fallback path.

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md (§CI migration plan, §stale-branch-guard design intent)
• Related PR: #4346 (console Billing Summary prod hotfix — blocked by this bug, merged separately once resolved/escalated)
• Related ADR: ADR-0115 (docs/architecture/adr/0115-develop-release-main-branching-model.md)