RCA — base-branch-lint false-positive on release promotions + migration-collision-check red on every PR

Incident ID: 2026-10-30-ci-gates-promotion-fp-and-migration-collision Date: 2026-10-30 Severity: SEV-3 (two CI gates failing for structural reasons; neither is a required check on develop, but one forced an admin merge on every release PR) Duration: gate 1 since the ADR-0115 promotion flow went live; gate 2 since the Woodpecker port (about 35 consecutive PRs, #4294 to #4812) Blast radius: every promote/release-*-to-main PR (#4809, #4817 confirmed); every PR touching console/migrations/versions/**, backend_v2/alembic/versions/** or the sqitch plan Author: sre-agent

Summary

base-branch-lint treats a commit that also lives on another remote branch as contamination. A release-promotion PR carries every develop commit by design, and every open feature branch cut from develop contains those same commits, so the gate fired on every promotion. Root cause established and fixed.

migration-collision-check had two stacked defects, both identified. (1) The base-SHA lookup depended on a GitHub token that is unusable on pull_request events. (2) Its replacement, the shared local resolver, assumes Woodpecker checks out the merge ref; in this Woodpecker the checkout is the PR head commit with no origin/* refs, so the resolver refused. The ci-pr gates that use the same resolver only work because their earlier base-branch-lint step fetches the target branch with a vault token. The gate now does the same fetch. Defect 2 was found by having the failing step post its own output to the PR, since Woodpecker logs are not readable.

Timeline (all times UTC)

• 2026-10-30 ~00:18 to 00:23 — Promotion PR #4817 (promote/release-2026.10.29-to-main, base main): ci/woodpecker/pr/ci-pr FAILURE; merged with admin override as #4809 had been.
• Gate 1 diagnosis — Reproduced locally against the real #4817 head with the pre-merge main tip as base: CONTAMINATED COMMIT e8fce6168 (develop tip), also present on three design-parity-* feature branches. The ci-pr step also passed only GITHUB_HEAD_REF, so the script silently compared every PR, including PRs into main, against origin/develop.
• Gate 2 diagnosis — The old check step's first non-install command was gh api repos/$GITHUB_REPOSITORY/pulls/$CI_COMMIT_PULL_REQUEST under set -eu. Woodpecker step logs were not readable (no API token; a credentialed read was not authorised), so the cause was initially inferred. Reproduced the old block under dash against a simulated merge ref with a gh that fails like an empty token: exit 4, checker never ran.
• ~04:28 — First fix pushed (local base SHA via resolve_pr_base_sha.sh). ci-pr and pii-scan green; migration-collision-check still FAILURE after about 29 s.
• ~04:41 — Second push added five temporary single-question probe workflows so the failing component could be read from commit statuses alone. Results: apt install of git and curl OK; pinned gh tarball download and run OK; a step declaring GH_TOKEN from the CI_FORGE_TOKEN secret runs OK; the same secret used for gh api repos/<repo> and gh pr list FAILED (token empty or rejected on pull_request events); the local resolver plus the checker with no gh and no secret FAILED. The checker passes on the same real merge ref locally under Python 3.11 and 3.14.
• Third push removed the probes and added the failure-report step and the defensive changes described below. It landed during a Woodpecker stall (the operator account's GitHub REST quota was exhausted from about 05:05Z to 05:30Z and no statuses were produced for pushes in that window); closing and reopening the PR re-fired CI as pipeline 19734.
• ~05:46 — The failing step's own PR comment gave the cause: HEAD is the single-parent PR head commit, CI_COMMIT_BEFORE is unset, origin/develop is not present, so resolve_pr_base_sha.sh exited with "could not determine a PR base SHA". CI_COMMIT_PULL_REQUEST and CI_COMMIT_TARGET_BRANCH were populated, which rules out the empty-variable theory.
• Fourth push (authorised by the coordinator after the three-push limit was reached): the step fetches the target branch with the vault read token and diffs against the merge-base. migration-collision-check went green (pipelines 19823 and 19828); ci-pr was red only on stale-branch-guard (develop had moved 16 commits ahead).
• Meanwhile #4863 merged to develop with the fetch-pr-base step and fetch_pr_base_ref.sh. The branch was rebased onto it and this gate's inline fetch was replaced by a call to that script (fifth push). See Resolution.

Impact

• Users affected: none (internal CI).
• User-visible symptoms: none. Release engineers saw red ci-pr on every promotion and had to admin-merge; migration-collision-check was red on every migration PR, so a real cross-PR collision would have been indistinguishable from the standing failure.
• Data integrity: ok.
• Revenue / billing: ok.

What went well

• resolve_pr_base_sha.sh (#4595) already existed with a fail-closed contract and tests; gate 2's first fix was a reuse, not a new design.
• Gate 1 reproduced exactly from public refs.
• Single-question probe workflows turned unreadable logs into readable commit statuses.

What didn't go well

• Gate 2 stayed red for about 35 PRs without anyone being able to see why: the failing step is a path-filtered, non-required pipeline, and its logs are not readable from the tooling available.
• #4613 changed the credential plumbing with no way to observe the effect, and an earlier comment in the pipeline asserted the check "has been passing on real migration PRs"; the statuses on all four cited PRs are FAILURE. A claim about CI health that was never checked against the statuses misdirected the next fix.
• The first fix was reasoned from inference alone and cost a push; the probes should have come first.
• Local tests ran the step's commands bare; they did not cover the exact shell Woodpecker uses (sh -e over an echo-wrapped script), nor the real CI environment.

Root cause analysis

• Gate 1, factor 1: the containment rule has no notion of an intentionally multi-branch history. ADR-0115 open question 5 anticipated the problem but assumed branch auto-delete would clear it; open feature branches defeat that assumption.
• Gate 1, factor 2: the Woodpecker step never passed the PR's real base to the script, so the script's develop default applied to every PR. Same hardcoded-base class as stale-branch-guard (2026-07-25).
• Gate 2, factor 1 (established): the base-SHA lookup depended on a credential that ordinary steps do not have on pull_request events. The CI_FORGE_TOKEN secret does not yield a working token there (probe result above). Five ci-pr gates had this exact shape and were fixed in #4595; this pipeline was not in that sweep.
• Gate 2, factor 2 (established): resolve_pr_base_sha.sh documents and tests a merge-ref checkout (branch a), which Woodpecker does not produce here; the only branch that works in CI is c (origin/<target> already present), and only because an earlier ci-pr step happens to fetch. A shared helper whose precondition was verified against a hand-fetched merge ref, not against a real Woodpecker checkout, was reused by a workflow that has no such fetch. The resolver's tests built the same merge-ref topology, so they could not see the difference. The related ci-pr gap (a PR that touches ios/** skips base-branch-lint, so a PR that also touches feature_flags.yaml or an API route ran the blocking resolver gates with no base) was fixed independently in #4863 with a fetch-pr-base step and scripts/ci/fetch_pr_base_ref.sh; this gate was reconciled onto that same script so there is one fetch mechanism, and the resolver now uses the same target-branch fallback chain as the script.
• Gate 2, factor 3: the pipeline triggers only on migration paths, so edits to its own definition never exercised it, and the fix for factor 1 (#4613) could not be verified from the PR that made it.

Detection

• What alerted us: operator report (admin merges on #4809 and #4817; 35 red migration-collision-check runs).
• How long between cause and detection: gate 1, one promotion cycle; gate 2, the whole life of the port.
• How to detect faster next time: the pipeline now self-triggers on its own YAML and scripts, and a red run explains itself in a PR comment.

Resolution

• Gate 1: scripts/ci/check_pr_base_clean.sh exits 0 with an EXEMPT: line when head matches promote/* AND base is main or release; base also falls back to CI_COMMIT_TARGET_BRANCH. .woodpecker/ci-pr.yaml exports GITHUB_BASE_REF from CI_COMMIT_TARGET_BRANCH, then CI_COMMIT_BRANCH, then develop. Gate still fully in force for feature PRs into develop, promote/* into develop, and any other head into main/release.
• Gate 2: .woodpecker/migration-collision-check.yaml fetches the target branch before resolving the base (vault GITHUB_API_READONLY_TOKEN, throwaway netrc deleted right after the fetch, then the shared scripts/ci/fetch_pr_base_ref.sh from #4863), resolves it with scripts/ci/resolve_pr_base_sha.sh (fail closed, CI_COMMIT_BEFORE ignored because on a PR it can be the previous head), and diffs against the merge-base so develop moving on does not look like PR changes. For the cross-PR comparison it uses the first of CI_FORGE_TOKEN and the vault read token that can actually run gh pr list; if neither can, a loud WARNING says the comparison was skipped and the gate does not fail. gh install is best-effort, the PR number is derived defensively, output is teed to .mcc-check.log, and a report-failure step (failure: ignore, status: failure only) posts that log as a sticky PR comment via scripts/ci/post_ci_failure_comment.py. The pipeline self-triggers on its own files.
• Validation: test_check_pr_base_clean.py (23 cases; the exemption cases fail against the old script, the gate-still-in-force cases pass on both), test_migration_collision_check_step.py (51 cases; the main fixture is the real CI shape: single-parent HEAD, no origin/* refs, a moved-on base that deleted a file the PR branch still has; its CI-shaped cases fail against the pipeline as it was before the base fetch), test_resolve_pr_base_sha.py (19 cases, including the target fallback chain shared with the fetch script), test_post_ci_failure_comment.py (22 cases). Live validation is the PR's own migration-collision-check run.

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md Failure modes E-7 and E-8
• Related incidents: docs/incidents/2026-07-25-ci-pr-stale-branch-guard-hardcoded-base.md
• ADRs: ADR-0115, ADR-0148
• Related PRs: #4595 (local base-SHA resolver), #4613 (from_secret attempt), #4809 and #4817 (admin-merged promotions), #4846 (this fix)