RCA — stale-branch-guard read a promote/* PR head's own parents as (base, head), failing every release promotion

Incident ID: 2026-10-31-ci-pr-stale-branch-guard-promote-head-read-as-merge-ref Date: 2026-10-31 Severity: SEV-3 (single required check false-red on release promotion PRs; no production impact) Duration: Latent since the resolver shipped (#4611). First observed on #4855 (promote/release-2026.10.30-to-main, opened 2026-10-30 13:04 UTC); reproduced on #4885, pipeline 20072, 2026-10-31 04:11 UTC. Blast radius: Every promote/* PR into main, and any PR whose branch has the base merged into it. Feature PRs were unaffected. Five other gates that share the resolver ran against the wrong base on those PRs. Author: sre-agent

Summary

scripts/ci/resolve_pr_base_sha.sh decided that HEAD was GitHub's merge ref whenever the event was pull_request and HEAD had two or more parents, then returned HEAD^1 as the base and HEAD^2 as the head. Woodpecker does not check out the merge ref: its clone step resets to CI_COMMIT_SHA, so HEAD is the PR head. A promote branch is origin/release with origin/main merged in, so its own parents are (release tip, main tip). The guard therefore compared the release tip against the main tip and reported "24 commits behind main" on a branch that fully contains main. Fixed by classifying the checkout by identity (HEAD vs CI_COMMIT_SHA) and taking the base from origin/<target> when HEAD is the PR head.

Timeline (all times UTC)

• 2026-10-30 13:04 — #4855 (promote/release-2026.10.30-to-main) opened; stale-branch-guard failed the same way (operator report). Merged 2026-10-31 01:24.
• 2026-10-31 04:11 — #4885 (promote/release-2026.10.31-to-main) opened; pipeline 20072, step 99225 fails: Base: origin/main (rev: 3aae7ca5e...) is the release tip, Head: aaf2f5066... is the main tip, fork point b267741ca, 24 commits behind.
• Diagnosis — the clone log of pipeline 20072 shows git fetch --no-tags origin +3a77af27...: and git reset --hard -q 3a77af27...: HEAD is the PR head (CI_COMMIT_SHA), not refs/pull/4885/merge. 3a77af272 has parents 3aae7ca5e (release tip) and aaf2f5066 (main tip), exactly the two revs the guard printed. Pipeline 20073 (a feature PR) has the identical clone shape with one parent, which is why feature PRs passed through the origin/<target> branch of the resolver.
• Fix authored and verified locally against the real #4885 head (see Resolution).

Impact

• Users affected: none (internal CI).
• User-visible symptoms: none.
• Data integrity: ok.
• Revenue / billing: ok.
• Engineering impact: required check ci/woodpecker/pr/ci-pr red on release promotions.
• Hidden second impact: b1-migration-path-lint, flag-promotion-check, route-migration-parity, smoke-surface-drift-check and migration-collision-check received the release tip as the base on those PRs. In #4885 the promote head's tree equals the release tip's (git diff between them is empty), so their diffs were empty and they passed without examining the promoted work. They now diff against origin/main. Run locally against the real #4885 head with the corrected base, all five pass (143 files changed; 4 new migrations 0094, 0319, 0320, 0321 and no collisions across 16 other open PRs).

What went well

• The resolver already failed closed and printed which branch it took, so the swapped revs were visible in the step log.
• check_stale_branch.sh accepts explicit revs, so the whole guard could be re-run locally on the real head.
• The Woodpecker log endpoint made the clone commands readable, which turned a hypothesis into an observation.

What didn't go well

• The resolver's header asserted that Woodpecker checks out the merge ref, "verified" against a hand-fetched refs/pull/4609/merge rather than a real pipeline checkout. #4846 corrected the prose for the single-parent case but left the two-parent branch (a) in place on the same wrong premise.
• No test modelled a PR head that is itself a merge commit, though promote branches are built that way on every release.
• The failure repeated on two consecutive promotions before anyone traced it.

Root cause analysis

• Contributing factor 1 — classification by shape, not identity: "two or more parents" was used as a proxy for "this is the synthetic merge". That is true of a real merge ref and of any PR head that is a merge commit. The system had a ground truth available the whole time (CI_COMMIT_SHA) and did not compare against it.
• Contributing factor 2 — an unverified premise about the checkout: the merge-ref branch was written for a checkout the CI system does not produce, so it was dead code for feature PRs and live, wrong code only for merge-shaped heads. Dead-on-the-happy-path code hides its bug until the rare shape arrives.
• Contributing factor 3 — shared resolver, no consumer-level test on a promotion shape: the stale guard and five other gates share one helper, and none exercised a head that already contains the base.

Detection

• What alerted us: red ci/woodpecker/pr/ci-pr on #4855 and #4885, reported by the operator.
• How long between cause and detection: latent since #4611; first visible on the first promotion after the guard began running against real heads.
• How to detect faster next time: the new tests build a promote-shaped repo and run the real guard on the resolver's output (see action item 2).

Resolution

• scripts/ci/resolve_pr_base_sha.sh: classify the pull_request checkout as pr-head (HEAD == CI_COMMIT_SHA) or merge-ref (HEAD^2 == CI_COMMIT_SHA); exit 1 if it is neither, if CI_COMMIT_SHA is not a commit in the checkout, or if CI_COMMIT_SHA is unset and HEAD is a merge commit. For pr-head the head is HEAD and the base is origin/<target>; CI_COMMIT_BEFORE is ignored on a PR. The merge-ref behaviour and every non-PR behaviour are unchanged.
• scripts/ci/tests/test_resolve_pr_base_sha.py: cases for a normal feature PR, a PR whose head already contains the base (promote shape and base-merged-into-branch shape), and a merge-ref checkout, each also run through the real check_stale_branch.sh. The promote fixture has 12 release-only commits, so the old parent reading fails the guard with "12 commits behind main" (asserted as a teeth test). Against the old resolver 12 of the 44 tests fail.
• scripts/ci/tests/test_migration_collision_check_step.py: a promote-shaped checkout through the real migration-collision-check block; the existing merge-ref case now sets CI_COMMIT_SHA as Woodpecker does.
• Comment and runbook corrections: .woodpecker/ci-pr.yaml, scripts/ci/check_stale_branch.sh, docs/ops/runbooks/ci-woodpecker.md (E-6 dimension 3).
• Validation: the fixed resolver against the original #4885 head 3a77af272 returns base aaf2f5066 (origin/main) and head 3a77af272; check_stale_branch.sh then reports 0 commits behind and exits 0.

Action items

Not filed as GitHub issues (standing instruction); tracked here. | # | Action | Owner | Due | Issue | |---|---|---|---|---| | 1 | Get the fix onto release, then onto each open promote/* branch (the PR pipeline runs the scripts of the branch under test, so #4885 keeps the old resolver until its branch contains this change) | operator | next promotion | n/a | | 2 | Add a nightly or post-merge assertion that the stale-branch guard exits 0 on the most recent promote-shaped head (release with main merged in), so this class is caught before a promotion | sre-agent | 2026-11-14 | n/a | | 3 | Audit the remaining gates that compute base..HEAD for the same "HEAD has two parents" assumption (grep for HEAD^1, HEAD^2, rev-list --parents) | sre-agent | 2026-11-14 | n/a |

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md (Failure mode E-6, dimension 3)
• Related RCAs: docs/incidents/2026-07-25-ci-pr-stale-branch-guard-hardcoded-base.md, docs/incidents/2026-07-30-ci-pr-stale-branch-guard-absolute-age.md
• Related PRs: #4611 (resolver), #4863 (fetch-pr-base-ref), #4846 (resolver corrected for the single-parent checkout); failing PRs #4855, #4885
• Woodpecker clone behaviour: woodpecker-ci/plugin-git, reset to the commit SHA when the commit is a full SHA