RCA — stale-branch-guard age check used absolute wall-clock age, false-failing every open PR

Incident ID: 2026-07-30-ci-pr-stale-branch-guard-absolute-age Date: 2026-07-30 Severity: SEV-2 (deploy pipeline blocked repo-wide — every open PR's required ci/woodpecker/pr/ci-pr check was red) Blast radius: Every open PR against develop (confirmed on #4351, #4368, #4369; also blocked the in-flight Paper Trading v1 wave). No production user impact. Author: sre-agent

Summary

The stale-branch-guard step's age check in scripts/ci/check_stale_branch.sh computed now - <fork-point commit timestamp>. For any PR branch cut from develop's current tip, the fork point IS develop's HEAD commit — so once develop had gone 48h+ without a merge, EVERY branch forked from it failed the age check, regardless of how many commits (zero) it was actually behind. develop's tip was ed2237f6d (#4347, merged 2026-07-25 10:52 -0400); by 2026-07-30 that fork point was ~118-153h old depending on when each PR branch was cut, well past the 48h threshold, failing the gate repo-wide. This is a distinct bug from the one fixed in #4347 (which corrected which branch the guard compares against — this incident is about what "age" means once compared against the correct branch).

Timeline (all times UTC)

• (ongoing, exact first-alert time not captured — discovered via operator report that ci/woodpecker/pr/ci-pr was red on #4351, #4368, #4369 and blocking the Paper Trading v1 wave)
• 13:12 — sre-agent confirms origin/develop HEAD is ed2237f6d (#4347), committed 2026-07-25 10:52:21 -0400 — 5 days with no develop merges.
• 13:15 — sre-agent reads .woodpecker/ci-pr.yaml stale-branch-guard step and scripts/ci/check_stale_branch.sh; confirms the age check computes FORK_TS=$(git log -1 --format="%ct" "${FORK_SHA}") where FORK_SHA=$(git merge-base origin/develop HEAD) — i.e., absolute commit date of the merge-base, not staleness relative to develop's progression.
• 13:20 — sre-agent creates a fresh branch off origin/develop HEAD, runs the (unmodified) guard locally: COMMITS_BEHIND=0 but fork-point age ~118h — confirms the hypothesis exactly (0 commits behind still fails the age check).
• 13:30 — Fix authored: age is now computed from the oldest commit the PR branch is missing from the base branch (git log --format=%ct FORK_SHA..origin/BASE | tail -1), and skipped entirely when COMMITS_BEHIND == 0.
• 13:35 — Fix validated locally against two scenarios: (a) branch fresh off develop HEAD — 0 commits behind, age check skipped, PASS; (b) branch 13 commits / ~150h behind develop — both checks correctly FAIL.
• 13:40 — Fix PR opened against develop; self-validates via its own .woodpecker/ci-pr.yaml (WP runs the PR branch's own pipeline definition).
• (see PR for merge timestamp) — Fix merged to develop.
• (see rebase report below) — Open Paper Trading wave PRs rebased onto fixed develop to pick up the corrected guard.

Impact

• Users affected: none (internal CI/CD only).
• User-visible symptoms: none.
• Data integrity: ok.
• Revenue / billing: ok.
• Engineering impact: every open PR's required status check was red; the Paper Trading v1 wave (#4351, #4368, #4369, plus an in-flight single-leg-options-correctness PR) was blocked from going green pending this fix.

What went well

• The #4347 fix (base-ref derivation) had already been applied, so isolating this as a genuinely separate dimension — "what does age mean" rather than "age against which branch" — was fast once the fork-point commit date was checked against wall-clock time.
• check_stale_branch.sh's existing structure (two independent checks, clearly separated) made the fix surgical: only the age-check block changed, the commit-count check and all surrounding logic were untouched.
• Local reproduction was straightforward: checking out a fresh branch off origin/develop HEAD and running the guard directly confirmed the bug in under 10 minutes, no CI round-trip needed.

What didn't go well

• The guard's age check has an inherent, unavoidable failure mode whenever a base branch (develop) genuinely stops receiving merges for longer than STALE_BRANCH_MAX_AGE_HOURS — a plausible and not-even-rare event (holidays, promotion freezes, focused single-PR sprints). Nothing caught this until it happened.
• No monitor exists for "how long since develop's last merge" — this would have given advance warning (e.g., alert at 36h of develop silence, well before the 48h gate trips for every open PR).
• This is the second stale-branch-guard bug found via repo-wide outage in 5 days (see #4347, 2026-07-25). The guard's design has now needed two independent corrections; a broader test suite for check_stale_branch.sh covering "0 commits behind + old fork point" and "N commits behind + fork point within threshold" as explicit cases would have caught this in review rather than production.

Root cause analysis

• Contributing factor 1 — age measured against the wrong reference point: the age check used now - merge_base_commit_date. For a branch forked from the base's current tip (the common case — most PRs cut recently), the merge-base commit IS the base branch's HEAD commit. This conflates two unrelated questions: "how long has the base branch been quiet" (a property of the base branch's own commit cadence) and "how far behind is this PR branch" (the actual thing the gate is supposed to measure). A branch cut yesterday from a base that hasn't merged in 6 days is not stale — it has zero missing commits — but the old logic failed it anyway.
• Contributing factor 2 — no test coverage for the "0 commits behind, old base" case: scripts/ci/tests/test_stale_branch_guard_1219.py (or equivalent) did not include a case where COMMITS_BEHIND == 0 but the fork-point commit itself is old. This is exactly the scenario a fast-moving-then-quiet trunk produces, and it went untested.

Detection

• What alerted us: operator report of ci/woodpecker/pr/ci-pr red on #4351/#4368/#4369, escalated to sre-agent in FIX mode. Not caught by an automated monitor.
• How long between cause and detection: the bug is latent in the guard's logic since it was written; it manifests only when the base branch goes quiet for >48h. In this instance, develop went quiet for ~5 days before the first PR branch tripped it — detection was same-day as the threshold was crossed, but only because an operator noticed the CI-wide red state, not because of a targeted alert.
• How to detect faster next time: add a low-priority monitor / daily check for "hours since last merge to develop" and alert at 36h (a margin before the 48h gate trips) — see action item #2.

Resolution

• What was changed:
• scripts/ci/check_stale_branch.sh — age is now computed from the oldest commit the PR branch is missing from the base branch (git log --format=%ct "${FORK_SHA}..origin/${BASE_BRANCH}" | tail -1), not from the fork-point commit's own date. When COMMITS_BEHIND == 0, the age check is skipped entirely (there is nothing missing to measure age against) and only the commit-count check applies.
• .woodpecker/ci-pr.yaml — added a comment block on the stale-branch-guard step documenting this as a second, independent fix layered on top of the 2026-07-25 base-ref fix.
• Validation:
• Local repro (unmodified script): branch cut fresh from origin/develop HEAD reports 0 commits behind but fails the age check (fork-point ~118-150h old depending on test time) — confirms the bug.
• Local repro (fixed script): same fresh branch reports 0 commits behind, age check explicitly skipped ("age check not applicable"), exits 0.
• Local repro (fixed script, genuinely-stale branch): branch 13 commits / ~150h behind develop still fails both checks — confirms the gate remains a real, hard gate for branches that ARE stale.
• Fix PR against develop runs its own .woodpecker/ci-pr.yaml per WP's PR pipeline execution model — confirmed ci-pr green on the fix PR itself (fresh-off-develop branch, 0 commits behind).
• Paper Trading wave PRs rebased onto the fixed develop and reverified green on ci-pr (see PR description / operator report for final state per PR).

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md
• Related RCA: docs/incidents/2026-07-25-ci-pr-stale-branch-guard-hardcoded-base.md (base-ref derivation fix — separate dimension of the same guard)
• Related PR: #4347 (base-ref fix), this incident's fix PR (age-check fix, see PR history)
• Related ADR: ADR-0115 (docs/architecture/adr/0115-develop-release-main-branching-model.md), ADR-0050 (staleness threshold tuning)