RCA — Console migration 0237 triple-claim collision

Incident ID: 2026-07-18-console-migration-0237-collision Date: 2026-07-18 (incident ID); the resolving merges landed 2026-07-19 UTC Severity: SEV-3 Duration: 0m total (detected pre-merge, resolved before any migration ran) Blast radius: develop branch only; no migration was applied to any database Author: sre-agent

Summary

Three separate PRs in the ticketing system chain each independently claimed console migration number 0237. PR #4276 (renumbered from 0233) and PRs #4277 and #4273 (both originally assigned 0237) would have produced a multi-head Alembic chain on develop, preventing alembic upgrade head from completing on the next deploy. The collision was detected via check_migration_collision.py before any PR merged and resolved by renumbering the three migrations into a linear chain (0237→0238→0239).

Timeline (all times UTC)

The times for detection and the fix were not recorded in the repo. An earlier draft listed 05:00 to 06:45 UTC for the whole sequence; that does not match the GitHub merge record, so only the verifiable times are given below.

1. PR chain merge session begins; migration-collision-check FAILURE observed on #4277 and #4273
2. Root cause identified: #4276 renumbered from 0233→0237 collides with the original 0237 claims on #4277 and #4273
3. Fix: #4276 keeps 0237 (down_revision=0236), #4277 renamed 0237→0238 (down_revision=0237), #4273 renamed 0237→0239 (down_revision=0238)
4. Local dry-run of check_migration_collision.py confirms no collisions
5. 2026-07-19 05:04:41Z, 05:05:46Z, 05:06:10Z — #4276 → #4277 → #4273 merged into develop in sequence (GitHub merge times); chain verified 0236→0237→0238→0239
6. 2026-07-19 05:10:46Z — this RCA committed

Impact

• Users affected: none
• User-visible symptoms: none (deploy would have failed at release phase if undetected)
• Data integrity: ok
• Revenue / billing: ok

What went well

• check_migration_collision.py and the CI migration-collision-check gate caught the collision before any PR merged
• The check_alembic_single_head.sh gate in ci-pr provides a second independent signal for multi-head states
• The merge dependency ordering (0237 in develop before 0238 can merge, 0238 before 0239) enforced chain integrity at merge time

What didn't go well

• Three PRs in the same feature wave were all assigned migration number 0237 by different authors working in parallel
• The PR template and migration creation tooling do not reserve numbers at PR-creation time — collision is detectable only when the CI gate runs against other open PRs

Root cause analysis

• Contributing factor 1: Parallel development — five console B1 migrations were developed simultaneously across six PRs. The migration numbering scheme is monotonic but not atomic: each author picks "next available" from develop at branch-cut time, but develop's tail can shift between branch-cut and CI run.
• Contributing factor 2: The collision-check CI gate runs at PR-open time against other open PRs, so a PR that renumbers an earlier collision (0233→0237) can create a secondary collision with PRs that already claimed 0237.
• Contributing factor 3: No locking or reservation mechanism exists between PR creation and merge. The check is detective (CI), not preventive (tooling).

Detection

• What alerted us: ci/woodpecker/pr/migration-collision-check: FAILURE on PRs #4277 and #4273
• How long between cause and detection: 0m (gate fires on every force-push)
• How to detect faster next time: already fast; the gate is the primary signal

Resolution

• What was changed: renamed 0237_promote_support_history_endpoints.py (on #4276, was originally 0233), 0237_promote_console_support_history.py → 0238_promote_console_support_history.py (on #4277), 0237_promote_support_portal_spa.py → 0239_promote_support_portal_spa.py (on #4273); updated revision= and down_revision= strings in all three files
• Validation: check_migration_collision.py --base-sha origin/develop returned No migration-number collisions detected. OK.; develop console migration head was linear 0236→0237→0238→0239 (confirmed via git show origin/develop:console/migrations/versions/023*.py | grep revision). The chain has since advanced (0240 onward) and the 0236 to 0240 revision/down_revision links are unchanged on develop as of 2026-10-30

Action items

Status of action item 1 (checked 2026-10-30 UTC): not done and past its due date. No reservation script or template exists on develop; scripts/ci/check_migration_collision.py and scripts/ci/check_alembic_single_head.sh remain the only (detective) controls.

References

• Runbook: docs/ops/runbooks/console-migrations.md
• Related incident: 2026-05-18 console deploy release-phase failures from a multi-head Alembic state (#2410, #2419), recorded in the "Last incident" line of the runbook above (there is no separate incident file)
• Gate script and its earlier collision (2026-06-05, migration 0145): scripts/ci/check_migration_collision.py
• PRs: #4276 (Raptor support-history endpoints, #4260), #4277 (console Support History panel, #4261), #4273 (support.raxx.app SPA skeleton, #655)