Console migrations runbook

System: Console Alembic migration chain (console/migrations/versions/00NN_*.py) Owner: sre-agent / operator (Kristerpher) Last incident: 2026-05-18 — two consecutive console deploy release-phase failures from multi-head Alembic state (#2410, #2419) Related runbook: docs/ops/runbooks/migration-gate.md (full CI gate detail)

What this runbook covers

• Adding a new Console migration
• Multi-head recovery (the primary incident class)
• Using the check_alembic_single_head.sh gate locally
• Why multi-head states happen and how CI now prevents them

For the Raptor (backend_v2) migration chain, see migration-gate.md.

Console migration chain overview

Console migrations live in console/migrations/versions/00NN_<slug>.py. They are Alembic Python migrations run against the Console Postgres database (DATABASE_URL env var on raxx-console-prod / raxx-console-staging).

The chain is linear: each migration has a single down_revision pointing to its predecessor. The current head is the highest-numbered file.

# Show current head(s) — should always print exactly one line ending in (head)
cd console && alembic -c migrations/alembic.ini heads

A healthy output looks like:

0106 -> (head) (0106_promote_backtest_peer_review_fixes)

A multi-head output (broken state) looks like:

0106 -> (head) (0106_promote_backtest_peer_review_fixes)
0105 -> (head) (some_other_migration)

Adding a new Console migration

1. Check the current head revision number:

bash ls console/migrations/versions/ | tail -3

2. Create the new file at the next sequential number:

console/migrations/versions/00NN_<slug>.py

Where NN is the next number after the current head. Copy the most recent 00NN_promote_*.py file as a template. Adjust: - revision: str = "00NN" - down_revision: Union[str, None] = "00(NN-1)" - The _FLAG, _TARGET_APP, _SOAK, upgrade(), downgrade() bodies

3. Verify the chain is linear before pushing:

bash bash scripts/ci/check_alembic_single_head.sh --console-only

Expected output: Console: OK (exactly 1 head)

4. If adding a feature flag promotion migration, the same PR must also add the flag to backend_v2/api/feature_flags.yaml. See docs/agents/adding_a_feature.md §1 for the B1 gate requirements.

5. Revision collision: if two PRs open at the same time each claim 00NN, the one that merges second will break the chain. After the first merges, rebase and renumber your file to 00(NN+1). See "Multi-head recovery" below.

Multi-head recovery

Symptom

Either of: - check_alembic_single_head.sh reports Console head count: 2 (or more) - alembic upgrade head fails with DETAIL: Multiple heads are present; please specify the target revision - alembic-heads-gate CI job fails on a PR

Cause

Two migration files in console/migrations/versions/ have the same down_revision value. This happens when two PRs are opened concurrently and both pick the same next revision number, and the second one merges without noticing the collision.

Historical examples: - 2026-05-18: 0082 dup from concurrent PRs — fixed by #2410 (rename + renumber) - 2026-05-18: 0083 broken down_revision pointer after renumber — fixed by #2419

Recovery steps

1. Identify the colliding files:

bash cd console && alembic -c migrations/alembic.ini heads

Each line ending in (head) is a branch tip. Note both revision IDs.

2. Find which two files share a down_revision:

bash grep -rn "^down_revision" console/migrations/versions/ | sort -t'"' -k2

The duplicated value will appear twice.

3. Choose the file to renumber — the one from the PR that merged later (higher PR number). Rename it to the next available number:

bash cd console/migrations/versions # e.g. rename 0083_foo.py → 0107_foo.py (where 0107 is next available) git mv 0083_foo.py 0107_foo.py

4. Update the file internals:

python revision: str = "0107" down_revision: Union[str, None] = "0106" # correct predecessor

5. Verify the chain is linear:

bash bash scripts/ci/check_alembic_single_head.sh --console-only

Expected: Console: OK (exactly 1 head)

6. Run the full migration round-trip locally if possible:

bash # Requires a local Postgres with DATABASE_URL set cd console && alembic -c migrations/alembic.ini upgrade head cd console && alembic -c migrations/alembic.ini downgrade -1 cd console && alembic -c migrations/alembic.ini upgrade head

7. Push and let CI confirm — the alembic-heads-gate and migration-gate jobs on the fix PR will both validate the chain.

The check_alembic_single_head.sh gate

Script: scripts/ci/check_alembic_single_head.sh

This script asserts that both the Console and Raptor Alembic chains each have exactly one head. It was added in PR #2420 to provide an early signal that the two 2026-05-18 incidents would have caught before merge.

Local usage:

# Check both chains (default)
bash scripts/ci/check_alembic_single_head.sh

# Check only Console
bash scripts/ci/check_alembic_single_head.sh --console-only

# Check only Raptor
bash scripts/ci/check_alembic_single_head.sh --raptor-only

No live DB required. The script supplies a dummy SQLite URL so that Alembic can enumerate the revision-file graph without connecting to a database. The alembic heads command reads revision / down_revision from the Python migration files, not from the alembic_version table.

Called from: - scripts/ci/run_smoke.sh — runs on every local smoke invocation - .github/workflows/ci-pr.yml job alembic-heads-gate — runs on every PR that touches console/migrations/** or backend_v2/alembic/versions/**

Exit codes: - 0 — all checked chains have exactly 1 head - 1 — one or more chains have 0 or multiple heads, alembic missing, or ini file not found

CI coverage summary

Both jobs must pass. The alembic-heads-gate fails faster (no DB spin-up) and serves as a pre-screen; migration-gate provides the full correctness guarantee.

Related references