Console migrations runbook

System: Console Alembic migration chain (console/migrations/versions/00NN_*.py) Owner: sre-agent / operator (Kristerpher) Last incident: 2026-05-18 — two consecutive console deploy release-phase failures from multi-head Alembic state (#2410, #2419) Related runbook: docs/ops/runbooks/migration-gate.md (full CI gate detail)

What this runbook covers

• Adding a new Console migration
• Multi-head recovery (the primary incident class)
• Using the check_alembic_single_head.sh gate locally
• Why multi-head states happen and how CI now prevents them

For the Raptor (backend_v2) migration chain, see migration-gate.md.

Console migration chain overview

Console migrations live in console/migrations/versions/00NN_<slug>.py. They are Alembic Python migrations run against the Console Postgres database (DATABASE_URL env var on raxx-console-prod / raxx-console-staging).

The chain is linear: each migration has a single down_revision pointing to its predecessor. The current head is the highest-numbered file.

# Show current head(s) — should always print exactly one line ending in (head)
cd console && alembic -c migrations/alembic.ini heads

A healthy output looks like:

0106 -> (head) (0106_promote_backtest_peer_review_fixes)

A multi-head output (broken state) looks like:

0106 -> (head) (0106_promote_backtest_peer_review_fixes)
0105 -> (head) (some_other_migration)

Adding a new Console migration

1. Check the current head revision number:

bash ls console/migrations/versions/ | tail -3

2. Create the new file at the next sequential number:

console/migrations/versions/00NN_<slug>.py

Where NN is the next number after the current head. Copy the most recent 00NN_promote_*.py file as a template. Adjust: - revision: str = "00NN" - down_revision: Union[str, None] = "00(NN-1)" - The _FLAG, _TARGET_APP, _SOAK, upgrade(), downgrade() bodies

3. Verify the chain is linear before pushing:

bash bash scripts/ci/check_alembic_single_head.sh --console-only

Expected output: Console: OK (exactly 1 head)

4. If adding a feature flag promotion migration, the same PR must also add the flag to backend_v2/api/feature_flags.yaml. See docs/agents/adding_a_feature.md §1 for the B1 gate requirements.

5. Revision collision: if two PRs open at the same time each claim 00NN, the one that merges second will break the chain. After the first merges, rebase and renumber your file to 00(NN+1). See "Multi-head recovery" below.

Safe migration rename / renumber procedure

When a collision is detected (or anticipated), follow these steps to renumber a migration file safely. The most common scenario is two open PRs that both claimed the same 00NN revision number.

1. Detect the collision

# Count heads — anything > 1 is broken
cd console && alembic -c migrations/alembic.ini heads | wc -l

# Or get the full picture
cd console && alembic -c migrations/alembic.ini heads

2. Pre-rename checklist — find every sibling reference

This is the step that cost 30 minutes of operator time on 2026-05-18 (#2419). Do it before touching any files.

# Find every migration that points at the revision you are about to rename.
# Run from repo root. Replace <old-rev> with the revision string being renamed
# (e.g. "0082").
grep -rn 'down_revision.*=.*"<old-rev>"' console/migrations/versions/

For each match found: - Update that file's down_revision to point at the new revision string. - If the matching file is on an open PR (not yet merged to main), comment on that PR with a rebase instruction: "Migration <old-rev> was renamed to <new-rev> — update your down_revision before merging."

Skipping this step produces a second multi-head the moment the sibling PR merges.

3. Rename the file

cd console/migrations/versions
# Replace old filename and new filename with the actual values
git mv 00NN_old_slug.py 00MM_old_slug.py

4. Update the file internals

Open the renamed file and update both constants:

revision: str = "00MM"                        # new revision string
down_revision: Union[str, None] = "00(MM-1)"  # correct predecessor

Also update _FLAG = "<flag_name>" if this is a B1 promotion migration — the B1 gate's regex looks for it within 500 chars of _INSERT_SQL.

5. Post-rename validation

# Single-head assertion (no DB required)
bash scripts/ci/check_alembic_single_head.sh --console-only

# Linear history (no DB required)
cd console && alembic -c migrations/alembic.ini history

# Full round-trip if a local Postgres is available
cd console
alembic -c migrations/alembic.ini upgrade head
alembic -c migrations/alembic.ini downgrade -1
alembic -c migrations/alembic.ini upgrade head

6. Common gotchas

7. Recovery: rollback a bad rename on a deployed environment

If the rename caused a partial deployment (migration ran under old name, then chain is broken):

1. Do not run alembic downgrade — it will attempt to walk the (now-broken) chain and may fail midway.
2. Identify the current alembic_version row in the Console DB:

bash heroku pg:psql -a raxx-console-prod -c "SELECT version_num FROM alembic_version;"

3. If the version_num points at the old revision that no longer exists as a file, manually update it to the new revision string:

bash heroku pg:psql -a raxx-console-prod -c \ "UPDATE alembic_version SET version_num = '00MM' WHERE version_num = '00NN';"

4. Verify the chain is now resolvable:

bash cd console && alembic -c migrations/alembic.ini current

5. Push the corrected files and let CI (alembic-heads-gate + migration-gate) validate before the next deploy.

Multi-head recovery

Symptom

Either of: - check_alembic_single_head.sh reports Console head count: 2 (or more) - alembic upgrade head fails with DETAIL: Multiple heads are present; please specify the target revision - alembic-heads-gate CI job fails on a PR

Cause

Two migration files in console/migrations/versions/ have the same down_revision value. This happens when two PRs are opened concurrently and both pick the same next revision number, and the second one merges without noticing the collision.

Historical examples: - 2026-05-18: 0082 dup from concurrent PRs — fixed by #2410 (rename + renumber) - 2026-05-18: 0083 broken down_revision pointer after renumber — fixed by #2419

Recovery steps

1. Identify the colliding files:

bash cd console && alembic -c migrations/alembic.ini heads

Each line ending in (head) is a branch tip. Note both revision IDs.

2. Find which two files share a down_revision:

bash grep -rn "^down_revision" console/migrations/versions/ | sort -t'"' -k2

The duplicated value will appear twice.

3. Choose the file to renumber — the one from the PR that merged later (higher PR number). Rename it to the next available number:

bash cd console/migrations/versions # e.g. rename 0083_foo.py → 0107_foo.py (where 0107 is next available) git mv 0083_foo.py 0107_foo.py

4. Update the file internals:

python revision: str = "0107" down_revision: Union[str, None] = "0106" # correct predecessor

5. Verify the chain is linear:

bash bash scripts/ci/check_alembic_single_head.sh --console-only

Expected: Console: OK (exactly 1 head)

6. Run the full migration round-trip locally if possible:

bash # Requires a local Postgres with DATABASE_URL set cd console && alembic -c migrations/alembic.ini upgrade head cd console && alembic -c migrations/alembic.ini downgrade -1 cd console && alembic -c migrations/alembic.ini upgrade head

7. Push and let CI confirm — the alembic-heads-gate and migration-gate jobs on the fix PR will both validate the chain.

The check_alembic_single_head.sh gate

Script: scripts/ci/check_alembic_single_head.sh

This script asserts that both the Console and Raptor Alembic chains each have exactly one head. It was added in PR #2420 to provide an early signal that the two 2026-05-18 incidents would have caught before merge.

Local usage:

# Check both chains (default)
bash scripts/ci/check_alembic_single_head.sh

# Check only Console
bash scripts/ci/check_alembic_single_head.sh --console-only

# Check only Raptor
bash scripts/ci/check_alembic_single_head.sh --raptor-only

No live DB required. The script supplies a dummy SQLite URL so that Alembic can enumerate the revision-file graph without connecting to a database. The alembic heads command reads revision / down_revision from the Python migration files, not from the alembic_version table.

Called from: - scripts/ci/run_smoke.sh — runs on every local smoke invocation - .github/workflows/ci-pr.yml job alembic-heads-gate — runs on every PR that touches console/migrations/** or backend_v2/alembic/versions/**

Exit codes: - 0 — all checked chains have exactly 1 head - 1 — one or more chains have 0 or multiple heads, alembic missing, or ini file not found

CI coverage summary

Both jobs must pass. The alembic-heads-gate fails faster (no DB spin-up) and serves as a pre-screen; migration-gate provides the full correctness guarantee.

Related references