RCA — First Tag-Gated Production Promotion (ADR-0115)

Incident ID: 2026-07-03-first-tag-gated-promotion Date: 2026-07-03 Severity: SEV-2 Duration: ~3h50m (18:48 UTC first gatekeeper failure → 22:35 UTC prod v177 live) Blast radius: Engineering-internal only. No user-visible downtime. Staging was degraded for ~2h while deploy fixes iterated. Author: sre-agent

Summary

The first end-to-end run of the ADR-0115 tag-gated release pipeline (develop → release → main → prod) exposed five latent structural failures that had not fired before because no release had been promoted through the new pipeline. Each failure was a consequence of the system not having been exercised under real-release conditions: a CI job that was always red, a GitHub App permission that was never needed before a bot-initiated push, two CI gates scoped too broadly, and a Heroku CLI auth path that works locally but not in CI. All five were diagnosed and fixed within one sprint. The release landed as prod v177 at 22:33 UTC with alembic head 0055 on both staging and prod.

Timeline (all times UTC)

• 17:26 — PR #4006 merged to develop: 31aa816f retires the SQLite-fallback coverage job (the latent-red job that was blocking the gatekeeper)
• 18:48 — Gatekeeper run 1802 triggers on develop CI green. Fails. Root cause: raxx-ops-bot GitHub App lacked Contents:write permission, so the bot could not push to the release branch.
• ~19:00 — Operator grants Contents:write to raxx-ops-bot App installation.
• ~19:15 — Gatekeeper re-runs; successfully creates release branch and pushes 1c9d11c3 (gatekeeper promote commit).
• 20:33 — Deploy-Heroku run 1803 triggers on release branch push. Fails. Root cause: heroku git:remote CLI auth fails in CI environment — the CLI attempts browser-based login when HEROKU_API_KEY is set but the runner cannot authenticate interactively.
• 20:42 — Fix attempt: PR commits 38996e5c / d4a24f71 (direct git remote-add bypassing heroku git:remote). Deploy run 1804 → FAILURE. Same root cause; deploy path still depended on CLI auth.
• 21:12 — Fix attempt: b3016f9c switches staging deploy to Source Build API + direct psql for alembic check, fully bypassing Heroku CLI auth. Deploy run 1805 → FAILURE. Migration release command on staging then errors out with ALTER ROLE raptor_app SET log_statement TO 'mod' → insufficient_privilege.
• 21:19–21:42 — Runs 1806–1810: release command failure cascades. Diagnostic commits land (7049bde4). Migration 0053 is identified as the source: it contained ALTER ROLE statements that require CREATEROLE, which the DATABASE_URL owner on Heroku Managed Postgres does not have (Postgres 16 enforcement).
• 21:53 — PR #4014 merged: 09e37083 removes ALTER ROLE from migration 0053; moves role-level log_statement to a post-deploy manual step documented in this RCA. Deploy run 1811 succeeds (staging up on alembic 0055).
• 22:14 — PR #4023 merged: f3b19377 scopes stale-branch and base-branch-cleanliness gates to non-main PRs. These two gates had been false-positiving on the release → main PR because the release branch is legitimately "behind" develop, and the comparison base for cleanliness is develop not main.
• 22:17 — Deploy run 1813 (release branch CI) passes clean. All PR gates green on #4021.
• 22:30 — PR #4021 merged to main.
• 22:31 — Deploy-Heroku run 1814 (workflow_dispatch on main) triggers. All jobs succeed.
• 22:33 — Heroku prod v177 live. Deploy 040c0bc0. /health → 200.
• 22:34 — GH Release tag raxx-api-prod-2026-07-03-223451-040c0bc created.
• 22:35 — Alembic parity confirmed: prod=0055, staging=0055.

Impact

• Users affected: 0
• User-visible symptoms: none (prod was never degraded; only staging was affected during the deploy-fix window)
• Data integrity: ok
• Revenue / billing: ok

What went well

• The gatekeeper → release → main pipeline structure held. Each failure was isolated to one stage and did not cascade to production.
• The fix-iteration cycle was fast (~3h total from first signal to prod live for a first-ever run of a new pipeline).
• Alembic migration 0055 applied cleanly on both staging and prod. No migration failures at the data layer.
• Deploy run 28686092210 (run 1814) passed all gates (smoke, PII scan, deploy freeze check) without manual override.
• Commit lineage confirmed: prod slug 040c0bc0 is the merge of PR #4021, which contains both 09e37083 and 31aa816f in its ancestry. The approved walkthrough copy (f15c34a8, PR #4005) is also in the slug.

What didn't go well

• Five structural failures fired on the first real release run, all of which were diagnosable in advance but were not caught by any pre-flight checklist.
• No documented remediation existed for any of these failure modes in the pipeline runbook at the time.
• The Heroku CLI auth failure required three fix iterations before the correct bypass (Source Build API + direct psql) was identified, consuming ~90 minutes.
• Migration 0053 included DDL (ALTER ROLE) that Heroku Managed Postgres's DATABASE_URL owner cannot execute. This was caught by the release command (alembic upgrade) at deploy time, not by CI, because SQLite tests do not exercise Postgres privilege constraints.

Root cause analysis

• Contributing factor 1: SQLite-fallback test job latently red — The gatekeeper's "ci.yml overall green" requirement blocked promotion as long as the SQLite coverage job was failing. This job was never meaningfully exercised (coverage gate moved to Postgres in #2125) but remained as a required check. The system allowed an always-red job to exist in CI without alerting, and the gatekeeper had no way to distinguish "latently red" from "legitimately red."

• Contributing factor 2: raxx-ops-bot App missing Contents:write scope — The GitHub App for raxx-ops-bot was provisioned without Contents:write on the repository. The gatekeeper's first push to the release branch required this scope. This permission gap was not surfaced by any pre-flight check for pipeline readiness.

• Contributing factor 3: Heroku CLI auth broken in CI for release branch deploys — The deploy workflow used heroku git:remote to configure the git remote, which requires interactive Heroku CLI auth. The HEROKU_API_KEY env var is present, but the CLI's auth helper does not use it for the git:remote subcommand in all versions. This path works when invoked locally because there is an active Heroku session; it fails in CI runners that have no prior session. The system had no end-to-end CI test of the staging deploy path before the first real release.

• Contributing factor 4: Migration 0053 ALTER ROLE requires CREATEROLE (Postgres 16) — Postgres 16 tightened ALTER ROLE ... SET: the executing user must have CREATEROLE + ADMIN OPTION on the target role, or be superuser. The Heroku Managed Postgres DATABASE_URL owner has neither. The migration was written against behavior that worked on older Postgres or with superuser access, and no integration test exercises the migration against a real Heroku Postgres instance in CI before the release command runs.

• Contributing factor 5: Stale-branch and base-branch-cleanliness gates scoped to all PRs — These gates compare a PR's branch against the repo's default branch (develop) to detect drift. When merging release → main, the release branch is intentionally ahead of develop and behind main in some dimensions, triggering false positives. The gate had no carve-out for the release → main merge path.

Detection

• What alerted us: CI run failure notifications and operator monitoring of the pipeline during the first release.
• How long between cause and detection: Each failure was detected within minutes of the run completing (CI notifications).
• How to detect faster next time: Pipeline pre-flight checklist (see Action Items). All five failure modes are detectable without running the pipeline.

Resolution

• Failure 1: PR #4006 retired the SQLite-fallback job; the gatekeeper's required-checks list no longer includes it.
• Failure 2: Operator granted Contents:write to raxx-ops-bot App. Documented in docs/ops/runbooks/github-app-provisioning.md.
• Failure 3: PRs #4010/#4013/#4014 replaced heroku git:remote with direct git remote-add using URL-embedded credentials, and replaced heroku run for alembic checks with direct psql via HEROKU_POSTGRESQL_*_URL.
• Failure 4: PR #4014 (09e37083) removed ALTER ROLE raptor_app SET log_statement TO 'mod' from migration 0053. The role-level log_statement setting is now a post-deploy manual step (see Action Items).
• Failure 5: PR #4023 (f3b19377) added if: github.base_ref != 'main' guards to the stale-branch and base-branch-cleanliness gates.
• Validation: Prod v177 deployed successfully. /health 200. Alembic 0055 on both environments. GH release tag created. Staging log-statement set to mod via heroku pg:settings:log-statement. Prod log-statement change is a pending operator action (see below).

Postgres log_statement — post-deploy status

ALTER ROLE raptor_app SET log_statement TO 'mod' cannot be executed by the DATABASE_URL owner on Heroku Managed Postgres because it requires the CREATEROLE attribute (Postgres 16 enforcement). The DATABASE_URL owner has ADMIN OPTION on raptor_app but not CREATEROLE.

The equivalent database-level setting is managed via heroku pg:settings:log-statement. State after this deployment:

audit_archiver role is not present on either environment as of alembic 0055. No migration through 0055 creates this role.

Operator action needed for prod: heroku pg:settings:log-statement DATABASE mod --app raxx-api-prod. This is a database-wide setting (applies to all connections, not only raptor_app). Authorize as database-wide if acceptable for the audit requirement, or defer until a role-scoped path is available.

Action items

References

• Runbook: docs/ops/runbooks/raptor.md, docs/ops/runbooks/raptor-postgres-roles.md
• PR #4006 (retire SQLite job): 31aa816f
• PR #4010 / #4013 / #4014 (Heroku CLI auth + migration 0053 fix): b3016f9c, 7049bde4, 09e37083
• PR #4021 (promote release → main): 040c0bc0
• PR #4023 (scope CI gates): f3b19377
• Prod release: raxx-api-prod v177, slug b2337ae4, tag raxx-api-prod-2026-07-03-223451-040c0bc
• GH deploy run: 28686092210 (conclusion: success, headSha: 040c0bc010a5cef0a215346b6d356bd5aef72b0a)
• Heroku feedback: feedback_heroku_pg_rds_password_gotcha.md