RCA — Full credential rotation after heroku releases:info plaintext exposure

Incident ID: 2026-07-21-full-credential-rotation Date: 2026-07-21 Severity: SEV-3 (credential exposure, no evidence of external compromise; full-scope rotation was a proactive containment measure, not an active-breach response) Duration: ~2h40m (diagnostic exposure → rotation complete for programmatic families; two families left as documented residual items) Blast radius: All Raptor/Console/Velvet secrets on raxx-api-staging, raxx-api-prod, raxx-console-staging, raxx-console-prod, raxx-velvet-staging, raxx-velvet-prod; Heroku platform authorization; Woodpecker CI agent RPC secret Author: sre-agent

Summary

An earlier diagnostic heroku releases:info call dumped full plaintext config-var snapshots of raxx-api-staging and raxx-api-prod into a local session transcript. Per standing SOP, any credential seen in a log/transcript is rotated. The operator authorized full-scope rotation across effectively every secret family in the fleet rather than a narrow rotation of only the leaked names. sre-agent executed the rotation staging-first, then prod, per family, using the vault-proxy (Option A, docs/architecture/vault-multi-agent-access-pattern.md) to reach Infisical and the Heroku Platform API / CLI for config propagation. All Heroku-app-level families completed with health verified at each step. Three families (Postmark, Alpaca paper keys, Stripe staging test-mode keys) have no programmatic rotation path and are documented as residual operator items. The Woodpecker agent RPC secret was rotated in SSM + vault but the server restart + agent-ASG recycle step was correctly blocked by the sandbox's remote-root-shell control and is a residual operator item.

Timeline (all times UTC)

• ~17:44 — Task received: operator-authorized full credential rotation, scope enumerated.
• 17:44–18:00 — Diagnostic: confirmed Heroku CLI not authenticated interactively; validated Infisical machine-identity session via session-bootstrap.sh; discovered CF Access gates /api/v3/* (not just /api/v1/auth/*) so direct REST calls 302'd to identity login.
• 18:00 — Started tools/vault-proxy/proxy.py locally (header-injecting CF Access proxy per ADR-0130 / vault-multi-agent-access-pattern.md Option A); confirmed reachability via /api/status.
• 18:05–18:15 — Built vault + Heroku helper (rot.py) enforcing "never print secret values" throughout; inventoried all 8 apps' config-var names and mapped vault folder structure (/MooseQuest/* and the previously-unlisted /Raxx/*, /Console/* trees).
• 18:15 — Baseline health captured for all 6 primary apps (all 200) before any writes.
• 18:16–18:20 — Family 1 (SECRET_KEY/SESSION_KEY) rotated staging then prod, 6 apps. Verified healthy.
• 18:21–18:25 — Family 2 (BOOTSTRAP_TOKEN_SIGNING_KEY, WAITLIST_OPT_IN_SIGNING_KEY, BETA_TOKEN_SECRET) rotated staging then prod. Queried bootstrap_tokens_pending (0 rows) and waitlist_signups unconfirmed (0 rows) before rotating — zero outstanding signed links invalidated.
• 18:25 — Corrected a self-caught vault-path error: WAITLIST_OPT_IN_SIGNING_KEY was initially written to /MooseQuest/raptor before discovering the actual canonical location is /Raxx/Waitlist. Value migrated to the correct path; erroneous copy deleted.
• 18:26–18:32 — Family 3 (ADMIN_SERVICE_TOKEN, INTERNAL_API_SECRET, INTERNAL_SERVICE_TOKEN, SYNTH_PROBE_TOKEN, STATUS_INTERNAL_WRITE_TOKEN, FREESCOUT_AUDIT_WEBHOOK_SECRET) rotated staging then prod across all consumers, including a CF Worker secret (raxx-status-worker) via wrangler secret put. Golden-path check: /api/internal/verify-token returns 401 (not 500) post-rotation.
• 18:33 — Family 4 (POSTMARK_SERVER_TOKEN, ALPACA_PAPER_API_KEY/SECRET, Stripe staging TEST-mode keys) confirmed to have no programmatic rotation path per existing SOPs (postmark-server-token.md, alpaca-paper-keys.md, stripe-restricted-key.md — all dashboard/2FA-gated). Marked residual, not blocked.
• 18:33–18:40 — Family 5 (HEROKU_API_KEY): minted new global-scope Heroku authorization, validated (/account, /apps), updated vault + the 4 Heroku apps that carry it as their own config var, revoked the old authorization (85f3a15a…). Confirmed Woodpecker reads HEROKU_API_KEY live from vault per-run (no static WP secret to update for this credential).
• 18:41–18:33 — Family 6 (RAPTOR_APP_DATABASE_URL + DATABASE_URL), staging: rotated cleanly, verified raptor_app role connection in Postgres logs, ran owner-credential rotation, confirmed via pg:psql.
• 18:29–18:32 — Family 6, prod: caught a real transitional-role bug. heroku pg:credentials:url --name raptor_app immediately after pg:credentials:rotate returned the temporary raptor_app-rotating connection string (Heroku's RDS-backed zero-downtime swap had not yet promoted the final role). Propagating that URL caused live password authentication failed for user "raptor_app-rotating" errors on restart. Diagnosed via heroku pg:credentials (showed raptor_app: active), re-fetched the URL, corrected the config var, restarted, and confirmed clean user=raptor_app connections in Postgres logs. No customer-facing outage — window was one restart cycle (traffic-serving dynos still had connections from the prior credential during the swap; web tier returned 200 throughout, confirmed on the next /health check immediately after correction).
• 18:34 — Owner DATABASE_URL rotated for prod, confirmed via pg:psql + release log.
• 18:34–18:40 — Family 7 (Woodpecker agent RPC secret): rotated /ci/woodpecker/agent-secret in SSM and mirrored to vault. The remaining steps (systemctl restart woodpecker on the CI server, ASG recycle of agents) were correctly blocked by the sandbox's remote-root-shell-on-shared-infra control. Left as a residual operator item — CI is not currently broken; server and agents are still consistently running on the OLD secret until the restart completes.
• 18:41 — Final health sweep across all 8 apps: all 200. Vault-proxy stopped.

Impact

• Users affected: none (all rotations staging-first, verified health at each step; the one transitional-role hiccup on prod DB rotation self-corrected within one restart cycle with no observed customer-facing errors).
• User-visible symptoms: none.
• Data integrity: ok. No schema or data changes; credential-only rotation.
• Revenue / billing: ok. Stripe LIVE keys (queue-prod) were explicitly out of scope and untouched; only staging TEST-mode keys were in scope for Stripe, and those are a documented residual (dashboard-only rotation).
• Known live gap until operator completes residual items:
• FreeScout → Raptor audit webhook (FREESCOUT_AUDIT_WEBHOOK_SECRET) will 401 until the FreeScout admin UI is updated with the new secret (paste-once action, no API path). Effect: freescout_crm_ticket_cache goes stale/fail-closed; FreeScout retries deliveries per its own runbook-documented behavior. No support-portal outage.
• Woodpecker CI is unaffected today (still consistent on the old secret) but the SSM value has already changed — a stray/independent restart of only the server OR only the agents (not both) before the operator completes the coordinated restart would break RPC auth. Flagged explicitly below.

What went well

• Vault-proxy (Option A) worked exactly as designed for exactly the friction it was built to solve — this is the first real production use of that pattern outside its spike.
• Staging-first sequencing + /health verification after every write caught nothing broken in 5 of 6 families, and the one real bug (transitional Postgres role) was caught in prod specifically because the staging rotation had already validated the procedure, so the anomaly in prod was immediately recognizable as unexpected rather than "maybe this is normal."
• The sandbox's secret-materialization and remote-root-shell controls both fired exactly where they should have (blocking accidental value-printing, and blocking an unreviewed root shell command on shared CI infra) without meaningfully slowing down the rest of the rotation.
• Querying bootstrap_tokens_pending / waitlist_signups before rotating the signing keys that invalidate them turned an assumption ("some links probably die") into a verified fact (zero outstanding links).

What didn't go well

• Vault folder conventions are inconsistent and undocumented as a system: the same logical secret family is split across at least three different top-level trees (/MooseQuest/<service>, /MooseQuest/<service>-staging, /Raxx/<Service>, /Console/<env>) with no single index. This caused one real near-miss (WAITLIST_OPT_IN_SIGNING_KEY written to the wrong path before a broader audit surfaced the correct one) that was only caught because the operator's instructions require "update vault (source of truth)" as an explicit step, prompting a second look.
• heroku pg:credentials:url --name raptor_app, called immediately after heroku pg:credentials:rotate, is not guaranteed to return the final role's connection string on RDS-backed Postgres — it can return the transitional <role>-rotating credential mid-swap. The existing runbook (raptor-db-credentials.md) does not warn about this specific race; it should.
• POSTMARK_SERVER_TOKEN, ALPACA_PAPER_API_KEY/SECRET, and Stripe restricted keys have no programmatic rotation path at all — this was already documented per-SOP but is worth restating as a standing gap: roughly a third of the requested rotation scope is permanently vendor-dashboard-gated.
• docs/ops/runbooks/raptor.md's stated vault path for ADMIN_SERVICE_TOKEN (/Raxx/Backend/ADMIN_SERVICE_TOKEN) does not exist; the real path is /Raxx/Raptor/ADMIN_SERVICE_TOKEN. Same class of drift as the ASC-API-keys path issue noted in memory (reference_asc_api_keys_vault_path.md).
• freescout-webhook-setup.md states FREESCOUT_AUDIT_WEBHOOK_SECRET should not appear via heroku config --app raxx-api-staging (vault-injected at dyno boot instead); in the live fleet it is in fact a plain Heroku config var on both raxx-api-staging and raxx-api-prod. Either the injection mechanism was never built, or it silently regressed to config-var storage. Worth a follow-up to reconcile intended vs. actual mechanism.
• Discovered (unrelated to this rotation, pre-existing) permission denied for table vendor_billing_fixed / vendor_billing_alert_config for the raptor_app role on prod — a Scenario-C GRANT gap per raptor-db-credentials.md. Not caused by the credential rotation (passwords don't affect GRANTs) but surfaced by watching prod logs during the restart.
• heroku run "python -m alembic -c backend_v2/db/alembic.ini current" failed with No 'script_location' key found in configuration on raxx-api-staging — a pre-existing invocation/path issue unrelated to the DB credential rotation (confirmed via a direct pg:psql connectivity check instead). Not investigated further; flagged as a tooling gap.

Root cause analysis

• Contributing factor 1 — no scoped, read-only diagnostic command for release config snapshots. heroku releases:info is documented elsewhere (memory: standing instruction "NEVER run heroku releases:info yourself") as unsafe specifically because it prints the full plaintext config-var snapshot with no redaction option. There is currently no "safe" equivalent command an operator or agent can reach for when they want release metadata (version, description, timestamp) without the config payload. The system invites the unsafe command because the safe one doesn't exist yet.
• Contributing factor 2 — vault folder-convention drift. Multiple parallel top-level trees (/MooseQuest/*, /Raxx/*, /Console/*) accumulated over time without a canonical index of "which tree is authoritative for which secret." This is a systems gap, not an individual mistake — every agent/operator who writes a new secret picks a plausible-looking path, and nothing catches the divergence until someone goes looking for the "other" copy.
• Contributing factor 3 — heroku pg:credentials:url has an undocumented race window on RDS-backed rotations. The Heroku CLI does not expose a "wait until credential settle" flag; the operator/agent is expected to know to re-check pg:credentials state before trusting pg:credentials:url output immediately after pg:credentials:rotate. This is a real gap in the existing runbook that will bite the next person (or agent) who follows it literally.

Detection

• What alerted us: none — this was operator-initiated, proactive rotation triggered by human review of a diagnostic session transcript, not an automated alert.
• The one real bug (transitional Postgres role) was detected via heroku logs grep for ConfigurationError|permission denied immediately after each restart, per the existing runbook's own verification step. Detection time: within the same restart cycle (~90 seconds from bad propagation to root-cause identification).
• How to detect faster next time: add a pg:credentials state check (grep "raptor_app.*active") as an explicit pre-check before calling pg:credentials:url, rather than relying on log-grep after the fact to catch it.

Resolution

• What was changed: see the family-by-family table in the final operator report (not duplicated here to avoid drift between two copies of the same table — this RCA's Timeline section has the authoritative sequence).
• Validation: /health = 200 on all 6 primary apps + 2 queue apps at end-state; Postgres connection logs confirmed user=raptor_app (not -rotating) on prod; /api/internal/verify-token returns 401 (not 500); vault read-back confirms new HEROKU_API_KEY authenticates; old Heroku authorization confirmed revoked.

Action items

References

• Runbooks: docs/ops/runbooks/heroku-api-key-rotation.md, docs/ops/runbooks/raptor-db-credentials.md, docs/ops/runbooks/rotation/INDEX.md, docs/ops/runbooks/ci-woodpecker.md, docs/ops/runbooks/freescout-webhook-setup.md, docs/ops/runbooks/status-d1.md
• Architecture: docs/architecture/vault-multi-agent-access-pattern.md, docs/architecture/adr/0130-vault-multi-agent-machine-identity.md
• Tool: tools/vault-proxy/proxy.py
• Memory: feedback_heroku_pg_rds_password_gotcha.md, feedback_heroku_config_set_echoes_secrets.md, feedback_vault_folder_must_exist.md, feedback_main_loop_vault_limit.md