Heroku API Key — Rotation Runbook

System: Heroku Platform API credential (HEROKU_API_KEY) Owner: sre-agent / operator (Kristerpher) Cadence: Every 90 days (quarterly) Detailed per-step SOP: docs/ops/runbooks/rotation/heroku-platform-token.md Drift recovery: docs/ops/runbooks/heroku-api-key-drift-recovery.md Issue: #251 Last rotated: 2026-07-21 UTC (sre-agent, operator-authorized full credential rotation — see docs/incidents/2026-07-21-full-credential-rotation.md) Next rotation due: 2026-10-19 UTC

What this key does

HEROKU_API_KEY is a global-scope Heroku OAuth authorization used by:

The billing-collector key (HEROKU_BILLING_API_KEY) is a separate read-scoped authorization rotated on its own schedule — it is NOT covered here.

Rotation procedure (summary)

Follow docs/ops/runbooks/rotation/heroku-platform-token.md for the full copy-pasteable commands. The high-level sequence is:

1. List existing authorizations — capture the OLD authorization ID.
2. Create new authorization via POST /oauth/authorizations (global scope). Description convention: raxx-ci-rotation-YYYY-MM.
3. Verify new token — GET /account must return kris@moosequest.net. Also verify app-list read (GET /apps). Abort if either fails.
4. Update all stores (in this order, never out of order): - Infisical vault: PATCH /api/v3/secrets/raw/HEROKU_API_KEY (path /MooseQuest/heroku, env prod) - The 4 Heroku app config vars that carry HEROKU_API_KEY as their own config var: raxx-api-staging, raxx-api-prod, raxx-console-staging, raxx-console-prod (heroku config:set HEROKU_API_KEY=<value> -a <app> for each). GitHub Actions deploys were retired 2026-07-10 (ADR-0135/ADR-0136) — there is no GHA secret consumer to update; see the consumer table above.
5. Re-verify vault — re-read the vault entry and confirm its suffix matches the new token. Abort if it does not.
6. Static-copy sweep (#4544) — run: bash grep -rn "from_secret: HEROKU" .woodpecker/ This must return zero hits. Any hit is a static WP org-secret copy of a Heroku credential that this rotation must also update (or, preferably, retire in favor of the run-time vault load — see #4544 for the precedent: HEROKU_PROD_API_KEY was exactly this kind of copy, went stale after the 2026-07-21 rotation, and broke prod deploys for 5 weeks before anyone noticed).
7. Revoke OLD authorization — heroku authorizations:revoke <OLD_ID>. Use the NEW key to make the revoke call. Verify the old authorization no longer appears in heroku authorizations.
8. Record audit entry — log secret.rotate.completed in the rotation log below and update Last rotated / Next rotation due in this file.

Guardrails (never skip)

• Verify the new token works BEFORE revoking the old one. No downtime gap.
• If you cannot identify the old authorization ID precisely via suffix matching, STOP and escalate rather than blind-revoke.
• Never print secret values to stdout/logs. Use suffix-only comparisons for validation.
• If more than one raxx-ci-rotation-YYYY-MM authorization is created (e.g. due to a retry), revoke all but the one written to vault before proceeding.

Rotation schedule

Scheduled reminder

A GitHub Actions cron should enforce the 90-day cadence. Until a dedicated credential-expiry-monitor workflow exists, track via a recurring GitHub issue or calendar event. Create a type:reliability issue due 1 week before each rotation date so it lands in the weekly sweep.

Suggested cron check (to be added to a future credential-monitor workflow):

# .github/workflows/credential-expiry-monitor.yml (future)
# Fires 7 days before each rotation due date; opens a reminder issue.
on:
  schedule:
    - cron: '0 8 12 10 *'  # 2026-10-12 08:00 UTC — one week before 2026-10-19
    - cron: '0 8 10 1 *'   # 2027-01-10 08:00 UTC — one week before 2027-01-17 (illustrative next cycle)

Until that workflow exists: set a calendar reminder for 2026-10-12 UTC so the rotation runs before the key reaches 90 days old (current authorization e4e7130b minted 2026-07-21 — see the rotation log below).

Rotation log

Authorization inventory (at 2026-07-01 rotation)

After the 2026-07-01 rotation, the following Heroku authorizations remain. Entries not maintained by this runbook are flagged for the operator.

Recommended operator cleanup (next maintenance window): Revoke 73f5fe49, 9d5ec3c4, f44d9427, and 99f3c0a4. These are not in active use.

Vault/session drift note (2026-07-01 discovery)

During the 2026-07-01 rotation, the vault key and the session-bootstrapped key were discovered to be different authorizations:

• Vault (source of truth): raxx-automation: dispatch + git push (b1e6320e) — rotated out
• Session (HEROKU_API_KEY_PROD): prod-deploy-key (1f970877) — still active

Both were valid at time of discovery. The vault key was rotated (new vault key is 85f3a15a). The prod-deploy-key (1f970877) was NOT revoked — it is still active. The operator's session HEROKU_API_KEY_PROD now points to a key that exists but is NOT the vault-canonical key.

Action required: After verifying session-bootstrap.sh reads from vault on refresh, revoke 1f970877 manually:

heroku authorizations:revoke 1f970877-9b2f-4899-ac67-030049fd7833

Only do this AFTER confirming the next session bootstrap reads the new vault key.

Drift detection

Between rotations, the following three stores must hold the same token (updated 2026-08-29, #4544 — matches heroku-api-key-drift-recovery.md):

1. Infisical vault: /MooseQuest/heroku/HEROKU_API_KEY (env: prod) — source of truth
2. The 4 Heroku app config vars: HEROKU_API_KEY on raxx-api-staging, raxx-api-prod, raxx-console-staging, raxx-console-prod
3. Session bootstrap: HEROKU_API_KEY exported by session-bootstrap.sh for agent sessions

There is no GitHub Actions secret in this picture — GHA deploys were retired 2026-07-10 (ADR-0135/ADR-0136) — and CI/CD (Woodpecker) has no static copy; it re-reads vault live on every pipeline run.

Drift symptom: CI deploy fails with Error: The token provided to HEROKU_API_KEY is invalid. See heroku-api-key-drift-recovery.md for recovery procedure.

Proactive check (monthly):

# Confirm vault key authenticates to Heroku (no value printed)
VAULT_KEY=$(infisical secrets get HEROKU_API_KEY \
  --env=prod --path=/MooseQuest/heroku --plain)
STATUS=$(curl -sS -o /dev/null -w "%{http_code}" \
  -H "Accept: application/vnd.heroku+json; version=3" \
  -H "Authorization: Bearer $VAULT_KEY" \
  https://api.heroku.com/account)
echo "Vault key HTTP status: $STATUS"   # expect 200

Escalation

Wake the operator when:

• A rotation attempt fails at the vault-write step (new token created but not stored — two valid tokens in flight; revoke the new one and restart).
• The old authorization cannot be identified by suffix matching (possible if the key was set via a mechanism that does not create a named authorization).
• Any rotation step returns an unexpected 5xx from Heroku.
• More than one rotation attempt leaves orphan authorizations and deduplication is unclear.

Contact: ops@raxx.app or Slack D0AJ7K184TV.

References

• Full per-step SOP: docs/ops/runbooks/rotation/heroku-platform-token.md
• Drift recovery: docs/ops/runbooks/heroku-api-key-drift-recovery.md
• Session bootstrap: docs/ops/runbooks/session-bootstrap.md
• Heroku OAuth docs: https://devcenter.heroku.com/articles/oauth
• Heroku Platform API: https://devcenter.heroku.com/articles/platform-api-reference
• Issue: #251