Raptor DB Credential Split — Two-URL SOP, Rotation, and Incident Response

System: Raptor (raxx-api-staging, raxx-api-prod) Owner: operator / sre-agent ADR: docs/architecture/adr/0099-raptor-app-least-privilege-role.md Issues: #1455 (origin finding), #2521 (SC-1 role provision), #2522 (SC-2 seed URL), #2523 (SC-3 engine init), #2524 (SC-4 GRANT/REVOKE migration), #2525 (this runbook) Last reviewed: 2026-05-20 UTC

1. The Two-URL Model

Raptor uses two distinct Postgres connection strings with different privileges. They must never be swapped.

The invariant: DATABASE_URL is never read by the Flask application process when FLAG_RAPTOR_APP_ROLE_SEPARATION=1. If it is, the append-only DDL-level REVOKEs protecting customer_audit_events are silently bypassed (threat T-PEN-2, ADR-0099).

Why Heroku's default pattern is insufficient

Heroku exports DATABASE_URL as the owner credential. Running application queries under the owner means a bug or compromised process can execute DELETE FROM customer_audit_events regardless of any REVOKE statement — REVOKE only restricts the role it targets, not the owner. The raptor_app role, connected via RAPTOR_APP_DATABASE_URL, is explicitly REVOKEd from UPDATE and DELETE on all audit and history tables.

Where each URL lives

• Canonical copy: Infisical at /MooseQuest/raxx/<env>/RAPTOR_APP_DATABASE_URL (both prod and staging environments).
• Heroku config var: RAPTOR_APP_DATABASE_URL on raxx-api-prod / raxx-api-staging. This is seeded from Infisical by the operator during provisioning (SC-2) and re-seeded after every rotation. The Heroku config var is the bootstrap copy; Infisical is authoritative.
• DATABASE_URL: Managed entirely by Heroku. Never written to Infisical. Never touched outside of migration context and owner-credential rotation.

Feature flag

FLAG_RAPTOR_APP_ROLE_SEPARATION controls which URL the application engine chooses at startup:

The fail-fast behaviour (ADR-0099 D4) is implemented in backend_v2/api/db.py (resolve_runtime_database_url). It prevents silent owner-credential connections in Postgres-mode environments.

2. Rotating RAPTOR_APP_DATABASE_URL (raptor_app credential)

Use this procedure when: - Velvet triggers a scheduled rotation for RAPTOR_APP_DATABASE_URL - A security incident requires immediate credential revocation - The raptor_app password is suspected to be compromised

heroku pg:credentials:rotate --name raptor_app generates a new password for the raptor_app role via Heroku's RDS-aware path. No DDL changes are made; the role's GRANTs/REVOKEs are preserved.

Step 1 — Rotate the credential on staging first

heroku pg:credentials:rotate DATABASE --name raptor_app -a raxx-api-staging

Heroku generates a new password. The old password is immediately invalid after this command completes.

Step 2 — Retrieve the new connection URL

heroku pg:credentials:url DATABASE --name raptor_app -a raxx-api-staging
# Copy the "Connection URL:" value (postgres://raptor_app:<newpass>@...).
# Do not print it to a shared terminal session.

Step 3 — Update Infisical (canonical copy)

Write the new URL to Infisical at:

/MooseQuest/raxx/staging/RAPTOR_APP_DATABASE_URL

Use the Infisical UI (vault.raxx.app) or the Infisical CLI. The folder must already exist; if it does not, create it via POST /api/v1/folders before writing the secret (per feedback_vault_folder_must_exist.md).

Step 4 — Update the Heroku config var (stdout silenced)

heroku config:set RAPTOR_APP_DATABASE_URL="<URL from Step 2>" -a raxx-api-staging >/dev/null 2>&1

The >/dev/null 2>&1 is mandatory. heroku config:set echoes the full value to stdout by default, which would expose the password in terminal history and any log forwarders (per feedback_heroku_config_set_echoes_secrets.md).

Step 5 — Verify the config var is set without revealing the value

heroku config:get RAPTOR_APP_DATABASE_URL -a raxx-api-staging | cut -c1-35
# Expect: postgres://raptor_app:...

Step 6 — Restart and verify

heroku restart -a raxx-api-staging
heroku logs -a raxx-api-staging --tail
# Watch for: "runtime connection using restricted raptor_app role" in startup logs.
# No "permission denied" or ConfigurationError messages.

Run the smoke suite:

scripts/ci/run_smoke.sh --env=staging

Step 7 — Repeat for prod (after staging soak)

Repeat Steps 1–6 for raxx-api-prod / /MooseQuest/raxx/prod/RAPTOR_APP_DATABASE_URL once staging has soaked cleanly for at least 30 minutes.

heroku pg:credentials:rotate DATABASE --name raptor_app -a raxx-api-prod
heroku pg:credentials:url DATABASE --name raptor_app -a raxx-api-prod
# Write new URL to Infisical /MooseQuest/raxx/prod/RAPTOR_APP_DATABASE_URL
heroku config:set RAPTOR_APP_DATABASE_URL="<URL>" -a raxx-api-prod >/dev/null 2>&1
heroku config:get RAPTOR_APP_DATABASE_URL -a raxx-api-prod | cut -c1-35
heroku restart -a raxx-api-prod
scripts/ci/run_smoke.sh --env=prod

Velvet rotation

RAPTOR_APP_DATABASE_URL is a managed secret in the Velvet rotation manifest. When the Velvet pipeline triggers this rotation, Velvet executes Steps 1–6 programmatically and updates Infisical as the canonical copy. The operator reviews the rotation job result in the console rotation UI before it is marked complete. Manual Steps 1–6 above are the fallback if Velvet is unavailable.

3. Rotating DATABASE_URL (owner credential)

Use this procedure when: - The Heroku Postgres owner credential is suspected to be compromised - A Heroku account is being deprovisioned or handed over - heroku pg:credentials:rotate (without --name) is triggered during a security incident

DATABASE_URL is the Heroku Postgres owner credential. Rotating it has no effect on the application runtime because the application uses RAPTOR_APP_DATABASE_URL (when FLAG_RAPTOR_APP_ROLE_SEPARATION=1). Migrations continue to run as the owner regardless.

Owner rotation does not restart the application

Because the application does not use DATABASE_URL at runtime, rotating it does not cause dyno restarts or application downtime. Only Alembic (run in the Heroku release dyno) uses DATABASE_URL.

Step 1 — Rotate the owner credential

heroku pg:credentials:rotate -a raxx-api-staging

Note: no --name flag. This rotates the default owner credential (DATABASE_URL).

Heroku automatically updates the DATABASE_URL config var on the app after rotation. No manual config:set is needed.

Step 2 — Verify the migration path still works

Trigger a no-op migration to confirm the release dyno can still connect with the new owner credential:

heroku run "python -m alembic -c backend_v2/db/alembic.ini current" -a raxx-api-staging
# Expect: current revision printed without error.

Step 3 — Verify the application is unaffected

heroku logs -a raxx-api-staging -n 20
# Expect: no new ConfigurationError or OperationalError lines.
# The application continues using RAPTOR_APP_DATABASE_URL (unchanged).

Step 4 — Repeat for prod

heroku pg:credentials:rotate -a raxx-api-prod
heroku run "python -m alembic -c backend_v2/db/alembic.ini current" -a raxx-api-prod

Infisical note

DATABASE_URL is not stored in Infisical. It is owned and managed entirely by Heroku. Do not write it to the vault.

4. Incident Response

Scenario A — Dyno crash-loop: RAPTOR_APP_DATABASE_URL is wrong or missing

Symptoms:

• heroku logs -a raxx-api-prod | grep "ConfigurationError" returns entries like: RAPTOR_APP_DATABASE_URL is required when FLAG_RAPTOR_APP_ROLE_SEPARATION=1 and DATABASE_URL resolves to a Postgres URL.
• All dynos are crash-looping. Heroku routes zero traffic to the app.
• heroku ps -a raxx-api-prod shows dynos in crashed or starting state cycling.

Cause options:

1. RAPTOR_APP_DATABASE_URL was never set on this app (SC-2 was not completed).
2. RAPTOR_APP_DATABASE_URL was accidentally unset (heroku config:unset RAPTOR_APP_DATABASE_URL).
3. RAPTOR_APP_DATABASE_URL is set to a malformed or stale URL (e.g., after a heroku pg:credentials:rotate --name raptor_app where Step 4 was not completed).

Immediate mitigation (< 2 minutes):

Option 1 — Disable the flag to fall back to DATABASE_URL (owner credential). This sacrifices least-privilege enforcement but restores service instantly:

heroku config:set FLAG_RAPTOR_APP_ROLE_SEPARATION=0 -a raxx-api-prod >/dev/null 2>&1
heroku restart -a raxx-api-prod
heroku logs -a raxx-api-prod -n 20
# Verify: normal request processing resumes within 30 seconds.

Option 2 — Re-set the correct URL from Infisical:

# Retrieve current value from Infisical vault at /MooseQuest/raxx/prod/RAPTOR_APP_DATABASE_URL
# (use vault.raxx.app or Infisical CLI)
heroku config:set RAPTOR_APP_DATABASE_URL="<URL from Infisical>" -a raxx-api-prod >/dev/null 2>&1
heroku restart -a raxx-api-prod
heroku logs -a raxx-api-prod -n 20

Preferred path: Option 2 preserves least-privilege enforcement. Use Option 1 only if Infisical is unreachable or the correct URL is unknown.

After recovery:

• If Option 1 was used: file a follow-up card to re-enable FLAG_RAPTOR_APP_ROLE_SEPARATION=1 once the root cause is resolved.
• Confirm Infisical /MooseQuest/raxx/prod/RAPTOR_APP_DATABASE_URL holds the valid URL.
• Confirm heroku config:get RAPTOR_APP_DATABASE_URL -a raxx-api-prod | cut -c1-35 starts with postgres://raptor_app:.

Scenario B — raptor_app role missing from Postgres

Symptoms:

FATAL: role "raptor_app" does not exist

in Heroku logs. The URL format is valid but the Postgres role itself has been dropped.

Cause: Someone ran DROP ROLE raptor_app or heroku pg:credentials:destroy --name raptor_app. This should never happen in normal operations.

Recovery:

Re-provision the role from scratch using the procedure in docs/ops/runbooks/raptor-postgres-roles.md. This requires: 1. heroku pg:credentials:create DATABASE --name raptor_app -a <app> 2. Retrieve the new URL and seed it (Steps 2–4 of Section 2 above). 3. Re-apply the GRANT/REVOKE matrix from backend_v2/db/migrations/031_audit_role_split.sql via heroku pg:psql. 4. Re-enable FLAG_RAPTOR_APP_ROLE_SEPARATION=1 and restart.

Do not drop the role on purpose unless decommissioning the entire environment.

Scenario C — permission denied for table in application logs

Symptoms:

sqlalchemy.exc.OperationalError: (psycopg2.errors.InsufficientPrivilege)
permission denied for table <table_name>

The raptor_app role exists and the URL is correct, but a specific table is missing a grant.

Cause: A migration added a new table without including an explicit GRANT ... TO raptor_app statement. GRANT ... ON ALL TABLES applies only to tables that exist at the time the statement runs; tables added later do not inherit permissions automatically.

Recovery:

heroku pg:psql -a raxx-api-prod -c "GRANT SELECT, INSERT, UPDATE, DELETE ON <table_name> TO raptor_app;"
# If the table has a sequence:
heroku pg:psql -a raxx-api-prod -c "GRANT USAGE ON SEQUENCE <table_name>_id_seq TO raptor_app;"

Then file a follow-up to add the explicit GRANT to the migration file for the new table. See the "Adding a new table after Phase 0" section in docs/ops/runbooks/raptor-postgres-roles.md.

Scenario D — RAPTOR_APP_DATABASE_URL was overridden by a config push

Symptoms: heroku config:get RAPTOR_APP_DATABASE_URL -a raxx-api-prod returns a value that does not start with postgres://raptor_app:. This can happen if a bulk config push (e.g., heroku config:push from a .env file) overwrote the var with DATABASE_URL's value.

Diagnosis:

heroku config:get RAPTOR_APP_DATABASE_URL -a raxx-api-prod | cut -c1-35
# If this starts with "postgres://raxx..." (the owner prefix), not "postgres://raptor_app...",
# the var has been overwritten with the owner URL.

Recovery: Re-seed from Infisical using Option 2 in Scenario A above.

5. Environment Variable Reference

6. Velvet Rotation Manifest

RAPTOR_APP_DATABASE_URL must be listed in docs/architecture/velvet/subscription-manifest.yml as a managed secret so Velvet can automate rotation on cadence and push the updated URL to both Infisical and the Heroku config var after each heroku pg:credentials:rotate --name raptor_app call.

As of 2026-05-20, the Velvet manifest does not yet include a RAPTOR_APP_DATABASE_URL entry. Adding it requires a separate PR to subscription-manifest.yml following the pattern of existing entries (e.g., PM_SERVER_MAIL). The rotation mode is operator-assisted because heroku pg:credentials:rotate does not have a fully programmable API path that Velvet can call autonomously — an operator must trigger the Heroku rotation step and then hand the new URL to Velvet for distribution.

Until the manifest entry ships, rotate manually using the procedure in Section 2.

A card to add RAPTOR_APP_DATABASE_URL to the Velvet manifest should be filed under the Velvet rotation pipeline work. Reference this runbook and ADR-0099 in that card.

7. Local Development

Local dev does not require RAPTOR_APP_DATABASE_URL. When DATABASE_URL is absent or resolves to an sqlite:// URL, resolve_runtime_database_url (in backend_v2/api/db.py) falls through to the SQLite path regardless of the flag state. No fail-fast is triggered.

If you want to test the Postgres path locally:

export DATABASE_URL=postgresql://localhost/raptor_dev
export RAPTOR_APP_DATABASE_URL=postgresql://localhost/raptor_dev
export FLAG_RAPTOR_APP_ROLE_SEPARATION=1

Both vars pointing to the same local database is acceptable for development. REVOKE enforcement is not tested locally — that runs in staging against the real role split.

8. CI Integration Tests

The CI integration-test job sets:

env:
  DATABASE_URL: postgresql://postgres:postgres@localhost/raptor_test
  RAPTOR_APP_DATABASE_URL: postgresql://postgres:postgres@localhost/raptor_test

Both variables point to the same ephemeral test database, and the CI Postgres user is not the restricted raptor_app role. This is intentional: the REVOKE verification test (SC-4, #2524) runs against staging with the real role split, not in CI. The CI integration tests verify that the engine initialises correctly and that application logic works; they do not verify REVOKE enforcement.

9. Fast Rollback

No code deploy required. One config:set + restart:

heroku config:set FLAG_RAPTOR_APP_ROLE_SEPARATION=0 -a raxx-api-prod >/dev/null 2>&1
heroku restart -a raxx-api-prod
heroku logs -a raxx-api-prod -n 20
# Verify: Raptor back up, using DATABASE_URL (owner credential).

The raptor_app role and its grants are preserved. Re-enabling is one config:set.

heroku config:set FLAG_RAPTOR_APP_ROLE_SEPARATION=1 -a raxx-api-prod >/dev/null 2>&1
heroku restart -a raxx-api-prod

10. Connects To

• ADR: docs/architecture/adr/0099-raptor-app-least-privilege-role.md
• Role provisioning + GRANT matrix: docs/ops/runbooks/raptor-postgres-roles.md
• Rotation index: docs/ops/runbooks/rotation/INDEX.md
• Engine init implementation: backend_v2/api/db.py (resolve_runtime_database_url)
• Migration: backend_v2/db/migrations/031_audit_role_split.sql
• Audit design §11: docs/architecture/customer-audit-unified/design.md
• Issues: #1455 (origin), #2521 (SC-1), #2522 (SC-2), #2523 (SC-3), #2524 (SC-4)
• Feedback: feedback_heroku_config_set_echoes_secrets.md, feedback_heroku_pg_rds_password_gotcha.md, feedback_vault_folder_must_exist.md