Raxx · internal docs

internal · gated

Departing Employee Deprovisioning Runbook

Owner: operator Scope: Audit infrastructure access — Heroku Postgres raptor_app credential + audit_archiver SSM credential Card: #1495 (SC-A15) Phase: Audit Phase 2 (operational runbook — lands in parallel with Phase 2 code cards) Depends on: SC-A2 (audit roles and credential model) Target window: All steps completable within 4 hours by a single operator SOC-2 ref: CC6.3 — Logical access termination Last verified: 2026-05-19 UTC (operator dry-run on staging) Next review due: 2026-08-19 UTC (quarterly — see SC-A16 attestation runbook)

Scope note: This runbook covers audit infrastructure access only: RBAC group removal, Console session revocation, Heroku Postgres raptor_app credential rotation, and audit_archiver SSM credential rotation. It does not cover broader offboarding (GitHub org, Heroku collaborator removal, Cloudflare allowlist removal, Google Workspace suspension, etc.). Those are separate procedures.

Runbook staleness: If Velvet takes over raptor_app credential rotation (planned post-v1), update Steps 2–3 to use the Velvet rotation flow and bump "Last verified". The SOC-2 quarterly attestation schedule (SC-A16) is the forcing function for review.

When to use this runbook

Execute within 4 hours of confirmed employee departure notice whenever the departing person held any of the following:

If in doubt, execute all six steps. The steps are idempotent — rotating credentials that the person did not hold causes no harm beyond a short service restart.

Prerequisites

Before starting:

Step 1 — Revoke Console + Antlers admin access

Estimated time: 10 minutes

1a. Remove RBAC group membership

Navigate to https://console.raxx.app/console/rbac/grants (prod) or https://console-staging.raxx.app/console/rbac/grants (staging dry-run).

  1. Search for the departing employee by email.
  2. For each group membership shown, click Revoke and confirm.
  3. Verify the employee no longer appears in any group after removal.

Alternatively, if the Console grants UI is unavailable, execute directly against the Console Postgres database (use heroku pg:psql -a raxx-console-prod):

-- Identify the user record
SELECT id, email FROM admins WHERE email = '<departing-email>';

-- Review current group memberships
SELECT g.name, gm.granted_at, gm.granted_by
FROM rbac_user_groups gm
JOIN rbac_groups g ON g.id = gm.group_id
WHERE gm.user_id = <admin_id>;

-- Remove all group memberships (run once per group_id shown above)
DELETE FROM rbac_user_groups WHERE user_id = <admin_id>;

-- Verify removal
SELECT COUNT(*) FROM rbac_user_groups WHERE user_id = <admin_id>;
-- Expected: 0

Revoking group membership automatically invalidates the RBAC session cache for the affected user on their next request. Proceed to Step 1b to force immediate session termination.

1b. Revoke active Console sessions

Navigate to https://console.raxx.app/console/admins/online (requires superadmin).

  1. Locate the departing employee in the active sessions list.
  2. For each active session shown, click Revoke and confirm.
  3. Verify no active sessions remain for the departing employee.

Alternatively, via direct DB:

-- List active sessions (Console Postgres: heroku pg:psql -a raxx-console-prod)
SELECT id, started_at, last_seen_at, user_agent
FROM admin_sessions
WHERE admin_id = <admin_id>
  AND revoked_at IS NULL
ORDER BY last_seen_at DESC;

-- Hard-revoke all sessions
UPDATE admin_sessions
SET revoked_at = NOW()
WHERE admin_id = <admin_id>
  AND revoked_at IS NULL;

-- Confirm
SELECT COUNT(*) FROM admin_sessions WHERE admin_id = <admin_id> AND revoked_at IS NULL;
-- Expected: 0

Checkpoint 1: Group memberships removed and all sessions revoked. The departing employee is locked out of Console and Antlers immediately.

Step 2 — Rotate the raptor_app Postgres credential

Estimated time: 30 minutes (including restart and smoke)

The raptor_app Heroku Postgres credential is stored in Infisical at:

/raxx/prod/raptor/RAPTOR_APP_DB_URL

Rotating this credential involves three sub-steps: generate a new password via Heroku's RDS-aware path, update Infisical, and restart Raptor dynos.

Important: Use heroku pg:credentials:rotate, not direct ALTER ROLE ... PASSWORD. Heroku Postgres runs on AWS RDS; the owner credential is not in rds_password and cannot set passwords via raw SQL. See docs/ops/runbooks/raptor-postgres-roles.md — Failure mode A.

2a. Rotate via Heroku CLI

# Rotate the raptor_app credential (this generates a new password in Heroku's RDS path)
heroku pg:credentials:rotate DATABASE --name raptor_app -a raxx-api-prod

# Retrieve the new connection URL (contains the new password)
heroku pg:credentials:url DATABASE --name raptor_app -a raxx-api-prod
# Copy the "Connection URL:" value — it begins with postgres://raptor_app:...

2b. Update Infisical at the canonical path

The raptor_app credential must be kept in sync in Infisical so that Velvet and any future automation can read it from the vault rather than the Heroku CLI. Update the secret at /raxx/prod/raptor/RAPTOR_APP_DB_URL:

# Write to Infisical via REST (replace <NEW_URL> with the value from 2a)
# Silence stdout — the URL contains the password.
curl -sf -X PATCH "https://vault.raxx.app/api/v3/secrets/raw/RAPTOR_APP_DB_URL" \
  -H "Authorization: Bearer $(cat ~/.raxx-vault-token)" \
  -H "Content-Type: application/json" \
  -d '{"workspaceId":"<workspace-id>","environment":"prod","secretPath":"/raxx/prod/raptor","secretValue":"<NEW_URL>"}' \
  >/dev/null

If vault access is unavailable (bootstrap circularity or token expiry), store the new URL temporarily in SSM at /raxx/prod/raptor/RAPTOR_APP_DB_URL and sync to Infisical at the next vault session. Do not leave the credential only in Heroku config — Infisical is the source of truth.

2c. Set the Heroku config var and restart

# Set the new URL as the Heroku config var (stdout silenced — contains password)
heroku config:set RAPTOR_APP_DATABASE_URL="<NEW_URL>" -a raxx-api-prod >/dev/null 2>&1

# Restart Raptor to pick up the new credential
heroku restart -a raxx-api-prod

# Tail logs to confirm clean startup
heroku logs -a raxx-api-prod --tail --num 30
# Watch for: no "password authentication failed for user raptor_app" errors
# Watch for: Raptor requests processing normally (200s in the log stream)

2d. Verify with smoke test

scripts/ci/run_smoke.sh --env=prod

Expected: all smoke checks pass. If the smoke test fails with a Postgres authentication error, re-check that the config var was set correctly:

heroku config:get RAPTOR_APP_DATABASE_URL -a raxx-api-prod | cut -c1-35
# Expected prefix: postgres://raptor_app:

Checkpoint 2: raptor_app credential rotated, Infisical updated, Raptor restarted, smoke green. The old credential is invalid.

Step 3 — Rotate the audit_archiver SSM credential

Estimated time: 20 minutes

Rotate only if the departing employee had direct Postgres access for archival operations. If the departing employee had Console-only access (no direct DB access), skip to Step 4.

The audit_archiver credential lives in AWS SSM Parameter Store under:

/raxx/prod/raptor/AUDIT_ARCHIVER_DB_URL

Per feedback_aws_workloads_use_ssm_not_vault.md, AWS-resident workload secrets live in SSM, not Infisical.

3a. Rotate via Heroku CLI

# Rotate the audit_archiver credential
heroku pg:credentials:rotate DATABASE --name audit_archiver -a raxx-api-prod

# Retrieve the new connection URL
heroku pg:credentials:url DATABASE --name audit_archiver -a raxx-api-prod
# Copy the "Connection URL:" value — begins with postgres://audit_archiver:...

3b. Update SSM

# Write new credential to SSM (stdout silenced — contains password)
aws ssm put-parameter \
  --name "/raxx/prod/raptor/AUDIT_ARCHIVER_DB_URL" \
  --type SecureString \
  --value "<NEW_URL>" \
  --region us-east-1 \
  --overwrite \
  >/dev/null 2>&1

# Verify the parameter was written without exposing the value
aws ssm get-parameter \
  --name "/raxx/prod/raptor/AUDIT_ARCHIVER_DB_URL" \
  --with-decryption \
  --region us-east-1 \
  --query "Parameter.Value" \
  --output text | cut -c1-35
# Expected prefix: postgres://audit_archiver:

3c. Update the Heroku Scheduler job (if applicable)

If the audit_archiver credential is referenced by a Heroku Scheduler job on raxx-api-prod, update the job's environment:

  1. Navigate to the Heroku dashboard → raxx-api-prod → Add-ons → Heroku Scheduler.
  2. Locate the archiver job (command: python jobs/audit_archiver.py).
  3. If the job reads AUDIT_ARCHIVER_DB_URL from an environment variable set directly on the dyno, update it:
heroku config:set AUDIT_ARCHIVER_DB_URL="<NEW_URL>" -a raxx-api-prod >/dev/null 2>&1
  1. Trigger a manual run to confirm the job succeeds with the new credential:
heroku run python jobs/audit_archiver.py --dry-run -a raxx-api-prod
# Expected: job exits 0 with no authentication errors

Checkpoint 3: audit_archiver SSM credential rotated, Heroku Scheduler job (if any) updated, manual dry-run confirmed green.

Step 4 — Verify: Raptor operates correctly with the new raptor_app credential

Estimated time: 5 minutes

Run the Raptor smoke suite against production and confirm zero Postgres authentication errors in the 10 minutes following the credential rotation.

# Full smoke suite
scripts/ci/run_smoke.sh --env=prod

# Tail prod logs for Postgres errors for 5 minutes
heroku logs -a raxx-api-prod --tail --num 100 | grep -i "authentication failed\|password\|FATAL\|ERROR" || echo "No Postgres auth errors found"

Expected: smoke suite exits 0, no Postgres auth errors in logs.

If smoke fails with a permission denied error (not an authentication error), check for grant drift per docs/ops/runbooks/raptor-postgres-roles.md § "Re-apply grants if drift occurs".

Checkpoint 4: Raptor production is operating normally on the new credential.

Step 5 — Verify: departing employee Console session is invalidated

Estimated time: 5 minutes

Confirm that no active Console sessions remain for the departing employee by re-querying the Console database (the same query as Step 1b):

heroku pg:psql -a raxx-console-prod -c "
SELECT COUNT(*) AS active_sessions
FROM admin_sessions
WHERE admin_id = (SELECT id FROM admins WHERE email = '<departing-email>')
  AND revoked_at IS NULL;
"
# Expected: active_sessions = 0

If the count is non-zero, return to Step 1b and revoke the remaining sessions.

Checkpoint 5: Zero active Console sessions for the departing employee.

Step 6 — Log the deprovisioning event in audit_archival_runs

Estimated time: 5 minutes

Record the deprovisioning event manually in audit_archival_runs (the audit infrastructure manifest table) to close the audit trail for this operation.

Connect to the Raptor prod Postgres as the owner credential (migration credential):

heroku pg:psql -a raxx-api-prod

Insert the deprovisioning record:

INSERT INTO audit_archival_runs (
    run_type,
    operator_email,
    freescout_ticket_id,
    notes,
    completed_at
)
VALUES (
    'deprovisioning',
    '<your-operator-email>',
    '<DEPROV-YYYY-MM-DD-ticket-id>',
    'Departing employee: <departing-email>. Steps completed: RBAC group removal, Console session revocation, raptor_app rotation, audit_archiver rotation (if applicable). Window: <start-time UTC> to <end-time UTC>.',
    NOW()
);

If audit_archival_runs does not yet have a deprovisioning run_type (SC-A13 may not have landed yet), insert with run_type = 'manual_deprovisioning' and note that the schema update is pending.

Checkpoint 6: Deprovisioning event logged in audit_archival_runs with FreeScout ticket reference and operator email. This row is append-only (INSERT + SELECT only for raptor_app; no UPDATE or DELETE).

Summary verification checklist

Complete all items before closing the FreeScout ticket:

Close the FreeScout ticket with a note: "Deprovisioning complete. Window: to . All 6 steps confirmed."

Timing guidance

Step Estimated time Cumulative
Step 1 — RBAC + session revocation 10 min 0:10
Step 2 — raptor_app rotation + smoke 30 min 0:40
Step 3 — audit_archiver rotation (if applicable) 20 min 1:00
Step 4 — Prod smoke verification 5 min 1:05
Step 5 — Session verification 5 min 1:10
Step 6 — Audit log entry 5 min 1:15

Total: ~75 minutes under normal conditions. The 4-hour window exists to accommodate interruptions, credential rotation delays, and escalation paths described below.

Escalation

Stop and escalate to Kristerpher if:

Rollback: if raptor_app rotation breaks production

The credential rotation is not directly reversible (the old password is gone from Heroku's RDS once rotated). Recovery path:

  1. Generate another rotation immediately (Step 2a–2c again).
  2. If Raptor is hard-down and can't wait for credential propagation, set FLAG_RAPTOR_APP_ROLE_SEPARATION=0 to fall back to the owner DATABASE_URL:
heroku config:set FLAG_RAPTOR_APP_ROLE_SEPARATION=0 -a raxx-api-prod >/dev/null 2>&1
heroku restart -a raxx-api-prod

This is a temporary fallback only. Re-enable the flag once the new raptor_app credential is confirmed working.

Phase 2+ note

When Velvet's Postgres credential rotation handler ships (planned post-v1), Steps 2–3 will be replaced by a Velvet rotation flow. At that point:

  1. Update this runbook to reference the Velvet SOP instead of the manual CLI steps.
  2. Bump "Last verified" and trigger a SOC-2 quarterly attestation dry-run.
  3. File the update as a follow-up to SC-A16.

Cross-references