Velvet Operator Runbook

Last verified against: Velvet v2 (2026-05-04 UTC) Parent epic: #907 Design doc: docs/architecture/velvet/v2-rotation-flows.md Handler-author guide: docs/architecture/velvet-handler-author-guide.md

Reading time: ~15 min

1. When to use Velvet vs. manual rotation

2. Pre-flight checklist

Complete every item before triggering a rotation. A stalled pre-flight is cheaper than a stalled distribute.

• [ ] Check Velvet health — both environments should return HTTP 200 with {"status": "ok"}:

curl -sf https://raxx-velvet-prod.herokuapp.com/healthz curl -sf https://raxx-velvet-staging.herokuapp.com/healthz

If either returns non-200 or times out, stop. Do not rotate against a degraded Velvet.

• [ ] Open a FreeScout ticket for this rotation. You will need the ticket ID at the revoke confirmation gate. Format: ROT-YYYY-CRED_NAME (example: ROT-2026-05-HK_PLATFORM_FULL).

• [ ] Confirm the feature flag is on:

curl -sf https://raxx-velvet-prod.herokuapp.com/flags | python3 -m json.tool | grep velvet_v2_rotation

Expected: "velvet_v2_rotation": true. If false, rotation endpoints return 503 and you must use the manual vendor SOP.

• [ ] Confirm you have the correct environment — look at the console environment banner. Red = prod, purple = staging. Do not rotate prod credentials against the staging Velvet app.

• [ ] Confirm the credential is listed in the manifest — inspect docs/architecture/velvet/subscription-manifest.yml or call:

curl -sf https://raxx-velvet-prod.herokuapp.com/tokens/HK_PLATFORM_FULL/subscribers

You should see the expected consumer list. If the list is empty or the endpoint returns 404, the credential is not registered.

• [ ] Check the current job history for recent failures on this credential:

curl -sf "https://raxx-velvet-prod.herokuapp.com/tokens/HK_PLATFORM_FULL/rotations?limit=5"

If the most recent job is in distribute_partial or revoke_failed, resolve it before starting a new job. Two overlapping rotation jobs for the same credential are not supported.

• [ ] Announce in #ops-internal (Slack) that a rotation is starting: credential name, job type (operational / revocation / testing), ticket ID.

3. Triggering a rotation

3a. Console UI (preferred)

1. Navigate to the console: https://raxx-console-prod.herokuapp.com/security/secrets
2. Locate the credential row in the Secrets table.
3. Click Rotate — this opens the Stage Wizard modal.
4. Follow the three-panel flow: Stage 1 (Verify) → Stage 2 (Mint + Distribute) → Stage 3 (Validate + Revoke).
5. Each stage requires explicit operator action before advancing. You can abort at any stage.

3b. API (for scripted or emergency use)

All endpoints require a rotate-scoped service token in the Authorization: Bearer <token> header.

Step 1 — Create the job:

POST https://raxx-velvet-prod.herokuapp.com/tokens/HK_PLATFORM_FULL/rotate
Content-Type: application/json

{
  "flow_type": "operational",
  "idempotency_key": "<uuid-v4>",
  "force_revoke": false
}

Response (202):

{ "job_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "status": "init" }

Step 2 — Run Stage 1 (Verify):

POST https://raxx-velvet-prod.herokuapp.com/tokens/HK_PLATFORM_FULL/rotations/<job_id>/stage
Content-Type: application/json

{ "action": "verify" }

Step 3 — Proceed to Mint + Distribute:

POST .../rotations/<job_id>/stage
{ "action": "proceed_mint" }

Step 4 — Proceed to Revoke (after validating all consumers):

POST .../rotations/<job_id>/stage
{ "action": "proceed_revoke" }

Polling for status:

GET https://raxx-velvet-prod.herokuapp.com/tokens/HK_PLATFORM_FULL/rotations/<job_id>

SSE stream (real-time status):

GET https://raxx-velvet-prod.herokuapp.com/tokens/HK_PLATFORM_FULL/rotations/<job_id>/stream
Accept: text/event-stream

4. Monitoring in-flight rotations

Console UI

The Stage Wizard shows live status via SSE. Each consumer row updates in real time. A spinner indicates in-progress; green check = succeeded; amber = in-progress; red X = failed.

API polling

Poll GET /tokens/{name}/rotations/{job_id} every 5 seconds. The response includes:

{
  "job_id": "...",
  "status": "distributing",
  "consumers": [
    { "consumer_id": "heroku-config-console-prod", "distribute_status": "succeeded" },
    { "consumer_id": "heroku-config-api-prod",     "distribute_status": "in_progress" },
    { "consumer_id": "github-actions-heroku-api-key", "distribute_status": "pending" }
  ],
  "created_at": "2026-05-04T06:00:00Z",
  "updated_at": "2026-05-04T06:00:42Z"
}

Heroku logs

heroku logs --tail --app raxx-velvet-prod | grep job_id=<your-job-id>

5. Job states reference

Every rotation_jobs row progresses through this state machine. The status column tells you exactly where a job is.

Revocation flow statuses:

6. Stuck job diagnosis and recovery

Definition of stuck: A job has been in the same status for more than 5 minutes without a state change, OR a job is in a terminal state (distribute_partial, revoke_failed, aborted) that requires operator action.

6a. Job stuck in verifying

The auth probe is timing out or being rate-limited.

1. Check Velvet logs: heroku logs --app raxx-velvet-prod | grep job_id=<id>
2. Look for ConnectionError, Timeout, or HTTP status code.
3. If the vendor API is rate-limiting: wait 2 minutes and click "Retry" in the console.
4. If the vendor API is returning 401: the current token is already invalid. Stop and use the vendor's manual revocation + re-issue process. File a FreeScout incident ticket.

6b. Job stuck in minting

1. Check Velvet logs for mint failed entries.
2. If the vendor returned 401 on the mint call, the old token drifted between Verify and Mint. This is rare but possible if two rotation jobs ran simultaneously. Abort this job; the old token is still valid.
3. If the vendor returned 5xx: retry once. If it fails again, wait 10 minutes (vendor-side transient issue) and retry.

6c. Job in distribute_partial

Some consumers received the new token; others did not. The old token is still valid.

Recovery options:

Option A (preferred): Retry failed rows in the console. Click "Retry failed" for each red row. Velvet will re-attempt the PATCH/INFISICAL_WRITE for those consumers only.

Option B (if retries keep failing): Identify the failing consumer(s) by their consumer_id in the job status. Manually push the new token to that consumer using the vendor's own interface. Once done, click "Manual confirm" in the console to mark that row as succeeded. After all rows are green, advance to Stage 3.

Option C (if you need to abort): Click Abort. The new token is now minted but distributed only to some consumers. You must manually delete the new token from vault and re-sync the affected consumers to the old token. The console shows the residual consumer list. File a FreeScout ticket and follow per-vendor SOP in docs/ops/runbooks/rotation/.

6d. Job in validate_partial or validate_failed

Healthchecks failed on one or more consumers after the new token was distributed.

1. Check which consumers are showing validate_status: failed in the job status response.
2. Confirm the consumer application has restarted and loaded the new config var. For Heroku apps: heroku ps --app <app-name> — if dyno is crashed, that's your answer.
3. Wait 60 seconds for dyno restart to complete, then click "Retry validation" in the console.
4. If a consumer has healthcheck_endpoint: null in the manifest, a "Manual confirm" button appears. Verify the consumer manually, then click confirm to mark it as validated.
5. If validation keeps failing: check whether the distribute step actually wrote the new token. Use the per-vendor SOP to verify the config var value was updated.

6e. Job in revoke_failed

The new token is distributed and validated. The old token has not been revoked.

1. Note the old_auth_id or equivalent from the job metadata (visible in the console audit summary and in Velvet logs).
2. Retry the revoke in the console ("Retry revoke" button).
3. If the vendor returns 404 on the revoke call, the old token was already deleted outside Velvet. Click "Mark manually revoked" and enter the FreeScout ticket ID. Velvet will advance to done.
4. If the vendor keeps returning errors: revoke the old token manually via the vendor dashboard or CLI. Then click "Mark manually revoked" with the FreeScout ticket ID.

6f. Job aborted from minted or distribute_partial

The new token exists in vault but the old token is still valid. Both tokens are now live simultaneously.

Cleanup required:

7. Rollback

Velvet does not support one-click rollback. Once the new token has been distributed and the old token revoked, there is no automated path back.

What is reversible:

• Before proceed_revoke: The old token is still valid. Abort the job. Manually roll back any consumers that received the new token to the old token. Delete the new token from vault.
• After done: The old token is revoked. Re-rotation is required: start a new operational rotation job to mint a fresh token.

What is NOT reversible:

• Revocation of a Cloudflare User API token: CF does not support re-activating a revoked token. You must create a new token in the CF dashboard.
• Revocation of a Heroku OAuth authorization: The authorization ID is gone. A new OAuth authorization must be minted.
• Any token that the vendor marks as single-use after deletion.

8. Rotating Velvet's own bootstrap credentials (Invariant I7)

Velvet's own credentials (INFISICAL_CLIENT_SECRET, INFISICAL_CLIENT_ID, HK_VELVET_BOOTSTRAP) are stored as Heroku config vars, not in vault, to break the bootstrap circularity. Velvet cannot rotate them itself.

Rotating INFISICAL_CLIENT_SECRET

1. In the Infisical dashboard, generate a new client secret for the Velvet machine identity.
2. Set the new value on both Heroku apps:

heroku config:set INFISICAL_CLIENT_SECRET=<new_value> --app raxx-velvet-prod >/dev/null 2>&1 heroku config:set INFISICAL_CLIENT_SECRET=<new_value> --app raxx-velvet-staging >/dev/null 2>&1

Note: always redirect to /dev/null 2>&1 — heroku config:set echoes config vars to stdout by default (feedback: heroku_config_set_echoes_secrets).

3. Verify Velvet restarts and /healthz returns 200 on both apps.
4. Revoke the old client secret in the Infisical dashboard.
5. Record the rotation in a FreeScout ticket.

Rotating HK_VELVET_BOOTSTRAP

This token is used by Velvet to authenticate its PATCH calls to Heroku config vars on behalf of consumer updates.

1. Use the Heroku Platform API or dashboard to create a new OAuth authorization for the Velvet machine user.
2. Set the new token:

heroku config:set HK_VELVET_BOOTSTRAP=<new_token> --app raxx-velvet-prod >/dev/null 2>&1 heroku config:set HK_VELVET_BOOTSTRAP=<new_token> --app raxx-velvet-staging >/dev/null 2>&1

3. Verify /healthz on both apps.
4. Revoke the old authorization via the Heroku dashboard or:

heroku authorizations:revoke <old-auth-id>

5. Update the companion secret HK_VELVET_BOOTSTRAP__AUTH_ID in Infisical with the new authorization UUID, so the next rotation can find it.

9. Failure modes by adapter

Postmark (PM_SERVER_MAIL)

Heroku (HK_PLATFORM_FULL)

Cloudflare (CF_DNS_EDIT_RAXX_APP, others)

Note: scripts/ops/probe_cf_token_perms.py reads Cloudflare token permissions directly from Infisical. It does not go through Velvet. This is intentional — it is a read-only diagnostic tool and has not been migrated to the Velvet bus.

AWS SSM (AWS_ACCESS_KEY_ID, password-class credentials)

10. SEV1 — rev_leaked response

If a revocation job reaches rev_leaked, one or more consumers returned a non-401 response after the old token was confirmed revoked at the vendor. This means at least one consumer still has a copy of the revoked token and may be accepting it.

Immediate steps:

1. You will have received a Slack DM on channel SL_BOT_NOTIFY within 30 seconds of the flag being set. The message includes the job_id, credential_name, and the list of leaking consumer_ids.
2. Open the Velvet console: the leaked consumers are highlighted in red with an "Investigate" button.
3. Click "Investigate" — this auto-creates a FreeScout ticket pre-filled with the consumer list.
4. For each leaking consumer: a. Determine whether the consumer is still actively serving traffic. b. If yes: immediately disable or restart the consumer to force it to reload config. c. Verify the consumer is no longer accepting the revoked token by re-running the healthcheck manually.
5. Once all consumers return 401, click "Confirm leak resolved" in the Velvet console. The job advances from rev_leaked to rev_done.
6. If any consumer cannot be forced to reject the token (e.g., a caching layer with a long TTL), escalate to a security incident per the security response runbook.

Root causes of rev_leaked:

• Consumer cached the token in memory and has not been restarted since rotation.
• Consumer received the new token via distribute but reverted to an old value from a local config file.
• Consumer's healthcheck endpoint is cached or proxied and is not reflecting the real auth state.

11. Staging vs. production

APP_ENV on each Heroku dyno controls which Infisical environment and SSM path prefix is used.

The subscription manifest uses env: prod and env: staging per consumer row. A rotation job on raxx-velvet-prod only fans out to consumers with env: prod.

The Heroku app consumer rows for staging config vars (raxx-console-staging, raxx-api-staging) are registered with env: prod in the manifest — this is intentional. The staging apps' config vars hold the same credential (the Heroku platform key), which is a single credential shared across environments.

12. Common operator mistakes and fixes

13. Slack DM notifications

Terminal events (job done, aborted, rev_leaked) trigger a Slack DM to the operator's channel.

Bot channel for automated alerts: SL_BOT_NOTIFY (configured in Velvet Heroku config vars). Operator DM for walk-away pings: D0AJ7K184TV (Kristerpher's DM channel).

rev_leaked alerts are additionally sent within 30 seconds to the ops alert inbox (ops@raxx.app).