Cloudflare API tokens runbook

System: Cloudflare API Tokens (User-level, zone-scoped) Owner: sre-agent Last incident: 2026-06-18 (BFM scope enumeration during CI→vault block; see docs/ops/incidents/2026-06-18-bfm-disabled-window.md) Last reviewed: 2026-06-18 UTC Related issues: #755 (provision CF_WAF_EDIT_RAXX_APP + CF_DNS_EDIT_GETRAXX_COM), #754 (token rename taxonomy), #81 (SDLC hardening epic)

Inventory

All tokens are stored in Infisical at /MooseQuest/cloudflare/ (env: prod). Each credential has three companion secrets:

Note on CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN: this is the broadest-scope token in the vault. It carries Zone:Bot Management:Write (CF group id 3b94c49258ec4573b06d51d99b6416c0) which is NOT present on any other vaulted CF token. It is the only token that can read or write the /zones/{zone_id}/bot_management endpoint. Use it for BFM enable/disable operations only; prefer the narrower CF_WAF_EDIT_RAXX_APP for WAF rule management.

Origin CA certificate issuance (added 2026-06-19): CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN has API Tokens:Write scope and can be used to mint short-lived tokens with Zone:SSL and Certificates:Write scope (CF permission group ID: c03055bc037c4ea9afb9a9f104b7b721). This is the correct path for issuing Cloudflare Origin Certificates for raxx.app zone origins (see freescout-cert-renewal.md Failure Mode E). No other vaulted token has SSL+Certificates:Write scope. CF_WAF_EDIT_RAXX_APP does NOT have this scope (confirmed 2026-06-19 incident — returned error 9109 on permission_groups endpoint).

Companion secret naming convention (per docs/archive/secrets/cf-token-taxonomy.md):

<TOKEN_NAME>__EXPIRES_AT   # ISO-8601 expiry timestamp
<TOKEN_NAME>__SCOPES       # Human-readable scope string
<TOKEN_NAME>__TOKEN_ID     # Cloudflare token ID (not the secret value)

Zone IDs:

The minting token (CLOUDFLARE_ACCESS_MGMT_TOKEN / CF_ACCESS_MGMT) has User:API Tokens:Edit (i.e., API Tokens Write) scope and is used to mint, roll, and revoke all other CF API tokens programmatically.

How to tell it's broken

• CF API call returns HTTP 403 with {"message":"request is not authorized"} — scope mismatch; see failure mode A below.
• CF API call returns HTTP 401 with {"code":1000,"message":"Invalid API token"} — token expired, revoked, or wrong value in vault.
• Console status poller shows a CF-dependent surface as degraded — check Heroku config var matches vault.
• A workflow using CF_WAF_EDIT_RAXX_APP returns 403 on the WAF entrypoint — token missing WAF:Edit scope or scoped to wrong zone.

How to diagnose (in order)

1. Identify which token the failing workflow uses. The workflow README or this runbook's consumer table is the source of truth.

2. Verify the token is active and verify its scope:

```bash # Read token from vault TOKEN=$(infisical secrets get CF_WAF_EDIT_RAXX_APP \ --path /MooseQuest/cloudflare --env prod --plain)

# Verify with Cloudflare curl -sS \ -H "Authorization: Bearer ${TOKEN}" \ "https://api.cloudflare.com/client/v4/user/tokens/verify" | python3 -m json.tool # Expect: {"success":true,"result":{"status":"active",...}} ```

3. Check the __EXPIRES_AT companion to see if the token has expired:

bash infisical secrets get CF_WAF_EDIT_RAXX_APP__EXPIRES_AT \ --path /MooseQuest/cloudflare --env prod --plain

4. If the token is active but the workflow still returns 403, verify you are calling an endpoint within the token's zone. CF_WAF_EDIT_RAXX_APP is scoped to raxx.app only — it will return 403 on any other zone's WAF endpoint.

5. If the token is expired or invalid, rotate it — see "Known failure modes" below.

Consumer map

Known failure modes

Failure mode A: 403 on CF API — wrong scope or wrong zone

Symptom: {"message":"request is not authorized"} on a WAF or DNS endpoint.

Cause: Token does not have the required permission, OR the request targets a zone that is outside the token's zone restriction.

Fix: - Confirm you are using the right token for the right zone (see inventory table above). - If the token is correct but 403 persists, check the token's policy in the CF dashboard or via:

bash MGMT_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \ --path /MooseQuest/cloudflare --env prod --plain) TOKEN_ID=$(infisical secrets get CF_WAF_EDIT_RAXX_APP__TOKEN_ID \ --path /MooseQuest/cloudflare --env prod --plain) curl -sS \ -H "Authorization: Bearer ${MGMT_TOKEN}" \ "https://api.cloudflare.com/client/v4/user/tokens/${TOKEN_ID}" | python3 -m json.tool

Confirm policies[].resources contains only the expected zone ID, and permission_groups includes WAF Write (ID: fb6778dc191143babbfaa57993f1d275) or DNS Write (ID: 4755a26eedb94da69e1066d98aa820be).

Verification: Repeat the failing API call and confirm HTTP 200.

Failure mode B: 401 — token invalid (expired or revoked)

Symptom: {"code":1000,"message":"Invalid API token"} on any CF API call.

Cause: Token expired (past expires_on), was revoked manually, or the vault value drifted from CF's actual token secret (e.g., a roll happened outside Velvet).

Fix:

1. Check __EXPIRES_AT companion to confirm expiry.

2. Roll the token via Velvet (preferred — see "Rotation via Velvet" section below), or use the programmatic roll path:

```bash MGMT_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \ --path /MooseQuest/cloudflare --env prod --plain) TOKEN_ID="d0cf0adeda107092f6e8c0646460f8f8" # CF_WAF_EDIT_RAXX_APP

NEW_VALUE=$(curl -sS -X PUT \ -H "Authorization: Bearer ${MGMT_TOKEN}" \ -H "Content-Type: application/json" \ "https://api.cloudflare.com/client/v4/user/tokens/${TOKEN_ID}/value" \ -d '{}' | python3 -c "import sys,json; print(json.load(sys.stdin)['result'])") # $NEW_VALUE is shown exactly once — do not echo it ```

3. Write the new value to vault (do NOT echo):

bash # Using Infisical API — omit stdout

See the rotation SOP at docs/ops/runbooks/rotation/cloudflare-user-api-token.md for the full propagate+verify+audit sequence.

Verification: GET /user/tokens/verify with the new value returns HTTP 200 and "status":"active".

Failure mode C: vault value missing or path not found

Symptom: infisical secrets get CF_WAF_EDIT_RAXX_APP ... returns 404.

Cause: Secret was deleted from Infisical, or path/env mismatch in the CLI call.

Fix:

1. Confirm the path is /MooseQuest/cloudflare and env is prod.
2. If genuinely missing: re-provision using the provisioning procedure below.

Failure mode D: CF_ACCESS_MGMT token itself is invalid

Symptom: All token minting/rolling operations fail with 401. The management token (CF_ACCESS_MGMT) is expired or revoked.

Cause: 90-day rotation cadence missed, or the token was revoked externally.

Fix: Escalate to operator. CF_ACCESS_MGMT must be rotated via the CF dashboard (dashboard fallback path in cloudflare-user-api-token.md) because it cannot roll itself — it requires a token with API Tokens Edit scope, which is itself.

Provisioning a new zone-scoped token

Use this procedure when a new zone-specific CF token is needed (e.g., a new zone is added to the account, or a new permission is required).

Prerequisites

• CF_ACCESS_MGMT (vault: /MooseQuest/cloudflare/CF_ACCESS_MGMT) with API Tokens Write scope — confirmed active.
• Zone ID for the target zone (see zone table above, or look up via GET /zones).
• Permission group IDs — retrieve fresh via GET /user/tokens/permission_groups.

Key permission group IDs (as of 2026-05-04)

Steps

1. Authenticate to Infisical and retrieve CF_ACCESS_MGMT token value.

2. Mint the token:

```bash MGMT_TOKEN="" ZONE_ID="" EXPIRES_AT="" # 90 days from today

RESP=$(curl -sS -X POST \ -H "Authorization: Bearer ${MGMT_TOKEN}" \ -H "Content-Type: application/json" \ "https://api.cloudflare.com/client/v4/user/tokens" \ -d "{ \"name\": \"CF__\", \"policies\": [{ \"effect\": \"allow\", \"resources\": {\"com.cloudflare.api.account.zone.${ZONE_ID}\": \"*\"}, \"permission_groups\": [ {\"id\": \"\"}, {\"id\": \"c8fed203ed3043cba015a93ad1616f1f\"} ] }], \"not_before\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\", \"expires_on\": \"${EXPIRES_AT}\" }") # Capture token ID and value — value shown once only TOKEN_ID=$(echo "$RESP" | python3 -c "import sys,json; print(json.load(sys.stdin)['result']['id'])") TOKEN_VALUE=$(echo "$RESP" | python3 -c "import sys,json; print(json.load(sys.stdin)['result']['value'])") ```

3. Verify the token against the target endpoint (e.g., WAF phase entrypoint or DNS records list).

4. Write to Infisical at /MooseQuest/cloudflare/: - <TOKEN_NAME> = token value - <TOKEN_NAME>__EXPIRES_AT = expiry timestamp - <TOKEN_NAME>__SCOPES = human-readable scope string - <TOKEN_NAME>__TOKEN_ID = CF token ID

5. Add a row to docs/ops/runbooks/rotation/INDEX.md.

6. Add an entry to this runbook's inventory table and consumer map.

Rotation via Velvet

CF User API tokens are rotated via the PUT /user/tokens/{id}/value endpoint. This is the OPERATOR_MANUAL flow in Velvet v2 because Cloudflare User API Tokens cannot be minted programmatically without a separate management token — Velvet uses the management token (CF_ACCESS_MGMT) to perform the roll.

When Velvet CF adapter is wired (#948):

1. Navigate to https://raxx-console-prod.herokuapp.com/security/secrets.
2. Locate the credential row (e.g., CF_WAF_EDIT_RAXX_APP).
3. Click Rotate to open the Stage Wizard.
4. Stage 1 (Verify): Velvet reads the current token from vault and calls GET /user/tokens/{id} to confirm it is active.
5. Stage 2 (Mint + Distribute): Velvet calls PUT /user/tokens/{id}/value using CF_ACCESS_MGMT. The rolled value is written to vault and propagated to registered consumers.
6. Stage 3 (Validate + Revoke): Velvet verifies each consumer healthcheck passes, then marks rotation as done. (CF token roll is atomic — the old value is already invalid after Stage 2.)

Manual fallback (until #948 ships):

Use the programmatic roll path documented in docs/ops/runbooks/rotation/cloudflare-user-api-token.md — Section 2 (Generate the new credential). Token IDs needed:

After rolling, update __EXPIRES_AT companion to 90 days from the rotation date.

Emergency stop

To immediately revoke a token (suspected compromise):

MGMT_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \
  --path /MooseQuest/cloudflare --env prod --plain)

# Delete the token — this revokes it immediately
curl -sS -X DELETE \
  -H "Authorization: Bearer ${MGMT_TOKEN}" \
  "https://api.cloudflare.com/client/v4/user/tokens/<token-id>"
# Expect: {"success":true,"result":{"id":"<token-id>"}}

After revocation: 1. Remove the secret from Infisical (or overwrite with REVOKED placeholder). 2. Identify all consumers from the consumer map above. 3. Remove or nullify the token from each consumer's config. 4. File a FreeScout incident ticket. 5. Provision a replacement token using the provisioning procedure above.

Escalation

Escalate to operator when:

• CF_ACCESS_MGMT itself is invalid (can't roll tokens without it — no automated path).
• A token appears active in CF but is returning 403 (scope corruption — rare but needs dashboard inspection).
• A token was deleted externally and consumers are live-impacted.
• The zone ID for a zone is not in the table above and cannot be confirmed via API.

CF dashboard (token management):

https://dash.cloudflare.com/profile/api-tokens