Billing read-only API tokens runbook

System: Billing data collectors (Heroku #760, Cloudflare #762, AWS #761) Owner: operator / sre-agent Issue: #759 Epic: #757 (Fixed-Cost Billing Dashboard) Last reviewed: 2026-05-20 UTC

Vault path for all billing credentials: /Raxx/Console/Billing/ (Infisical, env: prod)

The folder /Raxx/Console/Billing/ and its parent folders were created by sre-agent on 2026-05-11T07:05:01Z. Placeholder secrets are already present. Operator must replace each placeholder with a real token value using the steps below.

Overview — token inventory

Operator action required

All four secrets are currently set to PLACEHOLDER_OPERATOR_ACTION_REQUIRED. The collectors are gated behind feature flags (FLAG_HEROKU_BILLING_COLLECTOR, FLAG_CLOUDFLARE_BILLING_COLLECTOR, FLAG_AWS_BILLING_COLLECTOR) which are default OFF — no collector will run until both the flag is flipped AND a real token replaces the placeholder.

Work through each vendor section below in order. After replacing the placeholder, verify the token using the command shown, then move to the next vendor.

Heroku — HEROKU_BILLING_API_KEY

Why a separate key

Heroku has no billing-scoped API token. The global scope is required for GET /account/invoices. To limit blast radius, this key must be a separate named authorization distinct from the deploy key (HEROKU_API_KEY).

A compromised billing key leaks invoice data (low damage) but cannot deploy code or change config — both keys call the same API but are different authorization objects with different IDs, so the billing key can be revoked independently.

Step 1 — Create the authorization

Requires: Heroku CLI authenticated (heroku login) on a trusted workstation.

heroku authorizations:create \
  --description "raxx-billing-collector" \
  --scope "global"

Output includes: - Token: — the secret value (shown once; capture immediately) - ID: — the authorization ID (needed for future revocation)

Record the authorization ID somewhere safe (not in vault — it is not a secret, but is needed for the revocation step at rotation time).

REST alternative (if CLI unavailable):

curl -sS -n -X POST \
  -H "Accept: application/vnd.heroku+json; version=3" \
  -H "Content-Type: application/json" \
  https://api.heroku.com/oauth/authorizations \
  --data '{"description":"raxx-billing-collector","scope":["global"]}'

Step 2 — Verify the token

NEW_TOKEN="<value from step 1>"
curl -sS \
  -H "Accept: application/vnd.heroku+json; version=3" \
  -H "Authorization: Bearer ${NEW_TOKEN}" \
  "https://api.heroku.com/account/invoices" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'OK — {len(d)} invoice(s)')"
# Expected output: OK — N invoice(s)
# HTTP 401/403 = token scope problem; re-check the --scope flag

Step 3 — Write to Infisical

Do not paste the token value into any file, shell variable that gets logged, or heroku config:set (which echoes to stdout — see feedback_heroku_config_set_echoes_secrets.md).

Use the Infisical CLI:

infisical secrets set HEROKU_BILLING_API_KEY="$NEW_TOKEN" \
  --projectId="${INFISICAL_PROJECT_ID}" \
  --env=prod \
  --path="/Raxx/Console/Billing" >/dev/null 2>&1

Or via the Infisical web UI: navigate to vault.raxx.app, project → prod environment → /Raxx/Console/Billing/ → edit HEROKU_BILLING_API_KEY → paste value → save.

Step 4 — Inject into the Raptor console app

The billing collector reads HEROKU_BILLING_API_KEY from its process environment. Inject via Heroku config:

BILLING_KEY=$(infisical secrets get HEROKU_BILLING_API_KEY \
  --env=prod --path="/Raxx/Console/Billing" --plain)
heroku config:set HEROKU_BILLING_API_KEY="$BILLING_KEY" \
  --app raxx-console-prod >/dev/null 2>&1

The >/dev/null 2>&1 suppresses Heroku's stdout echo (which would log the token value to the terminal — see feedback_heroku_config_set_echoes_secrets.md).

Step 5 — Enable the flag

After verifying the token works, flip the collector on:

heroku config:set FLAG_HEROKU_BILLING_COLLECTOR=1 \
  --app raxx-console-prod >/dev/null 2>&1

Rotation

• Cadence: 90 days
• Mode: operator-assisted (same create-then-revoke pattern as HEROKU_API_KEY)
• At rotation: heroku authorizations:revoke <OLD_AUTH_ID> after the new token is live
• SOP: docs/ops/runbooks/rotation/heroku-platform-token.md (same procedure; use description raxx-billing-collector)

AWS — AWS_BILLING_ACCESS_KEY_ID + AWS_BILLING_SECRET_ACCESS_KEY

IAM policy

Create an IAM user with exactly these four permissions and nothing else:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BillingReadOnly",
      "Effect": "Allow",
      "Action": [
        "ce:GetCostAndUsage",
        "ce:GetCostForecast",
        "ce:GetDimensionValues",
        "cloudwatch:GetMetricStatistics"
      ],
      "Resource": "*"
    }
  ]
}

This policy is intentionally narrow: - ce:* — Cost Explorer read (charges $0.01/API call — the collector rate-caps at 4x/day) - cloudwatch:GetMetricStatistics — Lightsail NetworkOut egress metric only - NO EC2, S3, IAM, Lightsail management, RDS, or any write permission

A compromised key lets an attacker read cost data (low damage) and cannot provision resources, delete data, or escalate privileges.

Step 1 — Create the IAM user

Via AWS Console:

1. Navigate to console.aws.amazon.com/iam → Users → Create user
2. Username: raxx-billing-collector (suggested; any unique name is fine)
3. Select "Attach policies directly"
4. Click "Create policy" → JSON tab → paste the policy above → name it RaxxBillingReadOnly → create
5. Attach RaxxBillingReadOnly to the user
6. Create the user

Via AWS CLI (if preferred):

aws iam create-user --user-name raxx-billing-collector

aws iam put-user-policy \
  --user-name raxx-billing-collector \
  --policy-name RaxxBillingReadOnly \
  --policy-document '{
    "Version":"2012-10-17",
    "Statement":[{
      "Sid":"BillingReadOnly",
      "Effect":"Allow",
      "Action":[
        "ce:GetCostAndUsage",
        "ce:GetCostForecast",
        "ce:GetDimensionValues",
        "cloudwatch:GetMetricStatistics"
      ],
      "Resource":"*"
    }]
  }'

Step 2 — Generate the access key

Via AWS Console:

1. Navigate to the user → Security credentials tab → Access keys → Create access key
2. Select "Application running outside AWS"
3. Description tag: raxx-billing-collector-2026-05-11
4. Capture both the Access Key ID and Secret Access Key — the secret is shown once

Via AWS CLI:

aws iam create-access-key --user-name raxx-billing-collector

Output includes AccessKeyId and SecretAccessKey.

Step 3 — Verify the key

ACCESS_KEY_ID="<value from step 2>"
SECRET_ACCESS_KEY="<value from step 2>"

# Using AWS CLI with explicit credentials
AWS_ACCESS_KEY_ID="$ACCESS_KEY_ID" \
AWS_SECRET_ACCESS_KEY="$SECRET_ACCESS_KEY" \
aws ce get-cost-and-usage \
  --time-period "Start=$(date -u +%Y-%m-01),End=$(date -u +%Y-%m-%d)" \
  --granularity MONTHLY \
  --metrics "UnblendedCost" \
  --region us-east-1 \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK — cost:', d['ResultsByTime'][0]['Total']['UnblendedCost']['Amount'])"
# Expected: OK — cost: <dollar amount>
# AuthFailure / AccessDenied = policy not attached correctly; re-check step 1

Note: Cost Explorer is only available in us-east-1 regardless of where your resources live.

Step 4 — Write to Infisical

infisical secrets set \
  AWS_BILLING_ACCESS_KEY_ID="$ACCESS_KEY_ID" \
  AWS_BILLING_SECRET_ACCESS_KEY="$SECRET_ACCESS_KEY" \
  --projectId="${INFISICAL_PROJECT_ID}" \
  --env=prod \
  --path="/Raxx/Console/Billing" >/dev/null 2>&1

Or via vault web UI: edit both secrets at /Raxx/Console/Billing/.

Step 5 — Inject into the console app

KEY_ID=$(infisical secrets get AWS_BILLING_ACCESS_KEY_ID \
  --env=prod --path="/Raxx/Console/Billing" --plain)
SECRET=$(infisical secrets get AWS_BILLING_SECRET_ACCESS_KEY \
  --env=prod --path="/Raxx/Console/Billing" --plain)

heroku config:set \
  AWS_BILLING_ACCESS_KEY_ID="$KEY_ID" \
  AWS_BILLING_SECRET_ACCESS_KEY="$SECRET" \
  --app raxx-console-prod >/dev/null 2>&1

Step 6 — Enable the flag

heroku config:set FLAG_AWS_BILLING_COLLECTOR=1 \
  --app raxx-console-prod >/dev/null 2>&1

Rotation

• Cadence: 30 days (AWS IAM key rotation standard per docs/ops/runbooks/rotation/aws-iam-access-key.md)
• Mode: operator-assisted (create new key → vault → Heroku config → verify → delete old key)
• At rotation: aws iam delete-access-key --user-name raxx-billing-collector --access-key-id <OLD_KEY_ID>
• The IAM user itself does not change; only the key pair rotates

Cloudflare — CLOUDFLARE_BILLING_TOKEN

Why a separate token

This token is distinct from: - CF_DNS_EDIT_GETRAXX_COM (DNS write — different scope, different zone) - CF_PAGES_DEPLOY / CF_ACCESS_MGMT (Pages deploy / Access management)

The billing token needs two read-only scopes to call GET /zones/{zone_id}/subscription:

• Zone:Zone:Read (zone-scoped, all zones) — lets the token iterate zone metadata
• Account:Billing:Read (account-scoped, MooseQuest account) — required by the /subscription endpoint itself; without it the CF API returns HTTP 403 despite Zone:Zone:Read being present

Root cause of 2026-05-20 incident: the original spec listed only Zone:Zone:Read. The collector 403d on /zones/{zone_id}/subscription because that endpoint is account-gated on the billing permission, not zone-gated. Token e1feb102b0305acee7f22596511332b4 was updated via PUT /user/tokens/{id} on 2026-05-20T17:41Z UTC to add Account:Billing:Read.

Mixing billing with write-scope tokens still violates least-privilege — no write permission is granted by either scope above.

Token name in CF dashboard: raxx-billing-collector (suggested).

Step 1 — Create the token

Navigate to:

https://dash.cloudflare.com/profile/api-tokens

1. Click "Create Token" → "Custom token"
2. Token name: raxx-billing-collector
3. Permissions — both required: - Zone → Zone → Read - Account → Billing → Read ← required for /zones/{id}/subscription; omitting this causes HTTP 403
4. Zone resources: "All zones" (the collector queries raxx.app, raxx.io, getraxx.com, moosequest.net)
5. Account resources: "MooseQuest" account (for the Account:Billing:Read policy)
6. Do NOT add DNS Edit, Workers, Pages, or any write permission
7. Set an expiration date 90 days from today (optional but recommended)
8. Click "Continue to summary" → "Create token"
9. Capture the token value — shown once

Programmatic alternative using CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN (the automation token, which has User:API Tokens:Edit scope):

# Vault path for secrets below: /MooseQuest/cloudflare (env: prod)
AUTO_TOKEN=$(infisical secrets get CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN \
  --path /MooseQuest/cloudflare --env prod --plain 2>/dev/null)
CF_ACCOUNT_ID=$(infisical secrets get CLOUDFLARE_ACCOUNT_ID \
  --path /MooseQuest/cloudflare --env prod --plain 2>/dev/null)
EXPIRES_AT="$(python3 -c "from datetime import datetime,timedelta,timezone; print((datetime.now(timezone.utc)+timedelta(days=90)).strftime('%Y-%m-%dT00:00:00Z'))")"

# Permission group IDs (verified 2026-05-20):
#   Zone Read    : c8fed203ed3043cba015a93ad1616f1f  (zone-scoped, all zones)
#   Billing Read : 7cf72faf220841aabcfdfab81c43c4f6  (account-scoped)
RESP=$(curl -sS -X POST \
  -H "Authorization: Bearer ${AUTO_TOKEN}" \
  -H "Content-Type: application/json" \
  -A "raxx-sre-agent/1.0" \
  "https://api.cloudflare.com/client/v4/user/tokens" \
  -d "{
    \"name\": \"raxx-billing-collector\",
    \"policies\": [
      {
        \"effect\": \"allow\",
        \"resources\": {\"com.cloudflare.api.account.zone.*\": \"*\"},
        \"permission_groups\": [{\"id\": \"c8fed203ed3043cba015a93ad1616f1f\"}]
      },
      {
        \"effect\": \"allow\",
        \"resources\": {\"com.cloudflare.api.account.${CF_ACCOUNT_ID}\": \"*\"},
        \"permission_groups\": [{\"id\": \"7cf72faf220841aabcfdfab81c43c4f6\"}]
      }
    ],
    \"not_before\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
    \"expires_on\": \"${EXPIRES_AT}\"
  }")

TOKEN_VALUE=$(echo "$RESP" | python3 -c "import sys,json; print(json.load(sys.stdin)['result']['value'])")
TOKEN_ID=$(echo "$RESP" | python3 -c "import sys,json; print(json.load(sys.stdin)['result']['id'])")
# $TOKEN_VALUE is the secret; do not echo it
echo "Token ID: $TOKEN_ID"
echo "Expires: $EXPIRES_AT"

Note: CLOUDFLARE_ACCESS_MGMT_TOKEN (claude-cf-access-mgmt) does NOT have User:API Tokens:Edit scope and will return error 9109 on token CRUD calls. Use CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN for token management operations.

Step 2 — Verify the token

NEW_TOKEN="<value from step 1>"
curl -sS \
  -H "Authorization: Bearer ${NEW_TOKEN}" \
  "https://api.cloudflare.com/client/v4/user/tokens/verify" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('status:', d['result']['status'])"
# Expected: status: active

# Confirm zone read works (use raxx.app zone ID)
ZONE_ID="f12dbb5cac57d5591a5058874498a6d1"
curl -sS \
  -H "Authorization: Bearer ${NEW_TOKEN}" \
  "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/subscription" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); errs=d.get('errors',[]); code=errs[0].get('code') if errs else None; print('zone subscription result:', 'OK' if d.get('success') else ('free-tier zone (expected)' if code in (1017,1207) else d))"
# HTTP 200 with success:true = active subscription (paid zone)
# HTTP 404 with code:1017  = free-tier zone (expected for most zones) — this is fine
# HTTP 404 with code:1207  = "no active core subscription" — also free-tier, also fine
# HTTP 403                 = token missing Account:Billing:Read scope; re-check step 1

Note: HTTP 404 for a free-tier zone subscription is expected behavior (CF returns either code 1017 or 1207 depending on zone configuration). The collector treats both as $0 spend. HTTP 403 always indicates a scope problem — specifically a missing Account:Billing:Read policy on the token.

Step 3 — Write to Infisical

infisical secrets set CLOUDFLARE_BILLING_TOKEN="$NEW_TOKEN" \
  --projectId="${INFISICAL_PROJECT_ID}" \
  --env=prod \
  --path="/Raxx/Console/Billing" >/dev/null 2>&1

Store companion metadata (mirrors convention from cloudflare-tokens.md):

infisical secrets set \
  CLOUDFLARE_BILLING_TOKEN__TOKEN_ID="$TOKEN_ID" \
  CLOUDFLARE_BILLING_TOKEN__EXPIRES_AT="$EXPIRES_AT" \
  CLOUDFLARE_BILLING_TOKEN__SCOPES="Zone:Zone:Read (all zones) + Account:Billing:Read (MooseQuest account)" \
  --projectId="${INFISICAL_PROJECT_ID}" \
  --env=prod \
  --path="/Raxx/Console/Billing" >/dev/null 2>&1

Step 4 — Inject into the console app

CF_TOK=$(infisical secrets get CLOUDFLARE_BILLING_TOKEN \
  --env=prod --path="/Raxx/Console/Billing" --plain)
heroku config:set CLOUDFLARE_BILLING_TOKEN="$CF_TOK" \
  --app raxx-console-prod >/dev/null 2>&1

Step 5 — Enable the flag

heroku config:set FLAG_CLOUDFLARE_BILLING_COLLECTOR=1 \
  --app raxx-console-prod >/dev/null 2>&1

Rotation

• Cadence: 90 days
• Mode: operator-assisted (create new token → vault → Heroku config → verify → revoke old)
• SOP: docs/ops/runbooks/rotation/cloudflare-user-api-token.md (same procedure)
• At rotation: revoke the old token via DELETE /user/tokens/{old_token_id} using CF_ACCESS_MGMT
• Update CLOUDFLARE_BILLING_TOKEN__EXPIRES_AT companion after rotation

Verification checklist (post-provisioning)

After completing all three vendors, run through this checklist to confirm end-to-end health before enabling any flag in production:

[ ] GET /account/invoices returns HTTP 200 with HEROKU_BILLING_API_KEY
[ ] ce:GetCostAndUsage returns a valid response with AWS billing key pair
[ ] GET /zones/{zone_id}/subscription returns HTTP 200 or 404 (code 1017 or 1207) with CLOUDFLARE_BILLING_TOKEN — NOT 403
[ ] All four vault entries at /Raxx/Console/Billing/ show real values (not PLACEHOLDER_*)
[ ] raxx-console-prod Heroku config has HEROKU_BILLING_API_KEY, AWS_BILLING_ACCESS_KEY_ID,
    AWS_BILLING_SECRET_ACCESS_KEY, CLOUDFLARE_BILLING_TOKEN set (not echoed to terminal)
[ ] Feature flags remain OFF until verified — flip one at a time, check logs

Vault folder creation reference

The folder /Raxx/Console/Billing/ was created by sre-agent at 2026-05-11T07:05:01Z via three sequential POST /api/v1/folders calls:

1. / → Raxx (folder ID: 707e104a-6fc1-4436-a862-b4265a2b073f)
2. /Raxx → Console (folder ID: 3b94f735-576d-46dc-95b3-3c7c18f3da21)
3. /Raxx/Console → Billing (folder ID: e81171a5-7101-410d-a382-e48dad7ac971)

Infisical requires: POST /api/v1/folders with body {"workspaceId": "...", "environment": "prod", "path": "<parent>", "name": "<folder_name>"}. The path field is the parent directory; name is the new folder to create within it. If the parent does not exist, create it first (Infisical returns 404 otherwise — see feedback_vault_folder_must_exist.md).

Rotation matrix additions

Add the following rows to docs/ops/runbooks/rotation/INDEX.md after provisioning:

Security notes

• HEROKU_BILLING_API_KEY has global scope — mitigated by it being a separate authorization from the deploy key, revocable independently. Blast radius: billing data read access only.
• AWS_BILLING_ACCESS_KEY_ID / AWS_BILLING_SECRET_ACCESS_KEY — blast radius: cost data read access, no resource management or write permissions.
• CLOUDFLARE_BILLING_TOKEN — blast radius: zone metadata read + account billing data read, no DNS/WAF/Pages write. Two read-only scopes are required: Zone:Zone:Read (all zones) and Account:Billing:Read (MooseQuest account). Do not assign any write permission groups.
• All three token types are distinct from existing operator tokens; none share scopes with the deploy key (HEROKU_API_KEY), the CF management token (CF_ACCESS_MGMT), or the DNS edit token (CF_DNS_EDIT_GETRAXX_COM).

Escalation

Escalate to operator when: - A vendor returns a persistent auth error after a valid token has been stored - AWS IAM access key creation fails (may indicate IAM permission boundary or SCP restrictions in the account — Kristerpher needs to verify the account's Service Control Policies) - Cloudflare returns HTTP 403 on /zones/{zone_id}/subscription after confirming the token carries both Zone:Zone:Read AND Account:Billing:Read policies (may indicate zone ownership issue, account-level restriction, or CF API change)

Contact: ops@raxx.app / Slack MooseQuest workspace