CF Access runbook

System: Cloudflare Zero Trust Access — all Raxx-managed applications Owner: sre-agent / operator Last incident: 2026-06-12 (incomplete bypass set blocked beta invite flow; see 2026-06-12-beta-invite-cf-access-blocks.md) Last reviewed: 2026-06-13

Related runbooks

• docs/ops/runbooks/getraxx-launch-day-cf-access-removal.md — launch-day removal procedure for the getraxx.com beta gate
• docs/ops/runbooks/terraform-cf-access-state-imports.md — import commands for the terraform/cf-access/ stack (console, vault, freescout service tokens)
• docs/ops/runbooks/cf-access-service-token-provisioning.md — minting CF Access service tokens for machine-to-machine routes

How to tell CF Access is broken

• curl -I https://<gated-surface>/ returns HTTP/2 200 when it should return HTTP/2 302 (gate removed accidentally).
• curl -I https://<gated-surface>/ returns HTTP/2 302 after the gate was intentionally removed (gate not fully destroyed).
• terraform plan reports resources to add/destroy when you expect No changes (state drift).
• CF Zero Trust dashboard shows an application that Terraform thinks is gone, or vice versa.

How to diagnose (in order)

1. Check live CF Access state via API: bash export CLOUDFLARE_API_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \ --path /MooseQuest/cloudflare/ --plain) curl -sS -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \ "https://api.cloudflare.com/client/v4/accounts/22b5c35090724fbf05db6d4f501ac821/access/apps" \ | python3 -c "import sys,json; [print(a['id'], a['name'], a['domain']) for a in json.load(sys.stdin).get('result',[])]" Compare output to what Terraform state believes exists.

2. Check Terraform state: bash cd terraform/modules/cf-access-getraxx terraform state list

3. Run a plan against live state: bash export CLOUDFLARE_API_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \ --path /MooseQuest/cloudflare/ --plain) export TF_VAR_cf_access_account_id=$(infisical secrets get CF_ACCESS_ACCOUNT_ID_MOOSEQUEST \ --path /MooseQuest/cloudflare/ --plain) terraform plan A No changes plan means state is in sync. Anything else is drift.

Known failure modes

Failure mode A: Dashboard delete created Terraform state orphan (#2849)

Symptom: terraform plan shows resources to destroy that no longer exist in CF (e.g., cloudflare_zero_trust_access_application.getraxx_beta, cloudflare_zero_trust_access_policy.getraxx_beta_invitees, cloudflare_ruleset.www_to_apex_redirect). The CF dashboard shows these apps are gone.

Cause: The operator deleted one or more CF Access resources directly via the CF Zero Trust dashboard (or via direct API call), bypassing Terraform. Terraform state still references these resources; terraform plan plans to destroy things that are already gone, which would either no-op or error during apply.

Recommendation: terraform state rm (not terraform import) when the deletion was intentional and the resource should not be re-created. Use terraform import when the resource was accidentally deleted and must be re-created and re-imported.

Decision for #2849: The operator deleted the getraxx-beta CF Access app intentionally on 2026-05-27 UTC as part of the v1 launch procedure (Path B in docs/ops/runbooks/getraxx-launch-day-cf-access-removal.md). The module terraform/modules/cf-access-getraxx/ should be kept for reference but its state entries must be removed. Use the terraform state rm path below.

Fix — operator-action required (do NOT run autonomously):

Step 1: Source credentials

export CLOUDFLARE_API_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \
  --path /MooseQuest/cloudflare/ --plain)
export TF_VAR_cf_access_account_id=$(infisical secrets get CF_ACCESS_ACCOUNT_ID_MOOSEQUEST \
  --path /MooseQuest/cloudflare/ --plain)

Step 2: Confirm the resources are gone in CF

curl -sS -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  "https://api.cloudflare.com/client/v4/accounts/22b5c35090724fbf05db6d4f501ac821/access/apps" \
  | python3 -c "
import sys, json
apps = json.load(sys.stdin).get('result', [])
getraxx = [a for a in apps if 'getraxx' in a.get('name','').lower() or 'getraxx.com' in a.get('domain','')]
if getraxx:
    print('FOUND — not safe to rm:')
    for a in getraxx: print(' ', a['id'], a['name'])
else:
    print('Not found in CF — safe to terraform state rm')
"

Expected output if the delete succeeded: Not found in CF — safe to terraform state rm

If the resources are still present in CF, STOP. Do NOT run terraform state rm — the resource is live. Investigate why the dashboard delete did not propagate (CF rate limit, partial delete).

Step 3: Inspect current Terraform state

cd terraform/modules/cf-access-getraxx
terraform state list

Expected output (three orphaned resources):

cloudflare_ruleset.www_to_apex_redirect
cloudflare_zero_trust_access_application.getraxx_beta
cloudflare_zero_trust_access_policy.getraxx_beta_invitees

If the list is empty, state is already clean — skip to Step 5.

Step 4: Remove the orphaned resources from Terraform state

Run each terraform state rm command individually. There is no risk of live resource deletion — state rm only removes the entry from Terraform's local state file; it does not call the CF API.

terraform state rm cloudflare_zero_trust_access_policy.getraxx_beta_invitees
terraform state rm cloudflare_zero_trust_access_application.getraxx_beta
terraform state rm cloudflare_ruleset.www_to_apex_redirect

Order: remove the policy first (it depends on the application), then the application, then the ruleset (independent).

Step 5: Verify clean state

terraform plan

Expected output: No changes. Your infrastructure matches the configuration.

If Terraform still plans to destroy resources, check that the correct S3 backend state file is being used (terraform init may be needed if the backend was recently changed).

Step 6: Post-cleanup documentation

After successful terraform plan shows No changes:

1. Add a # ARCHIVED — resources removed 2026-05-27, state rm on YYYY-MM-DD comment to the top of terraform/modules/cf-access-getraxx/main.tf.
2. Update docs/security/web-surface-posture.md: set the getraxx.com row to WAF + rate limit (public).
3. Update docs/security/auth-posture.md §9: set the getraxx.com row to Public.

Failure mode B: terraform plan fails with authentication error

Symptom: terraform plan exits with Authentication error (10000) or invalid token when planning the terraform/modules/cf-access-getraxx/ module.

Cause: CF_ACCESS_MGMT token has expired, been revoked, or lost the Account:Zero Trust:Edit scope. Common after a vault rotation or token recreation.

Fix:

# Verify token status
CLOUDFLARE_API_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \
  --path /MooseQuest/cloudflare/ --plain)
curl -s -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  https://api.cloudflare.com/client/v4/user/tokens/verify | python3 -m json.tool
# Expect: "status": "active"

If inactive: follow docs/ops/runbooks/rotation/cloudflare-user-api-token.md to rotate the token, then update Infisical at /MooseQuest/cloudflare/CF_ACCESS_MGMT.

Verification: Re-run terraform plan after rotating the token.

Failure mode C: CF Access gate blocks legitimate service token traffic

Symptom: Machine-to-machine requests (e.g., Lambda → FreeScout) receive a 302 redirect to cloudflareaccess.com instead of being passed through.

Cause: CF Access service token policy uses decision = "allow" instead of decision = "non_identity". Per feedback_cf_access_service_token_needs_non_identity.md: allow requires an IdP identity that service tokens don't carry.

Fix: In terraform/cf-access/ (the main stack, not the getraxx module), set decision = "non_identity" on the service token policy resource, then apply. See docs/ops/runbooks/terraform-cf-access-state-imports.md Fix 1 for the freescout service token correction history.

Also check that the WAF skip rule is in place (Bot Fight Mode bypasses CF Access headers). See feedback_cf_access_does_not_bypass_bot_fight_mode.md.

Verification:

curl -v -H "CF-Access-Client-Id: <service-token-id>" \
     -H "CF-Access-Client-Secret: <service-token-secret>" \
     https://<gated-surface>/health
# Expect: 200 (not 302)

Emergency stop — re-gate a surface that was accidentally made public

If a CF Access application is accidentally removed and the surface needs to be immediately re-gated:

For getraxx.com (manual CF dashboard path — fastest): See "Re-adding the gate (rollback)" in docs/ops/runbooks/getraxx-launch-day-cf-access-removal.md.

For other surfaces (Terraform path):

cd terraform/cf-access   # or the relevant module directory
terraform apply           # re-creates from HCL definition

Escalation

Wake the operator when: - terraform apply proposes destroying any active CF Access application or policy protecting a live surface. - CF Access tokens cannot be retrieved from Infisical (vault access outage). - A surface that should be gated is publicly accessible and the cause is unknown. - terraform state rm would affect more than the expected resources listed above.

CF support: https://support.cloudflare.com/hc/en-us/requests/new

Bypass apps — complete inventory (raxx.app pre-launch gate)

The raxx.app root app (23bd313e) gates the entire domain. Every path that an anonymous browser or SSR fetch must reach requires its own bypass app (decision=bypass, include=everyone). This table is the canonical record — update after every bypass change.

Vault secret for the API token: CLOUDFLARE_ACCESS_MGMT_TOKEN at /MooseQuest/cloudflare/ (note: runbook diagnostic steps above reference CF_ACCESS_MGMT — the live key is CLOUDFLARE_ACCESS_MGMT_TOKEN).

Phase 1 — Beta walkthrough / preview (created 2026-06-10 / 2026-06-12)

Note: api.raxx.app root app (01b5423f) has a decision=bypass policy (71c61b7f) with include=everyone as its first policy, which currently bypasses the entire api.raxx.app domain for anonymous traffic. The per-path bypass apps above are belt-and-suspenders for when that root policy is eventually tightened.

Phase 2 — Beta join flow (created 2026-06-13, SC-5 #3550)

Verification probes run 2026-06-13 UTC (routes not yet built — probes confirm CF is not intercepting):

When SC-6 deploys the /beta/join/<token> page, re-run probe 1 and confirm non-302 (200 or expected Next.js response, not cloudflareaccess.com redirect).

Known gap — raxx.app/api/auth/register/* (operator action required before Phase 2 live)

Live probe 2026-06-13: raxx.app/api/auth/register/options returns 302 to cloudflareaccess.com for anonymous requests. This path is used by the WebAuthn registration flow (begin-with-token, verify-with-token) when the browser calls Raptor via raxx.app as the proxy host.

The corresponding api.raxx.app/api/auth/register/* path is unaffected — covered by the api.raxx.app root bypass policy 71c61b7f.

Impact now: zero — Phase 2 routes not yet live. SSR fetches on the join flow target api.raxx.app, which is bypassed.

Required before Phase 2 goes live: a bypass app for raxx.app/api/auth/register/* (decision=bypass, include=everyone). This is the same pattern as all other Phase 1/2 bypasses. Provision via CF API using CLOUDFLARE_ACCESS_MGMT_TOKEN from vault, following the Phase 2 provisioning pattern in docs/incidents/2026-06-12-beta-invite-cf-access-blocks.md Resolution section.

Checklist — CF Access bypass set for a public HMAC-signed flow

Lesson from docs/incidents/2026-06-12-beta-invite-cf-access-blocks.md: every anonymous dependency path needs its own bypass app. Use this checklist when adding a new public HMAC-signed flow.

For each new flow, ask:

• [ ] Page route bypassed? (raxx.app/<flow-path>/*)
• [ ] Next.js static assets bypassed? (raxx.app/_next/*) — shared; verify existing bypass covers
• [ ] Public dir assets bypassed? (raxx.app/<asset-dir>/*) — if the flow serves images or other public-dir files
• [ ] API routes on raxx.app bypassed? (raxx.app/api/<flow>/*)
• [ ] API routes on api.raxx.app bypassed? (api.raxx.app/api/<flow>/*) — SSR fetches target this host
• [ ] Auth/register paths bypassed if WebAuthn flow included? (raxx.app/api/auth/register/*) — currently gated; see Known gap above
• [ ] Verification probes run from a clean session (no CF cookie, browser UA, no CF Access credentials)?
• [ ] Root gate sanity check: raxx.app/dashboard still returns 302 to cloudflareaccess.com?

References

• Terraform module (getraxx): terraform/modules/cf-access-getraxx/
• Terraform stack (console/vault/freescout): terraform/cf-access/
• Launch-day removal runbook: docs/ops/runbooks/getraxx-launch-day-cf-access-removal.md
• State import runbook: docs/ops/runbooks/terraform-cf-access-state-imports.md
• Service token provisioning: docs/ops/runbooks/cf-access-service-token-provisioning.md
• Auth posture: docs/security/auth-posture.md
• Web surface posture: docs/security/web-surface-posture.md
• Incident #2849: getraxx.com dashboard delete
• Incident 2026-06-12: beta invite CF Access blocks — docs/incidents/2026-06-12-beta-invite-cf-access-blocks.md
• Phase 2 design doc: docs/architecture/beta-phase2-join-flow.md §6
• CF Zero Trust dashboard: https://one.dash.cloudflare.com/
• CF API token management: https://dash.cloudflare.com/profile/api-tokens