CF Access runbook

System: Cloudflare Zero Trust Access — all Raxx-managed applications Owner: sre-agent / operator Last incident: 2026-07-24 (CF Access on console.raxx.app blocked Heroku's ACM HTTP-01 renewal challenge for ~30 days, causing the origin cert to expire and console.raxx.app to 526 for authenticated sessions; see docs/incidents/2026-07-24-console-acm-cf-access-526.md and Failure mode E below). Prior: 2026-06-12 (incomplete bypass set blocked beta invite flow; see 2026-06-12-beta-invite-cf-access-blocks.md) Last reviewed: 2026-07-24

Related runbooks

• docs/ops/runbooks/getraxx-launch-day-cf-access-removal.md — launch-day removal procedure for the getraxx.com beta gate
• docs/ops/runbooks/terraform-cf-access-state-imports.md — import commands for the terraform/cf-access/ stack (console, vault, freescout service tokens)
• docs/ops/runbooks/cf-access-service-token-provisioning.md — minting CF Access service tokens for machine-to-machine routes

How to tell CF Access is broken

• curl -I https://<gated-surface>/ returns HTTP/2 200 when it should return HTTP/2 302 (gate removed accidentally).
• curl -I https://<gated-surface>/ returns HTTP/2 302 after the gate was intentionally removed (gate not fully destroyed).
• terraform plan reports resources to add/destroy when you expect No changes (state drift).
• CF Zero Trust dashboard shows an application that Terraform thinks is gone, or vice versa.

How to diagnose (in order)

1. Check live CF Access state via API: bash export CLOUDFLARE_API_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \ --path /MooseQuest/cloudflare/ --plain) curl -sS -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \ "https://api.cloudflare.com/client/v4/accounts/22b5c35090724fbf05db6d4f501ac821/access/apps" \ | python3 -c "import sys,json; [print(a['id'], a['name'], a['domain']) for a in json.load(sys.stdin).get('result',[])]" Compare output to what Terraform state believes exists.

2. Check Terraform state: bash cd terraform/modules/cf-access-getraxx terraform state list

3. Run a plan against live state: bash export CLOUDFLARE_API_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \ --path /MooseQuest/cloudflare/ --plain) export TF_VAR_cf_access_account_id=$(infisical secrets get CF_ACCESS_ACCOUNT_ID_MOOSEQUEST \ --path /MooseQuest/cloudflare/ --plain) terraform plan A No changes plan means state is in sync. Anything else is drift.

Known failure modes

Failure mode A: Dashboard delete created Terraform state orphan (#2849)

Symptom: terraform plan shows resources to destroy that no longer exist in CF (e.g., cloudflare_zero_trust_access_application.getraxx_beta, cloudflare_zero_trust_access_policy.getraxx_beta_invitees, cloudflare_ruleset.www_to_apex_redirect). The CF dashboard shows these apps are gone.

Cause: The operator deleted one or more CF Access resources directly via the CF Zero Trust dashboard (or via direct API call), bypassing Terraform. Terraform state still references these resources; terraform plan plans to destroy things that are already gone, which would either no-op or error during apply.

Recommendation: terraform state rm (not terraform import) when the deletion was intentional and the resource should not be re-created. Use terraform import when the resource was accidentally deleted and must be re-created and re-imported.

Decision for #2849: The operator deleted the getraxx-beta CF Access app intentionally on 2026-05-27 UTC as part of the v1 launch procedure (Path B in docs/ops/runbooks/getraxx-launch-day-cf-access-removal.md). The module terraform/modules/cf-access-getraxx/ should be kept for reference but its state entries must be removed. Use the terraform state rm path below.

Fix — operator-action required (do NOT run autonomously):

Step 1: Source credentials

export CLOUDFLARE_API_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \
  --path /MooseQuest/cloudflare/ --plain)
export TF_VAR_cf_access_account_id=$(infisical secrets get CF_ACCESS_ACCOUNT_ID_MOOSEQUEST \
  --path /MooseQuest/cloudflare/ --plain)

Step 2: Confirm the resources are gone in CF

curl -sS -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  "https://api.cloudflare.com/client/v4/accounts/22b5c35090724fbf05db6d4f501ac821/access/apps" \
  | python3 -c "
import sys, json
apps = json.load(sys.stdin).get('result', [])
getraxx = [a for a in apps if 'getraxx' in a.get('name','').lower() or 'getraxx.com' in a.get('domain','')]
if getraxx:
    print('FOUND — not safe to rm:')
    for a in getraxx: print(' ', a['id'], a['name'])
else:
    print('Not found in CF — safe to terraform state rm')
"

Expected output if the delete succeeded: Not found in CF — safe to terraform state rm

If the resources are still present in CF, STOP. Do NOT run terraform state rm — the resource is live. Investigate why the dashboard delete did not propagate (CF rate limit, partial delete).

Step 3: Inspect current Terraform state

cd terraform/modules/cf-access-getraxx
terraform state list

Expected output (three orphaned resources):

cloudflare_ruleset.www_to_apex_redirect
cloudflare_zero_trust_access_application.getraxx_beta
cloudflare_zero_trust_access_policy.getraxx_beta_invitees

If the list is empty, state is already clean — skip to Step 5.

Step 4: Remove the orphaned resources from Terraform state

Run each terraform state rm command individually. There is no risk of live resource deletion — state rm only removes the entry from Terraform's local state file; it does not call the CF API.

terraform state rm cloudflare_zero_trust_access_policy.getraxx_beta_invitees
terraform state rm cloudflare_zero_trust_access_application.getraxx_beta
terraform state rm cloudflare_ruleset.www_to_apex_redirect

Order: remove the policy first (it depends on the application), then the application, then the ruleset (independent).

Step 5: Verify clean state

terraform plan

Expected output: No changes. Your infrastructure matches the configuration.

If Terraform still plans to destroy resources, check that the correct S3 backend state file is being used (terraform init may be needed if the backend was recently changed).

Step 6: Post-cleanup documentation

After successful terraform plan shows No changes:

1. Add a # ARCHIVED — resources removed 2026-05-27, state rm on YYYY-MM-DD comment to the top of terraform/modules/cf-access-getraxx/main.tf.
2. Update docs/security/web-surface-posture.md: set the getraxx.com row to WAF + rate limit (public).
3. Update docs/security/auth-posture.md §9: set the getraxx.com row to Public.

Failure mode B: terraform plan fails with authentication error

Symptom: terraform plan exits with Authentication error (10000) or invalid token when planning the terraform/modules/cf-access-getraxx/ module.

Cause: CF_ACCESS_MGMT token has expired, been revoked, or lost the Account:Zero Trust:Edit scope. Common after a vault rotation or token recreation.

Fix:

# Verify token status
CLOUDFLARE_API_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \
  --path /MooseQuest/cloudflare/ --plain)
curl -s -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  https://api.cloudflare.com/client/v4/user/tokens/verify | python3 -m json.tool
# Expect: "status": "active"

If inactive: follow docs/ops/runbooks/rotation/cloudflare-user-api-token.md to rotate the token, then update Infisical at /MooseQuest/cloudflare/CF_ACCESS_MGMT.

Verification: Re-run terraform plan after rotating the token.

Failure mode D: Blackbox monitoring probe returns 403 for CF-fronted endpoints

Symptom: probe_success{instance="https://api.raxx.app/health"} == 0 in Prometheus; RaptorProdDown alert fires in Alertmanager even though the Heroku dyno is healthy (Heroku logs show continuous 200s from internal probes). curl with CF-Access-Client-Id/Secret headers still returns 403.

Cause: Cloudflare Bot Fight Mode (BFM) evaluates requests before CF Access. The monitoring EC2 (blackbox_exporter, prom/blackbox-exporter) sends a plain HTTP GET with no CF Access headers; BFM classifies it as bot traffic and returns 403 before CF Access can evaluate the bypass policy on the api.raxx.app root app. Even valid CF Access service-token headers do not bypass BFM — BFM runs at the WAF layer, CF Access runs after.

Mitigation (applied 2026-07-14): Probe was repointed to the Heroku origin URL https://raxx-api-prod-a60a19e5efbf.herokuapp.com/health which bypasses Cloudflare entirely. This correctly tests "is the Heroku dyno responding?" — the signal that matters for the RaptorProdDown alert. Alert cleared within one Prometheus scrape cycle.

Durable fix (pending — needs Zone:WAF:Edit scope): Add a Cloudflare WAF custom rule to skip SBFM for GET requests to api.raxx.app/health:

export CF_TOKEN=$(infisical secrets get CLOUDFLARE_ACCESS_MGMT_TOKEN \
  --path /MooseQuest/cloudflare/ --plain)
# Zone: f12dbb5cac57d5591a5058874498a6d1, custom ruleset: 17dc768ccadf4d02ae279e133b7b5bfd
curl -X POST "https://api.cloudflare.com/client/v4/zones/f12dbb5cac57d5591a5058874498a6d1/rulesets/17dc768ccadf4d02ae279e133b7b5bfd/rules" \
  -H "Authorization: Bearer $CF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "skip",
    "action_parameters": { "phases": ["http_request_sbfm"] },
    "expression": "http.host eq \"api.raxx.app\" and http.request.uri.path eq \"/health\"",
    "description": "sre: skip SBFM for monitoring probe on /health (RCA 2026-07-14)",
    "enabled": true
  }'

After adding the WAF skip rule, restore the probe target to https://api.raxx.app/health in terraform/ci-monitoring/templates/cloud-init.sh.tpl and apply live via SSM. Then run the idempotent inject script to add CF-Access headers to the blackbox http_2xx module:

bash /opt/monitoring/scripts/inject-cf-access-headers.sh "<cf-access-client-id>" "<cf-access-client-secret>"

The CLOUDFLARE_ACCESS_MGMT_TOKEN token must have Zone:WAF:Edit scope; the CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN only has Zone:Bot Management:Write and cannot create WAF custom rules.

Verification (after WAF rule applied):

curl -I https://api.raxx.app/health   # expect 200 from a clean IP, no CF Access headers

References: Incident docs/incidents/2026-07-14-raptor-prod-down-false-positive.md

Failure mode C: CF Access gate blocks legitimate service token traffic

Symptom: Machine-to-machine requests (e.g., Lambda → FreeScout) receive a 302 redirect to cloudflareaccess.com instead of being passed through.

Cause: CF Access service token policy uses decision = "allow" instead of decision = "non_identity". Per feedback_cf_access_service_token_needs_non_identity.md: allow requires an IdP identity that service tokens don't carry.

Fix: In terraform/cf-access/ (the main stack, not the getraxx module), set decision = "non_identity" on the service token policy resource, then apply. See docs/ops/runbooks/terraform-cf-access-state-imports.md Fix 1 for the freescout service token correction history.

Also check that the WAF skip rule is in place (Bot Fight Mode bypasses CF Access headers). See feedback_cf_access_does_not_bypass_bot_fight_mode.md.

Verification:

curl -v -H "CF-Access-Client-Id: <service-token-id>" \
     -H "CF-Access-Client-Secret: <service-token-secret>" \
     https://<gated-surface>/health
# Expect: 200 (not 302)

Failure mode E: CF Access blocks Heroku ACM renewal challenge — origin cert expires silently

Symptom: console.raxx.app (or any Heroku app with a broad CF Access application covering its whole hostname) returns Cloudflare 526 for authenticated CF Access sessions. Unauthenticated probes to / still return a normal 302 to the CF Access login page — CF Access itself never forwards the request to origin for anonymous traffic, so a plain curl -I against the root path looks completely healthy and does NOT reproduce the 526. The 526 only shows up once a request actually gets proxied to the (invalid) origin — i.e. for a browser with an active CF Access session, or for any path that has its own bypass (like the ACME challenge path itself, pre-fix). heroku certs:auto -a <app> shows acm_status: failed, reason "DNS redirect not forwarding path".

Cause: Heroku's Automatic Certificate Management renews via the ACME HTTP-01 challenge — Let's Encrypt (via Heroku) makes an unauthenticated GET to http(s)://<hostname>/.well-known/acme-challenge/<token>. If a CF Access application covers the whole hostname (e.g. console.raxx.app, no path-scoped exception), CF Access 302-redirects that unauthenticated probe to the Access login page instead of letting it reach the origin. The challenge can never complete, so ACM auto-renewal starts failing ~30 days before the current cert's expiry (acm_status: failed) and keeps failing silently — there is no page or email from Heroku when auto-renewal fails, only the eventual origin-cert-expired outage when the cert actually lapses. The origin's Full Strict TLS check then rejects the expired cert with an HTTP 526 for any request that does make it past CF Access.

Why detection was slow: the fleet TLS cert-expiry monitor (#2819) only had an edge probe for console.raxx.app — which sees Cloudflare's own Universal SSL edge cert (auto-renews fine, unrelated to this problem), not the origin cert. There was no origin probe for console.raxx.app (unlike vault.raxx.app / tickets.raxx.app), so nothing watched the cert that was actually expiring. Fixed in the same PR as this runbook update — see tls_connect_origin_console in terraform/ci-monitoring/templates/cloud-init.sh.tpl and docs/ops/runbooks/ci-monitoring.md §TLS cert-expiry monitoring.

Fix — scoped CF Access bypass on the ACME challenge path only:

Create a new, MORE SPECIFIC self-hosted Access application for <hostname>/.well-known/acme-challenge/* with a single bypass / include: everyone policy. CF Access matches on longest-path-prefix, so this app takes precedence over the broad hostname-root app for that one path only — every other path stays fully gated. Do NOT touch the main Access app, do NOT change the zone SSL mode.

export CLOUDFLARE_ACCESS_MGMT_TOKEN=$(infisical secrets get CLOUDFLARE_ACCESS_MGMT_TOKEN \
  --env=prod --path=/MooseQuest/cloudflare --plain)
ACCOUNT_ID="22b5c35090724fbf05db6d4f501ac821"

curl -sS -X POST \
  -H "Authorization: Bearer ${CLOUDFLARE_ACCESS_MGMT_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/access/apps" \
  -d '{
    "name": "<hostname> ACME challenge bypass (Heroku ACM renewal)",
    "domain": "<hostname>/.well-known/acme-challenge/*",
    "type": "self_hosted",
    "session_duration": "0s",
    "app_launcher_visible": false,
    "skip_interstitial": true,
    "policies": [
      {
        "name": "Bypass — Heroku ACM HTTP-01 challenge",
        "decision": "bypass",
        "include": [ { "everyone": {} } ]
      }
    ]
  }'

Then trigger the renewal and poll until it completes:

export HEROKU_API_KEY=$(infisical secrets get HEROKU_API_KEY \
  --env=prod --path=/MooseQuest/heroku --plain)
heroku certs:auto:refresh -a <app>
# Poll (typically completes within a few minutes):
heroku certs:auto -a <app>

Verification:

# Bypass is live: must NOT be a 302 to cloudflareaccess.com.
curl -sI "https://<hostname>/.well-known/acme-challenge/test"
# Before fix: 302 to moosequest.cloudflareaccess.com
# After fix (bypass live, origin cert still invalid): 526, or once renewed: 301 via: heroku-router

# Main gate untouched — root path still gated:
curl -sI "https://<hostname>/"
# Expected: 302 to cloudflareaccess.com (unchanged)

# Origin cert is fresh:
echo | openssl s_client -connect <herokudns-cname>:443 -servername <hostname> 2>/dev/null \
  | openssl x509 -noout -dates

Console (raxx-console-prod) instance of this fix, 2026-07-24: CF Access app 4f596651-ba67-4234-96cd-926fff78aeba ("Console ACME challenge bypass (Heroku ACM renewal)"), domain console.raxx.app/.well-known/acme-challenge/*, policy 3699df6b-246b-4be8-896f-e1bd20401c6f (bypass, include: everyone). See docs/incidents/2026-07-24-console-acm-cf-access-526.md.

Toil-reduction follow-up: any future Heroku app placed behind a hostname-root CF Access application needs this same ACME-path bypass provisioned proactively, before ACM's first renewal window (~60 days before expiry), not reactively after a renewal failure. Consider a Terraform module (terraform/modules/cf-access-acme-bypass/) taking a hostname and emitting both the bypass app and a companion origin TLS probe, so the pattern doesn't have to be hand-rolled per incident.

Emergency stop — re-gate a surface that was accidentally made public

If a CF Access application is accidentally removed and the surface needs to be immediately re-gated:

For getraxx.com (manual CF dashboard path — fastest): See "Re-adding the gate (rollback)" in docs/ops/runbooks/getraxx-launch-day-cf-access-removal.md.

For other surfaces (Terraform path):

cd terraform/cf-access   # or the relevant module directory
terraform apply           # re-creates from HCL definition

Escalation

Wake the operator when: - terraform apply proposes destroying any active CF Access application or policy protecting a live surface. - CF Access tokens cannot be retrieved from Infisical (vault access outage). - A surface that should be gated is publicly accessible and the cause is unknown. - terraform state rm would affect more than the expected resources listed above.

CF support: https://support.cloudflare.com/hc/en-us/requests/new

Bypass apps — complete inventory (raxx.app pre-launch gate)

The raxx.app root app (23bd313e) gates the entire domain. Every path that an anonymous browser or SSR fetch must reach requires its own bypass app (decision=bypass, include=everyone). This table is the canonical record — update after every bypass change.

Vault secret for the API token: CLOUDFLARE_ACCESS_MGMT_TOKEN at /MooseQuest/cloudflare/ (note: runbook diagnostic steps above reference CF_ACCESS_MGMT — the live key is CLOUDFLARE_ACCESS_MGMT_TOKEN).

Phase 1 — Beta walkthrough / preview (created 2026-06-10 / 2026-06-12)

Note: api.raxx.app root app (01b5423f) has a decision=bypass policy (71c61b7f) with include=everyone as its first policy, which currently bypasses the entire api.raxx.app domain for anonymous traffic. The per-path bypass apps above are belt-and-suspenders for when that root policy is eventually tightened.

Phase 2 — Beta join flow (created 2026-06-13, SC-5 #3550)

Verification probes run 2026-06-13 UTC (routes not yet built — probes confirm CF is not intercepting):

When SC-6 deploys the /beta/join/<token> page, re-run probe 1 and confirm non-302 (200 or expected Next.js response, not cloudflareaccess.com redirect).

Phase 3 — Heroku ACM renewal bypasses (created 2026-07-24, incident-driven)

Unlike Phase 1/2 (anonymous product flows), these bypass apps exist so Heroku's ACM HTTP-01 challenge can reach the origin through a CF Access-gated hostname. See Failure mode E above for the full mechanism.

Verification probe run 2026-07-24 UTC (post-fix): console.raxx.app/.well-known/acme-challenge/test returned 301 (via: heroku-router) once the origin cert was reissued — not a 302 to cloudflareaccess.com. console.raxx.app/ still returns 302 to cloudflareaccess.com (root gate intact).

Confirmed duplicate exposure — console-staging.raxx.app (NOT yet fixed, needs separate authorization): verified 2026-07-24 during this incident: console-staging.raxx.app uses the identical CF Access + Heroku ACM pattern (sibling app console-staging, fbacf797-9708-40eb-8c70-0ab558e96988) and is currently in the same failed state — heroku certs:auto -a raxx-console-staging shows acm_status: failed, reason "DNS redirect not forwarding path", origin cert notAfter=2026-07-23 22:42 UTC (already expired, same as prod was before this fix). This incident's authorized remediation was scoped explicitly to console.raxx.app (prod); creating a bypass app for console-staging.raxx.app was attempted and correctly blocked as an out-of-scope surface change — a separate authorization is needed to apply the identical fix to staging. Filed as a type:reliability action item rather than actioned unilaterally — see RCA docs/incidents/2026-07-24-console-acm-cf-access-526.md action item 3.

Known gap — raxx.app/api/auth/register/* (operator action required before Phase 2 live)

Live probe 2026-06-13: raxx.app/api/auth/register/options returns 302 to cloudflareaccess.com for anonymous requests. This path is used by the WebAuthn registration flow (begin-with-token, verify-with-token) when the browser calls Raptor via raxx.app as the proxy host.

The corresponding api.raxx.app/api/auth/register/* path is unaffected — covered by the api.raxx.app root bypass policy 71c61b7f.

Impact now: zero — Phase 2 routes not yet live. SSR fetches on the join flow target api.raxx.app, which is bypassed.

Required before Phase 2 goes live: a bypass app for raxx.app/api/auth/register/* (decision=bypass, include=everyone). This is the same pattern as all other Phase 1/2 bypasses. Provision via CF API using CLOUDFLARE_ACCESS_MGMT_TOKEN from vault, following the Phase 2 provisioning pattern in docs/incidents/2026-06-12-beta-invite-cf-access-blocks.md Resolution section.

Checklist — CF Access bypass set for a public HMAC-signed flow

Lesson from docs/incidents/2026-06-12-beta-invite-cf-access-blocks.md: every anonymous dependency path needs its own bypass app. Use this checklist when adding a new public HMAC-signed flow.

For each new flow, ask:

• [ ] Page route bypassed? (raxx.app/<flow-path>/*)
• [ ] Next.js static assets bypassed? (raxx.app/_next/*) — shared; verify existing bypass covers
• [ ] Public dir assets bypassed? (raxx.app/<asset-dir>/*) — if the flow serves images or other public-dir files
• [ ] API routes on raxx.app bypassed? (raxx.app/api/<flow>/*)
• [ ] API routes on api.raxx.app bypassed? (api.raxx.app/api/<flow>/*) — SSR fetches target this host
• [ ] Auth/register paths bypassed if WebAuthn flow included? (raxx.app/api/auth/register/*) — currently gated; see Known gap above
• [ ] Verification probes run from a clean session (no CF cookie, browser UA, no CF Access credentials)?
• [ ] Root gate sanity check: raxx.app/dashboard still returns 302 to cloudflareaccess.com?

References

• Terraform module (getraxx): terraform/modules/cf-access-getraxx/
• Terraform stack (console/vault/freescout): terraform/cf-access/
• Launch-day removal runbook: docs/ops/runbooks/getraxx-launch-day-cf-access-removal.md
• State import runbook: docs/ops/runbooks/terraform-cf-access-state-imports.md
• Service token provisioning: docs/ops/runbooks/cf-access-service-token-provisioning.md
• Auth posture: docs/security/auth-posture.md
• Web surface posture: docs/security/web-surface-posture.md
• Incident #2849: getraxx.com dashboard delete
• Incident 2026-06-12: beta invite CF Access blocks — docs/incidents/2026-06-12-beta-invite-cf-access-blocks.md
• Phase 2 design doc: docs/architecture/beta-phase2-join-flow.md §6
• CF Zero Trust dashboard: https://one.dash.cloudflare.com/
• CF API token management: https://dash.cloudflare.com/profile/api-tokens