RCA — console.raxx.app 526 (Heroku ACM renewal blocked by CF Access)

Incident ID: 2026-07-24-console-acm-cf-access-526 Date: 2026-07-24 Severity: SEV-1 (user-facing production outage — console.raxx.app) Duration: ~7m from remediation start to verified-fixed (root cause had been silently accumulating for ~30 days prior to detection) Blast radius: Operator/authenticated console.raxx.app sessions (Raxx Console — Heroku app raxx-console-prod). Note: during remediation, console-staging.raxx.app (Heroku app raxx-console-staging) was found to be independently affected by the identical failure mode (see Action item 3) — its fix is out of this incident's authorized scope and remains open. Author: sre-agent

Summary

console.raxx.app's origin TLS certificate (Heroku ACM, Let's Encrypt, CN=console.raxx.app) expired Jul 23 23:00:09 2026 GMT. Cloudflare's zone SSL mode is strict (Full Strict), so any request that Cloudflare Access proxied through to origin was rejected with an HTTP 526. Heroku's ACM auto-renewal had been failing silently since approximately 2026-06-24 (acm_status: failed, reason "DNS redirect not forwarding path") because the CF Access application "Raxx Console" gates the entire console.raxx.app hostname and 302-redirects every unauthenticated request — including Heroku's ACM HTTP-01 challenge probes to /.well-known/acme-challenge/* — to the CF Access login page, so the challenge could never complete. Fix: a new, path-scoped CF Access application for console.raxx.app/.well-known/acme-challenge/* with a single bypass policy (include: everyone), which takes precedence over the broader hostname-root app. This let Heroku's ACM challenge reach origin; a subsequent heroku certs:auto:refresh completed within ~2 minutes and issued a fresh cert (notAfter=Oct 22 16:59:45 2026 GMT). The main CF Access gate on console.raxx.app was not modified — every other path remains fully gated.

Timeline (all times UTC)

• ~2026-06-24 — Heroku ACM auto-renewal begins failing silently (acm_status: failed, "DNS redirect not forwarding path"). No monitoring existed to surface this — console.raxx.app had an edge TLS probe (sees Cloudflare's own Universal SSL cert, unaffected) but no origin probe (would have seen the Heroku ACM cert's actual expiry countdown).
• 2026-07-23 23:00 — Origin cert expires. Cloudflare Full Strict begins rejecting the origin with HTTP 526 for any request that reaches it (authenticated CF Access sessions; unauthenticated requests never reach origin at all — CF Access 302s them to login before proxying, so the outage was invisible to simple curl -I https://console.raxx.app/ checks).
• 2026-07-24 (incident start) — Operator reports console.raxx.app 526; authorizes the scoped-ACME-bypass + ACM-refresh remediation. Root cause already confirmed prior to dispatch (origin cert expired, ACM status failed, CF Access blocking the HTTP-01 challenge path).
• 17:51 — sre-agent session bootstrap; vault credentials for CLOUDFLARE_ACCESS_MGMT_TOKEN and HEROKU_API_KEY resolved via Infisical Universal Auth (CLI --token flag path did not work against this self-hosted instance in this environment; used the documented scripts/ci/wpc_load_vault_secrets.py REST pattern instead — no secret values printed).
• 17:53 — Confirmed before-state: origin cert notAfter=Jul 23 23:00:09 2026 GMT (direct TLS probe to the herokudns CNAME with SNI=console.raxx.app); heroku certs:auto -a raxx-console-prod → acm_status: failed, reason "DNS redirect not forwarding path"; existing CF Access app "Raxx Console" (0b55d01b-592b-4da4-b170-02d48a9f550d) confirmed unchanged (3 policies: operator-email allow, two service-token non_identity policies).
• 17:57:41 — Created CF Access application 4f596651-ba67-4234-96cd-926fff78aeba ("Console ACME challenge bypass (Heroku ACM renewal)"), domain console.raxx.app/.well-known/acme-challenge/*, single policy 3699df6b-246b-4be8-896f-e1bd20401c6f (bypass, include: everyone).
• 17:57:47 — Verified bypass is live: curl -I https://console.raxx.app/.well-known/acme-challenge/test returned 526 (not a 302 to cloudflareaccess.com) — confirms CF Access now proxies this path straight to origin instead of redirecting to login. Root path console.raxx.app/ still returned 302 (main gate untouched).
• 17:58:1x — Ran heroku certs:auto:refresh -a raxx-console-prod.
• 17:58–18:00 — Polled heroku certs:auto -a raxx-console-prod: DNS Verified → Cert issued within ~2 minutes.
• 18:00:29 — Verification: direct origin TLS probe shows fresh cert (notAfter=Oct 22 16:59:45 2026 GMT, issuer Let's Encrypt, CN=YR1); console.raxx.app/.well-known/acme-challenge/test now returns 301 with via: heroku-router (proxying to a healthy origin, no more 526); console.raxx.app/ returns 302 to moosequest.cloudflareaccess.com (normal, healthy CF Access login redirect — not a 526).
• Resolved.

Impact

• Users affected: operator only (console.raxx.app is CF Access-gated to kris@moosequest.net plus two service tokens — no public/customer traffic).
• User-visible symptoms: HTTP 526 for any authenticated console session reaching origin, since the cert expired 2026-07-23 23:00 UTC.
• Data integrity: ok (no data path involved — pure TLS/edge issue).
• Revenue / billing: ok.

What went well

• Root cause had already been correctly diagnosed before remediation started (origin cert expiry + ACM failure reason + CF Access blocking the challenge path) — no re-diagnosis time spent.
• The fix pattern (path-scoped bypass app, more specific than the root gate) was already established precedent in this runbook (Apple CDN / AASA .well-known bypass, beta-flow bypasses) — no novel CF Access mechanism had to be invented.
• Verification was layered correctly: confirmed the bypass was live via the ACME path itself before triggering the Heroku refresh, so if the bypass hadn't taken effect the failure mode would have been caught immediately rather than blamed incorrectly on Heroku.
• Total remediation time (bypass creation → verified fixed) was under 10 minutes once credentials were resolved.

What didn't go well

• Fleet TLS cert-expiry monitoring (#2819, built 2026-07-23 in response to the vault.raxx.app origin-cert incident) added an edge probe for console.raxx.app but not an origin probe — the one signal that would have caught this ~30 days before the actual outage. The monitoring build-out's own stated purpose ("edge cert hides origin-cert rot completely") applied to console.raxx.app just as much as vault/tickets, but console wasn't included in the origin-probe target list.
• Heroku ACM's acm_status: failed state produces no proactive page/email — it is only visible by manually running heroku certs:auto. Combined with the missing origin probe, there was no signal at all between the ~2026-06-24 first renewal failure and the 2026-07-23 actual expiry.
• The 526 was not reproducible with a simple unauthenticated curl -I https://console.raxx.app/ — CF Access intercepts and 302s anonymous traffic before it ever reaches origin, so the standard "is it down" check looked completely healthy. Only an authenticated session (or a bypassed path, like the ACME challenge path itself) would show the 526. This is a detection blind spot worth calling out explicitly in any future synthetic-probe design for CF Access-gated hosts.

Root cause analysis

• Contributing factor 1 — CF Access gates the ACME challenge path with no carve-out: The "Raxx Console" CF Access application covers the entire console.raxx.app hostname with no path exception. Heroku ACM's HTTP-01 challenge is, by protocol design, an unauthenticated GET to a well-known path — CF Access cannot distinguish "Let's Encrypt validation probe" from "anonymous browser hit" and correctly (per its own policy) redirects both to the login page. The system had no mechanism reconciling "gate everything" with "except the one path a third-party cert authority needs to validate against."
• Contributing factor 2 — monitoring build-out didn't generalize the origin-probe pattern to all Heroku-fronted, CF Access-gated hosts: The #2819 fleet cert-expiry monitor was built in direct response to a nearly identical origin-cert-expiry incident on vault.raxx.app one day earlier, and explicitly reasoned about "the origin cert can expire while the edge cert is fine" — but the origin-probe target list was populated from the two hosts already known to be origin-cert problems (vault, tickets; Lightsail-hosted, CF Origin CA certs) rather than from a systematic sweep of every CF Access-gated, Full-Strict hostname. console.raxx.app uses a different origin-cert mechanism (Heroku ACM / Let's Encrypt) and a different origin shape (Heroku CNAME, no static IP) than vault/tickets, which likely made it easy to overlook as "the same class of risk" during a fast, incident-driven build-out.
• Contributing factor 3 — no alerting on Heroku ACM renewal failure itself: Heroku's ACM system surfaces acm_status: failed only via manual CLI inspection (heroku certs:auto). There is no Heroku webhook, email, or API polling in place that would have surfaced the ~2026-06-24 renewal failure at the moment it happened — roughly 30 days before the cert actually expired, which would have been ample lead time to fix reactively even without the origin-probe gap.

Detection

• What alerted us: operator-reported outage (not an automated alert — the origin-probe gap meant no automated system caught this).
• How long between cause and detection: ~30 days (first ACM renewal failure ~2026-06-24) to cert expiry (2026-07-23), then undetected until operator report on 2026-07-24.
• How to detect faster next time: tls_connect_origin_console origin probe added in this incident's monitor PR closes the gap that mattered most — the actual cert-expiry countdown is now watched independently of CF's edge cert. TLSCertExpirySoon (< 30 days) / TLSCertExpiryUrgent (< 14 days) will now page well before a repeat of this failure mode reaches expiry. A second, still-open gap: Heroku ACM's acm_status: failed state itself is not polled — see action item below.

Resolution

• What was changed: 1. New CF Access application 4f596651-ba67-4234-96cd-926fff78aeba ("Console ACME challenge bypass (Heroku ACM renewal)"), domain console.raxx.app/.well-known/acme-challenge/*, single bypass / include: everyone policy 3699df6b-246b-4be8-896f-e1bd20401c6f. The existing "Raxx Console" app (0b55d01b-592b-4da4-b170-02d48a9f550d) and its three policies (operator-email allow, two service-token non_identity) were not modified. Zone SSL mode (strict) was not modified. 2. heroku certs:auto:refresh -a raxx-console-prod — triggered ACM to re-attempt the HTTP-01 challenge, now reachable through the bypass. 3. New Heroku-issued cert: CN=console.raxx.app, issuer Let's Encrypt, CN=YR1, notBefore=Jul 24 16:59:46 2026 GMT, notAfter=Oct 22 16:59:45 2026 GMT. 4. Added tls_connect_origin_console origin TLS probe to terraform/ci-monitoring/templates/cloud-init.sh.tpl (mirrors the existing tls_connect_origin_vault / tls_connect_origin_tickets pattern, targeting the herokudns CNAME with SNI forced to console.raxx.app; no ca_file override needed since the console origin cert is a publicly-trusted LE cert, not a CF Origin CA cert).
• Validation:
• Before fix: console.raxx.app/.well-known/acme-challenge/test → 302 to moosequest.cloudflareaccess.com; heroku certs:auto → acm_status: failed.
• Immediately after bypass creation, before refresh: console.raxx.app/.well-known/acme-challenge/test → 526 (proxying to origin now works; origin cert still expired, as expected pre-renewal).
• After certs:auto:refresh + ~2m: heroku certs:auto → Cert issued; direct origin TLS probe shows notAfter=Oct 22 16:59:45 2026 GMT; console.raxx.app/.well-known/acme-challenge/test → 301 via heroku-router (healthy origin, no 526); console.raxx.app/ → 302 to CF Access login (normal, gate intact, no 526).

Action items

References

• Runbook: docs/ops/runbooks/cf-access.md (Failure mode E)
• Runbook: docs/ops/runbooks/ci-monitoring.md (§TLS cert-expiry monitoring)
• Related incident (same failure class, one day earlier, different host): docs/incidents/2026-07-23 era vault.raxx.app origin cert expiry — see docs/ops/runbooks/ci-monitoring.md header for the vault/tickets precedent and the #2819 monitoring build-out this incident's fix extends.
• Related incident: docs/incidents/2026-06-19-freescout-526-ssl-strict.md (same 526-class failure — CF Full Strict rejecting an invalid/expired origin cert — different root cause: no origin cert installed at all, vs. here: valid mechanism blocked from renewing).
• Heroku ACM docs: https://devcenter.heroku.com/articles/automated-certificate-management
• CF Access apps API: https://developers.cloudflare.com/api/operations/access-applications-list-access-applications