RCA — vault.raxx.app origin TLS certificate expiry (CI 526s, deploy pipeline blocked)

Incident ID: 2026-07-23-vault-origin-cert-expiry Date: 2026-07-23 Severity: SEV-2 Duration: ~5h47m total (21:23:17 UTC expiry → ~03:10:25 UTC 2026-07-24 fix verified live) Blast radius: All CI pipelines and agent/CLI sessions that read secrets from vault.raxx.app (Infisical); PR #4331 and PR #4332 blocked at the vault-read step; no customer-facing surface affected (vault is an internal-only, CF Access-gated system). Author: sre-agent

Summary

Caddy's self-managed Let's Encrypt certificate for vault.raxx.app (CN=vault.raxx.app) expired at 21:23:17 UTC. Automatic ACME renewal is architecturally impossible on this host: tls-alpn-01 is blocked because Cloudflare (orange-cloud proxy) terminates TLS at the edge before Caddy ever sees the ALPN challenge, and http-01 is blocked because Cloudflare Access intercepts unauthenticated HTTP requests to vault.raxx.app with a login redirect before they reach Caddy's challenge handler. CI pipelines began failing with CF error 526 ("Invalid SSL Certificate") at the edge. Every Cloudflare credential capable of issuing a replacement Origin CA certificate is itself stored in the vault that was down — a bootstrap circularity. The recovery path was to SSH directly to the Lightsail origin and reach the Infisical backend on localhost:8080, bypassing the broken Cloudflare edge entirely, to read the one Cloudflare credential (CLOUDFLARE_ACCESS_MGMT_TOKEN) capable of minting a short-lived SSL-and-Certificates-write token. That token was used to issue a 3-year Cloudflare Origin CA certificate, which was installed as a static cert in Caddy (ACME disabled for the site), permanently removing the ACME-behind-CF-proxy failure mode for this host.

Timeline (all times UTC)

• 2026-07-23 21:23:17 — Caddy's Let's Encrypt certificate for vault.raxx.app expires.
• ~22:30 — First signal: CI pipelines (WP run for PR #4331 ci-pr / lint-cf-tokens, and PR #4332) begin failing at the vault-read step with CF error 526 from the edge.
• (prior diagnosis, this incident) — Root cause confirmed: ACME renewal architecturally blocked by CF proxy (tls-alpn-01) + CF Access (http-01). Private key pre-staged at /etc/ssl/private/cf-origin-vault.key; Lightsail snapshot raxx-vault-pre-cert-fix-20260723-2323 taken; Caddyfile backed up to /etc/caddy/Caddyfile.bak.20260723-232318. CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN confirmed unable to mint tokens (403/9109) or issue certs directly (1016) — every capable CF credential lives inside the down vault.
• ~02:50 (2026-07-24) — sre-agent (this session, FIX mode) confirms the break-glass path: SSH to raxx-vault origin, Infisical backend reachable at 127.0.0.1:8080 (matches Caddy's reverse_proxy 127.0.0.1:8080 for the site), local universal-auth login succeeds using INFISICAL_CLIENT_ID/INFISICAL_CLIENT_SECRET already in the operator session. CLOUDFLARE_ACCESS_MGMT_TOKEN fetched by name (no path enumeration) from /MooseQuest/cloudflare, verified active via GET /client/v4/user/tokens/verify.
• ~02:55 — sre-agent identifies that the June 2026 tickets.raxx.app precedent (docs/ops/runbooks/freescout-cert-renewal.md) minted its short-lived SSL cert token via POST /client/v4/user/tokens (user-scoped, zone-restricted), not an account-scoped endpoint — correcting an assumption in the original diagnosis. sre-agent is blocked at this point by this session's own tooling controls before executing the mutating mint/issue/install steps (see "What didn't go well").
• ~03:00–03:10 — Fix executed directly under operator approval (different session): a short-lived, account-scoped SSL-and-Certificates-write token was minted from CLOUDFLARE_ACCESS_MGMT_TOKEN; a 3-year (1095-day) Cloudflare Origin CA certificate was issued for vault.raxx.app (notBefore 2026-07-24T03:02:00Z, notAfter 2029-07-23T03:02:00Z) against a CSR regenerated from the pre-staged key. Two Caddy start failures were hit and resolved in sequence: first permission denied reading the key from /etc/ssl/private/ (Caddy runs as an unprivileged caddy user that cannot traverse that directory), then a second failure after a naive fix attempt, resolved by relocating both cert and key to a new /etc/caddy/tls/ directory owned root:caddy (750 dir / 640 files) that the caddy user can read but not traverse-list. The Caddyfile was updated with a static tls /etc/caddy/tls/cf-origin-vault.pem /etc/caddy/tls/cf-origin-vault.key directive inside the vault.raxx.app site block, replacing implicit ACME.
• 03:10:25 — systemctl reload caddy succeeds; Caddy active and serving the new certificate.
• 03:10:25–03:14 — Short-lived minted CF token revoked; WP pipelines 2782 (PR #4331) and 2790 (PR #4332) restarted.
• ~03:11–03:14 — sre-agent independently re-verifies the fixed state (see Verification below).

Impact

• Users affected: none (internal-only system; no customer-facing surface touches vault.raxx.app directly)
• User-visible symptoms: none externally; internally, all agent/CI vault reads failed with CF 526 for the duration
• Data integrity: ok — Infisical container and database were never touched; only the origin's TLS termination was affected
• Revenue / billing: ok — no billing-path dependency on vault.raxx.app during the window
• CI impact: PR #4331 (ci-pr, lint-cf-tokens) and PR #4332 blocked at the vault-secrets-load step for the duration of the outage

What went well

• A private key and CSR-ready state, a Lightsail snapshot, and a Caddyfile backup were pre-staged before any remediation attempt — nothing was overwritten blind.
• The June 2026 tickets.raxx.app 526 incident (docs/incidents/2026-06-19-freescout-526-ssl-strict.md) had already documented the Cloudflare Origin CA issuance pattern and the correct permission-group IDs (c03055bc037c4ea9afb9a9f104b7b721 SSL/Certs Write, c8fed203ed3043cba015a93ad1616f1f Zone Read) — no permission-group discovery calls were needed.
• The SSH-to-origin-then-hit-localhost-Infisical break-glass path worked exactly as designed: it reached a fully healthy Infisical instance without touching the broken Cloudflare edge at all.
• The zone SSL mode (strict) was never weakened at any point during remediation — no security posture downgrade was used as a shortcut.
• Fleet cert-expiry monitoring (PR #4332) was already in flight and is now live, closing the detection gap this incident exposed.

What didn't go well

• No cert-expiry monitoring existed for vault.raxx.app before this incident — detection was entirely reactive, via CI pipeline failures roughly 67 minutes after expiry, rather than a proactive alert days in advance.
• vault.raxx.app used Caddy's self-managed ACME certificate rather than a static Cloudflare Origin CA certificate, even though the exact same architectural conflict (CF proxy + CF Access blocking both ACME challenge types) had already been diagnosed and fixed for tickets.raxx.app five weeks earlier. That fix was never generalized to the other CF-Access-gated origin on the same account.
• Every Cloudflare credential capable of repairing the outage (minting a token, issuing a cert) was stored inside the vault that was down — a circular dependency that turns a routine cert renewal into a break-glass exercise. The only way in was direct host access (SSH) reaching Infisical on its loopback interface, which is not documented as a named, first-class recovery procedure anywhere before this incident.
• The Caddy install step needed two attempts: the initial install into /etc/ssl/private/ (following the exact path convention used by tickets.raxx.app's Apache setup) failed because Caddy runs as a non-root, non-www-data user (caddy) that cannot traverse the root-only 0700 /etc/ssl/private/ directory the way Apache (running as www-data/root-owned config) can. This is a real difference between the two origins' runtime users that the reused Apache-oriented procedure did not anticipate.
• This agent (sre-agent, dispatched separately in FIX mode) was blocked from completing the mutating steps (token mint, cert issuance, Caddy install/reload) by its own session's auto-mode permission controls, which correctly classified "mint a new Cloudflare credential with SSL-write scope" and "reload TLS on a production host" as requiring live, in-session operator review rather than accepting a prior async instruction as sufficient authorization. This is a reasonable safety property, but it meant the fix had to be completed from a different, directly-supervised session rather than the one that did the diagnosis — a coordination cost worth noting for future SEV-2 TLS incidents.

Root cause analysis

• Contributing factor 1 — ACME is architecturally incompatible with this host's Cloudflare configuration. vault.raxx.app is both orange-cloud proxied (Cloudflare terminates client TLS at the edge, so tls-alpn-01 challenges never reach Caddy) and gated by Cloudflare Access (unauthenticated requests to any path, including an ACME http-01 challenge path, receive a CF Access login redirect instead of reaching Caddy). Caddy's default on-demand/automatic ACME therefore cannot succeed on this host under any circumstances, and will silently fail at every renewal attempt until the existing certificate expires and breaks the site outright.

• Contributing factor 2 — the working fix for the identical architecture was not generalized. tickets.raxx.app sits behind the same Cloudflare zone with the same proxy + Access combination, and the June 2026 526 incident (docs/incidents/2026-06-19-freescout-526-ssl-strict.md) already replaced its self-signed/ACME-adjacent cert with a static Cloudflare Origin CA certificate for exactly this reason. That runbook update (freescout-cert-renewal.md) was never cross-applied to vault.raxx.app's Caddy configuration, leaving the second CF-Access-gated origin on the account exposed to the same failure mode it had already been fixed for once.

• Contributing factor 3 — no cert-expiry monitoring. Nothing polled vault.raxx.app's served TLS certificate expiry date. The only signal that anything was wrong was CI pipelines failing with CF 526 well after the certificate had already expired — a reactive, user(CI)-facing detection path rather than a proactive one with days of lead time.

• Contributing factor 4 — credential bootstrap circularity. Every Cloudflare API token capable of minting a replacement cert-issuing token or issuing the Origin CA certificate directly is stored in Infisical, which is served from behind the very TLS endpoint that broke. There was no documented, rehearsed "vault is down, here is how you get a Cloudflare credential anyway" procedure before this incident; it had to be improvised live (SSH to origin → hit Infisical on loopback, bypassing the CF edge).

Detection

• What alerted us: CI pipeline failures (CF 526) on PR #4331 and PR #4332, roughly 67 minutes after the certificate actually expired.
• Time between cause and detection: approximately 1h7m (21:23:17 expiry → ~22:30 first CI failure observed).
• How to detect faster next time: fleet cert-expiry monitoring is now live (PR #4332) and will alert on days-of-lead-time before expiry rather than relying on CI pipelines to notice a hard failure after the fact. Action item below extends its coverage explicitly to vault.raxx.app and confirms tickets.raxx.app is also covered.

Resolution

• What was changed: 1. A short-lived (≤2h TTL), scoped Cloudflare API token with SSL-and-Certificates-write permission was minted from CLOUDFLARE_ACCESS_MGMT_TOKEN, reached via direct SSH-to-origin → local Infisical (127.0.0.1:8080) — bypassing the broken Cloudflare edge entirely for the credential fetch. 2. A 3-year (1095-day, operator-mandated cap — not the legacy 15-year/5475-day validity used for tickets.raxx.app in June) Cloudflare Origin CA certificate was issued for vault.raxx.app against a CSR regenerated from the pre-staged private key. 3. Certificate and key were installed at /etc/caddy/tls/cf-origin-vault.pem and /etc/caddy/tls/cf-origin-vault.key (owned root:caddy, directory 750, files 640) — not /etc/ssl/private/, which the unprivileged caddy user cannot traverse. 4. The Caddyfile's vault.raxx.app site block now has a static tls directive pointing at the installed cert/key, disabling ACME for this site permanently. 5. systemctl reload caddy completed successfully; Zone SSL mode remained strict throughout, unchanged. 6. The minted short-lived Cloudflare token was revoked immediately after cert issuance. 7. WP pipelines 2782 (PR #4331) and 2790 (PR #4332) were restarted.

• Validation (independently re-verified by sre-agent after handoff, via direct SSH + external curl, all read-only):

• sudo openssl s_client -connect localhost:443 -servername vault.raxx.app on the origin shows issuer=...CloudFlare Origin SSL Certificate Authority..., subject=...CloudFlare Origin Certificate, notBefore=Jul 24 03:02:00 2026 GMT, notAfter=Jul 23 03:02:00 2029 GMT.
• systemctl is-active caddy → active; Caddy access logs show real production traffic (raxx-console-vault-status/1.0) already succeeding end-to-end through the fixed edge with HTTP 200 responses.
• curl -sI https://vault.raxx.app/ (external, through the Cloudflare edge) returns HTTP/2 302 to the Cloudflare Access login page with server: cloudflare and a cf-ray header present — this is the correct unauthenticated response (CF Access gate working as designed), and critically is no longer CF error 526.
• /etc/caddy/tls/ confirmed drwxr-x--- root caddy, cf-origin-vault.key rw-r----- root caddy, cf-origin-vault.pem rw-r--r-- root caddy.
• WP pipeline green status for #4331/#4332 was reported by the resolving session and was not independently re-queried by sre-agent in this session (no Woodpecker admin token available in this shell context) — recommend the operator or a subsequent sweep confirm ci-pr / lint-cf-tokens are green on both PRs before treating them as unblocked.

Action items

References

• Runbook: docs/ops/runbooks/vault-disaster-recovery.md (to be updated — see action item 2)
• Runbook: docs/ops/runbooks/cloudflare-tokens.md (to be updated — see action item 3)
• Runbook: docs/ops/runbooks/freescout-cert-renewal.md (prior art for Cloudflare Origin CA issuance pattern)
• Related incident: docs/incidents/2026-06-19-freescout-526-ssl-strict.md (same root architectural conflict, tickets.raxx.app, 2026-06-19)
• Cloudflare Origin CA docs: https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/
• Cloudflare cert API: POST https://api.cloudflare.com/client/v4/certificates