RCA — TLSOriginProbeDown false positive for vault.raxx.app (Cloudflare Origin CA not trusted)

Incident ID: 2026-07-24-vault-origin-probe-ca-false-positive Date: 2026-07-24 Severity: SEV-3 Duration: ~9h (2026-07-24 03:10 origin cert install → ~10:38 fix applied and verified) Blast radius: ci-monitoring alerting only. vault.raxx.app itself was healthy throughout (CF Access edge returned 302 clean, ~70ms). No customer- or CI-facing impact. Author: sre-agent

Summary

TLSOriginProbeDown fired continuously for vault.raxx.app after the 2026-07-23 origin-cert-expiry incident was fixed by installing a static 3-year Cloudflare Origin CA certificate on the Lightsail origin (docs/incidents/2026-07-23-vault-origin-cert-expiry.md). The tls_connect_origin_vault blackbox module validates the origin cert with insecure_skip_verify: false against the default system CA trust store, which does not (and should not) trust Cloudflare's private Origin CA. The alert was a false positive: the origin was healthy and the cert was valid through 2029-07-23. Fix: pin the blackbox origin modules' tls_config.ca_file to Cloudflare's published Origin CA root, restoring both probe_success=1 and the probe_ssl_earliest_cert_expiry metric this monitor exists to provide.

Timeline (all times UTC)

• 2026-07-24 03:10:25 — vault.raxx.app origin cert replaced with a 3-year Cloudflare Origin CA cert (notAfter=2029-07-23T03:02:00Z) as the fix for the prior day's expiry outage. TLSOriginProbeDown{instance="vault.raxx.app"} begins firing from this point (origin cert now signed by an untrusted-by-default CA).
• 10:33:40 — This incident opened. TLSOriginProbeDown confirmed firing in Prometheus for both vault.raxx.app and tickets.raxx.app.
• 10:33:40 — Pre-change state preserved: curl https://vault.raxx.app/ confirmed edge healthy (302 to CF Access login, TLS clean, ~70ms). blackbox.yml backed up to blackbox.yml.pre-2819-ca-fix.bak on the box. Confirmed probe_ssl_earliest_cert_expiry{instance="34.207.47.85:443"} returned no data (metric not populating — a failed handshake has no chain to read an expiry from).
• 10:33:40 — Root cause confirmed via blackbox_exporter debug probe (GET /probe?target=34.207.47.85:443&module=tls_connect_origin_vault&debug=true): level=error msg="Error dialing TCP" err="tls: failed to verify certificate: x509: certificate signed by unknown authority" TCP+TLS dial itself succeeded (ruling out the firewall/reachability hypothesis) — failure was purely chain verification.
• 10:33:40 — Secondary hypothesis (monitoring box egress IP firewalled off the vault origin) ruled out: the debug log shows a completed TLS dial, not a timeout/refused connection. raxx-vault's Lightsail firewall allows 443/0.0.0.0/0 (documented, unlike tickets).
• 10:34–10:36 — Fetched Cloudflare's published Origin CA RSA root (https://developers.cloudflare.com/ssl/static/origin_ca_rsa_root.pem) and verified the vault origin leaf cert's issuer matches this root exactly (openssl verify -CAfile origin_ca_rsa_root.pem vault-leaf.pem → OK).
• 10:36–10:38 — terraform/ci-monitoring/templates/cloud-init.sh.tpl updated: the CA root baked in as a static file write ($MONITORING_DIR/blackbox/certs/cloudflare-origin-ca-rsa-root.pem), both tls_connect_origin_vault and tls_connect_origin_tickets modules gained tls_config.ca_file pointing at it. terraform validate passed; bash -n on the rendered template passed; the extracted blackbox.yml block parsed as valid YAML.
• 10:38:52 — Same change hot-applied to the live instance via SSM Run Command (no SSH): cert file written + chmod 644, blackbox.yml patched with ca_file on both origin modules via a Python/PyYAML edit (same idiom as the existing inject-cf-access-headers.sh), docker compose restart blackbox-exporter.
• 10:39:08 — Debug probe re-run: msg="Successfully dialed", probe_success 1, probe_ssl_earliest_cert_expiry 1.87947012e+09 (= 2029-07-23T03:02:00Z, matching the cert exactly), TLS 1.3, leaf subjectalternative="vault.raxx.app".
• ~10:39:48 (after next Prometheus scrape) — Prometheus confirms probe_success{job="blackbox-tls-origin",instance="vault.raxx.app"} == 1, cert-expiry days-remaining ≈ 1094.68 (matches notAfter), and vault.raxx.app no longer appears in /api/v1/alerts under TLSOriginProbeDown. tickets.raxx.app remains probe_success=0 (10.000s — a timeout, not a cert error) and pending/firing in Prometheus, unchanged and expected: its origin probe is separately non-functional due to a documented Lightsail firewall allowlist gap, not the CA issue this incident addresses.
• Resolved.

Impact

• Users affected: none (internal monitoring signal only; vault itself was up throughout)
• User-visible symptoms: none
• Data integrity: ok
• Revenue / billing: ok
• Secondary effect (now also resolved): probe_ssl_earliest_cert_expiry for the vault origin was blank for the ~9h duration — the 30/14-day expiry lead-time this monitor was built to provide (in direct response to the 2026-07-23 outage) was itself silently unavailable for the origin that incident was about.

What went well

• The pre-existing debug-probe workflow (/probe?...&debug=true) gave the exact failure string immediately — no guessing between cert-expiry, reachability, and chain-verification failure modes.
• The firewall/reachability hypothesis was ruled out with hard evidence (a completed TLS dial in the debug log) rather than assumed away.
• The fix was verified against Cloudflare's actual published root, not assumed: openssl verify against the downloaded PEM confirmed a byte-level chain match to the live origin leaf cert before touching any config.
• The existing "hot-apply via SSM, bake into cloud-init.sh.tpl" pattern from #2819 applied cleanly to this fix — no new deployment mechanism needed.
• insecure_skip_verify: true was never used as a shortcut. The fix narrows trust to exactly the CA in use; it does not disable verification.
• The unrelated, already-documented tickets.raxx.app firewall gap was left untouched and unaffected — the same ca_file was applied to its module (correct and ready for when the firewall gap closes) without attempting to fix or paper over the separate, known, operator-decision-pending issue.

What didn't go well

• The #2819 build-out (PR #4332, merged the same day as this incident) shipped an origin-probe design that only worked against the origin cert state that existed at build time (a Let's Encrypt cert, about to expire). It did not anticipate that the very incident it was built to detect would itself change the origin's cert issuer to a CA outside the public trust store, and had no test case covering "origin serves a valid CF Origin CA cert."
• There was a ~9h gap between the origin cert being replaced (03:10) and this false positive being triaged (10:33) — the alert fired the whole time but was not looked at immediately, because it looked like a continuation of the already-known, still-being-worked prior incident rather than a distinct new condition.

Root cause analysis

• Contributing factor 1 — the origin probe validates against the public CA trust store by default, but both origins it watches use a private CA. vault.raxx.app and tickets.raxx.app intentionally use static Cloudflare Origin CA certificates (not publicly-trusted ones) specifically because they sit behind Cloudflare Full Strict and don't need public trust — only Cloudflare's edge needs to trust them. blackbox_exporter's tls_config has no way to know that without an explicit ca_file; its default behavior (system trust store) is correct for probing an arbitrary public host but wrong for probing these two specific private-CA origins.

• Contributing factor 2 — the #2819 monitoring build-out was validated against a cert that was mid-incident, not the target end state. The origin-probe design and its "verification evidence" in PR #4332 was written and tested while vault.raxx.app's origin cert was the old, expired Let's Encrypt cert — a state that happened to produce probe_success=0 for the right reason (expiry) but never exercised the code path against the post-fix Cloudflare Origin CA cert, which produces probe_success=0 for the wrong reason (untrusted issuer) unless ca_file is set.

Detection

• What alerted us: TLSOriginProbeDown firing continuously in Prometheus/Alertmanager.
• How long between cause (cert install) and detection/triage: ~9h27m.
• How to detect faster next time: this class of failure is now covered by Failure mode I in the runbook, with the debug-probe command as the first diagnostic step; a future improvement would be to add a synthetic post-deploy check whenever an origin's TLS cert issuer changes (cert rotation event) that immediately re-runs the origin probe and treats a new failure within minutes of a known cert change as high-priority to triage rather than assuming it's a continuation of the prior incident.

Resolution

• What was changed: 1. Cloudflare's published Origin CA RSA root certificate baked into terraform/ci-monitoring/templates/cloud-init.sh.tpl as a static file write at $MONITORING_DIR/blackbox/certs/cloudflare-origin-ca-rsa-root.pem. 2. tls_connect_origin_vault and tls_connect_origin_tickets blackbox modules gained tls_config.ca_file pointing at that file. insecure_skip_verify unchanged (false). 3. docs/ops/runbooks/ci-monitoring.md updated with Failure mode I and a cross-reference in the §TLS cert-expiry monitoring section. 4. Same change hot-applied to the live ci-monitoring EC2 instance via SSM Run Command (cert file write, blackbox.yml patch, docker compose restart blackbox-exporter) so the fix is live now, not just on next replacement.

• Validation:

• Before: debug probe → tls: failed to verify certificate: x509: certificate signed by unknown authority; probe_success 0; probe_ssl_earliest_cert_expiry absent; TLSOriginProbeDown{instance="vault.raxx.app"} firing.
• After: debug probe → Successfully dialed; probe_success 1; probe_ssl_earliest_cert_expiry = 2029-07-23T03:02:00Z (exact match to the cert's real notAfter); Prometheus confirms probe_success{job="blackbox-tls-origin",instance="vault.raxx.app"}==1 and days-remaining ≈1094.7; vault.raxx.app no longer present under TLSOriginProbeDown in /api/v1/alerts.
• terraform validate and bash -n (on the rendered template) both pass. The extracted blackbox.yml block parses as valid YAML with ca_file present on both origin modules.
• tickets.raxx.app origin probe remains probe_success=0 post-fix, but now for the pre-existing, documented reason (10s timeout — Lightsail firewall, not a cert error) — confirmed unaffected/not regressed by this change.

Action items

No GitHub issue filed for these — per standing operator policy ("fix, don't file — no unsolicited GH issues"), tracked here in the RCA instead.

References

• Runbook: docs/ops/runbooks/ci-monitoring.md (§TLS cert-expiry monitoring, Failure mode I)
• Related incident: docs/incidents/2026-07-23-vault-origin-cert-expiry.md (origin cert replaced with CF Origin CA cert — the change that caused this false positive)
• Related incident: docs/incidents/2026-06-19-freescout-526-ssl-strict.md (prior CF Origin CA precedent, tickets.raxx.app)
• PR that introduced the origin probes: #4332 (feat(o11y): fleet TLS cert-expiry monitoring)
• Cloudflare Origin CA docs: https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/
• Cloudflare Origin CA RSA root: https://developers.cloudflare.com/ssl/static/origin_ca_rsa_root.pem