RCA — RaptorProdDown false-positive: BFM blocks monitoring probe on api.raxx.app/health

Incident ID: 2026-07-14-raptor-prod-down-false-positive Date: 2026-07-14 Severity: SEV-3 (monitoring false-positive; Raptor prod healthy throughout; no user impact) Duration: ~24h 27m (first signal 2026-07-13T16:49:51Z → alert cleared 2026-07-14T17:17Z) Blast radius: Internal ops only — RaptorProdDown alert firing in Alertmanager, ops@ email spam Author: sre-agent

Summary

The RaptorProdDown Alertmanager alert fired for ~24 hours while Raptor prod was fully healthy. The blackbox_exporter probe was sending a bare GET to https://api.raxx.app/health (the CF-fronted URL). Cloudflare Bot Fight Mode (BFM) evaluated the request before CF Access and returned 403, causing probe_success == 0. Raptor itself logged continuous GET /health → 200 from internal health checks throughout. Fix: repointed the probe to the Heroku origin URL https://raxx-api-prod-a60a19e5efbf.herokuapp.com/health, which bypasses Cloudflare and correctly tests dyno health. Alert cleared within one Prometheus scrape cycle (30s).

Timeline (all times UTC)

2026-07-13 16:49:51 — RaptorProdDown first fires; probe_success{instance="https://api.raxx.app/health"} == 0 for 2 consecutive minutes; Alertmanager begins emailing ops@
2026-07-13 16:49+ — Raptor prod healthy throughout (Heroku logs: continuous GET /health → 200, no R14/H12/crash)
2026-07-14 ~17:10 — Operator reports "Raptor is alerting" and dispatches sre-agent with evidence of false-positive
2026-07-14 17:10 — sre-agent: confirmed alert via SSM → Alertmanager: single active RaptorProdDown
2026-07-14 17:10 — Identified root cause: BFM blocks probe at WAF layer, before CF Access bypass policy evaluates
2026-07-14 17:10 — Confirmed CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN has Zone:Bot Management:Write but NOT Zone:WAF:Edit (cannot create WAF skip rule)
2026-07-14 17:16 — SSM Run Command: updated /opt/monitoring/prometheus/prometheus.yml (new target URL) and /opt/monitoring/prometheus/rules/alerts.yml (updated expression + description); Prometheus config validated via promtool; Prometheus reloaded (HTTP 200)
2026-07-14 17:17 — Verified: probe_success{instance="https://raxx-api-prod-a60a19e5efbf.herokuapp.com/health"} = 1; Alertmanager active alerts: []
2026-07-14 17:17 — Updated terraform/ci-monitoring/templates/cloud-init.sh.tpl to reflect new probe target (IaC durability)
2026-07-14 17:17 — Updated docs/ops/runbooks/cf-access.md with new Failure Mode D

Impact

Users affected: none
User-visible symptoms: none (Raptor prod healthy throughout)
Data integrity: ok
Revenue / billing: ok
Internal: ops@ received repeated RaptorProdDown CRITICAL emails for ~24h; alert fatigue risk

What went well

Operator's initial triage correctly identified false-positive before escalating
Heroku logs and the operator's evidence clearly showed healthy dyno (no real outage)
SSM access to monitoring instance worked immediately; no SSH key / VPN required
Prometheus --web.enable-lifecycle allowed hot-reload without container restart
promtool check config validation caught any config errors before reload
Fix applied and verified within 7 minutes of agent dispatch

What didn't go well

Alert fired for ~24h before operator noticed — no alert deduplication / suppression
The monitoring probe was configured to hit the CF-fronted URL without CF Access credentials or a WAF skip rule, making it structurally broken from day one
CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN does not have Zone:WAF:Edit scope — agent could not create the durable WAF skip rule autonomously; that work is pending operator action
No runbook entry for "BFM blocks monitoring probe" existed; had to diagnose from first principles

Root cause analysis

Contributing factor 1: BFM evaluates before CF Access — Cloudflare Bot Fight Mode is enforced at the WAF layer, which runs before CF Access policies. The api.raxx.app root CF Access app has a decision=bypass policy covering all traffic, but this policy never evaluates because BFM returns 403 first. Any bare HTTP GET from a non-browser UA (including prom/blackbox-exporter) is classified as a bot and blocked.
Contributing factor 2: Probe configured without considering BFM — When the blackbox-http scrape job was set up in the cloud-init template, the probe target was https://api.raxx.app/health (the CF-fronted URL) with the plain http_2xx module. No CF Access service-token headers were added, and no WAF skip rule was documented as a prerequisite. The probe was structurally broken from the first scrape.
Contributing factor 3: No runbook entry for this failure mode — docs/ops/runbooks/cf-access.md had three failure modes documented, none covering monitoring probe 403s. The agent had no documented remediation to follow.
Contributing factor 4: Alert fatigue over 24h — The alert fired for ~24 hours before the operator noticed. With repeat_interval of 1h for critical alerts, this generated ~24 emails to ops@. No alert suppression or routing rules exist to catch repeated false-positives.

Detection

What alerted us: operator noticed RaptorProdDown alert and reported it
How long between cause and detection: ~24h (probe was broken from deployment on 2026-07-08)
How to detect faster next time: See action item #1 (Alertmanager inhibition rule for false-positives when Heroku dyno is confirmed healthy via a separate probe)

Resolution

What was changed: 1. Live: /opt/monitoring/prometheus/prometheus.yml — target changed from https://api.raxx.app/health to https://raxx-api-prod-a60a19e5efbf.herokuapp.com/health 2. Live: /opt/monitoring/prometheus/rules/alerts.yml — alert expression updated to match new instance label; description updated to clarify probe bypasses Cloudflare 3. IaC: terraform/ci-monitoring/templates/cloud-init.sh.tpl — both prometheus.yml and alerts.yml sections updated for future instance replacement 4. Docs: docs/ops/runbooks/cf-access.md — Failure Mode D added with WAF skip rule procedure
Validation: probe_success = 1 for new target; Alertmanager returns [] (zero active alerts)

Action items

#	Action	Owner	Due	Issue
1	Add WAF custom skip rule for SBFM on `api.raxx.app/health` using `CLOUDFLARE_ACCESS_MGMT_TOKEN` (needs `Zone:WAF:Edit` scope); then restore probe target to CF-fronted URL	operator	2026-07-21	file separately
2	Ensure `CLOUDFLARE_ACCESS_MGMT_TOKEN` in Infisical has `Zone:WAF:Edit` scope (or provision a dedicated monitoring-probe token); document scope in `docs/ops/runbooks/cf-access.md`	operator	2026-07-21	file separately
3	Add a new blackbox-http probe for the Heroku origin separately from the CF-fronted probe so monitoring covers both the dyno (Heroku URL) and the full stack (CF URL with WAF skip rule)	sre-agent	after #1	file separately
4	Add Alertmanager inhibition/silence rule that auto-suppresses `RaptorProdDown` if Heroku API confirms the dyno is running — reduces false-positive noise during CF layer incidents	sre-agent	2026-07-28	file separately

References

Runbook: docs/ops/runbooks/cf-access.md (Failure Mode D)
Runbook: docs/ops/runbooks/raptor.md
IaC: terraform/ci-monitoring/templates/cloud-init.sh.tpl
Memory: feedback_cf_access_does_not_bypass_bot_fight_mode.md
Memory: feedback_cf_access_service_token_needs_non_identity.md
Alertmanager fingerprint: 81cf144a04015499