RCA — ForgeConnectivityDown: monitoring-stack restart caused false-positive forge alert

Incident ID: 2026-07-14-forge-connectivity-down Date: 2026-07-14 Severity: SEV-3 Duration: ~10m (estimated 16:53–17:08 UTC, pending→firing→resolving) Blast radius: Internal only. No pipelines blocked. Forge auth to GitHub was never interrupted. Alert was a monitoring-plane false-positive caused by the probe target (blackbox-exporter) being restarted. Author: sre-agent

Summary

The ForgeConnectivityDown Prometheus alert fired on 2026-07-14 at approximately 17:03 UTC. The alert condition (probe_success{job="blackbox-forge"} == 0 for: 10m) was met because the blackbox-exporter container on the ci-monitoring EC2 (i-0a656c6631ddc45b3) was restarted twice by a prior SRE agent — at 16:53:49Z and 17:03:32Z — while it was injecting Cloudflare Access headers into the Raptor probe config. During the ~10-minute window between restarts, consecutive Prometheus scrapes (5m interval) found the forge probe failing. The Woodpecker server's actual forge connectivity to GitHub was unaffected throughout: GitHub webhooks continued to arrive and WP processed them normally. The alert cleared by ~17:10 UTC. No pipelines were lost.

Timeline (all times UTC)

16:30:39 — (Prior day context) First blackbox-exporter restart recorded in container logs (2026-07-13); agent was already making config changes to monitoring stack.
16:53:49 — SRE agent SSM command (command 3463848e-2b73-47b2-a96b-5c03079d94ba) restarted blackbox-exporter to inject CF Access headers into http_2xx module for Raptor probe. Agent verified Raptor probe returned 403 (CF Access creds issue), logged exit.
~16:53–17:03 — Prometheus scrape of blackbox-forge job (5m interval) returned probe_success=0 for two consecutive cycles. Alertmanager entered pending state.
17:03:32 — Agent's second SSM command restarted blackbox-exporter again (same inject pattern, same verification).
~17:03–17:08 — After second restart, forge probe recovered (probe_success=1). Alertmanager resolved alert.
~17:08 — ForgeConnectivityDown alert cleared (auto-resolved by Prometheus + Alertmanager).
17:23:48 — sre-agent investigated; confirmed probe_success=1 via direct blackbox probe query; Alertmanager shows []; WP server logs show no GitHub auth errors.
17:30 — Incident declared resolved; RCA drafted.

Impact

Users affected: none (internal tooling only)
User-visible symptoms: none
Pipelines blocked: none (WP forge auth was healthy throughout)
Data integrity: ok
Revenue / billing: ok
Alert sent to ops@raxx.app: yes (email via Alertmanager → Postmark)

What went well

The for: 10m threshold on ForgeConnectivityDown absorbed single container restarts — only a sustained failure over two scrape cycles triggered the alert. A shorter for duration would have produced more noise.
WP server logs were clean — no 401/403/rate-limit errors — making it fast to rule out actual forge auth failure.
The SSM log retained the full prior agent command payload, making root-cause attribution unambiguous.
Alert cleared automatically once the probe recovered; no manual Alertmanager silence needed.

What didn't go well

The ForgeConnectivityDown alert rule and the Raptor probe CF-Access injection workflow both involve the same blackbox-exporter container. The system allowed one remediation action (CF-Access header inject + restart) to trigger an unrelated alert (forge probe down).
The CF-Access injection script ran twice because its own verification step saw probe_success 0 for api.raxx.app/health (403 on Raptor probe), which prompted the agent to retry — but that 403 was unrelated to the forge probe, and the second restart extended the forge probe's outage window.
The forge_connectivity alert group (PR #4213) is only in the live alerts.yml on the monitoring instance. It was NOT backfilled into terraform/ci-monitoring/templates/cloud-init.sh.tpl. Next rebuild of the monitoring instance would lose this alert rule.
Similarly, the RaptorProdDown alert expression was changed live (from api.raxx.app/health to raxx-api-prod-a60a19e5efbf.herokuapp.com/health) but the Terraform template was not updated. Two live config drifts exist.

Root cause analysis

Contributing factor 1: Blackbox-exporter restart caused forge probe gap. The blackbox-forge Prometheus job scrapes api.github.com/zen via the blackbox-exporter container. Restarting the container creates a window during which the probe returns a connection error. With a 5m scrape interval, the probe failure from the 16:53 restart persisted in Prometheus until the next scrape at ~16:58. The system did not distinguish between "forge unreachable" and "monitoring probe container restarted."
Contributing factor 2: Second restart extended failure window past for: 10m. The prior agent restarted the container a second time at 17:03 (10 minutes after the first), timing the second restart close enough to the alert's for threshold that the consecutive probe failures crossed the firing threshold.
Contributing factor 3: IaC drift — live alert rules not in Terraform. The forge connectivity alert group was added post-deploy via SSM, not via a Terraform apply. The Terraform cloud-init template does not include this alert. Drift means: (a) the next instance rebuild loses the alert; (b) there is no PR history for the alert rule; (c) the RaptorProdDown probe target change is also lost on rebuild.

Detection

What alerted us: Alertmanager email alert to ops@raxx.app (ForgeConnectivityDown, severity=critical)
How long between cause and detection: ~10m (probe failed at 16:53, alert fired at ~17:03)
How to detect faster next time: The for: 10m threshold is appropriate for genuine forge connectivity loss. The issue is distinguishing a probe-container restart from genuine forge loss. Action item #2 adds a guard to the alert expression.

Resolution

Automatic: after the second blackbox-exporter restart at 17:03:32Z, the forge probe recovered to probe_success=1 on the next Prometheus scrape. Alertmanager resolved the alert automatically. No manual intervention was required.
Validation:
Alertmanager active alerts: []
Prometheus query probe_success{job="blackbox-forge"}: value 1 at 17:23:48Z
WP server logs: zero forge auth errors; GitHub webhooks received and processed normally
Direct probe query to blackbox at 17:23:48Z: 200 OK in 47ms, probe_success 1

Action items

#	Action	Owner	Due	Notes
1	Backfill `forge_connectivity` alert group into `terraform/ci-monitoring/templates/cloud-init.sh.tpl`	sre-agent	2026-07-21	Also backfill Raptor probe target change and `WoodpeckerNoWorkersWithBacklog` rule from `woodpecker-backlog-alert.yml`.
2	Guard `ForgeConnectivityDown` expr against blackbox-exporter restart noise	sre-agent	2026-07-21	Change expr to `probe_success{job="blackbox-forge"} == 0 and on() (up{job="blackbox-http"} == 1)`. Fires only if blackbox is itself reachable but forge probe fails — eliminates restarts from alert pool.
3	Add idempotency guard to CF-Access inject SSM script	sre-agent	2026-07-21	Check if `CF-Access-Client-Id` already present before restarting container. Skip restart if unchanged.
4	Add failure mode B-3 (ForgeConnectivityDown false-positive via monitoring-stack restart) to ci-woodpecker runbook	sre-agent	2026-07-14	Completed in this incident.
5	File Terraform PR to backfill all live monitoring-stack drift	operator review	2026-07-21	sre-agent opens PR; operator merges.

References

Runbook: docs/ops/runbooks/ci-woodpecker.md (Failure mode B-3, added by this RCA)
Prior genuine forge connectivity incident: docs/incidents/2026-07-09-wp-forge-context-deadline.md
Alert rule (live): forge_connectivity group in /opt/monitoring/prometheus/rules/alerts.yml on i-0a656c6631ddc45b3
SSM command that caused restart: command ID 3463848e-2b73-47b2-a96b-5c03079d94ba at 16:53 UTC, visible in /var/log/amazon/ssm/amazon-ssm-agent.log on i-0a656c6631ddc45b3