RCA — CI Alertmanager paged the operator during an authorized maintenance window, then kept paging for an unrelated stale target

Incident ID: 2026-08-07-ci-alertmanager-maintenance-noise Date: 2026-08-07 Severity: SEV-3 (non-blocking degradation, single-system alert-fidelity impact — no customer-facing outage) Duration: Notification-noise window ~06:20-06:43 UTC (the #4416 maintenance window itself); stale-target false-positive re-paging ~06:22-08:32 UTC (~2h10m) until caught and fixed during this response Blast radius: Operator notification channel only (email via Postmark → kris@moosequest.net). No CI or production functional impact — this incident is about alert-signal quality, not the underlying #4416 outage (which has its own RCA). Author: sre-agent

Summary

The authorized #4416 Woodpecker-server-replace maintenance window (~06:20-06:43 UTC, see docs/incidents/2026-08-07-woodpecker-server-replace-pip-cache-activation.md) fired real Alertmanager email notifications (NodeDown, WoodpeckerHealthzDown) to the operator for outage conditions that were expected and authorized. There was no mechanism to say "this specific, time-boxed page is expected" before starting the work. Separately, and more seriously, the replace left a stale Prometheus scrape target: the ci-monitoring Terraform root (a separate root from infra/ci) hardcodes the WP server's private IP as a static scrape target for node_exporter and WP's own /metrics endpoint. When the WP server was replaced, this target was never updated, so node-woodpecker kept trying to reach the destroyed instance's dead IP — connection refused — and the resulting critical NodeDown alert re-fired every repeat_interval (1h) for approximately two hours after the maintenance window closed, with no maintenance excuse at all. Both problems were fixed during this response: a new CI-scoped Alertmanager silence tool (scripts/ops/ci-maintenance-silence.sh) now lets planned CI maintenance suppress exactly the alerts it should, and the stale target was fixed forward (Prometheus config updated + reloaded, node_exporter reinstalled on the new WP server instance) and the underlying Terraform defaults corrected so future replacements start from the right baseline.

Timeline (all times UTC)

• 06:19:39-06:42:46 — #4416 Woodpecker server replace executes (see that incident's own RCA for full detail). Old instance (i-082ee835595d90ae0) destroyed ~06:20:54; ALB target group and agent launch template out of sync until ~06:36-06:42.
• ~06:22:45 — Critical NodeDown{instance="ci-woodpecker-server"} fires (node_exporter target unreachable — old instance destroyed). Alertmanager routes to email-ops per group_wait: 30s. This is the alert that reached the operator during the maintenance window.
• ~06:22-06:24 (est.) — Critical WoodpeckerHealthzDown also fires (ALB target group had zero healthy targets from the destroy until the target-group-attachment fix-forward around 06:36-06:39). Second email to the operator during the same window.
• 06:42:46 — #4416's golden-path validation completes; WP server, ALB routing, and agent fleet all confirmed healthy.
• 06:42:46 onward — WoodpeckerHealthzDown and any transient WoodpeckerNoWorkersWithBacklog risk resolve as the golden path recovers. NodeDown{instance="ci-woodpecker-server"} does NOT resolve — the Prometheus scrape target still points at the destroyed instance's IP (10.42.10.143); the replace never touched this separate Terraform root's config.
• Operator feedback received 2026-08-07 (this session): "we don't want alarms if CI is paused or away — I'm getting notifications" — reporting the paging from the #4416 window.
• ~08:22-08:25 — Investigation for this RCA confirms NodeDown is still active in Alertmanager (GET /api/v2/alerts), startsAt = 06:22:45, continuously firing since the replace — roughly 2 hours of unremediated, genuinely-false critical paging, on top of the authorized-window pages. MemoryPressureHigh{instance="ci-monitoring"} also active (startsAt=07:19:15) — separate, unrelated, real resource-pressure signal (see Action items).
• 08:29 — Confirmed root cause: curl http://localhost:9090/api/v1/targets showed node-woodpecker / woodpecker jobs both scraping 10.42.10.143 (destroyed instance); live WP server private IP is 10.42.10.157. terraform/ci-monitoring/variables.tf wp_server_private_ip default was still 10.42.10.143.
• 08:30-08:32 — Built and verified scripts/ops/ci-maintenance-silence.sh end-to-end against live Alertmanager: created a CI-scoped test silence, confirmed it suppressed the currently-firing NodeDown and MemoryPressureHigh alerts (silencedBy populated, state: suppressed), expired it, confirmed both returned to active.
• 08:32 — Fix-forward: sed-replaced 10.42.10.143 → 10.42.10.157 in the live /opt/monitoring/prometheus/prometheus.yml (via SSM Run Command), docker compose kill -s HUP prometheus to reload without restart.
• 08:32 — Targets still down — connection refused, not resolved by the IP fix alone. Root cause #2: node_exporter was never installed on the new WP server instance (i-0563f92028b656333) — it's a fresh EC2 from the replace, not carrying over the old instance's manually-installed node_exporter. Ran the documented node_exporter_install_command (v1.8.2, arm64) against the new instance via SSM.
• 08:33 — Confirmed node_exporter active, curl localhost:9100/metrics returns 200 on the new WP instance.
• 08:33 (after next scrape) — node-woodpecker target up in Prometheus; GET /api/v2/alerts on Alertmanager shows NodeDown no longer present (only the unrelated MemoryPressureHigh remains active).
• 08:33-08:40 — Updated terraform/ci-monitoring/variables.tf defaults (wp_server_private_ip → 10.42.10.157, new wp_server_instance_id var → i-0563f92028b656333) and outputs.tf (de-hardcoded the node_exporter_install_command output's instance ID). terraform plan confirmed zero infrastructure changes (the box's own lifecycle { ignore_changes = [user_data_base64] } makes this purely a docs/state-level correction); terraform apply completed, 0 added, 0 changed, 0 destroyed.
• Resolved.

Impact

• Users affected: none (internal operator notification channel only; no customer-facing or CI-functional impact).
• User-visible symptoms: none.
• Data integrity: ok.
• Revenue / billing: ok.
• Operator impact: multiple unwanted email notifications during an authorized maintenance window, followed by ~2 hours of a genuinely false critical page (NodeDown) re-arriving hourly with no way to distinguish it from a real new problem — this is the alert-fatigue risk the SRE playbook explicitly warns against ("every alert that can't be acted on by this agent is toil... silence or auto-resolve noise before it trains the on-call to ignore alerts").

What went well

• The underlying #4416 replace itself was correctly pre-flighted (zero pipelines running before the apply) and fully validated against the golden path, not just instance-running status — see that incident's own RCA.
• The alert rules that fired (NodeDown, WoodpeckerHealthzDown) were correct in the sense that the conditions they describe were true during the window — the gap was suppression capability, not false alarm logic, for the authorized-window portion of this incident.
• Alertmanager's existing routing (group_by: [alertname, severity, instance], group_wait: 30s, group_interval: 5m, repeat_interval: 1h critical / 4h default) is already reasonable and was not the actual root cause — investigated and confirmed rather than assumed (see "Routing posture" analysis below and in docs/ops/runbooks/ci-monitoring.md §Maintenance silences).
• The stale-target root cause was found and fixed with hard evidence at every step (raw lastError from Prometheus's targets API showing connection refused, not guessed at) rather than assumed from the symptom alone.
• The maintenance-silence tool was verified end-to-end against live Alertmanager before being declared done — created a real test silence, confirmed it actually suppressed the real, currently-firing alerts (not synthetic ones), then expired it and confirmed alerts re-armed.

What didn't go well

• No mechanism existed to tell Alertmanager "this specific window is expected downtime" — the #4416 runbook procedure had seven validation steps but no step 0 (silence) or final step (expire + verify re-armed).
• The WP-server-replace procedure's "dependent resources" enumeration (ALB target group attachment, agent launch template — both added to the runbook after the #4416 RCA) did not include a third dependent that lives in a wholly separate Terraform root: terraform/ci-monitoring's hardcoded wp_server_private_ip/scrape targets. Because this dependency crosses Terraform root boundaries, terraform plan in infra/ci (where the replace happens) can never surface it — there is no automated linkage between the two roots at all, only a comment and a runbook step a human has to remember.
• node_exporter on the WP server was originally a one-time, manual post-apply operator action (documented in ci-monitoring.md §Post-apply configuration, "Step 1"), never captured as something the WP server's own boot template (infra/ci/server.tf) provisions itself. Every future WP server replace will silently drop node_exporter again unless this runbook step is followed exactly, every time — the system has no self-check that would catch "this box came up without node_exporter."
• The NodeDown alert firing continuously for ~2 hours produced no escalation beyond the initial page and the 1h repeat — nothing flagged "this specific alert has been active continuously since a known maintenance window started and never resolved," which is a stronger, more specific signal than a generic repeat-interval reminder.

Root cause analysis

• Contributing factor 1 — no maintenance-silence capability existed: Alertmanager natively supports silences for exactly this purpose, but nothing in this repo wired up a CI-scoped way to create one. The standalone-replace runbook procedure (docs/ops/runbooks/ci-build-cache.md) was written and iterated on twice already (2026-08-07, two separate PRs) without this gap being caught, because the runbook's focus was "make the replace itself work," not "make the replace itself quiet."
• Contributing factor 2 — dependent-resource enumeration was scoped to a single Terraform root: the #4416 RCA's own "what didn't go well" section explicitly calls out that -target scoping missed direct dependents within infra/ci (ALB target group, agent launch template). It did not — and structurally could not, from a single-root terraform plan — surface the terraform/ci-monitoring root's hardcoded reference to the same instance's IP. This is the same class of gap (identity/IP- based dependents instead of stable references) recurring across a root boundary that no tooling currently spans.
• Contributing factor 3 — no automated check for "a critical alert has been active longer than any plausible maintenance window": Alertmanager alone cannot distinguish "still legitimately down" from "should have resolved by now" — that requires either a maximum-alert-age monitor or a human noticing. Neither existed; detection here was incidental (an operator complaint about paging, not the stale alert itself surfacing).

Detection

• What alerted us: the operator's direct feedback about receiving notifications during the #4416 window. The stale-target false-positive (the more serious of the two problems) was NOT independently detected — it was found while investigating the operator's complaint, by directly querying Alertmanager's /api/v2/alerts for the current state.
• How long between cause and detection: ~2h10m for the stale-target issue (06:22 fire → ~08:32 fix), all of it silent except for the hourly re-notification itself.
• How to detect faster next time: the maintenance-silence discipline (step 0 / last step in the runbook) makes the authorized portion of this self-documenting — a silence with a start/end time is itself a durable, queryable record of "this was expected, for this long." For the stale-target class specifically, action item #3 below proposes an automated check.

Resolution

• What was changed: 1. New tool: scripts/ops/ci-maintenance-silence.sh — CI-scoped Alertmanager silence create/list/expire/expire-all, wrapping amtool over SSM Run Command. Closed allowlist of CI-only alertnames; does not touch RaptorProdDown or any TLSCertExpiry*/TLSOriginProbeDown alert. 2. Fix-forward on the live ci-monitoring box: prometheus.yml node-woodpecker/woodpecker scrape targets updated from 10.42.10.143 to 10.42.10.157 (the live WP server's current private IP); docker compose kill -s HUP prometheus to reload. 3. Fix-forward on the live WP server instance (i-0563f92028b656333): installed node_exporter v1.8.2 (arm64) via the existing documented SSM Run Command, which had never been run against this (replaced) instance. 4. terraform/ci-monitoring/variables.tf: corrected wp_server_private_ip default to 10.42.10.157; added new wp_server_instance_id variable (default i-0563f92028b656333) with staleness-caveat documentation in both variables' descriptions. 5. terraform/ci-monitoring/outputs.tf: node_exporter_install_command now references var.wp_server_instance_id instead of a hardcoded, already-stale instance ID. 6. terraform apply in terraform/ci-monitoring/ — 0 added, 0 changed, 0 destroyed (confirmed inert to live infrastructure; only the docs-only output value and variable defaults recorded in state). 7. docs/ops/runbooks/ci-build-cache.md: added step 0 (create silence) and step 8 (expire silence + verify re-armed) to the standalone- replace procedure; added the ci-monitoring Prometheus-target update as a fourth documented dependent resource in step 5; added Failure mode H (stale target after replace). 8. docs/ops/runbooks/ci-monitoring.md: new §Maintenance silences section (tool usage, CI-scoped allowlist rationale, and an explicit routing-posture analysis concluding the existing group_wait/ group_interval/repeat_interval config is already reasonable and should NOT be converted to a digest — see that section for the full argument); Failure mode J (cross-references build-cache runbook); punch-list entries for the residual staleness risk and the unrelated MemoryPressureHigh finding.
• Validation:
• scripts/ops/ci-maintenance-silence.sh create/list/expire/expire-all each exercised against the live Alertmanager instance. create produced a real silence ID; list showed it with the correct CI-scoped matcher; a direct GET /api/v2/alerts query during the silence showed both currently-firing CI alerts (NodeDown, MemoryPressureHigh) with status.state: "suppressed" and silencedBy containing the new silence's ID; expire returned both alerts to state: "active"; expire-all verified against a second test silence.
• Prometheus GET /api/v1/targets: node-woodpecker job transitioned from health: "down" (lastError: "connect: connection refused") to health: "up" after the fix-forward.
• Alertmanager GET /api/v2/alerts: NodeDown no longer present after the fix (only the separate, genuine MemoryPressureHigh remains active).
• terraform plan/terraform validate/terraform fmt -check all clean against the corrected variables.tf/outputs.tf; terraform apply confirmed 0 added, 0 changed, 0 destroyed.

Action items

References

• Runbook: docs/ops/runbooks/ci-monitoring.md (§Maintenance silences, Failure mode J)
• Runbook: docs/ops/runbooks/ci-build-cache.md (standalone-replace procedure steps 0/5/8, Failure mode H)
• Script: scripts/ops/ci-maintenance-silence.sh
• Related incident (the maintenance window itself): docs/incidents/2026-08-07-woodpecker-server-replace-pip-cache-activation.md
• Related incident (silent-outage precedent for why CI alerts must not be digested away): docs/incidents/2026-07-13-ci-spot-wedge-silent-outage.md
• CI notification posture (governs a different surface — GH Actions/cron Slack digest, not this Alertmanager): docs/architecture/ci-notification-posture.md
• Terraform: terraform/ci-monitoring/variables.tf, terraform/ci-monitoring/outputs.tf
• Alertmanager silence docs: https://prometheus.io/docs/alerting/latest/silence/