ADR-0141 — On-Call Claude Agent: Alertmanager-Triggered Triage with Slack Escalation

Status: Accepted Date: 2026-07-14 UTC Deciders: Kristerpher (operator); software-architect (design) Scope: terraform/ci-monitoring/ stack (Alertmanager), new on-call-agent Docker container, Slack workspace DM channel D0AJ7K184TV, Infisical/SSM secret stores Refs: ADR-0134 (CI EC2 runners), ADR-0135 (WP migration), ADR-0136 (WP HA), ADR-0137 (WP config service), docs/ops/runbooks/ci-monitoring.md, docs/ops/runbooks/ci-woodpecker.md, .claude/agents/sre-agent.md

1. Context

The observability stack (terraform/ci-monitoring/) runs Prometheus, Alertmanager, Grafana, and blackbox_exporter on a dedicated EC2 t4g.small. Today, Alertmanager routes all firing alerts to Postmark → kris@moosequest.net. The 2026-07-13 spot-fleet wedge incident (#4039) exposed the core problem: an email in an inbox at 00:41 UTC is easy to miss; the detection gap was 5 hours.

The desired outcome is: when any alert fires, Kristerpher receives a Slack DM within seconds, and within 30 seconds receives a second in-thread message containing triage context — what is broken, why (correlated from Prometheus metrics and the relevant runbook), and what to do next. No autonomous action is taken in v1. The human decides; the agent provides context.

This ADR makes three binding decisions:

1. Runtime model — how the agent actually executes when an alert fires.
2. Dead-man's switch — how the operator is paged even when the agent is down.
3. Phase gating — what ships in each phase and when auto-remediation is allowed.

2. Invariants

The following constraints are non-negotiable before any design choice:

• No stored credentials. The on-call agent receives no credentials it can replay against financial or user-identity systems. Its toolset is read-only: Prometheus query API, runbook files (read), Heroku log drain (read). It holds no write tokens to any system that manages money, auth, or user data.
• Agent is additive, never the sole alert path. Alertmanager MUST retain a direct Slack route that fires for every alert, independent of the Claude layer. If the agent container crashes, the operator still gets a Slack notification.
• v1 is advisory only. The agent triages and explains. It does not restart services, scale fleets, change config, merge PRs, or issue commands. Auto-remediation is a fenced v2 feature behind an explicit allow-list.
• Least-privilege identity. Per ADR-0131 and feedback_handoff_agents_least_privilege, the agent is provisioned a scoped read-only identity at dispatch time. No destructive prod credentials. No Heroku admin token. No GitHub write token.
• Audit every page. Every escalation the agent posts to Slack is written to a structured audit log. The log records: alert name, severity, timestamp, Prometheus values queried, runbook consulted, and the text sent to Slack.
• Kill-switch. The on-call agent container can be stopped without stopping Alertmanager or the direct Slack route. Stopping it degrades to Phase 1 behavior (raw alert in Slack, no enrichment). The kill-switch is docker stop on-call-agent on the monitoring EC2 box via SSM Session Manager.

3. Decision

Chosen runtime: Alertmanager webhook → always-on Python receiver (6th Docker container on ci-monitoring EC2) → Claude Agent SDK triage loop → Slack.

The receiver runs as on-call-agent in the existing docker-compose.yml on the ci-monitoring EC2 instance alongside Prometheus, Alertmanager, and Grafana. It listens on localhost:9099 for Alertmanager webhook POSTs. On receipt it: 1. Stores the Slack thread state (ts) from the direct-page message. 2. Spawns a Claude Agent SDK session with scoped tools and posts the triage in-thread (~15–30 seconds after the direct-page message fires).

Alertmanager is configured with two receivers in parallel: - slack-direct — native Alertmanager Slack receiver; fires first (~2 seconds); the guaranteed floor. - claude-enricher — HTTP webhook to http://on-call-agent:9099/webhook (Docker bridge network); fires in parallel; provides AI triage in-thread.

If the on-call-agent container is down, Alertmanager's webhook POST times out and Alertmanager logs the error, but slack-direct already delivered. The operator is paged regardless.

Why this model over the alternatives:

• Option B (native Slack + async enrichment triggered externally): The async enrichment still needs a trigger — either a second webhook (same architecture) or polling. Polling is not event-driven and introduces up-to-60-second detection latency. Option B collapses into this architecture when implemented properly.

• Option C (headless Claude Code polling Alertmanager /api/v2/alerts): Polling misses flash events, adds up to 60-second detection lag, and the polling loop can fail silently without adding another watchdog. Event-driven webhook has cleaner failure semantics: Alertmanager logs each failed POST; polling failure is invisible.

Language: Tier 2 — Python. See §Language choice rationale.

4. Language Choice Rationale

Service: on-call-agent (Python FastAPI webhook receiver + Claude Agent SDK orchestrator)

Language tier: - [x] Tier 2 — Python

Rationale for this classification: This service is an orchestration layer. It receives one HTTP POST per alert firing, queries Prometheus, reads runbook files, and calls the Claude API. None of the Tier 1 criteria from docs/architecture/language-tier-policy.md apply: - Not on a security-critical hot path (C-1: no passkeys, session tokens, or crypto). - No sub-5ms latency requirement; "enrichment within 60 seconds" is the SLO (C-2). - Throughput is at most one webhook per alert event (C-3). - RSS will be negligible for a small FastAPI process (C-4).

Python is correct for v1 and well beyond this scale.

API contract portability (Tier 2): Yes. The receiver exposes a single POST /webhook endpoint accepting the standard Alertmanager webhook JSON payload. Slack and Prometheus clients are isolated behind thin adapter interfaces. A future port would implement the same HTTP contract without redesign.

5. Architecture

5.1 Component Map

┌─────────────────────────────────────────────────────────────────┐
│  EC2 ci-monitoring (t4g.small, us-east-2a, private subnet)     │
│                                                                  │
│  ┌────────────┐  ┌──────────────┐  ┌──────────────┐            │
│  │ Prometheus │  │  Grafana     │  │  blackbox    │            │
│  │ :9090      │  │  :3000       │  │  :9115       │            │
│  └─────┬──────┘  └──────────────┘  └──────────────┘            │
│        │ rule eval                                               │
│        ▼                                                         │
│  ┌──────────────────────────────┐                               │
│  │  Alertmanager :9093          │                               │
│  │  receiver: slack-direct  ───────────────────────────────▶ Slack API
│  │  receiver: claude-enricher ─────┐                            │
│  └──────────────────────────────┘  │                            │
│                                     │                            │
│  ┌──────────────────────────────────▼──────────────────────┐   │
│  │  on-call-agent :9099  (new container, Tier 2 Python)    │   │
│  │                                                          │   │
│  │  POST /webhook                                           │   │
│  │    1. ack to Alertmanager (200 OK, < 50ms)               │   │
│  │    2. enqueue triage task (async)                        │   │
│  │                                                          │   │
│  │  Triage worker (async):                                  │   │
│  │    a. classify severity + derive runbook path            │   │
│  │    b. query Prometheus /api/v1/query (localhost:9090)    │   │
│  │    c. read runbook from /runbooks/ (read-only volume)    │   │
│  │    d. invoke Claude Agent SDK session (2-tool budget)    │──▶ Claude API
│  │    e. POST triage to Slack in-thread                     │──▶ Slack API
│  │    f. write structured audit log → CloudWatch            │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

External (independent liveness):
  GHA watchdog workflow → ci.moosequest.net/api/version probe
  (outside VPC; unaffected by EC2-internal failures)

5.2 Sequence Diagram (CRIT alert, happy path)

sequenceDiagram
    participant P  as Prometheus
    participant AM as Alertmanager
    participant SD as slack-direct receiver
    participant OC as on-call-agent
    participant CL as Claude API
    participant SL as Slack (DM D0AJ7K184TV)

    P  ->> AM: alert rule fires (t=0)
    AM -->> SD: POST (native Slack receiver, parallel)
    AM -->> OC: POST /webhook (claude-enricher, parallel)
    SD ->> SL: raw DM page (t ≈ 2s)
    SL -->> SD: response includes Slack ts
    OC ->> AM: 200 OK (ack, synchronous, < 50ms)

    Note over OC: triage task runs async
    OC ->> P:  GET /api/v1/query?query=<expr>
    P  -->> OC: current metric values
    OC ->> OC: read runbook section from /runbooks/ mount
    OC ->> CL: Claude SDK session — triage prompt + tools
    CL ->> P:  tool: prometheus_query(range, 15m)
    P  -->> CL: metric history
    CL -->> OC: triage text + recommended action
    OC ->> SL: thread reply on DM (t ≈ 30s)
    OC ->> OC: write audit log (CloudWatch /ci/on-call-agent)

5.3 On-Call Agent Tool Set (v1 — read-only)

No write access to Prometheus, Alertmanager, Heroku config, AWS, GitHub, or any data store holding user data or financial state.

The Slack bot token is scoped to chat:write in the MooseQuest workspace only.

6. The Triage Loop

On receiving a webhook POST, the agent executes the following for each alert group:

Step 1 — Classify: - Parse alert.labels.severity → determines Slack routing - Parse alert.labels.alertname → key for runbook lookup - Parse alert.annotations.runbook → file path in docs/ops/runbooks/ - Routing: severity=critical → DM D0AJ7K184TV; severity=warning → #ops-alerts (OQ-1)

Step 2 — Fetch: - Query Prometheus for the raw expression from the alert rule at current time - Query a 15-minute range at 30s intervals to show trend direction - If alert.labels.instance exists, also fetch host-level metrics (node_cpu, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes)

Step 3 — Read: - Load the runbook file identified in annotations.runbook - Extract the "Known failure modes" section matching alertname

Step 4 — Reason (Claude Agent SDK):

System prompt (condensed):

"You are an on-call SRE. You have received a Prometheus alert. Available read-only tools: prometheus_query, read_runbook, heroku_log_tail. v1 constraint: advisory only — do not issue commands or take action. Produce a triage message under 400 words: severity, what fired, the most likely cause with confidence level, the runbook section to follow, and the single most important action for the operator."

The SDK session runs with a hard 2-tool-call budget per alert (prevents runaway cost) and a 45-second wall-clock timeout. If the session exceeds the timeout, the agent posts whatever it has assembled so far with a [triage incomplete — timeout] suffix.

Step 5 — Escalate: - Format Slack message (see §7) - POST to Slack as thread reply to the direct-page message - Write structured audit log entry (see §9.3)

Step 6 — Cool-down: - The same alertname within 30 minutes skips the enrichment step (de-dup window). Alertmanager's own repeat_interval governs re-paging; the cool-down only prevents duplicate enrichment posts for the same firing event.

7. Slack Escalation UX

7.1 Direct-Page Message (Phase 1 — slack-direct receiver)

Posted by native Alertmanager Slack receiver. No AI. Fires in ~2 seconds. Always fires regardless of agent state.

[CRIT] WoodpeckerNoWorkersWithBacklog
Fired: 2026-07-14 03:41 UTC  |  Duration firing: 10m+
Value: 3 pending pipelines, 0 workers
Runbook: docs/ops/runbooks/ci-monitoring.md
Grafana: https://console.moosequest.net/grafana/d/ci-woodpecker

CRIT: posted to DM D0AJ7K184TV. WARNING: posted to #ops-alerts channel (OQ-1 — channel ID to confirm).

7.2 In-Thread Triage (Phase 2 — on-call-agent enrichment)

Posted in-thread under the direct-page message. Target: ~30 seconds after alert fires.

Claude triage [advisory — no action taken]:

Likely cause: Spot fleet wedge — ASG desired >= 1, InService = 0.
Confidence: HIGH (pattern matches #4039, 2026-07-13).

Prometheus at alert time:
  woodpecker_pending_pipelines = 3
  woodpecker_workers = 0
  15m trend: workers dropped to 0 at 03:31 UTC; pending grew 1 → 3

Runbook: ci-monitoring.md § WoodpeckerNoWorkersWithBacklog
  1. Check ASG desired vs InService (AWS console or aws autoscaling describe-*)
  2. Check scale_arbiter CloudWatch logs (/aws/lambda/ci-scale-arbiter)
     for WEDGE DETECTED / WEDGE REFRESH log lines
  3. If arbiter stuck: set ASG desired=0 then desired=1 to reset

Single most important action: verify ASG InService count first.

7.3 Thread State Handoff

The native Alertmanager slack-direct receiver posts the raw DM and obtains the Slack ts (thread timestamp). The on-call-agent needs this ts to reply in-thread rather than opening a new message. Mechanism: slack-direct is replaced with a thin Alertmanager webhook that posts to on-call-agent's POST /slack-page endpoint, which (a) forwards to Slack, (b) stores the returned ts in a shared volume at /tmp/oncall-state/<alertname>-<fingerprint>.json, and (c) the enrichment task reads this file to post in-thread. This replaces the Alertmanager native Slack receiver with a webhook. Benefit: single path from Alertmanager → on-call-agent, which serializes the page and enrichment on one service. Fallback: if the on-call-agent is down, the email-ops receiver (still configured with continue: true) fires independently.

7.4 Phase 3 — Interactive Buttons (design-only in this ADR)

Phase 3 adds action buttons to the triage message. Designed now; built in Phase 3.

Phase 3 requires a Slack app with chat:write and actions scope, and an interaction endpoint reachable from Slack's servers (OQ-3).

Silence 4h is the only write call allowed in Phase 3; it calls Alertmanager's silence API on localhost. It does not change infrastructure state.

8. Dead-Man's Switch / Failover

Critical invariant: the Claude layer is enrichment on top of a guaranteed path. It must never be the sole path to the operator.

8.1 Guaranteed Alert Paths (always active)

Alertmanager
  ├─ receiver: slack-page (on-call-agent /slack-page, posts raw DM + stores ts)
  │    → if on-call-agent is DOWN: falls through to email-ops
  └─ receiver: email-ops  (Postmark → kris@moosequest.net)
       → configured with continue: true; fires for every alert
       → backup if Slack API is also down

Both paths are configured in Alertmanager with continue: true so the claude enrichment (/webhook) fires as a third receiver, not instead of the others.

8.2 Watchdog for the On-Call Agent

A Prometheus alert monitors the agent health endpoint:

# prometheus/rules/alerts.yml — added in Phase 2 sub-card
- alert: OnCallAgentDown
  expr: probe_success{job="blackbox-http",instance="http://on-call-agent:9099/healthz"} == 0
  for: 5m
  labels:
    severity: warning
    team: ops
  annotations:
    summary: "On-call agent enrichment service is unreachable"
    description: >-
      The on-call-agent container is not responding. Direct Slack + email
      paging remain active. Restart: SSM Session Manager to ci-monitoring,
      then: docker compose restart on-call-agent
    runbook: "docs/ops/runbooks/on-call-agent.md"

OnCallAgentDown routes via the email-ops receiver (no enrichment — the enrichment service is what's down). The operator gets an email telling them the enrichment agent is down. Once Phase 1 Slack is live, this also gets a direct Slack post.

8.3 Failure Mode Table

The external GHA watchdog (.github/workflows/woodpecker-healthcheck.yml) is independent — it probes from outside the VPC and is unaffected by EC2-internal failures.

9. Security + Least Privilege

9.1 Credentials Held by on-call-agent

No credentials that can mutate infrastructure, billing, user identity, or financial state. No GitHub token. No AWS write IAM. No Alertmanager write API access in v1/v2.

9.2 Network Exposure

The on-call-agent binds to 0.0.0.0:9099 on the Docker monitoring bridge network. Alertmanager reaches it as http://on-call-agent:9099. The monitoring box's security group has no inbound rule for port 9099; the container is not internet-accessible. No new security group rules required in Phase 2.

9.3 Audit Trail

Every triage event writes a structured log line:

{
  "event": "oncall_triage",
  "alert_name": "WoodpeckerNoWorkersWithBacklog",
  "severity": "critical",
  "fingerprint": "abc123",
  "fired_at": "2026-07-14T03:41:00Z",
  "triaged_at": "2026-07-14T03:41:28Z",
  "prometheus_queries": ["woodpecker_pending_pipelines", "woodpecker_workers"],
  "runbook_read": "docs/ops/runbooks/ci-monitoring.md",
  "claude_tokens_used": 1842,
  "claude_tool_calls": 2,
  "slack_channel": "D0AJ7K184TV",
  "slack_thread_ts": "1721234567.123456",
  "outcome": "posted_triage"
}

Logs shipped to CloudWatch (/ci/on-call-agent) via Docker awslogs log driver. Retention: 90 days. The ci-monitoring EC2 IAM role already has logs:CreateLogStream and logs:PutLogEvents for its existing log groups; extend the policy to include /ci/on-call-agent in the Phase 2 sub-card.

9.4 GDPR / PII

Alert payloads contain no user PII. They contain infrastructure metric values, hostnames (EC2 private IPs), and pipeline IDs. None are personal data under GDPR. No DSR mechanism required for this service. No retention limit beyond operational log policy (90 days).

10. Phased Build Plan

Each phase is independently shippable as its own PR to develop.

Phase 1 — Guaranteed Slack Route (ships with HA work, no AI)

Scope: Add native Alertmanager Slack receiver and SSM secret for the Slack bot token. No on-call-agent container. No Claude API key.

Changes (terraform/ci-monitoring only): - cloud-init.sh.tpl — add slack-direct receiver to Alertmanager config. - ssm.tf — add slack_bot_token SSM parameter (SecureString, operator-seeded). - docs/ops/runbooks/ci-monitoring.md — add Slack alert format and mute procedure.

Outcome: Operator receives Slack DM within 2 seconds of any CRIT alert firing. Eliminates the email-only detection gap that caused the 5-hour miss on 2026-07-13. Email route (email-ops) remains active in parallel.

Phase 2 — Claude Enrichment (new on-call-agent container)

Scope: Python service, Claude triage loop, in-thread enrichment post, watchdog alert.

Changes: - services/on-call-agent/ — FastAPI app, Dockerfile, requirements.txt. - cloud-init.sh.tpl — add on-call-agent to docker-compose.yml; add claude-enricher Alertmanager receiver; mount docs/ops/runbooks/ read-only. - ssm.tf — add claude_api_key SSM parameter. - terraform/ci-monitoring/iam.tf — extend EC2 role to allow logs:* on /ci/on-call-agent CloudWatch log group. - prometheus/rules/alerts.yml — add OnCallAgentDown alert. - docs/ops/runbooks/on-call-agent.md — new runbook for the service itself.

Cost estimate: ~$0.003–0.01 per alert page (Claude Sonnet: ~2K input + 400 output tokens). At 10 CRIT alerts/month: < $0.10/month.

Outcome: Enriched in-thread triage within 30 seconds. Estimated MTTR reduction: 30–60 minutes per CRIT incident (operator no longer manually correlates Prometheus with runbooks during wakeup).

Phase 3 — Interactive Ack / Silence

Scope: Slack app with Ack and Silence 4h buttons. No infra mutation beyond Alertmanager silence API (localhost).

Changes: - Slack app registration in MooseQuest workspace (operator action — needs button-capable bot token). - services/on-call-agent/ — add /slack/actions interaction handler. - Expose /slack/actions externally (Cloudflare Tunnel preferred; OQ-3).

Gate: Phase 2 in production for 14+ days with zero false-positive enrichment posts.

Phase 4 — Documented-Fix Auto-Remediation (fenced)

Status: Design-only. Not scheduled until: 1. Phase 3 has been live 30+ days. 2. Every candidate remediation has a documented runbook section with verified manual steps. 3. Operator has explicitly approved each allow-listed remediation in a GitHub issue comment. 4. Every remediation is idempotent with a documented rollback.

Candidate allow-list entries: - WoodpeckerNoWorkersWithBacklog → set ASG desired=0 then desired=1 (idempotent; rollback = manual ASG resize; documented in ci-monitoring.md) - OnCallAgentDown → docker restart on-call-agent (self-healing; idempotent)

Not on allow-list (Phase 4 or never): - Any remediation that changes Heroku app config or environment variables. - Any remediation that modifies Prometheus rules or Alertmanager config. - Any remediation that deletes or modifies data.

11. Cost + Outcomes

Phase 1 alone eliminates the primary detection gap. Phase 2 eliminates the manual triage gap. Both are additive to the existing email route.

Alternatives Considered

Alternative A — Lambda triggered by Alertmanager webhook via API Gateway

Would require internet-exposing the Alertmanager webhook (the monitoring EC2 is in a private subnet with no public IP). Also adds Lambda cold-start latency and API Gateway cost and configuration overhead. Revisit only if the monitoring EC2 is decommissioned in favor of a managed observability stack.

Alternative B — Claude Code headless polling Alertmanager /api/v2/alerts

Polling every 60 seconds introduces up to 60-second detection lag and can fail silently without adding another watchdog (the polling process dying is invisible without its own health check). Event-driven webhook has cleaner semantics: Alertmanager logs each failed POST; polling failure is invisible.

Alternative C — Alertmanager native Slack + separate async enrichment process

This is the same architecture as the chosen model when implemented correctly. The "separate async enrichment process" still needs a trigger — webhook (this ADR's approach) or poll (Alternative B). Not a distinct alternative.

Consequences

Positive

• Eliminates the 5-hour detection gap pattern from #4039.
• Operator receives correlated context (not just a raw page) within 30 seconds.
• Dead-man's switch means agent failure never means silent alerts.
• Phase 1 ships with zero new containers or AI cost.
• v1 advisory-only posture means agent error blast radius is zero.

Negative / risks

• on-call-agent is an additional service to monitor. Mitigated by OnCallAgentDown alert and Phase 1 email fallback.
• Claude API is an external dependency; outage silently skips enrichment. Mitigated by direct Slack page firing first.
• Agent triage may be incorrect (wrong runbook match, stale metric interpretation). Mitigated by advisory-only framing — operator decides all actions.
• Slack bot token is a new credential surface. Mitigated by chat:write-only scope; rotation does not require redeploy.

Neutral

• services/on-call-agent/ is a new top-level directory consistent with the future services pattern (alongside Velvet, Queue scaffold).
• The runbook volume mount gives the agent read access to all runbooks in docs/ops/runbooks/. This is intentional.

Security / GDPR Checklist

• PII collected: None. Alert payloads are infrastructure metrics (hostnames, metric values, timestamps). No user data.
• Retention period: Audit log 90 days (CloudWatch). Slack messages per workspace policy (operator-controlled).
• Deletion on DSR: Not applicable — no user PII.
• Audit trail: Every triage event logged to CloudWatch /ci/on-call-agent as structured JSON. Includes alert name, severity, Prometheus queries, runbook read, Claude tokens used, Slack thread ID, outcome. Retained 90 days.
• Stored credentials: Slack bot token and Claude API key in SSM at /ci/monitoring/slack_bot_token and /ci/monitoring/claude_api_key. Neither can impersonate a user, access financial flows, or replay auth material.
• Breach notification path: If Slack bot token is compromised, attacker can post Slack messages (chat spam, no data exfil). Rotation: revoke in Slack app settings, update SSM parameter, docker restart on-call-agent. No user data exposure.
• Secrets location + rotation: SSM /ci/monitoring/ prefix. All rotatable without container rebuild.
• Kill-switch: docker stop on-call-agent via SSM Session Manager on ci-monitoring EC2. Degrades immediately to Phase 1 (email). No downstream system impact.

Open Questions

OQ-1 and OQ-2 must be resolved before SC-2 can be claimed. OQ-3 only blocks SC-3.

Revisit When

• Monitoring EC2 is replaced with a managed observability stack (Grafana Cloud, Datadog): re-evaluate; CloudWatch Alarm + Lambda may be more natural.
• Team grows beyond 1 operator: add on-call rotation support (PagerDuty or Opsgenie in place of direct DM to D0AJ7K184TV).
• Phase 3 has 30 clean production days: evaluate Phase 4 allow-list candidates.