CI Monitoring runbook

System: ci-monitoring (Prometheus + Alertmanager + Grafana + blackbox_exporter) Owner: sre-agent / Kristerpher Terraform root: terraform/ci-monitoring/ Card: #1022 ADR: ADR-0134 (Woodpecker CI) Last reviewed: 2026-07-08 Last incident: 2026-07-08 (see RCA — cloud-init curl crash + Grafana secrets perms)

Overview

A dedicated EC2 t4g.small (arm64, us-east-2) running a docker-compose monitoring stack:

None of these ports are open to the internet. Access is via SSM port forwarding.

This stack complements — it does NOT replace — the external GHA watchdog (.github/workflows/woodpecker-healthcheck.yml). The GHA watchdog provides independent-vantage liveness detection from outside the VPC.

Accessing the UI

Pre-requisite: AWS SSM Session Manager plugin installed locally.

brew install session-manager-plugin   # macOS

Grafana (dashboards)

aws ssm start-session \
  --region us-east-2 \
  --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["3000"],"localPortNumber":["3000"]}'
# then open http://localhost:3000 in your browser
# username: admin
# password: retrieve from SSM:
aws ssm get-parameter --region us-east-2 \
  --name /ci/monitoring/grafana_admin_password \
  --with-decryption --query Parameter.Value --output text

The instance ID is in the Terraform output instance_id.

Prometheus

aws ssm start-session \
  --region us-east-2 \
  --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["9090"],"localPortNumber":["9090"]}'
# then open http://localhost:9090

Alertmanager

aws ssm start-session \
  --region us-east-2 \
  --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["9093"],"localPortNumber":["9093"]}'
# then open http://localhost:9093

Post-apply configuration (one-time operator actions)

After terraform apply completes, complete these steps in order:

Step 1 — Install node_exporter on the Woodpecker server

Copy-paste the node_exporter_install_command from terraform output and run it. It uses SSM Run Command to install node_exporter v1.8.2 on the WP server without SSH.

Monitor the command execution:

COMMAND_ID=<output from send-command>
aws ssm list-command-invocations \
  --region us-east-2 \
  --command-id "$COMMAND_ID" \
  --details \
  --query "CommandInvocations[0].CommandPlugins[0].Output"

Step 2 — Configure Woodpecker to expose /metrics

Retrieve the Prometheus bearer token:

WP_TOKEN=$(aws ssm get-parameter \
  --region us-east-2 \
  --name /ci/monitoring/wp_prometheus_token \
  --with-decryption \
  --query Parameter.Value \
  --output text)

On the Woodpecker server, add these two environment variables to the WP server configuration (wherever WOODPECKER_* vars are currently set — typically the systemd unit file or the docker-compose that runs the WP server):

WOODPECKER_METRICS_SERVER_ADDR=:9001
WOODPECKER_PROMETHEUS_AUTH_TOKEN=<value of $WP_TOKEN>

Then restart the Woodpecker server. The Prometheus target woodpecker will transition from DOWN to UP within 30 seconds.

Step 3 — Verify all Prometheus targets are UP

Open Prometheus (port 9090 via SSM forward) → Status → Targets. Expected state after post-apply setup:

Step 4 — Fire a test alert

Send a test alert through Alertmanager to verify email delivery:

# SSM forward to port 9093 first, then:
curl -s -X POST http://localhost:9093/api/v2/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "instance": "ci-monitoring-test"
    },
    "annotations": {
      "summary": "Test alert from ci-monitoring runbook",
      "description": "This is a manual test alert. Resolve by sending an empty array to the same endpoint."
    },
    "endsAt": "'"$(date -u -v+5M '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || date -u -d '+5 minutes' '+%Y-%m-%dT%H:%M:%SZ')"'"
  }]'

Expect an email at kris@moosequest.net within 30 seconds. Check spam if it doesn't arrive. To resolve the test alert early: POST an empty array [] to the same endpoint.

Step 5 — Retire the GHA watchdog (deferred)

Keep the GHA watchdog (.github/workflows/woodpecker-healthcheck.yml) running until this stack's alerting is verified end-to-end (Steps 3 + 4 complete, at least one real alert received and resolved). Once verified, the watchdog can be disabled or left as a cheap second opinion from outside the VPC.

How to tell it's broken

• No email on a known outage: Alertmanager email pipeline is broken. Check SMTP config, Postmark token validity.
• Prometheus targets showing DOWN: See failure modes below.
• Grafana 502 Bad Gateway: Grafana container crashed. Check logs.
• Bootstrap didn't complete: /var/log/ci-monitoring-bootstrap.log is empty or stops mid-way. Check for curl(23) (output directory missing) or set -e exit. Instance role may also lack SSM GetParameter permission.
• Grafana keeps restarting: /run/secrets/grafana_admin_password: Permission denied in grafana container logs. Fix: chmod 644 /opt/monitoring/secrets/grafana_admin_password then docker compose up -d grafana.

How to diagnose (in order)

1. Check container status:

# SSM shell session:
aws ssm start-session --region us-east-2 --target <instance-id>

# On the box:
cd /opt/monitoring
docker compose ps
docker compose logs --tail=50 prometheus
docker compose logs --tail=50 alertmanager
docker compose logs --tail=50 grafana

2. Check bootstrap log:

cat /var/log/ci-monitoring-bootstrap.log

3. Check Prometheus targets: Prometheus UI → Status → Targets

4. Check alert rules: Prometheus UI → Alerts

5. Check Alertmanager config:

curl -s http://localhost:9093/api/v2/status | python3 -m json.tool

Known failure modes

Failure mode A: Woodpecker /metrics target DOWN

Symptom: woodpecker job shows DOWN in Prometheus targets.

Cause A1: WOODPECKER_METRICS_SERVER_ADDR is not set on WP server. Fix A1: Set env var :9001 on WP server and restart it.

Cause A2: Bearer token mismatch. Fix A2: Retrieve the token from SSM and compare with what's set on the WP server:

aws ssm get-parameter --region us-east-2 \
  --name /ci/monitoring/wp_prometheus_token \
  --with-decryption --query Parameter.Value --output text

Cause A3: SG rule not applied — monitoring SG lacks inbound access to port 9001 on WP SG. Fix A3: terraform apply should have created aws_security_group_rule.wp_metrics_from_monitoring. Verify:

aws ec2 describe-security-groups --region us-east-2 --group-ids sg-044249808bc11eaa7 \
  --query "SecurityGroups[0].IpPermissions[?FromPort==\`9001\`]"

Verification: Prometheus target woodpecker transitions to UP within 30s.

Failure mode B: node_exporter target DOWN on WP server

Symptom: node-woodpecker job shows DOWN.

Cause B1: node_exporter not installed on WP server. Fix B1: Run the node_exporter_install_command from terraform output.

Cause B2: node_exporter installed but not running. Fix B2:

aws ssm send-command \
  --region us-east-2 \
  --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters 'commands=["systemctl status node_exporter; journalctl -u node_exporter -n 20"]'

Cause B3: SG rule missing (port 9100 not open from monitoring SG to WP SG). Fix B3: Verify aws_security_group_rule.wp_node_exporter_from_monitoring exists:

aws ec2 describe-security-groups --region us-east-2 --group-ids sg-044249808bc11eaa7 \
  --query "SecurityGroups[0].IpPermissions[?FromPort==\`9100\`]"

If missing, run terraform apply from terraform/ci-monitoring/.

Verification: node-woodpecker target UP.

Failure mode C: Alertmanager not sending email

Symptom: Alert fires in Prometheus, appears in Alertmanager, but no email arrives.

Cause C1: Postmark token expired or invalid. Fix C1:

# Check current token
aws ssm get-parameter --region us-east-2 \
  --name /ci/monitoring/postmark_server_token \
  --with-decryption --query Parameter.Value --output text

# Update if rotated:
aws ssm put-parameter --region us-east-2 \
  --name /ci/monitoring/postmark_server_token \
  --type SecureString \
  --value "NEW_TOKEN" \
  --overwrite >/dev/null

# Regenerate alertmanager.yml with new token and reload:
# (on the monitoring box)
cd /opt/monitoring
# Edit alertmanager/alertmanager.yml and replace the smtp_auth_password
docker compose exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml
docker compose kill -s HUP alertmanager

Cause C2: Email in spam / Postmark inactive sender signature. Fix C2: Check Postmark dashboard for delivery status. Ensure no-reply@raxx.app has an active sender signature.

Verification: amtool alert add sends a test, email arrives within 60s.

Failure mode D: Grafana restart loop (secrets permission denied)

Symptom: docker compose ps shows grafana in Restarting state. Logs show /run/secrets/grafana_admin_password: Permission denied.

Cause: The secrets file /opt/monitoring/secrets/grafana_admin_password is owned by root with mode 600. The Grafana container runs as UID 472 (non-root) and cannot read it.

Fix:

chmod 644 /opt/monitoring/secrets/grafana_admin_password
cd /opt/monitoring && docker compose up -d grafana

Verification: docker compose ps grafana shows Up (not Restarting).

Failure mode E: Grafana shows "Data source not found"

Symptom: Grafana dashboards show "No data" or datasource error.

Cause: Prometheus container restarted with a new container IP, breaking the http://prometheus:9090 URL.

Fix: Restart grafana — Docker's internal DNS will re-resolve:

cd /opt/monitoring && docker compose restart grafana

Verification: Dashboards show data within 60s.

Failure mode F: Monitoring box ran out of disk

Symptom: DiskUsageHigh alert fires on instance=ci-monitoring. Prometheus may stop ingesting (writes fail when disk is full).

Fix:

# Check retention setting (currently 15 days):
docker compose exec prometheus \
  sh -c 'du -sh /prometheus && df -h /prometheus'

# Clear old data if urgent (preserves last 7 days):
docker compose exec prometheus sh -c \
  'find /prometheus -name "*.tmp" -delete'

# Long-term: reduce retention in docker-compose.yml:
#   --storage.tsdb.retention.time=7d
# then: docker compose up -d prometheus

Verification: DiskUsageHigh resolves.

Failure mode G: cloud-init crash — bootstrap stops at package install

Symptom: /var/log/ci-monitoring-bootstrap.log stops partway through — e.g. curl(23) error or abrupt stop during docker-compose install. Stack never starts. The instance was tainted and re-provisioned, but the bug is in cloud-init.sh.tpl.

Cause: set -euo pipefail at the top of the bootstrap causes any non-zero exit to abort the entire script. Known trigger: curl writing to a path where the parent directory doesn't exist (e.g. /usr/local/lib/docker/cli-plugins/ before mkdir).

Fix: In terraform/ci-monitoring/templates/cloud-init.sh.tpl, the docker-compose install now uses dnf install -y docker-compose-plugin as primary method (available on AL2023), with mkdir -p before the curl fallback. This was fixed in PR #4084.

To re-run bootstrap on existing instance without taint (for testing):

# SSM shell on the box:
bash /var/lib/cloud/instance/scripts/part-001

Verification: Bootstrap log ends with [ci-monitoring] completed at ....

Updating configs

Configs live in /opt/monitoring/ on the instance. To update:

1. Make config change via SSM shell or update cloud-init.sh.tpl + re-provision.
2. For Prometheus config changes (new scrape target, rule update):

cd /opt/monitoring
# Edit prometheus/prometheus.yml or prometheus/rules/alerts.yml
docker compose exec prometheus sh -c \
  'kill -HUP 1'  # sends SIGHUP to reload config without restart

3. For Alertmanager config changes:

cd /opt/monitoring
# Edit alertmanager/alertmanager.yml
docker compose kill -s HUP alertmanager

4. For Grafana changes (new dashboard, datasource), place JSON in /opt/monitoring/grafana/dashboards/ and wait up to 30s for Grafana's provisioning watcher to pick it up.

Emergency stop

# SSM session on monitoring box:
cd /opt/monitoring
docker compose down

# To prevent auto-restart on reboot:
systemctl disable ci-monitoring.service

This does not affect the Woodpecker CI server.

Cost estimate

t4g.small On-Demand: $0.0184/hr × 730h = ~$13.43/mo in us-east-2 (2026). An additional Reserved Instance (1-year, no upfront) reduces this to ~$8.30/mo.

Escalation

Wake the operator (kris@moosequest.net) when: - All Prometheus targets are DOWN (monitoring box itself is unreachable) - Alertmanager has been unable to send email for > 1 hour - The monitoring box has been unreachable via SSM for > 15 minutes - Disk is > 95% on the monitoring box

Known open items (punch-list)

References

• Terraform root: terraform/ci-monitoring/
• Woodpecker CI runbook: (TBD — create if Woodpecker-specific failures recur)
• Postmark dashboard: https://account.postmarkapp.com
• Prometheus docs: https://prometheus.io/docs
• Alertmanager docs: https://prometheus.io/docs/alerting/latest/alertmanager/
• Woodpecker metrics: https://woodpecker-ci.org/docs/administration/server-config#metrics
• ADR-0134: Woodpecker CI adoption