CI Monitoring runbook

System: ci-monitoring (Prometheus + Alertmanager + Grafana + blackbox_exporter) Owner: sre-agent / Kristerpher Terraform root: terraform/ci-monitoring/ Card: #1022 ADR: ADR-0134 (Woodpecker CI) Last reviewed: 2026-08-13 — memory-pressure hardening (see Last incident below): per-container mem_limit on all 6 Docker services, MemoryHigh/ MemoryMax on the previously-uncapped devpi-server systemd unit, a 2 GB swap file, and journald caps, baked into cloud-init.sh.tpl and applied live. Prior reviewed 2026-08-09 — three independent changes landed same day: (1) #4432: user_data_server.sh.tpl now installs node_exporter automatically at boot, going forward from the server's next replace (Post-apply Step 1, Failure mode B); (2) #4433: wp_server_private_ip/ wp_server_instance_id are no longer hand-maintained Terraform variable defaults — both now read live from infra/ci's state via terraform_remote_state (§Cross-root remote state, Failure mode J); (3) #4436: §Maintenance silences updated with the enforced scripts/ops/ci-asg-refresh.sh pre-flight wrapper for agent ASG instance-refreshes. Last incident: 2026-08-13 — MemoryPressureHigh{instance="ci-monitoring"} had been active continuously since 2026-08-07 (~6 days, filed as #4435, unactioned). Root-caused to devpi-server (systemd, not Docker — added 2026-08-05) running with no memory ceiling; its RSS grew to 1.17 GB (63% of the box's 2 GB) with no swap backstop. See RCA docs/incidents/2026-08-13-ci-monitoring-devpi-memory-pressure.md and §Memory-pressure hardening below. Prior: 2026-08-07 — planned #4416 WP server replace fired real Alertmanager pages with no way to suppress an authorized maintenance window, plus left a stale Prometheus scrape target (node-woodpecker/woodpecker jobs pointed at the destroyed instance's IP) re-paging a critical NodeDown hourly for ~2h afterward. See RCA docs/incidents/2026-08-07-ci-alertmanager-maintenance-noise.md and the new §Maintenance silences section below. Prior: 2026-07-26 — TLSOriginProbeDown{instance="tickets.raxx.app"} cleared: the raxx-tickets Lightsail firewall was additively opened on port 443 to the ci-monitoring box's egress IP (18.217.254.146/32), operator-authorized in-session; the tls_connect_origin_console module + scrape target (merged to develop in #4344 but never hot-applied) was also hot-applied to the live monitoring box, closing that IaC-vs-live drift. See §TLS cert-expiry monitoring below (both "Known gap" notes below are now resolved) and PR that codifies the firewall CIDR in terraform/freescout/main.tf. Prior: 2026-07-24 — console.raxx.app origin (Heroku ACM) cert expired silently for ~30 days because it had no origin probe (only the edge probe, which never saw the problem). Root cause was a CF Access application on console.raxx.app blocking Heroku's HTTP-01 renewal challenge; fixed via a scoped CF Access bypass on /.well-known/acme-challenge/*. Added tls_connect_origin_console to close the monitoring gap — see RCA docs/incidents/2026-07-24-console-acm-cf-access-526.md. Prior: 2026-07-24 — TLSOriginProbeDown false positive for vault.raxx.app: the origin's new Cloudflare Origin CA cert (installed 2026-07-24, see prior incident below) failed chain verification against blackbox_exporter's default CA bundle. See RCA docs/incidents/2026-07-24-vault-origin-probe-ca-false-positive.md and Failure mode I below. Prior: 2026-07-23 — vault.raxx.app origin TLS certificate expired (notAfter=Jul 23 21:23:17 2026 GMT) with no cert-expiry monitoring in place to catch it. Fleet TLS cert-expiry monitoring (#2819) was built in direct response — see §TLS cert-expiry monitoring below. Prior: 2026-07-13 (see RCA docs/incidents/2026-07-13-ci-spot-wedge-silent-outage.md — scale_arbiter wedged spot, 5h silent CI outage)

Overview

A dedicated EC2 t4g.small (arm64, us-east-2) running a docker-compose monitoring stack:

None of these ports are open to the internet. Access is via SSM port forwarding.

This stack complements — it does NOT replace — the external GHA watchdog (.github/workflows/woodpecker-healthcheck.yml). The GHA watchdog provides independent-vantage liveness detection from outside the VPC.

TLS cert-expiry monitoring (#2819)

Why this exists: on 2026-07-23 the vault.raxx.app origin TLS certificate (Let's Encrypt, CN=vault.raxx.app) expired at Jul 23 21:23:17 2026 GMT with no monitoring in place to catch it, causing a live vault outage. This section was built during that incident.

The core problem this solves: for Cloudflare-proxied hosts, a plain TLS probe against the public hostname sees the Cloudflare-managed edge cert (Universal SSL, auto-renews every 90 days). That edge cert hides origin-cert rot completely — the edge can be perfectly healthy while the origin cert behind it has been expired for hours. vault.raxx.app and tickets.raxx.app are the two hosts where we manage the origin cert ourselves (Lightsail, behind CF Full Strict), so they get an additional direct-to-origin probe with the correct SNI servername, alongside the edge probe every host gets.

Probes

Both jobs carry a probe_type label (edge / origin) so Grafana and alert annotations can disambiguate which layer a reading is about.

insecure_skip_verify: false on the origin modules is intentional: an invalid/expired origin cert must show up as probe_success=0, matching what Cloudflare Full Strict itself would reject with an HTTP 526. This makes blackbox-tls-origin the "526-class detector" — see docs/ops/runbooks/waf.md and docs/incidents/2026-06-19-freescout-526-ssl-strict.md for the prior incident where exactly this class of failure (origin cert rejected by Full Strict) took tickets.raxx.app down with no automated detection.

Both Lightsail origin modules (tls_connect_origin_vault, tls_connect_origin_tickets) also carry ca_file: /etc/blackbox_exporter/certs/cloudflare-origin-ca-rsa-root.pem (added 2026-07-24, see Failure mode I below) — both origins serve a Cloudflare Origin CA certificate rather than a publicly-trusted one, and without this the default system trust store rejects a perfectly valid origin cert with certificate signed by unknown authority, producing a false TLSOriginProbeDown and silently blanking the cert-expiry metric. This does not weaken insecure_skip_verify: false — it only adds the one CA these origins actually use to the trust set the probe validates against. tls_connect_origin_console (added 2026-07-24, post-incident — see below) does not carry this ca_file: the console origin is Heroku ACM / Let's Encrypt, a publicly-trusted CA, so the default trust store is correct.

console.raxx.app origin probe (added 2026-07-24, post-incident): until this date, console.raxx.app had an edge probe (blackbox-tls) but no origin probe — the exact gap that let its Heroku ACM origin cert expire silently for ~30 days (see docs/incidents/2026-07-24-console-acm-cf-access-526.md). Unlike vault/tickets, the console origin is a Heroku dyno with no static IP, so the probe target is the app's herokudns CNAME (animated-bass-9kbgm8srm32uu4q97xz0hzu3.herokudns.com:443) with server_name: console.raxx.app forced via tls_config so Heroku's router presents the correct SNI-routed cert. No firewall-allowlist caveat applies here (unlike the tickets.raxx.app gap below) — Heroku's router accepts inbound 443 from anywhere.

IaC-vs-live drift closed 2026-07-26: tls_connect_origin_console merged to terraform/ci-monitoring/templates/cloud-init.sh.tpl in #4344 (2026-07-24) but was never hot-applied to the running instance, so console.raxx.app's origin cert went unmonitored for ~2 days despite the config existing on develop. Hot-applied via the same SSM Run Command + python3/PyYAML idiom used for #4338 (idempotent patch of blackbox.yml's modules: map + a surgical text insertion of the new static_configs target into prometheus.yml, docker compose restart blackbox-exporter, then kill -HUP 1 on the prometheus container to reload without restart). Verified: probe_success{job="blackbox-tls-origin",instance="console.raxx.app"} == 1, target up in Prometheus, cert expiry ~88d out (matches the Oct 2026 Heroku ACM renewal), no TLSOriginProbeDown firing. Lesson: a terraform merge to cloud-init.sh.tpl only lands on the next terraform apply / instance replacement — it does not retroactively apply to the running box. Any monitoring-config PR that needs to close a live gap immediately (not at the next planned replacement) must include an explicit hot-apply step, not just the template change.

Resolved 2026-07-26 — tickets.raxx.app origin probe: the raxx-tickets Lightsail instance's firewall restricted inbound port 443 to Cloudflare's published IP ranges only — a legitimate hardening measure — and the ci-monitoring box's probes originate from its NAT gateway EIP (18.217.254.146), which was not in that allowlist, so the origin probe timed out for ~46h (confirmed via debug probe: TCP dial timeout, not a cert or DNS error). Operator authorized (in-session, 2026-07-26) an additive widening of the raxx-tickets port-443 allow-list: 18.217.254.146/32 added via aws lightsail open-instance-public-ports (additive — does not replace the existing rule set the way put-instance-public-ports would), preserving the CF-range CIDRs and the port 80/22 rules unchanged. Verified: probe_success{job="blackbox-tls-origin",instance="tickets.raxx.app"} == 1, cert (CN=CloudFlare Origin Certificate, SAN=tickets.raxx.app) valid to 2041-06-15, TLSOriginProbeDown{instance="tickets.raxx.app"} absent from both Prometheus /api/v1/alerts and Alertmanager. The CIDR is now also codified in terraform/freescout/main.tf (aws_lightsail_instance_public_ports.freescout) as an additive port_info block alongside the CF dynamic block, so a future terraform apply on that (declarative, non-additive per-resource) config does not revert the CLI-applied change. By contrast, raxx-vault's Lightsail firewall allows 443 from 0.0.0.0/0 — that's why the vault origin probe has always worked without this allowlist gap; whether vault should be tightened to the same CF-only posture as tickets remains a separate, out-of-scope observation for whoever owns the vault host fix.

Alert rules

All three route through the existing email-ops Alertmanager receiver (Postmark → kris@moosequest.net). Per the standing exception, TLS cert CRIT is a prod-incident-class page and is not subject to the pre-launch alert digest — these fire per-event, same as today.

Grafana

The "Blackbox Probes" dashboard (ci-blackbox UID) gained a Certificate Expiry table panel: host, edge/origin, days remaining, sorted ascending. A missing row for an origin probe means the TLS handshake itself failed (expired/invalid cert or unreachable) — cross-check TLSOriginProbeDown and the "Probe Status" panel.

A note on IPv4/IPv6

The tls_connect module previously had no preferred_ip_protocol set, which defaults blackbox_exporter to IPv6-first. The monitoring box's private subnet has no IPv6 route, so any CF-proxied host with an AAAA record (i.e. most of them) failed with dial tcp6 ...: cannot assign requested address — a false probe_success=0 unrelated to the actual cert. This had been latent in the pre-existing ci.moosequest.net probe (which happens to have no AAAA record, so it never surfaced) until the #2819 build-out added CF-proxied hosts and hit it immediately. Fixed by setting preferred_ip_protocol: "ip4" + ip_protocol_fallback: false on all tls_connect* modules.

Accessing the UI

Pre-requisite: AWS SSM Session Manager plugin installed locally.

brew install session-manager-plugin   # macOS

Grafana (dashboards)

aws ssm start-session \
  --region us-east-2 \
  --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["3000"],"localPortNumber":["3000"]}'
# then open http://localhost:3000 in your browser
# username: admin
# password: retrieve from SSM:
aws ssm get-parameter --region us-east-2 \
  --name /ci/monitoring/grafana_admin_password \
  --with-decryption --query Parameter.Value --output text

The instance ID is in the Terraform output instance_id.

Dashboards provisioned on this host include Blackbox Probes, Host Metrics, Woodpecker CI HA Health, Woodpecker Pipeline Health, and (2026-07) AWS KMS Request Volume & Cost Attribution — see docs/ops/runbooks/aws-kms.md and docs/ops/aws-kms-cost-model.md for the latter.

Prometheus

aws ssm start-session \
  --region us-east-2 \
  --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["9090"],"localPortNumber":["9090"]}'
# then open http://localhost:9090

Alertmanager

aws ssm start-session \
  --region us-east-2 \
  --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["9093"],"localPortNumber":["9093"]}'
# then open http://localhost:9093

Post-apply configuration (one-time operator actions)

After terraform apply completes, complete these steps in order:

Step 1 — Install node_exporter on the Woodpecker server

As of #4432 (merged 2026-08-09): automatic going forward. infra/ci/templates/user_data_server.sh.tpl now installs and enables node_exporter v1.8.2 (arm64) at boot, so this manual step is no longer needed for a WP server instance replaced after #4432 lands (user_data changes only take effect on the server's next replace — lifecycle { ignore_changes = [user_data] } means the currently-running instance was not affected by merging #4432). Use this step only to fix-forward an instance still running a pre-#4432 user_data, or to verify node_exporter came up correctly on a fresh replace (see Step 3 below).

Copy-paste the node_exporter_install_command from terraform output and run it. It uses SSM Run Command to install node_exporter v1.8.2 on the WP server without SSH.

Monitor the command execution:

COMMAND_ID=<output from send-command>
aws ssm list-command-invocations \
  --region us-east-2 \
  --command-id "$COMMAND_ID" \
  --details \
  --query "CommandInvocations[0].CommandPlugins[0].Output"

Step 2 — Configure Woodpecker to expose /metrics

Retrieve the Prometheus bearer token:

WP_TOKEN=$(aws ssm get-parameter \
  --region us-east-2 \
  --name /ci/monitoring/wp_prometheus_token \
  --with-decryption \
  --query Parameter.Value \
  --output text)

On the Woodpecker server, add these two environment variables to the WP server configuration (wherever WOODPECKER_* vars are currently set — typically the systemd unit file or the docker-compose that runs the WP server):

WOODPECKER_METRICS_SERVER_ADDR=:9001
WOODPECKER_PROMETHEUS_AUTH_TOKEN=<value of $WP_TOKEN>

Then restart the Woodpecker server. The Prometheus target woodpecker will transition from DOWN to UP within 30 seconds.

Step 3 — Verify all Prometheus targets are UP

Open Prometheus (port 9090 via SSM forward) → Status → Targets. Expected state after post-apply setup:

Step 4 — Fire a test alert

Send a test alert through Alertmanager to verify email delivery:

# SSM forward to port 9093 first, then:
curl -s -X POST http://localhost:9093/api/v2/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "instance": "ci-monitoring-test"
    },
    "annotations": {
      "summary": "Test alert from ci-monitoring runbook",
      "description": "This is a manual test alert. Resolve by sending an empty array to the same endpoint."
    },
    "endsAt": "'"$(date -u -v+5M '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || date -u -d '+5 minutes' '+%Y-%m-%dT%H:%M:%SZ')"'"
  }]'

Expect an email at kris@moosequest.net within 30 seconds. Check spam if it doesn't arrive. To resolve the test alert early: POST an empty array [] to the same endpoint.

Step 5 — Retire the GHA watchdog (deferred)

Keep the GHA watchdog (.github/workflows/woodpecker-healthcheck.yml) running until this stack's alerting is verified end-to-end (Steps 3 + 4 complete, at least one real alert received and resolved). Once verified, the watchdog can be disabled or left as a cheap second opinion from outside the VPC.

Maintenance silences

Rule: step 0 of any planned CI pause (WP server replace, agent ASG instance refresh, ci-monitoring box maintenance) is creating a silence. The last step is expiring it and confirming alerts re-armed clean. This is not optional — see docs/incidents/2026-08-07-ci-alertmanager-maintenance-noise.md, where a correctly-authorized, planned #4416 outage paged the operator with no way to say "I know, this is expected."

For the agent ASG instance-refresh leg specifically, this rule is enforced, not just documented (#4436) — see §Enforced agent ASG instance-refresh below. WP server replaces and ci-monitoring box maintenance remain process-adherence (documented, not enforced by tooling).

The tool

scripts/ops/ci-maintenance-silence.sh wraps amtool silence add/query/expire over AWS SSM Run Command against the ci-monitoring box (Alertmanager's port 9093 is VPC-private — this is the same access pattern as the rest of this runbook, just non-interactive).

# Create (before starting planned work):
scripts/ops/ci-maintenance-silence.sh create \
  --duration 30m \
  --comment "Woodpecker server replace, #4416, kris" \
  --author "kris"

# List active silences:
scripts/ops/ci-maintenance-silence.sh list

# Expire early (after confirming the golden path — see runbook step "verify"):
scripts/ops/ci-maintenance-silence.sh expire <silence-id>
scripts/ops/ci-maintenance-silence.sh expire-all   # all silences this tool created

Set --duration to what you actually expect, not a padded guess — if the work runs long, create another silence rather than front-loading a large window "just in case." Long-lived silences are themselves a reliability risk (a real, unrelated problem can hide behind an over-broad or over-long silence).

Enforced agent ASG instance-refresh (scripts/ops/ci-asg-refresh.sh, #4436)

Starting an agent-fleet instance-refresh (aws autoscaling start-instance-refresh --auto-scaling-group-name ci-woodpecker-agents-asg) is itself a maintenance window on this alert surface — the drain + reboot can trip WoodpeckerNoWorkersWithBacklog if it runs long (see docs/ops/runbooks/ci-build-cache.md, step 5 and Failure mode G). Rather than relying on remembering step 0 above every time, always start the refresh through the wrapper:

# Validate the silence check without starting a refresh:
scripts/ops/ci-asg-refresh.sh --dry-run

# Start the refresh for real:
scripts/ops/ci-asg-refresh.sh

ci-asg-refresh.sh calls scripts/ops/ci-maintenance-silence.sh list --json first and REFUSES (non-zero exit, actionable error naming the missing step) to call start-instance-refresh unless it finds at least one active silence carrying this script's [ci-maintenance-silence] comment tag. It does not create a silence on your behalf — run ci-maintenance-silence.sh create first, deliberately, same as any other planned CI pause:

scripts/ops/ci-maintenance-silence.sh create \
  --duration 15m \
  --comment "agent ASG instance-refresh, #<issue>, <your name>" \
  --author "<your name>"
scripts/ops/ci-asg-refresh.sh

This is scoped to exactly the one ASG action — it is not a general-purpose "any AWS mutation needs a silence" gate, and it does not change the ASG's min_size/desired capacity or refresh strategy.

CI-scoped only — this is a closed allowlist, not "silence everything"

This Alertmanager instance carries BOTH CI-infra alerts AND prod/customer- facing alerts on the same box (RaptorProdDown, TLSCertExpirySoon, TLSCertExpiryUrgent, TLSOriginProbeDown — covers raxx.app, api.raxx.app, console.raxx.app, vault.raxx.app, tickets.raxx.app, getraxx.com, and staging hosts). The script's matcher is a hardcoded, closed allowlist of CI-only alertnames:

WoodpeckerDown | NodeDown | DiskUsageHigh | MemoryPressureHigh |
WoodpeckerPipelineQueueStuck | WoodpeckerHealthzDown |
WoodpeckerNoWorkersWithBacklog | ForgeConnectivityDown |
ForgeConnectivitySlowApproachingTimeout

It deliberately excludes RaptorProdDown and all three TLS-cert alerts — those protect customer-facing production surfaces and must page immediately regardless of any CI maintenance window. Do not widen the matcher in the script without updating both its header comment and this section. See the script's header comment for the known limitation around NodeDown/ DiskUsageHigh/MemoryPressureHigh (currently CI-scoped only because the only two node_exporter targets on this Prometheus are CI infra — revisit if a third, non-CI node target is ever added).

Routing posture — does this conform to the pre-launch digest policy?

No, and it shouldn't. docs/architecture/ci-notification-posture.md governs routine GitHub Actions/cron Slack notifications (a different notification surface entirely) — daily-digesting those removes noise without removing signal, because they're mostly routine successes. This Alertmanager instance's alert set is different in kind: WoodpeckerNoWorkersWithBacklog exists specifically because a 2026-07-13 wedged-spot outage went undetected for 5 hours (see docs/incidents/2026-07-13-ci-spot-wedge-silent-outage.md) — digesting it would reintroduce that exact failure mode. RaptorProdDown and the TLS-cert alerts are explicitly prod-incident-class (this runbook's own alert-rule comments already say so: "Not subject to the pre-launch alert digest — cert CRIT is a prod-incident-class page per standing exception").

The actual routing config (route: in terraform/ci-monitoring/templates/ cloud-init.sh.tpl) is already reasonable, not "per-scrape spam": alerts are grouped by (alertname, severity, instance) with group_wait: 30s / group_interval: 5m (one email per new distinct problem, not one per Prometheus evaluation cycle), and repeat_interval is 4h default / 1h for critical (a periodic reminder while genuinely unresolved, not a flood). What was actually missing on 2026-08-07 was (1) no way to say "this specific, time-boxed page is expected" — now fixed by the silence tool above — and (2) a genuinely broken, unrelated stale scrape target that kept re-firing a real critical alert for ~2h with no maintenance excuse at all (see docs/ops/runbooks/ci-build-cache.md Failure mode H). Neither problem is a routing-posture problem; converting critical CI/prod alerts to a digest would trade a known, purpose-built 10-minute detection SLA for an unbounded one.

Memory-pressure hardening (#4435, 2026-08-13)

This box is a t4g.small (2 GB RAM). As of 2026-08-13 it carries four layers of memory hardening, all baked into cloud-init.sh.tpl (so a future box replacement starts hardened) and applied live to the currently-running instance (required separately — see the lifecycle { ignore_changes = [user_data_base64] } caveat in §Cross-root remote state above, same class of gap as Failure mode J):

1. Per-container mem_limit on all 6 Docker Compose services (docker-compose.yml): prometheus 384m, alertmanager 128m, grafana 256m, blackbox-exporter 128m, node-exporter 64m, registry-mirror 384m. Sized with 4-7x headroom above observed steady-state usage — these are safety ceilings against runaway growth, not tight production allocations. A container that hits its limit is OOM-killed and restarted (restart: unless-stopped) rather than starving the whole host.
2. MemoryHigh=768M / MemoryMax=1024M on devpi-server (/etc/systemd/system/devpi-server.service) — this is the actual fix for the 2026-08-13 incident (see Failure mode K below). devpi-server is NOT a Docker container (see §11b context in cloud-init.sh.tpl), so it needed systemd's own resource-control directives rather than compose's mem_limit. MemoryHigh throttles/reclaims under cgroup pressure without killing the process; MemoryMax is the hard backstop.
3. 2 GB swap file (/swapfile, vm.swappiness=10) — last-resort cushion against a sudden allocation spike from any component, so a spike degrades to slower service rather than an unpredictable kernel OOM-kill.
4. journald caps (SystemMaxUse=200M, RuntimeMaxUse=64M) — prevents log volume from becoming an unrelated second source of memory/disk pressure on a box that runs close to its ceiling.

Sizing note: these limits total more than the box's physical 1846 MB if every container and devpi-server were simultaneously at its own ceiling (1088 MB Docker + 1024 MB devpi = 2112 MB) — this is normal and intentional for resource limits (safety nets against any one runaway consumer), not a guarantee that the sum fits concurrently. Actual combined steady-state usage after this hardening landed was ~900 MB (57.5% of the box). If sustained combined usage climbs back toward these ceilings over time (more scrape targets, larger devpi/registry-mirror caches as more packages/images are proxied), that is itself the signal to revisit sizing or the resize proposal in #4435 — not something these limits alone will silently absorb forever.

Not applied to the Woodpecker CI server (infra/ci, separate Terraform root): the 2 GB swap file and journald caps were applied live there too (same box class, proactive hardening — that box was healthy, not part of this incident). Container mem_limits for woodpecker-server/ wp-config-svc were attempted but reverted — see Failure mode K below and docs/incidents/2026-08-13-ci-monitoring-devpi-memory-pressure.md Action item #3.

How to tell it's broken

• No email on a known outage: Alertmanager email pipeline is broken. Check SMTP config, Postmark token validity.
• Prometheus targets showing DOWN: See failure modes below.
• Grafana 502 Bad Gateway: Grafana container crashed. Check logs.
• Bootstrap didn't complete: /var/log/ci-monitoring-bootstrap.log is empty or stops mid-way. Check for curl(23) (output directory missing) or set -e exit. Instance role may also lack SSM GetParameter permission.
• Grafana keeps restarting: /run/secrets/grafana_admin_password: Permission denied in grafana container logs. Fix: chmod 644 /opt/monitoring/secrets/grafana_admin_password then docker compose up -d grafana.

How to diagnose (in order)

1. Check container status:

# SSM shell session:
aws ssm start-session --region us-east-2 --target <instance-id>

# On the box:
cd /opt/monitoring
docker compose ps
docker compose logs --tail=50 prometheus
docker compose logs --tail=50 alertmanager
docker compose logs --tail=50 grafana

2. Check bootstrap log:

cat /var/log/ci-monitoring-bootstrap.log

3. Check Prometheus targets: Prometheus UI → Status → Targets

4. Check alert rules: Prometheus UI → Alerts

5. Check Alertmanager config:

curl -s http://localhost:9093/api/v2/status | python3 -m json.tool

Known failure modes

Failure mode A: Woodpecker /metrics target DOWN

Symptom: woodpecker job shows DOWN in Prometheus targets.

Cause A1: WOODPECKER_METRICS_SERVER_ADDR is not set on WP server. Fix A1: Set env var :9001 on WP server and restart it.

Cause A2: Bearer token mismatch. Fix A2: Retrieve the token from SSM and compare with what's set on the WP server:

aws ssm get-parameter --region us-east-2 \
  --name /ci/monitoring/wp_prometheus_token \
  --with-decryption --query Parameter.Value --output text

Cause A3: SG rule not applied — monitoring SG lacks inbound access to port 9001 on WP SG. Fix A3: terraform apply should have created aws_security_group_rule.wp_metrics_from_monitoring. Verify:

aws ec2 describe-security-groups --region us-east-2 --group-ids sg-044249808bc11eaa7 \
  --query "SecurityGroups[0].IpPermissions[?FromPort==\`9001\`]"

Verification: Prometheus target woodpecker transitions to UP within 30s.

Failure mode B: node_exporter target DOWN on WP server

Symptom: node-woodpecker job shows DOWN.

Cause B1: node_exporter not installed on WP server. As of #4432, this should only occur on an instance still running a pre-#4432 user_data — fresh replaces install it automatically at boot (see Post-apply Step 1). Fix B1: Run the node_exporter_install_command from terraform output.

Cause B2: node_exporter installed but not running. Fix B2:

aws ssm send-command \
  --region us-east-2 \
  --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters 'commands=["systemctl status node_exporter; journalctl -u node_exporter -n 20"]'

Cause B3: SG rule missing (port 9100 not open from monitoring SG to WP SG). Fix B3: Verify aws_security_group_rule.wp_node_exporter_from_monitoring exists:

aws ec2 describe-security-groups --region us-east-2 --group-ids sg-044249808bc11eaa7 \
  --query "SecurityGroups[0].IpPermissions[?FromPort==\`9100\`]"

If missing, run terraform apply from terraform/ci-monitoring/.

Verification: node-woodpecker target UP.

Failure mode C: Alertmanager not sending email

Symptom: Alert fires in Prometheus, appears in Alertmanager, but no email arrives.

Cause C1: Postmark token expired or invalid. Fix C1:

# Check current token
aws ssm get-parameter --region us-east-2 \
  --name /ci/monitoring/postmark_server_token \
  --with-decryption --query Parameter.Value --output text

# Update if rotated:
aws ssm put-parameter --region us-east-2 \
  --name /ci/monitoring/postmark_server_token \
  --type SecureString \
  --value "NEW_TOKEN" \
  --overwrite >/dev/null

# Regenerate alertmanager.yml with new token and reload:
# (on the monitoring box)
cd /opt/monitoring
# Edit alertmanager/alertmanager.yml and replace the smtp_auth_password
docker compose exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml
docker compose kill -s HUP alertmanager

Cause C2: Email in spam / Postmark inactive sender signature. Fix C2: Check Postmark dashboard for delivery status. Ensure no-reply@raxx.app has an active sender signature.

Verification: amtool alert add sends a test, email arrives within 60s.

Failure mode D: Grafana restart loop (secrets permission denied)

Symptom: docker compose ps shows grafana in Restarting state. Logs show /run/secrets/grafana_admin_password: Permission denied.

Cause: The secrets file /opt/monitoring/secrets/grafana_admin_password is owned by root with mode 600. The Grafana container runs as UID 472 (non-root) and cannot read it.

Fix:

chmod 644 /opt/monitoring/secrets/grafana_admin_password
cd /opt/monitoring && docker compose up -d grafana

Verification: docker compose ps grafana shows Up (not Restarting).

Failure mode E: Grafana shows "Data source not found"

Symptom: Grafana dashboards show "No data" or datasource error.

Cause: Prometheus container restarted with a new container IP, breaking the http://prometheus:9090 URL.

Fix: Restart grafana — Docker's internal DNS will re-resolve:

cd /opt/monitoring && docker compose restart grafana

Verification: Dashboards show data within 60s.

Failure mode F: Monitoring box ran out of disk

Symptom: DiskUsageHigh alert fires on instance=ci-monitoring. Prometheus may stop ingesting (writes fail when disk is full).

Fix:

# Check retention setting (currently 15 days):
docker compose exec prometheus \
  sh -c 'du -sh /prometheus && df -h /prometheus'

# Clear old data if urgent (preserves last 7 days):
docker compose exec prometheus sh -c \
  'find /prometheus -name "*.tmp" -delete'

# Long-term: reduce retention in docker-compose.yml:
#   --storage.tsdb.retention.time=7d
# then: docker compose up -d prometheus

Verification: DiskUsageHigh resolves.

Failure mode G: cloud-init crash — bootstrap stops at package install

Symptom: /var/log/ci-monitoring-bootstrap.log stops partway through — e.g. curl(23) error or abrupt stop during docker-compose install. Stack never starts. The instance was tainted and re-provisioned, but the bug is in cloud-init.sh.tpl.

Cause: set -euo pipefail at the top of the bootstrap causes any non-zero exit to abort the entire script. Known trigger: curl writing to a path where the parent directory doesn't exist (e.g. /usr/local/lib/docker/cli-plugins/ before mkdir).

Fix: In terraform/ci-monitoring/templates/cloud-init.sh.tpl, the docker-compose install now uses dnf install -y docker-compose-plugin as primary method (available on AL2023), with mkdir -p before the curl fallback. This was fixed in PR #4084.

To re-run bootstrap on existing instance without taint (for testing):

# SSM shell on the box:
bash /var/lib/cloud/instance/scripts/part-001

Verification: Bootstrap log ends with [ci-monitoring] completed at ....

Failure mode I: TLSOriginProbeDown false positive — origin cert signed by CF Origin CA

Symptom: TLSOriginProbeDown fires for vault.raxx.app or tickets.raxx.app even though the origin is healthy and the origin cert is unexpired. The edge (https://<host>/) responds normally through Cloudflare.

Cause: vault.raxx.app and tickets.raxx.app both terminate TLS at a Lightsail origin with a static Cloudflare Origin CA certificate (not a publicly-trusted CA — see docs/incidents/2026-07-23-vault-origin-cert-expiry.md for why vault moved to this pattern on 2026-07-24, and docs/incidents/2026-06-19-freescout-526-ssl-strict.md for the original tickets.raxx.app precedent). The tls_connect_origin_vault / tls_connect_origin_tickets blackbox modules validate the origin cert against a CA bundle (insecure_skip_verify: false, by design — see §TLS cert-expiry monitoring above). Before this fix, that bundle was blackbox_exporter's default system trust store, which correctly does not trust a private CF Origin CA. Debug output showed the real error:

level=error msg="Error dialing TCP" err="tls: failed to verify certificate: x509: certificate signed by unknown authority"

This produced probe_success=0 on a perfectly healthy origin — a false positive — and, as a side effect, probe_ssl_earliest_cert_expiry stopped populating for that origin (a failed TLS handshake returns no chain to read an expiry date from), silently disabling the 30/14-day expiry lead-time this monitor exists to provide.

Fix: Point the origin modules' tls_config.ca_file at Cloudflare's published Origin CA root (RSA), baked into cloud-init.sh.tpl as a static heredoc ($MONITORING_DIR/blackbox/certs/cloudflare-origin-ca-rsa-root.pem, mounted into the container via the existing ./blackbox:/etc/blackbox_exporter:ro bind — no compose change needed) and referenced from both origin modules:

tls_connect_origin_vault:
  tcp:
    tls_config:
      insecure_skip_verify: false
      server_name: vault.raxx.app
      ca_file: /etc/blackbox_exporter/certs/cloudflare-origin-ca-rsa-root.pem

insecure_skip_verify stays false — this narrows trust to the one CA that issues these origins' certs, it does not disable verification. A genuinely expired/invalid/wrong-CN origin cert still fails chain verification and still reports probe_success=0 correctly.

To hot-apply on the live box without waiting for a terraform apply / instance replacement (same SSM Run Command pattern used to land #2819 live):

# 1. Write the CA root (fetch fresh from Cloudflare, or copy the block baked
#    into cloud-init.sh.tpl) to /opt/monitoring/blackbox/certs/cloudflare-origin-ca-rsa-root.pem
#    on the instance, chmod 644.
# 2. Patch blackbox.yml (python3 + PyYAML, same idiom as
#    inject-cf-access-headers.sh) to add ca_file under both origin modules'
#    tls_config.
# 3. cd /opt/monitoring && docker compose restart blackbox-exporter

Verification:

curl -s "http://localhost:9115/probe?target=34.207.47.85:443&module=tls_connect_origin_vault&debug=true" | grep -E "probe_success|Successfully dialed"
# probe_success 1

Then confirm in Prometheus: probe_success{job="blackbox-tls-origin",instance="vault.raxx.app"} returns 1, (probe_ssl_earliest_cert_expiry{job="blackbox-tls-origin",instance="vault.raxx.app"}-time())/86400 returns a sane day count (matches the cert's actual notAfter), and TLSOriginProbeDown{instance="vault.raxx.app"} is absent from /api/v1/alerts. Cross-check the Grafana "Certificate Expiry" panel shows a row for the origin again.

At the time this fix landed (2026-07-24), only vault.raxx.app could be verified end-to-end this way — the tickets.raxx.app origin probe was separately non-functional due to the Lightsail firewall allowlist gap (see §TLS cert-expiry monitoring → "Resolved 2026-07-26 — tickets.raxx.app origin probe"), so its probe_success stayed 0 (10s timeout, not a cert error) until that firewall change was authorized. The ca_file fix was applied to its module at the time and was confirmed correct once the firewall gap closed on 2026-07-26 — probe_success{instance="tickets.raxx.app"} == 1 with the Cloudflare Origin CA cert chain verifying cleanly.

Updating configs

Configs live in /opt/monitoring/ on the instance. To update:

1. Make config change via SSM shell or update cloud-init.sh.tpl + re-provision.
2. For Prometheus config changes (new scrape target, rule update):

cd /opt/monitoring
# Edit prometheus/prometheus.yml or prometheus/rules/alerts.yml
docker compose exec prometheus sh -c \
  'kill -HUP 1'  # sends SIGHUP to reload config without restart

3. For Alertmanager config changes:

cd /opt/monitoring
# Edit alertmanager/alertmanager.yml
docker compose kill -s HUP alertmanager

4. For Grafana changes (new dashboard, datasource), place JSON in /opt/monitoring/grafana/dashboards/ and wait up to 30s for Grafana's provisioning watcher to pick it up.

Cross-root remote state (#4433)

terraform/ci-monitoring/main.tf declares data.terraform_remote_state.infra_ci to read the Woodpecker CI server's private IP + instance ID directly from infra/ci's Terraform state, instead of hand-maintained variable defaults (var.wp_server_private_ip / var.wp_server_instance_id, removed in #4433). This eliminates the class of drift that caused the 2026-08-07 phantom NodeDown re-paging (Failure mode J below / docs/ops/runbooks/ci-build-cache.md Failure mode H).

Operator prerequisites before running terraform plan/apply on terraform/ci-monitoring: - s3:GetObject on moosequest-tf-state-us-east-2's ci/woodpecker/terraform.tfstate object — this is a different state backend (bucket + region) from terraform/ci-monitoring's own (raxx-iac-state-prod, us-east-1, per versions.tf). Both stacks live in the same AWS account (521228113048) as of 2026-08; if that ever changes, the remote-state config needs a cross-account role_arn. - The referenced infra/ci output (woodpecker_server_private_ip, woodpecker_server_instance_id) must already exist in infra/ci's applied state. If infra/ci has pending .tf changes that add or rename an output this stack depends on, apply infra/ci first — a plan here will fail with Error: Unsupported attribute ... This object does not have an attribute named "..." until infra/ci is re-applied.

Deploy order after a WP server replace: infra/ci apply (which performs the replace) always updates its own outputs as part of that same apply — no extra step needed there. Then terraform/ci-monitoring will resolve the new values on its next plan/apply automatically. See Failure mode J below and docs/ops/runbooks/ci-build-cache.md Failure mode H for the full standalone-replace procedure, including why the live ci-monitoring box's prometheus.yml still needs a manual fix-forward step regardless.

Failure mode H: WoodpeckerNoWorkersWithBacklog alert fires (wedged spot fleet)

Symptom: WoodpeckerNoWorkersWithBacklog critical alert fires. Pipelines are stuck in pending with 0 connected workers for 10+ minutes.

Cause: A spot interruption killed the CI agent. The ASG kept desired >= 1 but could not place a replacement (spot capacity unavailable in the AZ). This is the failure mode from the 2026-07-13 00:41 UTC outage (#4039).

Diagnosis:

1. Check the scale_arbiter CloudWatch logs for wedge log lines:

aws logs filter-log-events \
  --region us-east-2 \
  --log-group-name /aws/lambda/ci-scale-arbiter \
  --filter-pattern "WEDGE" \
  --start-time $(date -u -v-1H +%s000 2>/dev/null || date -u -d '1 hour ago' +%s000) \
  --query "events[].message" \
  --output text

2. Check ASG instance state:

aws autoscaling describe-auto-scaling-groups \
  --region us-east-2 \
  --auto-scaling-group-names <ci-agents-asg-name> \
  --query "AutoScalingGroups[0].{Desired:DesiredCapacity,Instances:Instances[].{Id:InstanceId,State:LifecycleState}}" \
  --output json

If Desired >= 1 and Instances is empty → spot is wedged. The scale_arbiter should self-heal via its escalation ladder (bump desired → instance refresh). If it has not, proceed to manual fix.

Fix (manual override):

ASG_NAME="<your-ci-agents-asg-name>"
# Reset to idle:
aws autoscaling set-desired-capacity --region us-east-2 \
  --auto-scaling-group-name "$ASG_NAME" \
  --desired-capacity 0 --honor-cooldown false
# Let ASG pick a new spot in a different AZ:
aws autoscaling set-desired-capacity --region us-east-2 \
  --auto-scaling-group-name "$ASG_NAME" \
  --desired-capacity 1 --honor-cooldown false

Wait 3-5 minutes for the spot to be placed, instance to boot, and agent to register. Check woodpecker_workers in Prometheus or WP admin UI.

Verification: Alert resolves. Pending pipelines drain. woodpecker_workers >= 1.

Escalate to operator if: The manual desired toggle does not result in a placed spot within 10 minutes (AZ or region capacity issue; may need instance-type override added to the launch template, or a switch to on-demand as a temporary measure).

Failure mode I: TLSOriginProbeDown or TLSCertExpiry{Soon,Urgent} fires

Symptom: Email alert TLSOriginProbeDown, TLSCertExpirySoon, or TLSCertExpiryUrgent for one of the fleet TLS hosts (#2819 — see §TLS cert-expiry monitoring above for the full probe/alert design).

Diagnosis — determine edge vs origin first (the alert's probe_type label and the Grafana Certificate Expiry table tell you which):

• Edge probe alert (probe_type="edge"): the public-facing Cloudflare cert for that hostname is expiring or already invalid. This is almost always a Cloudflare Universal SSL problem (rare — it auto-renews) or the DNS record was switched from proxied to DNS-only. Check: bash curl -sI https://<host>/ | grep -i "cf-ray\|server" Absence of cf-ray/server: cloudflare means the record is no longer proxied — re-enable the orange cloud on the DNS record.

• Origin probe alert (probe_type="origin", only vault/tickets): the cert we manage on the Lightsail origin is expiring, invalid, or the origin is unreachable. Confirm directly (read-only, safe during an incident): bash # Replace IP/servername with the affected host's origin IP + hostname: # vault.raxx.app -> 34.207.47.85 # tickets.raxx.app -> 54.146.13.200 echo | openssl s_client -connect <origin-ip>:443 -servername <hostname> 2>&1 \ | openssl x509 -noout -subject -issuer -dates 2>/dev/null || \ echo | openssl s_client -connect <origin-ip>:443 -servername <hostname> If it shows verify error:num=10:certificate has expired (or similar), the origin cert needs renewal — this is the exact class of failure from the 2026-07-23 vault outage. Renewal is host-specific and out of scope for this runbook (this is the monitoring layer, not the cert-issuance layer); escalate to whoever owns TLS for that host if no cert-renewal runbook exists for it. If the connection times out entirely rather than producing a TLS error, the origin is unreachable — check the Lightsail firewall (aws lightsail get-instance-port-states --instance-name <instance>) and instance state before assuming a cert problem.

Known non-actionable state (documented, not a bug): TLSOriginProbeDown for tickets.raxx.app fires continuously because the raxx-tickets Lightsail firewall only allows inbound 443 from Cloudflare's IP ranges — the monitoring box's egress IP (18.217.254.146) isn't in that allowlist. See §TLS cert-expiry monitoring above for the exact fix and why it's blocked on an operator decision rather than applied automatically.

Verification: alert resolves once probe_ssl_earliest_cert_expiry for that instance is comfortably above the 30-day WARN threshold (edge/origin cert-expiry alerts), or probe_success{job="blackbox-tls-origin"} returns to 1 for 10+ minutes (origin-down alert).

Failure mode J: NodeDown/woodpecker targets keep re-paging hours after an unrelated WP server replace

Status (as of #4433, merged 2026-08-09): the hand-maintained-default half of this is fixed. terraform/ci-monitoring no longer hardcodes the WP server's private IP / instance ID as Terraform variable defaults — both are now read live from infra/ci's Terraform state via a terraform_remote_state data source (terraform/ci-monitoring/main.tf). A fresh terraform apply on terraform/ci-monitoring after any infra/ci WP server replace now always resolves the current private IP/instance ID — there is no longer a second Terraform default to remember to bump by hand. The live-box fix-forward half is unchanged and still required — see Fix below. Symptom: Critical NodeDown (or MemoryPressureHigh/DiskUsageHigh) for instance=ci-woodpecker-server keeps firing/re-notifying long after a infra/ci Woodpecker server replace, even though the WP golden path itself (UI, agents, pipelines) is fine. Cause: This Prometheus's scrape targets for the WP server track whatever private IP was baked into the already-running ci-monitoring box's rendered prometheus.yml at its own last boot. That box's lifecycle { ignore_changes = [user_data_base64] } (terraform/ci-monitoring/main.tf) means a WP server replace's new IP never reaches the live prometheus.yml automatically — Terraform is only "correct" for a future ci-monitoring box, not the one currently running. First hit 2026-08-07 (#4416 replace) — ~2 hours of hourly false-positive critical pages before caught. Fix + verification: full procedure lives in docs/ops/runbooks/ci-build-cache.md Failure mode H (same underlying cause, documented there alongside the standalone-replace procedure it's a step of) — don't duplicate it here, keep that copy authoritative. As of #4433, the "update the Terraform default" sub-step there is gone (automatic via remote state); the sed + docker compose kill -s HUP prometheus fix-forward on the live box is still a required manual step.

Failure mode K: MemoryPressureHigh — devpi-server RSS growth uncapped

Status (as of #4435, fixed 2026-08-13): see §Memory-pressure hardening above for the full fix. This entry documents the symptom/diagnosis for the next time memory pressure returns (a new hog, or these caps proving insufficient over time).

Symptom: MemoryPressureHigh{instance="ci-monitoring"} active in Alertmanager, free -m shows usage climbing toward 90%+ with little available.

Diagnose (in order):

# 1. Which process, not just which container:
free -m
docker stats --no-stream --format 'table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'
ps aux --sort=-%mem | head -15
# If the top consumer is NOT one of the 6 docker-compose services, check for
# an uncontainerized systemd service (devpi-server is the known example —
# systemctl status devpi-server shows its cgroup Memory: line directly).

# 2. Confirm it's not a sudden spike (OOM already happened) vs. slow growth:
dmesg | grep -i oom
journalctl -k --since '14 days ago' | grep -iE 'out of memory|oom-kill|killed process'

# 3. Confirm the trend (Prometheus has the history even if the alert is new):
curl -s 'http://localhost:9090/api/v1/query?query=%281+-+%28node_memory_MemAvailable_bytes%7Binstance%3D%22ci-monitoring%22%7D+%2F+node_memory_MemTotal_bytes%7Binstance%3D%22ci-monitoring%22%7D%29%29+%2A+100&time=<7-days-ago-epoch>'
# Compare against the current-time value (omit &time=...) to see growth rate.

Cause (2026-08-13 instance): devpi-server (systemd, not Docker) had no MemoryHigh/MemoryMax — see §Memory-pressure hardening above for the fix and root-cause writeup, and docs/incidents/2026-08-13-ci-monitoring-devpi-memory-pressure.md for the full RCA.

Fix: Create a maintenance silence first (this box's alertname allowlist covers MemoryPressureHigh — see §Maintenance silences above), then: - If a Docker container is the hog: add/raise its mem_limit in /opt/monitoring/docker-compose.yml, docker compose up -d to apply (recreates that container). - If a systemd-managed process is the hog: add/raise MemoryHigh/MemoryMax in its unit file, systemctl daemon-reload && systemctl restart <unit>. A restart of a cache-only service (devpi-server, registry-mirror) is cheap — on-disk cache persists, only in-process RSS resets. - If nothing is individually over a reasonable limit and total usage is still high: that's the resize signal, not a tuning problem — escalate to the operator with current usage numbers rather than lowering limits below what components actually need (which just trades MemoryPressureHigh for container OOM-kill-restart-loops). Expire the silence and verify GET /api/v2/alerts returns [] (not just that the silence expired) before declaring done — same discipline as Failure mode J / ci-build-cache.md Failure mode H.

Verification: free -m / Prometheus query shows usage back under the 90% WARN threshold with real margin (not just barely under); the specific process/container that was the hog is now running within its new cap.

Emergency stop

# SSM session on monitoring box:
cd /opt/monitoring
docker compose down

# To prevent auto-restart on reboot:
systemctl disable ci-monitoring.service

This does not affect the Woodpecker CI server.

Cost estimate

t4g.small On-Demand: $0.0184/hr × 730h = ~$13.43/mo in us-east-2 (2026). An additional Reserved Instance (1-year, no upfront) reduces this to ~$8.30/mo.

Escalation

Wake the operator (kris@moosequest.net) when: - All Prometheus targets are DOWN (monitoring box itself is unreachable) - Alertmanager has been unable to send email for > 1 hour - The monitoring box has been unreachable via SSM for > 15 minutes - Disk is > 95% on the monitoring box

Known open items (punch-list)

References

• Terraform root: terraform/ci-monitoring/
• Woodpecker CI runbook: (TBD — create if Woodpecker-specific failures recur)
• Postmark dashboard: https://account.postmarkapp.com
• Prometheus docs: https://prometheus.io/docs
• Alertmanager docs: https://prometheus.io/docs/alerting/latest/alertmanager/
• Woodpecker metrics: https://woodpecker-ci.org/docs/administration/server-config#metrics
• ADR-0134: Woodpecker CI adoption
• Fleet TLS cert-expiry monitoring: card #2819 (this build-out)
• FreeScout TLS/cert architecture (origin cert model, precedent for the 526-class detector): docs/ops/runbooks/freescout-cert-renewal.md
• Prior origin-cert-rejected-by-Full-Strict incident: docs/incidents/2026-06-19-freescout-526-ssl-strict.md
• blackbox_exporter module reference (preferred_ip_protocol, tls_config.server_name): https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md
• Maintenance silence tool: scripts/ops/ci-maintenance-silence.sh
• CI build-cache / standalone-replace procedure: docs/ops/runbooks/ci-build-cache.md
• 2026-08-07 maintenance-noise RCA: docs/incidents/2026-08-07-ci-alertmanager-maintenance-noise.md
• 2026-08-13 memory-pressure hardening RCA: docs/incidents/2026-08-13-ci-monitoring-devpi-memory-pressure.md
• amtool docs: https://prometheus.io/docs/alerting/latest/silence/
• systemd resource control (MemoryHigh/MemoryMax): https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
• infra/ci user_data byte-budget guard: infra/ci/scripts/check_user_data_size.sh (#4467, ruling #4453)