RCA — ci-monitoring box memory pressure root-caused to uncapped devpi-server; fleet hardened

Incident ID: 2026-08-13-ci-monitoring-devpi-memory-pressure Date: 2026-08-13 Severity: SEV-3 (non-blocking degradation — monitoring/alerting infrastructure at risk, no customer-facing or CI-functional outage occurred) Duration: MemoryPressureHigh{instance="ci-monitoring"} active continuously 2026-08-07 09:37 UTC → 2026-08-13 12:41 UTC (~6 days) before this response; hardening applied and verified within ~25 minutes of a single maintenance window Blast radius: ci-monitoring EC2 (Prometheus/Alertmanager/Grafana/blackbox-exporter/node-exporter/registry-mirror/devpi-server) — the CI observability and build-cache stack. No impact to the Woodpecker CI server, pipelines, or any customer-facing surface. Author: sre-agent

Summary

Operator escalation on unactioned issue #4435 ("ci-monitoring box memory pressure ~95% used") triggered a full fleet review. Root cause: devpi-server (the PyPI caching mirror added 2026-08-05 as part of the CI build-cache cost-reduction pass, #4393) runs directly on the host as a systemd service with no memory ceiling, and its in-process RSS grew unboundedly to 1.17 GB (63% of the box's 2 GB total) over its first ~8 days of uptime — dwarfing the combined ~190 MB footprint of the box's six Docker containers. No swap existed as a backstop. Applied non-disruptive hardening (per-container mem_limit, a MemoryHigh/MemoryMax systemd cap on devpi-server, a 2 GB swap file, and journald caps) inside a single CI-scoped Alertmanager maintenance silence; the devpi-server restart this required reset its RSS from 1.17 GB to ~540 MB and the box's memory usage from 93.9% to 57.5%. The Woodpecker CI server (i-0563f92028b656333) was investigated in parallel and found healthy (23% memory used, no pressure) — it received only non-disruptive hygiene (swap, journald caps) applied live; a proposed IaC change to add container mem_limits there was reverted after it broke the server's tight 12 KB raw user_data byte budget (see "What didn't go well"). MooseQuest-LLC/project-avenue was confirmed to share this same Woodpecker instance, agent fleet, and — critically — the same global devpi/registry-mirror caches (the pip-index wiring is server-level, not per-repo), so Avenue's own package/image pulls do proportionally contribute to the cache growth that drove this pressure; no evidence of Avenue-caused OOM kills, wedge events, or queue contention was found in the same window.

Timeline (all times UTC)

• 2026-08-07 09:37:15 — MemoryPressureHigh{instance="ci-monitoring"} (warning) first fires. Filed as issue #4435 the same day per the 2026-08-07 maintenance-noise RCA's action item #4. Never actioned.
• 2026-08-13 (this response) — Operator directive: review and harden the CI server fleet given #4435 unactioned, the WP server replace (#4416, 2026-08-07) having added more on-box services, and a report that Project-Avenue (sibling product, same AWS account) may also be contributing load.
• ~12:32 — Confirmed via GET /api/v2/alerts: only MemoryPressureHigh{instance="ci-monitoring"} active (93.9% used), startsAt=2026-08-07T09:37:15Z — continuously firing, unresolved, for ~6 days.
• ~12:33 — docker stats + ps aux --sort=-%mem on ci-monitoring: all 6 Docker containers combined use ~191 MB RSS. devpi-server (systemd, not containerized) is running at 1,194,008 KB RSS (1.17 GB, 63.1% of total box memory) — by a wide margin the single largest process on the box.
• ~12:34 — free -m confirms 0/0/0 swap configured. dmesg/journalctl -k show zero OOM-kill events in the last 14 days — the box has been running at the edge without yet crossing it. Disk healthy (root 19% used, /mnt/ci-cache 7% used, devpi's own on-disk cache only 580 MB — RSS bloat is NOT proportional to on-disk cache size, confirming it is in-process index/metadata overhead, not disk-cache size).
• ~12:35 — Prometheus range query confirms the growth trend: memory usage was 62.6% seven days prior (≈2026-08-06, consistent with devpi-server/registry-mirror's 2026-08-05 addition), 94.0% one day prior, 93.9% now — rapid growth followed by a plateau at the ceiling, not a sudden spike.
• ~12:34 — Woodpecker CI server (i-0563f92028b656333) checked in parallel: 420/1846 MB used (23%), 1252 MB available, 0 OOM events, healthy. Not the box firing the alert; confirmed NOT a contributing factor to #4435.
• ~12:36 — Confirmed MooseQuest-LLC/project-avenue (repo id 2) is active on this same Woodpecker instance alongside raxx-app/TradeMasterAPI (repo id 1). Reviewed Avenue's last 30 pipelines (2026-08-06 through 2026-08-13): normal success/failure mix; 3 killed statuses, one anomalously long (pipeline #187, ran 4.4h before being killed, 2026-08-06 13:28–17:55) — flagged for Avenue-owner follow-up, not further investigated here (out of this repo's scope). aws logs filter-log-events against /aws/lambda/ci-scale-arbiter for WEDGE in the full 2026-08-05→13 window: zero matches — no evidence of Avenue-caused agent-fleet contention. Current ASG state: Desired=0, no running agents (idle, expected off-hours state).
• 12:41:49 — Created CI-scoped Alertmanager maintenance silence (scripts/ops/ci-maintenance-silence.sh create --duration 45m) before any container-restarting change, per the mandatory runbook bookend from the 2026-08-07 maintenance-noise RCA.
• 12:42 — Applied 2 GB swap file + vm.swappiness=10 on ci-monitoring (no restarts involved).
• 12:42 — Applied journald caps (SystemMaxUse=200M, RuntimeMaxUse=64M) on ci-monitoring; systemctl restart systemd-journald.
• 12:42–12:43 — Added mem_limit to all 6 services in /opt/monitoring/docker-compose.yml (prometheus 384m, alertmanager 128m, grafana 256m, blackbox-exporter 128m, node-exporter 64m, registry-mirror 384m); docker compose config -q validated; docker compose up -d recreated all 6 containers — confirmed Up within 9 seconds.
• 12:43:32 — Added MemoryHigh=768M / MemoryMax=1024M to /etc/systemd/system/devpi-server.service; systemctl daemon-reload && systemctl restart devpi-server. RSS reset from 1.17 GB to 579 MB immediately on restart, settling to ~540 MB. curl http://localhost:3141/root/pypi/+simple/ returns 200.
• 12:43 — Post-change verification: free -m shows 906/1846 MB used (was 1601/1846); Prometheus targets all up except the pre-existing, already-documented-DEFERRED woodpecker /metrics job (unrelated — WP native metrics wiring was never activated, tracked separately in ci-monitoring.md's punch-list) and blackbox-forge transiently unknown (5-minute scrape interval, had not yet re-scraped since Prometheus's own restart — expected, not investigated further).
• 12:43 — GET /api/v2/alerts returns [] — zero active alerts, including the (now-expired) silence's own matchers. Confirms the underlying condition resolved, not merely masked.
• ~12:44 — Applied 2 GB swap + journald caps to the Woodpecker CI server (i-0563f92028b656333) live — no restarts, no container changes; docker ps confirms both woodpecker-woodpecker-server-1 and woodpecker-wp-config-svc-1 unaffected, Up 6 days (healthy) throughout.
• Expired the maintenance silence (ci-maintenance-silence.sh expire-all); re-confirmed GET /api/v2/alerts returns [] with no silence active.
• Attempted to add matching mem_limits to the Woodpecker CI server's Terraform boot template (infra/ci/templates/user_data_server.sh.tpl) for future-replace parity; sh infra/ci/scripts/check_user_data_size.sh failed at 13,358 bytes (over the 12,100-byte guard threshold, itself already only 188 bytes below AWS's hard 12,288-byte cap). Reverted this specific addition — see "What didn't go well" and Action items.
• Final confirmed state (Prometheus direct query): ci-monitoring memory usage 57.5% (down from 93.9%), 6/6 containers running within their new limits, devpi-server capped, zero active alerts.
• Resolved.

Impact

• Users affected: none (internal CI observability infrastructure only).
• User-visible symptoms: none.
• Data integrity: ok — devpi-server's on-disk cache at /mnt/ci-cache/devpi was never touched; only in-process RSS was reset by the restart.
• Revenue / billing: ok.
• Operational risk (the actual thing being fixed): for ~6 days, this box ran with no process-level memory ceiling on its single largest consumer and no swap backstop. A sudden allocation spike (e.g. a burst of concurrent pipeline activity driving more devpi/registry-mirror traffic) could have triggered a kernel OOM-kill with no predictable victim — plausibly Alertmanager (breaking the very paging pipeline meant to catch incidents) or Prometheus (breaking all CI-fleet visibility) rather than the actual offender.

What went well

• The root cause was found with direct evidence at every step (docker stats, ps aux --sort=-%mem, dmesg/journalctl -k, a 7-day Prometheus trend query) rather than guessed from the alert's face value — the original #4435 filing correctly suspected "six containers" as the driver; the actual data showed the containers were never the problem, an uncontainerized systemd service was.
• The mandatory maintenance-silence bookend (create before, expire-after-verify after) from the 2026-08-07 RCA was followed exactly, and it worked as designed: GET /api/v2/alerts returned [] both immediately after the silence expired (confirming a real fix, not a masked one) and the changes needed only a single 45-minute window.
• The fix was verified against the actual mechanism the alert measures (Prometheus's own node_memory_MemAvailable_bytes query), not just "the box looks less full now" — 93.9% → 57.5%, a number directly comparable to the WARN(90%) threshold that had been breached.
• The Woodpecker CI server was checked in parallel rather than assumed healthy or assumed to share the problem — it turned out to be genuinely unaffected, which mattered for scoping the fix correctly (no unnecessary disruption to the CI golden path).
• The infra/ci user_data byte-budget guard (check_user_data_size.sh, #4467) did exactly its job: caught an over-budget change at plan-check time via a local dry run, before it could fail a real terraform plan/CI gate on the next WP server replace. The change was reverted rather than force-fit by trimming comments elsewhere in a file that already carries a documented, intentional 298-byte margin.
• No changes were made to Prometheus retention/TSDB settings or Grafana despite being explicitly in scope to consider — both were checked (55 MB and 57 MB RSS respectively) and found not to be the hog. Not touching a healthy component that wasn't causing the problem avoided an untested, unnecessary change.

What didn't go well

• 4435 sat open and unactioned for 6 days despite being correctly filed, correctly labeled type:reliability, and describing a real, ongoing condition — there was no mechanism (comparable to the "critical alert firing with no maintenance silence on record" idea already tracked in #4434) that would have escalated a MemoryPressureHigh warning-severity alert continuously active for multiple days into something more urgent than routine backlog. #4434 is currently scoped to the CI-scoped alertname allowlist and critical-severity focus; it would not by default have caught this specific warning-severity, multi-day case.

• devpi-server was added to this box (2026-08-05, #4393) as a systemd service with no MemoryMax/MemoryHigh from day one — there was no equivalent-of-mem_limit review for the one build-cache component that isn't a Docker container, even though the box's existing node_exporter precedent (also systemd, not Docker) was explicitly cited as the reason to add it that way. Container-style resource limits were only applied to things that look like containers.
• The Woodpecker CI server's user_data byte budget (infra/ci/templates/user_data_server.sh.tpl) was already thin (298 bytes of headroom, by design, pending a gzip-compression follow-up flagged in #4453) before this incident — a reasonable, unrelated hardening addition (mem_limits for 2 lightweight containers) could not be added without either implementing that gzip follow-up in the same PR or reverting. This is a pre-existing, already-tracked constraint (not new to this incident), but it meant the WP-server-side IaC hardening had to be scoped down to what fits, leaving that box's future-replace state slightly less hardened than ci-monitoring's.
• There is still no automated way to detect "an uncontainerized systemd service on this box is consuming most of its memory" before it shows up as a host-level MemoryPressureHigh — the fix here (an explicit MemoryHigh/MemoryMax on devpi-server) is a targeted patch for this one process, not a structural guard against the next one added the same way.

Root cause analysis

• Contributing factor 1 — devpi-server has no built-in cache-size or memory ceiling, and none was added when it was deployed: devpi-server 6.20.3 has no native max-cache-size/eviction setting; its in-process RSS grows with every distinct package/version it proxies and was never observed to plateau below the box's physical ceiling until this fix capped it externally via systemd.
• Contributing factor 2 — resource-limit review at deployment time (#4393, 2026-08-05) covered Docker containers but not the one systemd-managed process added in the same change: the box's existing node_exporter precedent (systemd, not Docker — cited explicitly in the build-cache runbook as the reason devpi-server was built the same way) meant there was already a non-containerized service on this box, but "add resource limits" was implicitly treated as a Docker-only concern.
• Contributing factor 3 — no escalation path exists for a warning-severity alert that stays continuously active for multiple days: MemoryPressureHigh fired correctly and immediately (2026-08-07 09:37), but a severity: warning alert with a 4-hour repeat_interval blends into routine noise over days in a way a severity: critical alert with a 1-hour repeat_interval does not — #4434 (still open) is the right shape of fix for this class of gap but is currently scoped narrower than what would have caught this specific case.
• Contributing factor 4 — no swap existed on either t4g.small box as a last-resort cushion: this did not cause the incident, but it meant the box had zero margin between "94% used, plateaued" and an actual kernel OOM-kill — any component's normal, non-buggy allocation spike could have been the trigger.

Detection

• What alerted us: MemoryPressureHigh fired correctly and on time (2026-08-07 09:37 UTC) — the alerting itself worked. What failed was the response: the resulting issue (#4435) sat in the backlog for 6 days until this operator-directed fleet review.
• How long between cause and detection: 0 (the alert fired the same window the pressure crossed 90%) — but 6 days between detection and response.
• How to detect faster next time: the alert itself already detects this correctly at the box level. What's missing is escalation of an unactioned, continuously-active alert — see Action items (extending #4434, or a lighter-weight variant scoped to warning-severity CI alerts specifically).

Resolution

• What was changed (all applied live on ci-monitoring, i-0a656c6631ddc45b3, inside a single 45-minute CI-scoped Alertmanager silence): 1. 2 GB swap file (/swapfile, vm.swappiness=10) — safety-net backstop against any future allocation spike. 2. journald caps (SystemMaxUse=200M, RuntimeMaxUse=64M) — prevents log volume from becoming an unrelated second source of pressure. 3. mem_limit added to all 6 Docker Compose services in /opt/monitoring/docker-compose.yml (prometheus 384m, alertmanager 128m, grafana 256m, blackbox-exporter 128m, node-exporter 64m, registry-mirror 384m), sized with 4-7x headroom above observed steady-state usage; applied via docker compose up -d (all 6 containers recreated, confirmed Up within 9 seconds). 4. MemoryHigh=768M / MemoryMax=1024M added to /etc/systemd/system/devpi-server.service (the actual root cause — an uncontainerized systemd process with no prior ceiling); systemctl restart devpi-server reset RSS from 1.17 GB to ~540 MB, well under the new cap. 5. On the Woodpecker CI server (i-0563f92028b656333, healthy throughout, not the alerting box): 2 GB swap + journald caps applied live only — no container changes, no restarts of woodpecker-server/wp-config-svc. 6. terraform/ci-monitoring/templates/cloud-init.sh.tpl updated to bake all four ci-monitoring-side changes (swap, journald caps, per-container mem_limit, devpi-server MemoryHigh/MemoryMax) into the boot template, so a future ci-monitoring box replacement starts hardened. This box's lifecycle { ignore_changes = [user_data_base64] } guard means this does NOT retroactively apply to the already-hardened live box — the live-apply above was required regardless, matching the established pattern from the 2026-08-07 RCA. 7. infra/ci/templates/user_data_server.sh.tpl was NOT changed — an attempted swap+journald+mem_limit addition there broke the server's check_user_data_size.sh guard (13,358 bytes vs. the 12,100-byte fail threshold); reverted rather than force-fit. See Action items.
• Validation:
• Prometheus direct query (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes): 93.9% → 57.5% memory usage on ci-monitoring.
• docker stats: all 6 containers running comfortably within their new limits (highest observed: grafana at 34.78% of its 256m cap).
• systemctl status devpi-server: Memory: 541.9M (high: 768.0M max: 1.0G), active (running).
• curl http://localhost:3141/root/pypi/+simple/ → 200 (devpi still serving correctly post-restart).
• GET /api/v1/targets: all targets up except the pre-existing, separately-tracked woodpecker /metrics DEFERRED item and a transient blackbox-forge unknown (resolved on next 5-min scrape).
• GET /api/v2/alerts: [] both immediately after expiring the silence and on a follow-up check — confirms a genuine fix, not a masked one.
• Woodpecker CI server: docker ps showed both containers Up 6 days (healthy) before and after — zero disruption to the CI golden path.
• sh infra/ci/scripts/check_user_data_size.sh (WP server template, unchanged from before this incident): 12,078 bytes, WARN (pre-existing, documented, non-blocking) — confirms the revert left this file byte-for-byte equivalent to its pre-incident state (git diff empty).

Action items

References

• Runbook: docs/ops/runbooks/ci-monitoring.md (§Memory-pressure hardening, updated punch-list)
• Runbook: docs/ops/runbooks/ci-build-cache.md (devpi-server MemoryHigh/MemoryMax context)
• Prior incident: docs/incidents/2026-08-07-ci-alertmanager-maintenance-noise.md (maintenance-silence tool + bookend discipline used here)
• Issue: #4435 (original filing, findings posted as a comment)
• Issue: #4434 (critical-alert-continuously-firing detection — extension proposed here)
• Issue: #4453 (adjudication ruling on user_data byte budget, gzip follow-up)
• Issue: #4467 (the check_user_data_size.sh guard that caught the over-budget WP-server change in this incident)
• Script: scripts/ops/ci-maintenance-silence.sh
• Script: infra/ci/scripts/check_user_data_size.sh
• Terraform: terraform/ci-monitoring/templates/cloud-init.sh.tpl, infra/ci/templates/user_data_server.sh.tpl (unchanged)
• devpi-server docs (no native cache-size/eviction setting confirmed as of 6.20.3): https://doc.devpi.net/latest/
• systemd resource control (MemoryHigh/MemoryMax): https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html