CI build cache runbook

System: ci-build-cache (Docker registry pull-through mirror + PyPI caching proxy) Owner: operator (sre-agent for maintenance) Card: #4393 ADR: ADR-0134 (Woodpecker CI baseline) — this is a cost-reduction addition, not an architecture change Last incident: 2026-08-13 — devpi-server's in-process RSS grew unboundedly (no MemoryHigh/MemoryMax since its 2026-08-05 deployment) to 1.17 GB (63% of the ci-monitoring box's 2 GB), driving a 6-day-long MemoryPressureHigh alert (#4435). Fixed via systemd resource limits + compose mem_limits on the other 5 containers + swap; see docs/incidents/2026-08-13-ci-monitoring-devpi-memory-pressure.md and docs/ops/runbooks/ci-monitoring.md §Memory-pressure hardening, Failure mode K. Prior: 2026-08-07 — two incidents same day: (1) server replace to activate pip cache ran ~23min vs 3-5min accepted downtime, see docs/incidents/2026-08-07-woodpecker-server-replace-pip-cache-activation.md; (2) that same replace fired real Alertmanager pages for the authorized maintenance window, and left a stale Prometheus scrape target re-paging hourly for ~2h afterward, see docs/incidents/2026-08-07-ci-alertmanager-maintenance-noise.md Last reviewed: 2026-08-09 — three independent changes landed same day: (1) #4426/#4432: user_data_server.sh.tpl now performs the ECR login (Failure mode E) and installs node_exporter (Failure mode H, node_exporter half only) automatically at boot, going forward from the server's next replace; (2) #4433: terraform/ci-monitoring no longer hand-maintains wp_server_private_ip/wp_server_instance_id as Terraform variable defaults — both now read live from infra/ci's state via terraform_remote_state (Failure mode H, scrape-target-IP half — see below); (3) #4436: added scripts/ops/ci-asg-refresh.sh, an enforced pre-flight wrapper for agent ASG instance-refreshes, replacing the bare aws autoscaling start-instance-refresh calls in step 5 and Failure mode G with a call that hard-refuses without an active CI-scoped maintenance silence.

Context

The ci-woodpecker agent fleet (spot ASG, infra/ci/agents.tf) pushed ~1.49 TB/mo through the CI NAT Gateway with no build cache — every ephemeral agent instance re-pulled the same Docker Hub base images (python:3.11-slim referenced 131x across .woodpecker/*.yaml, node:20-slim, postgres:15, etc.) and re-downloaded the same pip packages on every fresh instance, because agents are spot/ephemeral by design (ADR-0134) and carry no persistent local cache across replacement. This was the single largest line item in the account's AWS bill ($66.89/mo of $502.06 in July 2026 — see docs/business/bookkeeper/2026-08-aws-cost-breakdown.md).

Two services address this, both hosted on the existing ci-monitoring EC2 instance (i-0a656c6631ddc45b3, us-east-2a) — reusing capacity that's already paid for rather than provisioning new EC2:

1. registry-mirror — registry:2.8.3 running as a Docker Hub pull-through cache (terraform/ci-monitoring/templates/cloud-init.sh.tpl, docker-compose service). Agents' Docker daemons point at it via registry-mirrors in /etc/docker/daemon.json (infra/ci/templates/user_data_agent.sh.tpl) — this requires zero .woodpecker/*.yaml changes because the mirror faithfully proxies unmodified docker.io/... image references.
2. devpi-server — raw systemd service (not dockerized — no actively-maintained official devpi Docker image; mirrors the node_exporter precedent already on this box) providing a PEP 503-compliant PyPI caching mirror at the built-in root/pypi index. Wired via Woodpecker's server-level WOODPECKER_ENVIRONMENT setting (injects PIP_INDEX_URL + PIP_TRUSTED_HOST into every pipeline step's container automatically) — also zero .woodpecker/*.yaml changes.

Both live behind ci-monitoring-sg, ingress-gated to ci-agent-sg only (see terraform/ci-monitoring/security-groups.tf) — the WP server itself does not run pipeline steps and has no need to reach either port.

Why NOT ECR pull-through cache: AWS's native ECR pull-through cache requires the client to reference a rewritten image path (<account>.dkr.ecr.<region>.amazonaws.com/<prefix>/<image>) — it cannot be the target of Docker's native registry-mirrors daemon setting for unmodified docker.io/... references. Using it would have required rewriting the image: line in ~80 pipeline files (131 references to python:3.11-slim alone) — the opposite of the least-invasive wiring this system was built for.

Current deployment status (as of 2026-08-07)

Pip-cache-hit evidence (2026-08-07): devpi-server access log (journalctl -u devpi-server on ci-monitoring, i-0a656c6631ddc45b3) shows real GET /root/pypi/+simple/<pkg>/ and GET /root/pypi/+f/<hash>/<wheel> requests from a live pipeline run, with sub-second response times — the same cache-hit signature used to verify the registry mirror (see Verification section below).

Standalone-replace path (used 2026-08-07, activation card #4416)

The Woodpecker server's aws_instance has lifecycle { ignore_changes = [user_data] } (see infra/ci/server.tf) so that unrelated template changes don't trigger an unwanted replace. This is the exact procedure used to intentionally replace the server and pick up new user_data (pip cache, wp-config-svc, or any other future template change), including the full dependent-resource set discovered during the 2026-08-07 incident — see docs/incidents/2026-08-07-woodpecker-server-replace-pip-cache-activation.md for the full RCA.

0. Create a maintenance silence FIRST, before touching anything. This window will trip real Alertmanager alerts (NodeDown, WoodpeckerHealthzDown, possibly WoodpeckerNoWorkersWithBacklog if the agent-refresh leg runs long) — that's expected and correct behavior for the probes, not a bug. Silence them for the duration you actually expect, not longer: bash scripts/ops/ci-maintenance-silence.sh create \ --duration 30m \ --comment "Woodpecker server replace, #<issue>, <your name>" \ --author "<your name>" This is CI-scoped only — it does not touch RaptorProdDown or the TLS-cert-expiry alerts, which stay live throughout. See docs/ops/runbooks/ci-monitoring.md §Maintenance silences for the full alertname allowlist and rationale. Do not skip this step even for a "should be fast" replace — the 2026-08-07 incident's accepted estimate was 3-5 minutes and actual was 23.
1. Preserve state before touching anything: - docker ps -a and cat /opt/woodpecker/docker-compose.yml on the live server (via SSM AWS-RunShellScript). - docker inspect wp-config-svc (or any other out-of-band container not in the compose file) — capture image, env, ports, healthcheck. Anything running out-of-band will NOT survive the replace as-is; you need this baseline to validate/fix-forward afterward.
2. Confirm no pipelines are mid-run (GET /api/repos/{id}/pipelines, check for status: pending|running). Re-check immediately before the apply — don't trust a check from more than a minute or two earlier.
3. Remove the ignore_changes block in infra/ci/server.tf (comment it out with a note pointing at the authorizing issue), then: terraform plan -target=aws_instance.woodpecker_server -out=replace.plan Confirm the plan shows exactly one -/+ resource "aws_instance" "woodpecker_server" { ... } must be replaced, forced by a user_data hash change, and nothing else. If the plan shows other resources changing, stop and investigate before applying.
4. Apply the targeted plan: terraform apply replace.plan. This takes under 2 minutes. This step alone is NOT sufficient — continue to step 5 before declaring success. Re-add the ignore_changes guard to server.tf immediately after.

5. Apply the direct dependents that -target does NOT cover (discovered the hard way on 2026-08-07 — do these every time, not just when something looks broken): - terraform apply -target=aws_lb_target_group_attachment.woodpecker_server — the ALB target group's target_id references the instance ID directly; it does not update itself when the instance is replaced. Without this, the public URL (ci.moosequest.net) stays unreachable even after the new server's containers are healthy. - terraform apply -target=aws_launch_template.woodpecker_agent — the agent launch template's user_data bakes in aws_instance.woodpecker_server.private_ip at apply time (static-mode server discovery, see infra/ci/agents.tf). Without this, any currently-running agent (and any new agent launched from the stale template version) keeps trying to reach the destroyed instance's dead IP — pipelines queue but are never claimed. - The agent ASG's instance_refresh block does not auto-trigger from a bare launch-template content update (the ASG resource's own launch_template.version = "$Latest" is a literal string that doesn't show a diff). Start one through the enforced wrapper, not a bare aws autoscaling start-instance-refresh call (see #4436 — this instance refresh is itself a second maintenance window on the same alert surface, and the wrapper hard-refuses if the step-0 silence has already lapsed instead of silently proceeding unsilenced): bash scripts/ops/ci-asg-refresh.sh --dry-run # validates the silence is still active scripts/ops/ci-asg-refresh.sh # starts the refresh for real scripts/ops/ci-asg-refresh.sh wraps aws autoscaling start-instance-refresh --auto-scaling-group-name ci-woodpecker-agents-asg --preferences '{"MinHealthyPercentage":0,"InstanceWarmup":180}' and REFUSES (non-zero exit, actionable error) to call it unless scripts/ops/ci-maintenance-silence.sh list --json shows at least one active CI-scoped silence — see the script's header for the full rationale. It polls nothing on its own; after it starts the refresh, poll aws autoscaling describe-instance-refreshes until Successful as before. If the silence from step 0 is close to lapsing before this step finishes, extend it (ci-maintenance-silence.sh create again, same duration) — the wrapper will refuse the refresh once the silence it's checking against has expired. - terraform/ci-monitoring's Prometheus scrape targets for node-woodpecker and woodpecker are a fourth dependent, in a separate Terraform root. As of #4433 (merged 2026-08-09), the WP server's private IP / instance ID are no longer hand-maintained Terraform variable defaults — terraform/ci-monitoring/main.tf reads both live from infra/ci's state via a terraform_remote_state data source, so a plain terraform apply in terraform/ci-monitoring always resolves the current value automatically. That consuming instance's own lifecycle { ignore_changes = [user_data_base64] } (terraform/ci-monitoring/main.tf) still means the resolved value does NOT propagate to the already-running box's rendered prometheus.yml — the live-box fix-forward step below is still required every time: ```bash # 1. Get the new WP server private IP: aws ec2 describe-instances --region us-east-2 \ --filters "Name=tag:Name,Values=ci-woodpecker-server" \ --query "Reservations[].Instances[].PrivateIpAddress" --output text

   # 2. Fix-forward on the live ci-monitoring box (replace OLD_IP/NEW_IP): aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \ --document-name AWS-RunShellScript \ --parameters '{"commands":["sed -i \"s/OLD_IP/NEW_IP/g\" /opt/monitoring/prometheus/prometheus.yml"]}' aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \ --document-name AWS-RunShellScript \ --parameters '{"commands":["cd /opt/monitoring && docker compose kill -s HUP prometheus"]}'

   # 3. Re-install node_exporter on the NEW WP server instance — only # needed for an instance that boots from a pre-#4432 user_data (see # Failure mode H). Instances replaced after #4426+#4432 land install # node_exporter automatically at boot; skip this step for those. # The instance ID is resolved automatically (terraform_remote_state, # #4433) — no variable to update first: cd terraform/ci-monitoring && terraform apply && terraform output node_exporter_install_command Step 4 ("update the Terraform defaults so a future ci-monitoring box replacement bakes in the right values") from before #4433 no longer exists — there is nothing to hand-edit anymore. **Note:** the `terraform apply` above requires `infra/ci`'s state to already reflect the new instance (i.e. run this *after* the `infra/ci` replace in steps 3-4 has completed) and s3:GetObject on `infra/ci`'s state object (`moosequest-tf-state-us-east-2`, bucket in a different region/backend from this stack's own — see `terraform/ci-monitoring/main.tf` comment). **Missed entirely on 2026-08-07** — this produced a real critical `NodeDown` page re-firing hourly for ~2 hours after the replace, because no alert rule watches "is this Prometheus target itself stale," only "is the currently-configured target reachable." See `docs/incidents/2026-08-07-ci-alertmanager-maintenance-noise.md`. 6. **Fix forward on `wp-config-svc` / any other out-of-band container** — only needed for an instance that boots from a pre-#4426 `user_data` (see Failure mode E). As of #4426, `user_data_server.sh.tpl` wires an ECR `docker login` as `ExecStartPre=` on `woodpecker.service`, so instances replaced after #4426 lands do not hit this. If you're validating an older instance and it crash-loops with `no basic auth credentials`, fix forward manually: aws ssm send-command --region us-east-2 --instance-ids \ --document-name AWS-RunShellScript \ --parameters '{"commands":["aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 521228113048.dkr.ecr.us-east-2.amazonaws.com","systemctl restart woodpecker"]}' 7. **Validate the golden path, not just health checks:** - `curl` or `GET /api/user` with the admin token against `https://ci.moosequest.net` — confirms ALB routing, not just the instance being `running`. - `GET /api/agents` and check for a recent `last_contact`, or (more reliably) trigger one real pipeline and confirm it moves from `pending` to `running` within the agent's boot time — confirms the agent fleet actually reconnected, not just that the ASG shows `InService`. - For pip-cache changes specifically: `journalctl -u devpi-server` on `ci-monitoring` during the pipeline run, looking for real `GET /root/pypi/...` traffic — more reliable than pipeline step logs, which are often `--quiet` and don't show the resolved index URL. - Prefer triggering a narrow, single-workflow pipeline over a full manual trigger on `develop` if one is available — a full manual trigger re-runs every workflow in the monorepo definition (60+ workflows, including `deploy-*` steps), which is a much larger blast radius than intended for a verification step. 8. **Expire the maintenance silence and verify alerts re-armed.** Do this last, only after step 7 confirms the golden path — do not leave the silence running past the actual window, and do not let it silently expire on its own without checking that nothing is left broken underneath it:bash scripts/ops/ci-maintenance-silence.sh list # confirm the silence you created scripts/ops/ci-maintenance-silence.sh expire-all # or expire <id> for a specific one Then confirm the underlying alert state is actually healthy now that nothing is masking it:bash aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \ --document-name AWS-RunShellScript \ --parameters '{"commands":["curl -s http://localhost:9093/api/v2/alerts"]}' `` Expect zero CI-scoped alerts in the response (seedocs/ops/runbooks/ci-monitoring.md§Maintenance silences for the allowlist). If anything CI-scoped is stillactive`, you have a real, unmasked problem — diagnose it now, don't let the silence's expiry be the thing that "resolves" it by going stale itself.

How to tell it's broken

• Symptom 1: docker pull from a CI agent hangs or falls back to slow/direct Docker Hub fetches — check /etc/docker/daemon.json on the agent for the registry-mirrors entry; confirm the mirror is reachable.
• Symptom 2: NAT data-processing cost trends back up toward pre-cache levels (~$67/mo) — Cost Explorer, EC2 - Other service, NatGateway-Bytes usage type, tag-filtered to Component=ci-woodpecker.
• Symptom 3: registry-mirror or devpi-server container/service not running on ci-monitoring (i-0a656c6631ddc45b3).
• Symptom 4: pip installs in pipeline steps error with an SSL/trusted-host warning once WOODPECKER_ENVIRONMENT is activated — trusted-host misconfiguration.

How to diagnose (in order)

1. Confirm the registry mirror is up: aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \ --document-name AWS-RunShellScript \ --parameters '{"commands":["curl -s -o /dev/null -w httpcode:%{http_code} http://localhost:5000/v2/"]}' Expected: httpcode:200.

2. Confirm devpi is up: aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \ --document-name AWS-RunShellScript \ --parameters '{"commands":["curl -s -o /dev/null -w httpcode:%{http_code} http://localhost:3141/root/pypi/+simple/"]}' Expected: httpcode:200.

3. Confirm an agent's Docker daemon has the mirror configured: aws ssm send-command --region us-east-2 --instance-ids <agent-instance-id> \ --document-name AWS-RunShellScript \ --parameters '{"commands":["cat /etc/docker/daemon.json","docker info | grep -A2 \"Registry Mirrors\""]}' Expected: registry-mirrors points at http://10.42.10.76:5000 and docker info confirms it's active.

4. Cross-check NAT data-processing trend in Cost Explorer (tag Component=ci-woodpecker, usage type NatGateway-Bytes) — should trend down over the following billing cycle.

Known failure modes

Failure mode A: registry-mirror container not running

Symptom: docker pull on an agent times out or falls through to slow direct Docker Hub fetch (no error — Docker silently continues without the mirror if it's unreachable, just without the caching benefit). Cause: registry-mirror container crashed or ci-monitoring instance rebooted without the container restarting (should not happen — restart: unless-stopped — but check). Fix:

aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["cd /opt/monitoring && docker compose up -d registry-mirror","docker compose ps registry-mirror"]}'

Verification: docker compose ps registry-mirror shows Up; curl http://localhost:5000/v2/ returns 200.

Failure mode B: devpi-server systemd unit not running

Symptom: pip installs (once WOODPECKER_ENVIRONMENT is live) fail with connection refused to 10.42.10.76:3141. Fix:

aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["systemctl restart devpi-server","systemctl status devpi-server --no-pager"]}'

Verification: systemctl status devpi-server shows active (running); curl http://localhost:3141/root/pypi/+simple/ returns 200.

Memory note (#4435, 2026-08-13): as of this date devpi-server.service carries MemoryHigh=768M/MemoryMax=1024M (baked into cloud-init.sh.tpl §11b, applied live). Prior to this, devpi-server's RSS was uncapped and grew to 1.17 GB (63% of the box) over its first ~8 days, driving a 6-day MemoryPressureHigh alert. If devpi-server crash-loops under the new cap (systemctl status devpi-server shows Main PID exited, code=killed, status=9/KILL repeatedly), that means its steady-state working set now genuinely exceeds 1024M — raise the cap rather than removing it; see docs/ops/runbooks/ci-monitoring.md §Memory-pressure hardening and Failure mode K, and docs/incidents/2026-08-13-ci-monitoring-devpi-memory-pressure.md.

Failure mode C: build-cache EBS volume didn't attach on instance replacement

Symptom: /mnt/ci-cache doesn't exist; registry-mirror and devpi fall back to writing on the root volume (still functions, just not on the dedicated volume — see build-cache.tf header comment and the cloud-init warning message). Cause: aws_volume_attachment.ci_cache didn't reattach after an instance replacement (EBS volumes are NOT part of the instance's launch template — they must be explicitly reattached if ci-monitoring is ever replaced). Fix: terraform apply in terraform/ci-monitoring/ reattaches the existing volume (it is NOT destroyed on instance replacement, only detached). Verification: df -h /mnt/ci-cache shows the 30 GB volume mounted.

Failure mode D: cache serving stale/vulnerable base image layers

Symptom: A security patch to a base image (e.g. python:3.11-slim) isn't showing up in CI builds days after Docker Hub published it. Cause: registry:2.8.3 does NOT support proxy.ttl via environment variable (confirmed via a startup warning: Ignoring unrecognized environment variable REGISTRY_PROXY_TTL — see terraform/ci-monitoring/templates/cloud-init.sh.tpl for the full note). Cached tags are effectively cached indefinitely until the underlying content digest changes are surfaced through a HEAD-check, which itself may not force a re-pull under all conditions. Fix: Force a full cold cache:

aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["cd /opt/monitoring && docker compose down -v registry-mirror && docker compose up -d registry-mirror"]}'

This wipes the entire cache volume-backed data for the container (safe — pure cache, no source-of-truth data lost) and starts fresh; the next pull per-image will be a genuine cache miss and re-fetch from Docker Hub. Verification: docker pull <image> on an agent, check registry-mirror logs for a fresh Challenge established with upstream entry for that image.

Failure mode E: WP server crash-loops after a template-driven replace — no basic auth credentials

Status (as of #4426, merged 2026-08-09): fixed at the template level going forward. user_data_server.sh.tpl now wires aws ecr get-login-password | docker login as ExecStartPre= on the woodpecker.service systemd unit, so it re-runs on every service start (boot, daily restart, manual restart) — not just first boot, since ECR auth tokens expire ~12h. This applies automatically starting with the server's next replace (user_data changes are guarded by lifecycle { ignore_changes = [user_data] } — the currently-running instance was not replaced by #4426 and does not carry this fix until it next boots from a fresh user_data). The failure mode and manual fix-forward below remain accurate for any instance still running the pre-#4426 template.

Symptom: After a standalone server replace, journalctl -u woodpecker.service shows repeated restarts (Scheduled restart job, restart counter is at N) with docker[...]: Error response from daemon: Head "https://<account>.dkr.ecr.us-east-2.amazonaws.com/v2/ raxx-wp-config-svc/manifests/latest": no basic auth credentials. Cause: Pre-#4426, user_data_server.sh.tpl did not perform an ECR login before docker compose up -d. If the previously-running wp-config-svc container was authenticated out-of-band (manually, or by some process not captured in the template), the fresh instance had no ECR credentials cached and the pull failed, taking woodpecker-server down with it (both services are declared in the same docker compose up invocation, so one pull failure blocks both). Fix:

aws ssm send-command --region us-east-2 --instance-ids <instance-id> \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 521228113048.dkr.ecr.us-east-2.amazonaws.com","systemctl restart woodpecker"]}'

Verification: docker ps shows both woodpecker-server and wp-config-svc containers Up ... (healthy). Note: aws ecr get-login-password issues a short-lived (12h) IAM token, not a stored secret — acceptable as an interim fix-forward on a pre-#4426 instance.

Failure mode F: server replaced but public URL still down — ALB target group empty

Symptom: New instance is running and docker ps shows healthy containers, but https://ci.moosequest.net times out or the ALB shows a 5xx, and aws elbv2 describe-target-health on ci-woodpecker-server-tg returns an empty TargetHealthDescriptions list. Cause: A -target=aws_instance.woodpecker_server-scoped terraform apply does not cascade to aws_lb_target_group_attachment.woodpecker_server even though it directly references the instance's ID — the attachment resource is a separate node in the dependency graph and is out of scope for a single-resource -target apply. Fix:

terraform apply -target=aws_lb_target_group_attachment.woodpecker_server

Verification: aws elbv2 describe-target-health shows the new instance ID with TargetHealth.State = healthy; GET /api/user against https://ci.moosequest.net with the admin token returns 200.

Failure mode G: pipelines queue but never run after a server replace

Symptom: New pipelines sit in pending indefinitely; GET /api/agents shows no agent with a recent last_contact. Cause: infra/ci/agents.tf bakes aws_instance.woodpecker_server.private_ip into the agent launch template's user_data at apply time (static-mode server discovery). A -target-scoped server replace changes the instance's private IP but does not update the agent launch template, so any already-running (or newly spot-launched, until the LT is fixed) agent keeps trying to reach the destroyed instance's dead IP over gRPC. Fix:

terraform apply -target=aws_launch_template.woodpecker_agent
scripts/ops/ci-asg-refresh.sh

Note: the ASG's instance_refresh block does NOT auto-trigger from the launch-template update alone — version = "$Latest" is a literal string that produces no diff on the aws_autoscaling_group resource itself, so the refresh must be started explicitly. Use scripts/ops/ci-asg-refresh.sh (#4436) rather than a bare aws autoscaling start-instance-refresh — it wraps the same call but REFUSES to run unless scripts/ops/ci-maintenance-silence.sh shows an active CI-scoped silence (create one first if this fix is being applied outside an already-silenced maintenance window: scripts/ops/ci-maintenance-silence.sh create --duration 15m --comment "agent refresh fix-forward, #<issue>, <name>" --author "<name>"). ci-asg-refresh.sh --dry-run validates the silence check without starting a refresh. Verification: aws autoscaling describe-instance-refreshes reaches Successful; trigger a pipeline and confirm it transitions from pending to running within the agent's boot time (~2-3 min).

Failure mode H: NodeDown keeps re-paging hours after a server replace that otherwise succeeded

Status (as of #4432 + #4433, both merged 2026-08-09): both halves of this failure mode have a structural fix; one manual step remains. - node_exporter-not-installed half — fixed at the template level going forward. user_data_server.sh.tpl now installs and enables node_exporter v1.8.2 (arm64) on every boot, mirroring terraform/ci-monitoring/outputs.tf's node_exporter_install_command exactly, so a fresh WP server instance no longer needs the manual post-apply SSM step. Applies starting with the server's next replace (see the ignore_changes caveat in Failure mode E above — the currently-running instance does not carry this fix yet). - Hand-maintained-scrape-target-IP half — fixed. terraform/ci-monitoring no longer hardcodes var.wp_server_private_ip / var.wp_server_instance_id as Terraform variable defaults (#4433). Both are read live from infra/ci's state via a terraform_remote_state data source (terraform/ci-monitoring/main.tf), so a terraform apply on terraform/ci-monitoring always resolves the current WP server identity — no second hardcoded default to remember to bump. - Still manual: the already-running ci-monitoring box's rendered prometheus.yml does not update itself — that instance's own lifecycle { ignore_changes = [user_data_base64] } means a resolved Terraform value never reaches the live box automatically. The sed + docker compose kill -s HUP prometheus fix-forward below is still required after every WP server replace, by design (see this card's non-goal note — force-refreshing the live box's already-running Prometheus config was explicitly out of scope for #4433).

Symptom: The golden path is fine (WP UI reachable, agents reconnected, pipelines running), but a critical NodeDown (or MemoryPressureHigh/ DiskUsageHigh) alert for instance=ci-woodpecker-server keeps firing/ re-notifying every repeat_interval (1h for critical) long after the replace window closed. Cause: The already-running ci-monitoring box's Prometheus config was rendered from whatever WP server private IP was live at that box's own last boot — a static scrape target baked into prometheus.yml at that time. A WP server replace changes the WP server's IP, but nothing in the replace procedure pushes the new IP to the already-running monitoring box (its lifecycle { ignore_changes = [user_data_base64] } guard means even a terraform apply that resolves the correct new value via terraform_remote_state does not reach the live box's rendered config), nor does it re-run the node_exporter install on the new WP instance (it's a fresh EC2; node_exporter is not baked into the AMI, except on instances replaced after #4426/#4432 land). The target keeps trying to reach the destroyed instance's dead IP — connection refused, not a timeout, since nothing is listening at that address on the new network path — indefinitely, until someone notices and fixes it manually. This is exactly what happened 2026-08-07: caught ~2h after the replace via a live Alertmanager query, not via any automated detection. Fix: See step 5's fourth dependent-resource block above (fix-forward on the live box: update prometheus.yml target IP + docker compose kill -s HUP prometheus; reinstall node_exporter on the new WP instance via the node_exporter_install_command Terraform output, whose instance ID now resolves automatically via terraform_remote_state — no variable to edit first). Verification: curl http://localhost:9090/api/v1/targets (via SSM, port 9090 not exposed) shows node-woodpecker and woodpecker jobs up; curl http://localhost:9093/api/v2/alerts shows no active NodeDown for ci-woodpecker-server. Why this wasn't caught by the maintenance silence: the silence (step 0) was correctly scoped to the window — it expires. This bug outlives the window because nothing re-checks the target's correctness after the silence expires; the silence masked it during the authorized outage, then it kept firing for real afterward. Step 8's "verify alerts re-armed to a genuinely clean state, not just that the silence expired" check exists specifically to catch this class.

Verification performed at deployment (2026-08-05)

Confirmed cache-hit behavior via a controlled test on the live agent fleet (i-039120644f75aabde):

1. docker rmi python:3.11-slim && docker pull python:3.11-slim (cold) — each blob GET logged Challenge established with upstream + Adding new scheduler entry on registry-mirror (genuine fetch from Docker Hub).
2. docker rmi python:3.11-slim && docker pull python:3.11-slim (warm) — NO upstream challenge on the second pull; blob response durations dropped 4-39x (e.g. a 14.4 MB blob: 410ms → 49ms; a 29.7 MB blob: 436ms → 102ms).

This is the definitive cache-hit signature: identical bytes served locally without a second round-trip to Docker Hub.

Emergency stop

To disable the mirror and fall back to direct Docker Hub / PyPI access (NAT egress, no caching, but always works):

# On the agent(s): remove registry-mirrors from daemon.json and restart docker
aws ssm send-command --region us-east-2 --instance-ids <agent-instance-id> \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["rm -f /etc/docker/daemon.json","systemctl restart docker","systemctl restart woodpecker-agent"]}'

Or set docker_registry_mirror_url = "" in infra/ci/terraform.tfvars and terraform apply — the next agent instance refresh will boot without the registry-mirrors config (see the %{ if docker_registry_mirror_url != "" } guard in infra/ci/templates/user_data_agent.sh.tpl).

Escalation

Wake the operator when: - NAT data-processing cost trends back toward pre-cache levels for more than one billing cycle (cache is silently not working). - The build-cache EBS volume fills (df -h /mnt/ci-cache > 85%) — grow the volume (cache_volume_size_gb in terraform/ci-monitoring/variables.tf, online resize, no downtime) before adding an eviction policy. - Any change here would require touching the WP server's live config — escalate rather than SSM-patching production; use the documented intentional-replace path in infra/ci/server.tf instead.

Contact: ops@raxx.app