CI agent instance sizing (#4545)

Symptom

The Woodpecker CI heavy-build agent class (c7a.xlarge / c6a.xlarge / c5a.xlarge, all 4 vCPU / 8 GB) OOMs on memory-bound compile lanes even with WOODPECKER_MAX_WORKFLOWS=1 and a 6 GiB per-container Docker memory cap (WOODPECKER_BACKEND_DOCKER_LIMIT_MEM / ..._MEM_SWAP, applied live 2026-08-29 as a stopgap).

Evidence

• dmesg on a live agent instance showed a single rustc process (compiling the avenue-api test binary, MooseQuest-LLC/project-avenue) peaking at anon-rss 7.29 GB, with no co-scheduling — the box has only 7.7 GB total.
• Project Avenue's actual working-set requirement for that lane is ~8 GiB total: ~7.5 GiB for the single rustc process plus ~0.5 GiB for postgres/cargo overhead. The current 6 GiB container cap fails deterministically (safely, but every time) for this lane.
• Project Avenue independently measured CARGO_BUILD_JOBS=1 only gets peak RSS down to 6.3 GiB, at 3.5x wall time — codegen-unit tuning changes nothing, because this is frontend/monomorphization memory, not parallelism. No repo-side env change alone fixes this reliably.
• TradeMasterAPI's own Queue C++ container build is the other memory-heavy lane sharing this same agent fleet.

Conclusion: this needs either swap or a larger instance class. A swapfile alone is a backstop, not a fix — it trades OOM-kills for a documented 3.5x-wall-time-class slowdown once the working set spills past physical RAM (the CARGO_BUILD_JOBS=1 data point above is the closest proxy for what swap thrashing would feel like on this same compile). An EBS volume by itself is not a fix either — disk isn't RAM.

Pricing (us-east-2, priced 2026-08-29/31 via aws pricing + `aws ec2

describe-spot-price-history`; spot fluctuates, treat as a live reference point not a locked-in number)

"Monthly @ 2x on-demand floor" is a costing reference point per the task brief, not the actual fleet policy — the fleet keeps its documented mixed policy (on_demand_base_capacity = 1, spot burst above the floor, price-capacity-optimized allocation); the "1x OD + 1x spot" column is closer to today's real steady-state cost shape.

Recommendation

Leanest: m6a.xlarge (4 vCPU / 16 GB). Keeps the same vCPU count as the current fleet (no parallel-compile regression for the C++ build), and its 16 GB fixes the OOM with real margin: cap the container at 14 GiB (16 GB − 2 GiB host reserve for dockerd/agent/SSM), which sits ~6 GiB above the observed ~8 GiB avenue-api working set.

• agent_instance_types = ["m6a.xlarge"]
• agent_docker_limit_mem_bytes = 15032385536 (14 GiB) — sets both WOODPECKER_BACKEND_DOCKER_LIMIT_MEM and ..._MEM_SWAP

One step up: r6a.xlarge (4 vCPU / 32 GB). Same vCPU as "lean," double the RAM. Cap the container at 30 GiB (32 GB − 2 GiB host reserve), ~22 GiB above the working-set target — headroom for the working set to grow, or for eventually raising WOODPECKER_MAX_WORKFLOWS above 1 once co-scheduling is otherwise addressed.

• agent_instance_types = ["r6a.xlarge"]
• agent_docker_limit_mem_bytes = 32212254720 (30 GiB)

Both options are wired as a paired, documented two-line change in infra/ci/terraform.tfvars.example (commented "option A: lean" / "option B: headroom" blocks) and in infra/ci/variables.tf (agent_instance_types and agent_docker_limit_mem_bytes descriptions). Picking one is a two-line tfvars edit; no other files need to change.

r6a.large (2 vCPU / 16 GB) was considered — it's the cheapest option that clears the memory bar — but it halves vCPU count vs. the current 4-vCPU fleet, which risks slowing the TradeMasterAPI C++ build's parallel compilation. Rejected in favor of keeping vCPU count constant. c6a.2xlarge (8 vCPU / 16 GB) was considered and rejected: it pays for double the vCPU count for a workload that's memory-bound, not CPU-bound, without adding meaningfully more RAM headroom than m6a.xlarge.

Swapfile backstop (either option)

Not required for either recommended option — both fit the ~8 GiB working set with real headroom. If added anyway as a defense-in-depth backstop against future working-set growth, a 4-8 GB swapfile would need to be added in infra/ci/templates/user_data_agent.sh.tpl (this instance class has none today) — e.g., a fallocate + mkswap + swapon step alongside the existing Docker install step, before the agent's .env/compose files are written. Be honest with the operator about the tradeoff: swap is a backstop, not a substitute for RAM sizing — Project Avenue's own CARGO_BUILD_JOBS=1 measurement (3.5x wall time to shave ~1 GB off peak RSS) is the closest proxy for how much a swap-driven slowdown would cost here if the working set spills past physical RAM under either recommended class.

Rollout (deferred — NOT part of this PR)

The actual cutover is a separate, later, operator-scheduled action:

1. Announce a ci-maintenance-silence window.
2. Land one of the two tfvars option blocks above (uncomment, comment out the "current" block).
3. terraform plan — expect a launch template replacement and an ASG instance refresh (instance_refresh block already handles rolling replacement, min_healthy_percentage = 0).
4. terraform apply during the window.
5. Confirm agents register, run a canary build of each heavy lane (queue C++ build, avenue-api cargo build), watch CloudWatch for OOM events.
6. Close the silence window.

This PR does not perform any of the above — it only lands the priced options and the tfvars/variable plumbing to make picking one a two-line change when the operator schedules the cutover.

Follow-ups

• infra/ci Terraform still shows min_size = 1 while the live ASG floor was corrected to MinSize=2/Desired=2 by a separate scale-arbiter fix tonight — tracked on #4545, out of scope here, but terraform plan will show this drift.
• Consider revisiting WOODPECKER_MAX_WORKFLOWS back above 1 once the chosen class's RAM headroom is confirmed stable across a few weeks of real builds — that's a toil-reduction/throughput win but should be its own follow-up change, not bundled into this sizing fix.