Raxx · internal docs

internal · gated

ADR 0134 — CI compute: Woodpecker CI control plane + spot EC2 agents

Status: Accepted Date: 2026-07-03 UTC Decision date: 2026-07-03 UTC Deciders: Kristerpher (operator); software-architect (design) Scope: CI infrastructure; all 85 .github/workflows/*.yml files; runner compute Refs: #726 (billing posture), #728 (Ubicloud trigger card), #1595 (CodeBuild card), ADR-0033 (superseded), docs/architecture/ci-self-hosted-ec2-decision-2026-07-03.md

Context

ADR-0033 (April 2026) set a threshold: migrate to Ubicloud managed runners when monthly CI burn exceeds 5,000 min/month for two consecutive months. Since then, the workflow inventory grew from ~34 to 85 files. A 2026-07-03 run-density sample estimates current burn at ~25,700 min/month (~$186/month), well above the trigger.

The operator directive (2026-07-03) escalates beyond Ubicloud (a managed runner service): the requirement is to move compute to operator-owned EC2 at ci.moosequest.net, production-ready, with a load balancer, defense-in-depth, and a phased build plan. Two candidates were evaluated (see the decision package linked above):

The invariants relevant to this decision: - Ephemeral runners (no stored credentials between jobs) - AWS-resident workload secrets in SSM Parameter Store, not Infisical - moosequest.net DNS stays on Oracle Dyn; no CF Access for ci.moosequest.net - claude-infisical-bootstrap IAM user must not be widened

Decision

Adopt Candidate B: replace the CI control plane entirely with Woodpecker CI (Apache 2.0) running on EC2, with a spot agent fleet in the same VPC.

Operator rationale (recorded 2026-07-03): Full independence from the GitHub Actions control plane is the primary driver. Sovereignty over the entire CI stack is valued over the 6-9 week migration cost. The operator explicitly accepted the migration cost and the ~$107/month running cost.

Locked constraints (operator-confirmed, not open for re-evaluation)

  1. iOS: DEFERRED entirely. ios-ci.yml is disabled now (separate card). No macOS runners anywhere in the new stack. iOS CI will be re-evaluated post-migration independently.
  2. ALB auth: Google Workspace OIDC. The Woodpecker UI is gated by ALB OIDC using the operator's existing Google Workspace account (passkey enforced at Google account level). No alternative IdP.
  3. AWS region: us-east-2. Existing AWS account; no new account provisioned.
  4. Cost envelope: ~$60-100/month target; operator accepted the ~$107/month Candidate B estimate (slightly exceeds envelope; accepted explicitly).

Production architecture (Candidate B)

Networking: - VPC 10.42.0.0/16 (us-east-2) - Public subnets A+B: ALB only - Private subnets A+B: Woodpecker control plane (server + state DB) + agent fleet + NAT egress

ALB: - ACM TLS certificate for ci.moosequest.net (DNS validation via Dyn CNAME) - HTTP→HTTPS redirect - Listener port 443: - Default rule: Google OIDC authenticate → forward to Woodpecker UI (port 8000) - Path rule /hook*: bypass OIDC → forward to Woodpecker server; Woodpecker validates its own HMAC signature on all webhook payloads - AWS WAF: CommonRuleSet + LinuxRuleSet + rate limit (100 req/5min/IP non-hook; 500 req/5min/IP on /hook) - Session cookie: 8-hour duration; scopes: openid email

Woodpecker server: - t4g.small EC2 instance (ARM64, ~$12/month on-demand; not spot — control plane must stay up during agent job execution) - Docker Compose: woodpeckerci/woodpecker-server:latest + state DB sidecar - State DB: smallest RDS (db.t4g.micro, Postgres 15) OR litestream SQLite→S3 — engineering team picks based on operational preference; both are valid - GitHub OAuth App registered at github.com/settings/developers for SCM integration (operator browser step: create app, paste client ID + secret into SSM) - Woodpecker server env vars sourced from SSM Parameter Store at startup

Agent fleet: - Spot Auto Scaling Group: c5.xlarge / c5a.xlarge, min=0, max=10, ephemeral, scale-to-zero - Heavy-job label (size:heavy): c5.2xlarge on-demand, min=0, max=2 - Amazon Linux 2023 + Docker (latest AMI at apply time) - Woodpecker agent token from SSM; no inter-job credential persistence (agent container restarts between pipelines; workspace volume mounted per-pipeline then deleted) - IMDSv2 required (HttpTokens = required); no SSH key pair; SSM Session Manager only

SSM Parameter Store (SecureString, KMS aws/ssm CMK): - /ci/woodpecker/github-client-id — GitHub OAuth App client ID - /ci/woodpecker/github-client-secret — GitHub OAuth App client secret - /ci/woodpecker/agent-secret — shared agent-to-server token - /ci/woodpecker/webhook-secret — Woodpecker HMAC secret for GitHub webhook - /ci/oidc/client-id — Google OIDC client ID (ALB) - /ci/oidc/client-secret — Google OIDC client secret (ALB)

CloudWatch: - Log groups: /ci/woodpecker-server, /ci/agent-fleet, /ci/waf (all 90-day retention) - Alarm: agent ASG at max capacity for 10+ minutes → ops@ daily digest - Alarm: EC2 daily spend > $10 → ops@ digest

IaC: infra/ci/ in this repo; Terraform; migrates to raxx-infra repo at Phase 5 when a second MooseQuest repo onboards.

Explicit operator-action checkpoints (Phase 1 card — not separate operator-action cards)

  1. Dyn CNAMEs (two): _acm-challenge.ci.moosequest.net → <ACM validation CNAME> after first terraform apply; then ci.moosequest.net → <ALB DNS name> after ACM issues. Estimate 15 minutes in Oracle Dyn console.
  2. GitHub OAuth App (browser): Create OAuth App at github.com/settings/developers (Authorization callback URL: https://ci.moosequest.net/authorize); copy client ID and secret into SSM parameters /ci/woodpecker/github-client-id and /ci/woodpecker/github-client-secret. Estimate 5 minutes.

Phased Build Plan

Phase 0 — Design sign-off (complete)

This document. Operator reviewed candidates, confirmed Candidate B (2026-07-03). Constraints locked.

Phase 1 — Terraform baseline

Card: #4012 (amended for Candidate B scope). Full Terraform for VPC, ALB, WAF, IAM, SSM stubs, Woodpecker server on t4g.small, state DB, agent spot ASG.

Gate: Woodpecker UI reachable at https://ci.moosequest.net after both operator Dyn CNAME steps and GitHub OAuth App creation.

Rollback: terraform destroy. No workflows migrated yet; zero blast radius on CI.

Phase 2 — Woodpecker pilot (3 low-risk cron workflows)

Card filed. Onboard 3 cron workflows (no service containers, no deploy consequences, no required-status-check entries). 48-hour soak. Success = green runs + GitHub commit status checks reported back via Woodpecker GitHub integration.

Rollback: re-enable the GHA cron trigger; zero blast radius.

Phase 3a — PR-gate and push-CI workflow migration

Card filed. Translate PR-gate workflows to .woodpecker/ format. Port all 5 composite actions to Woodpecker plugin/step equivalents (load-vault-secrets is the critical path). Wire required-status-checks to Woodpecker GitHub status integration.

Phase 3b — Heavy CI migration (backend-tests-postgres)

Card filed. Depends on AL2023+Docker postgres:15 service container smoke (OQ-6) passing. Migrate ci.yml backend-tests-postgres (90-min class) and related PR/console CI.

Phase 4 — Deploy/release pipeline (highest risk)

Card filed. Migrate gatekeeper, release pipeline, and all deploy workflows. Per-workflow rollback notes in card body.

Phase 5 — Decommission + /onboard-ci skill

Card filed. Disable all remaining GHA workflows (ios-ci.yml already disabled). Drop GHA spending limit to $20/month (iOS buffer). Build /onboard-ci skill per decision doc Section 9. Close #728; update #726.

iOS-defer — Disable ios-ci.yml now

Card filed. Independent of migration; implements the iOS DEFERRED constraint.

Language choice rationale

Woodpecker server + agents: Pre-built container images (woodpeckerci/woodpecker-server, woodpeckerci/woodpecker-agent). Operator does not author this code; Go is the upstream language. Not in Raxx language tier policy (infrastructure-layer, not operator-authored).

State DB: Postgres 15 (or SQLite via litestream). Engineering-team choice; both natively supported by Woodpecker.

IaC: Terraform HCL. Infrastructure definition; outside language tier policy scope.

/onboard-ci skill: Shell script (scripts/skills/onboard-ci.sh). Appropriate for a one-off idempotent orchestration script with no latency or memory-safety requirements.

Consequences

Positive

Negative / risks

Neutral

Alternatives considered

Candidate A — GitHub Actions control plane + EC2 runners

Evaluated. Rejected because: - Control plane remains GitHub-hosted. A GH Actions outage blocks all CI. This violates the sovereignty requirement the operator stated as the primary driver. - Migration effort is lower (3-4 days), but the sovereignty gap is not closed. - Saves $42/month more than Candidate B, but operator explicitly accepted the cost differential.

Buildkite (SaaS control plane + self-hosted agents)

Excluded: Buildkite control plane is a SaaS service; pipeline scheduling and logs transit through Buildkite infrastructure. Does not satisfy the full-sovereignty requirement.

AWS CodeBuild native (issue #1595)

Evaluated. AWS CodeBuild routes jobs through managed compute. Sovereignty is partial; CodeBuild is a managed control plane. Rejected on same grounds as Candidate A.

Ubicloud managed runners (ADR-0033 trigger)

Superseded. Ubicloud is a managed SaaS runner service. Does not satisfy the full independence requirement per the operator directive.

Security / GDPR checklist

Revisit when