CI Self-Hosted EC2 — Decision Package

Date: 2026-07-03 UTC Author: software-architect Status: Proposed — awaiting operator decision ADR: 0134-ci-ec2-self-hosted-runner-fleet.md Supersedes: ADR-0033 (Ubicloud trigger threshold, now clearly exceeded) Parent issues: #726 (billing posture), #728 (Ubicloud migration card), #1595 (CodeBuild migration)

1. Context

GitHub Actions is the current CI platform. Since ADR-0033 was written (April 2026), the workflow inventory has grown from ~34 to 85 files. Monthly minutes are now estimated at 15,000–26,000 (basis: measured run density below), well past the 5,000 min/month Ubicloud trigger threshold set in ADR-0033. The operator wants to migrate compute to ci.moosequest.net on EC2 with defense-in-depth, a load balancer in front, and the design decision made here, not in a sprint.

The billing API returns 404/410 for this repo configuration. Monthly cost is estimated from run-history timestamps (GH API, 2026-07-03 sample). Billing numbers below are estimates, not actuals; OQ-5 flags how to close this.

2. Measured Workload (Ground Truth)

Workflow inventory (85 files)

Note: counts overlap because most workflows have multiple triggers.

Runner types

Service containers

postgres:15 appears in at least 10 workflows (ci.yml, ci-pr.yml, alembic-version-cron, billing-collector-cron, freescout-backup, mbt-* crons, review-app-console, bcp-smoke-monthly). Any solution must support Docker service containers on the runner.

Heaviest jobs

Composite actions (5 actions, 22 workflows depend on load-vault-secrets)

• ./.github/actions/load-vault-secrets — Infisical vault.raxx.app via CF Access headers
• ./.github/actions/notify-deploy-status
• ./.github/actions/emit-audit-event
• ./.github/actions/cloudflare-retry
• ./.github/actions/check-deploy-freeze

Load-bearing release pipeline

Estimated monthly burn

From 2026-07-03 GH API sample (100 runs, ~6.5-hour window):

• Observed density: ~15 runs/hour across all trigger types
• Median run duration: ~2 min (most lint/monitor workflows run 1-2 min)
• Heavy jobs (backend-tests-postgres): ~2 runs/day × 78 min = 156 min/day
• Short runs: ~350/day × 2 min = 700 min/day
• Estimated daily: ~856 min/day; monthly: ~25,700 min/month

At GitHub Pro ($4/mo, 3,000 min included, $0.008/min overage): - Overage: 22,700 min × $0.008 = ~$182/mo + $4 base = ~$186/mo

This estimate may be ±40%. OQ-5 asks the operator to confirm from the billing page.

3. Invariants That Apply

• No stored credentials. Runner instances must not persist any credential between jobs. Enforced by ephemeral-runner design (each EC2 instance terminates after one job).
• Passkeys / WebAuthn only for Raxx product surfaces. ci.moosequest.net is an internal ops tool on the moosequest.net domain, not a Raxx customer surface. Its auth layer must enforce 2FA at the IdP but is not required to be WebAuthn-native at the ALB layer. See OQ-3.
• Credentials into infra, not code. GH App private key, webhook HMAC, and OIDC client secret live in SSM Parameter Store only. Nothing in Terraform files or workflow YAML.
• Audit trail for state changes affecting money, permissions, or data access. CloudWatch logs for all runner lifecycle events; retention 90 days; structured JSON. Build logs are developer-artifact data, not customer PII.
• AWS-resident workload secrets in SSM, not Infisical. The runner fleet is AWS-resident. Its secrets go into SSM Parameter Store. Runners still call Infisical at build time via load-vault-secrets, unchanged from today.
• moosequest.net DNS stays on Oracle Dyn. ci.moosequest.net is a Dyn CNAME to the ALB DNS name. No zone migration. No CF Access available for this hostname.

4. Candidate A — GHA Orchestration + EC2 Runner Fleet

GitHub Actions remains the CI control plane. Compute migrates to self-hosted ephemeral EC2 runners managed by the philips-labs/terraform-aws-github-runner autoscaler.

ci.moosequest.net becomes the runner fleet observability dashboard (CloudWatch dashboards, spend tracking, runner utilization) — not a CI control plane UI.

Flow

GitHub webhook (workflow_job.queued event)
  → API Gateway → Lambda (webhook processor)
    → SQS → Lambda (runner provisioner)
      → EC2 spot instance spins up (ephemeral)
        → runner registers via JIT token (job-scoped, one-time)
          → job runs
            → runner deregisters → EC2 instance terminates

YAML migration scope

None required. runs-on: ubuntu-latest continues to route to the self-hosted fleet once philips-labs is configured with that label mapping. A separate mechanical PR can add a custom label ([self-hosted, linux, x64]) after the fleet is stable — this does not affect required status check names.

iOS carve-out

ios-ci.yml uses macos-14. EC2 Mac Dedicated Hosts require a 24-hour minimum allocation (~$650+/month). For one iOS job running ~5 min several times per week, this is not economical. ios-ci.yml stays on GH-hosted macOS indefinitely. iOS minutes (macOS-14 at $0.08/min) remain as a small fixed GitHub bill (~$1-2/month).

Cost estimate (Candidate A)

Savings vs current ~$186/mo GHA bill: ~$121/month. Infrastructure build effort (12–20 engineering hours) breaks even within the first month.

Migration effort

Risks

5. Candidate B — Woodpecker CI (Full Platform on EC2)

Woodpecker CI (Apache 2.0, active community fork of Drone CI) replaces the GitHub Actions control plane entirely. The Woodpecker server runs on EC2 in the private subnet. ci.moosequest.net becomes the Woodpecker UI.

Buildkite evaluated and excluded: Buildkite has a SaaS control plane that pipeline logs and scheduling metadata transit through — it is not fully self-hosted. For an operator requirement of full infrastructure ownership at ci.moosequest.net, Buildkite does not qualify. Woodpecker is the correct Candidate B.

What changes

• All 85 .github/workflows/*.yml files require full rewrite to .woodpecker.yml syntax (Woodpecker uses Docker-per-step model; no uses: composite action equivalent)
• 5 composite actions must be ported to Docker images or shell init steps
• Console Ops Dispatch must change from workflow_dispatch GitHub API to Woodpecker API
• Required status check names change (Woodpecker posts via GitHub Checks API but names differ from GHA job names)
• iOS CI hybrid: ios-ci.yml must remain on GH Actions (macOS runner dependency); routing two CI systems from one repo adds permanent complexity

iOS carve-out

Same as Candidate A — macos-14 stays GH-hosted. This makes Candidate B a permanent hybrid: Woodpecker for all Linux workflows, GH Actions for ios-ci.yml only.

Cost estimate (Candidate B)

Savings vs current ~$186/mo: ~$79/month — vs $121/month for Candidate A. Candidate B saves $42/month less while requiring 15-20x more migration effort.

Migration effort

Risks

6. Decision Matrix

Recommendation: Candidate A.

Candidate A saves $42/month more than Candidate B while requiring 15-20x less engineering effort. The migration risk difference is decisive: Candidate A changes nothing in 85 workflow files; Candidate B requires a full rewrite under a live release pipeline. Candidate A's rollback is a one-line PR reverting runs-on. Candidate B's rollback during a 6-9 week migration window is complex. The operator retains full compute ownership while keeping GitHub as the well-understood CI control plane.

7. Production Architecture (Candidate A)

VPC Layout

Region: us-east-1

VPC cidr: 10.42.0.0/16
│
├── Public subnet A  10.42.0.0/24  (us-east-1a)  — ALB only
├── Public subnet B  10.42.1.0/24  (us-east-1b)  — ALB only
│
├── Private subnet A 10.42.10.0/24 (us-east-1a)  — Lambda, EC2 runners
└── Private subnet B 10.42.11.0/24 (us-east-1b)  — Lambda, EC2 runners (AZ failover)

Internet Gateway → public subnets
NAT Gateway (in public subnet A) → private subnet egress
  → GitHub API, Infisical vault.raxx.app, Heroku API, Cloudflare API, PyPI, npm

The ALB sits in front of the observability dashboard only. The GH Actions control plane is GitHub-hosted; there is no private backend behind the ALB for CI scheduling. The Lambda webhook processor is in the private subnet with NAT egress.

Security Groups

No port 22 on any resource. SSM Session Manager is the only shell-access path. IMDSv2 required on all EC2 instances (HttpTokens = required in launch template).

ALB Configuration

• TLS termination: ACM certificate for ci.moosequest.net
• DNS: Oracle Dyn CNAME ci.moosequest.net → <alb-dns-name> (operator adds this record; ACM validation also requires a Dyn CNAME — see OQ-1)
• Port 80 listener: redirect to HTTPS (no backend)
• Port 443 listener:
• Default rule: OIDC authenticate → forward to dashboard target group (CloudWatch embedded dashboard or a lightweight observability shim)
• Path-based rule /webhook*: skip OIDC → forward to API Gateway → Lambda HMAC validation happens in Lambda before any runner action

Webhook auth bypass:

sequenceDiagram
    participant GH as GitHub
    participant ALB
    participant Lambda
    participant SQS
    participant EC2

    GH->>ALB: POST /webhook (X-Hub-Signature-256: sha256=...)
    Note over ALB: Path rule: skip OIDC
    ALB->>Lambda: forward (no auth)
    Lambda->>Lambda: validate HMAC(payload, secret from SSM)
    alt HMAC invalid
        Lambda-->>GH: 401
    else HMAC valid
        Lambda->>SQS: enqueue runner job
        SQS->>Lambda: provision EC2 runner
        EC2->>GH: register via JIT token
        GH->>EC2: dispatch job
    end

AWS WAF (attached to ALB)

WAF logs → CloudWatch → daily ops digest (pre-launch digest posture).

Runner Fleet (philips-labs/terraform-aws-github-runner)

Key parameters:

# Standard Linux runners (ubuntu-latest equivalent)
runner_instance_types = ["c5.xlarge", "c5a.xlarge"]   # 4 vCPU / 8 GB
runner_enable_spot    = true
runner_max_size       = 10   # hard cap; tune after pilot
runner_min_size       = 0    # zero idle cost

# Heavy runner for backend-tests-postgres (90-min job, spot interruption risk)
# Separate runner configuration in Terraform:
heavy_instance_types  = ["c5.2xlarge"]   # 8 vCPU / 16 GB
heavy_enable_spot     = false            # on-demand; can't afford mid-job interruption
heavy_max_size        = 2

# Ephemeral: terminate after one job
runner_enable_ephemeral  = true

# JIT registration (token is job-scoped, single-use)
runner_enable_jit_config = true

# Runner group: private, scoped to raxx-app org
runner_runner_group_name = "ec2-self-hosted"

AMI: Amazon Linux 2023 with Docker pre-installed. Monthly AMI refresh via SSM Automation document (new launch template version → apply). Ephemeral runners always pick up the latest AMI at launch.

Secrets in SSM Parameter Store

All CI infrastructure secrets are AWS-resident; SSM SecureString encrypted with aws/ssm KMS key:

Nothing in Terraform .tf files or workflow YAML. Terraform references SSM paths by ARN; values are never in tfvars or state in plaintext.

GitHub App (raxx-ci-runner)

A new GitHub App, separate from raxx-ops-bot and raxx-pm-bot, following the existing bot identity pattern. Least-privilege permissions:

Events subscribed: workflow_job only.

Installation scope: single repository (TradeMasterAPI) for Phase 1. Expand to org-wide when additional repos onboard in Phase 5.

App private key stored in SSM only. Never in repository files, Terraform state, or environment variables (Lambda reads from SSM at cold start).

IAM (two new roles; claude-infisical-bootstrap is NOT modified)

Role: ci-runner-fleet-role (Lambda execution role)

ssm:GetParameter, ssm:GetParameters          — /ci/* only
ec2:RunInstances                             — runners only (condition on Name tag)
ec2:TerminateInstances                       — runners only
ec2:Describe*                                — runners only (tag filter)
ec2:CreateTags                               — runners only
sqs:SendMessage, sqs:ReceiveMessage          — runner queue only
sqs:DeleteMessage, sqs:GetQueueAttributes
cloudwatch:PutMetricData                     — /CI/* namespace only
logs:CreateLogGroup, logs:PutLogEvents       — /ci/* log groups only

Role: ci-runner-instance-role (EC2 instance profile for runners)

ssm:GetParameter*                            — /ci/runner/* only (not App key or HMAC)
ssm:StartSession                             — self (SSM Session Manager break-glass)
logs:CreateLogGroup, logs:PutLogEvents       — /ci/runner-output/* only
ecr:GetDownloadUrlForLayer, ecr:BatchGetImage — if private Docker images added later

IaC placement: infra/ci/ in this repo for Phase 1; migrate to raxx-infra repo when a second repo onboards (see OQ-4).

Ops

Patching: Ephemeral runners always launch from the latest AMI in the Terraform launch template. Monthly SSM Automation document updates the AMI ID; terraform apply propagates it. Lambda functions are stateless; redeploy on schedule.

Backup / restore: No self-hosted control-plane state to back up (GH holds pipeline history). SSM parameters are versioned (20 versions retained). CloudWatch logs retained 90 days. No RDS.

Capacity alarms: - SQS queue depth > 5 for > 10 minutes → CloudWatch alarm → ops@ daily digest - EC2 cost tag purpose=ci-runner daily spend > $10 → billing alarm → ops@ digest

Kill-switch: Set runner_max_size = 0 + terraform apply (< 2 minutes). Or disable the GitHub App webhook in GitHub App settings (30-second path). All queued jobs wait in GH Actions queue; no jobs run. To resume: re-enable webhook or scale up.

8. Phased Build Plan

Phase 0 — Design sign-off

This document. Operator reviews candidates, confirms Candidate A, answers OQ-1 through OQ-5. No infra deployed.

Rollback: N/A.

Phase 1 — Terraform baseline

Sub-card: infra/ci Terraform (VPC, subnets, IGW, NAT, security groups, ALB, ACM, WAF, SSM parameter stubs, IAM roles, philips-labs runner module).

Gate: ACM cert validated (OQ-1 DNS record added). 5+ successful runner registrations visible in GitHub → Settings → Actions → Runners.

Rollback: terraform destroy. No traffic yet; zero blast radius.

Phase 2 — Pilot on low-risk cron workflows

Target: ci-digest-cron.yml, postmark-delivery-monitor.yml, queue-zero-dyno-monitor.yml (short, no service containers, no secrets, cron-only).

Soak: 48 hours, 10+ successful runs per workflow. Confirm cold-start p95 < 90 seconds.

Rollback: revert runner label → GH-hosted instant resume.

Phase 3 — PR gates + heavy CI

Target: ci.yml (backend-tests-postgres 90-min job), ci-pr.yml, antlers-next-ci.yml, ci-console.yml.

Gate before Phase 3: validate postgres:15 service container on AL2023 runner AMI (OQ-6 smoke test).

Soak: 7 days, 20+ backend-postgres successful runs on heavy on-demand runner.

Rollback: one-PR revert of runs-on change. < 5-minute blast radius.

Phase 4 — Deploy + release workflows (highest risk)

Sequence within Phase 4: 1. Staging-only deploys first (deploy-heroku to staging, deploy-antlers-next-staging) 2. 5-day soak on staging 3. Prod deploys (deploy-heroku prod, deploy-console, deploy-queue, deploy-velvet) 4. Release pipeline last (gatekeeper-develop-to-release, promote-release-to-main)

Rollback: revert runner label per-workflow. Because workflows are stateless, any single workflow can revert independently in under 5 minutes.

Phase 5 — Decommission + expand

• Set GitHub Actions spending limit to $20/month (iOS macOS buffer only)
• Close #728 (Ubicloud card superseded), update #726 (billing posture doc)
• Supersede ADR-0033 with ADR-0134
• Onboard additional MooseQuest repos via /onboard-ci skill (Section 9)

9. /onboard-ci Skill (1-page sketch)

Modeled on the /onboard-vault skill pattern. Script at scripts/skills/onboard-ci.sh.

/onboard-ci <github-org>/<repo> [--runner-label <label>] [--canary-workflow <name>]

Steps (idempotent — re-runs are safe):

1. App install check: verify raxx-ci-runner GitHub App is installed on target repo. If not, output install URL and exit 1.

2. Webhook registration: POST to GitHub API POST /repos/{owner}/{repo}/hooks with workflow_job event and ALB /webhook endpoint + HMAC secret. Skip if identical webhook already exists (check by URL match).

3. Runner group repo scope: update runner_runner_group_repos in Terraform (or via GH API PUT /orgs/{org}/actions/runner-groups/{id}/repositories/{id}) to include the new repo. Apply if changed.

4. Status check audit: read branch protection rules via GitHub API; print any required status check names that will change if runner labels change. No auto-update; operator confirms.

5. Canary run: trigger lowest-risk workflow via workflow_dispatch (default: the first workflow_dispatch-enabled workflow in the repo). Poll for completion. Confirm the run's runner_name contains ip-10.42 (self-hosted runner IP range).

6. Output: Repo: raxx-app/TargetRepo GH App: INSTALLED Webhook: REGISTERED (id: 99887766) Runner group: UPDATED (ec2-self-hosted) Canary run: #42 SUCCESS (runner: ip-10-42-10-12.ec2.internal, 52s cold start) Status: ONBOARDED

The skill has no persistent state. Language: shell (Tier 2 classification applies; no Tier 1 criteria met — not a latency-critical path, no auth credential handling).

10. Security / GDPR Checklist

• PII collected: Build logs may contain developer usernames and email addresses from git log output. No Raxx customer PII in CI build logs. CloudWatch log groups are IAM-gated (ops@ only).
• Retention period: CloudWatch logs: 90 days. SSM parameter versions: 20 (AWS default). No long-term CI log archival needed.
• Deletion on DSR: CI logs contain no customer PII. If a developer DSR is filed, delete the relevant CloudWatch log group streams. No automated path needed.
• Audit trail: CloudWatch Logs: Lambda invocations (JSON), runner lifecycle events, WAF allow/block decisions. Retained 90 days. DPA-ready format.
• Stored credentials: None on runners. JIT tokens are job-scoped and expire after the job. GH App private key in SSM SecureString (KMS-encrypted); Lambda reads at cold start, never logs it. Ephemeral runners write nothing to EBS between jobs (root volume terminates with the instance).
• Breach notification: Exposed GH App key → revoke in GitHub App settings + rotate in SSM; no customer PII at risk; ops@ alert within 1 hour. HMAC secret exposure → rotate in GitHub webhook settings + SSM. Neither is a GDPR personal data breach.
• Secrets location + rotation: SSM Parameter Store SecureString; KMS aws/ssm CMK. All secrets rotatable without redeploy (Lambda reads SSM at cold start; ALB OIDC secret reads at listener-rule update). Zero-downtime rotation path.
• Kill-switch: runner_max_size = 0 + terraform apply (< 2 minutes) or GitHub App webhook disable (30 seconds). All deploy and live-execution workflows stop immediately. Satisfies paper-first gating kill-switch requirement at CI layer.

11. Open Questions (required before sub-cards can be claimed)

OQ-1 (blocking Phase 1): Oracle Dyn DNS records for ACM and ci subdomain ACM needs a DNS validation CNAME at _acm-challenge.ci.moosequest.net. After Terraform apply outputs the ALB DNS name, a second Dyn CNAME ci.moosequest.net → <alb-dns> is required. Both are manual steps in Oracle Dyn. Estimate: 15 minutes. Operator does this once after Phase 1 Terraform first-apply.

OQ-2 (blocking Phase 1): OIDC IdP choice for ALB Proposed: Google Workspace OAuth 2.0 (operator already uses Google; 2FA + passkey enforced at Google account level). Alternative: GitHub OAuth (simpler but weaker isolation). Confirm choice so Terraform configures the ALB OIDC rule and the Google Cloud Console OAuth app is created.

OQ-3 (design clarification): WebAuthn invariant boundary for ci.moosequest.net The Raxx product invariant (passkeys/WebAuthn only) applies to raxx.app. ci.moosequest.net is an internal ops tool. ALB OIDC → Google (with passkey at Google account level) is proposed as consistent with the spirit of the invariant. Operator confirms this interpretation, or instructs otherwise.

OQ-4 (Phase 1 scoping): IaC repo placement Options: (a) infra/ci/ in this repo (simpler for Phase 1), (b) new raxx-infra repo (better secrets isolation; Terraform state stays separate from application code; better for Phase 5 multi-repo expansion). Recommendation: start in infra/ci/ and move to raxx-infra before Phase 5. Operator confirms.

OQ-5 (cost verification): Confirm actual monthly GH Actions minutes The estimate (25,700 min/month, ~$186/mo) was derived from a run-density sample. Actual figure is in GitHub Settings → Billing → Actions usage. Operator provides the number (or confirms the estimate is in the right order of magnitude) to validate the savings case.

OQ-6 (Phase 3 gate): postgres:15 service container smoke on AL2023 AMI Before migrating ci.yml, a 1-hour smoke test must confirm that postgres:15 service containers start correctly on Amazon Linux 2023 with Docker. This is an engineering verification, not an operator decision — flagged here as a required gate before Phase 3.