ADR-0135 — Woodpecker CI: full GHA migration execution plan (Phases 3–5)

Status: Accepted Date: 2026-07-04 UTC Deciders: Kristerpher (operator); software-architect (design) Scope: All 86 .github/workflows/*.yml files; 5 composite actions; branch protection; deploy/release pipeline; agent capacity; decommission Refs: ADR-0134 (infra decision, Candidate B accepted); docs/architecture/ci-self-hosted-ec2-decision-2026-07-03.md; docs/architecture/woodpecker-migration-table-2026-07-04.md (per-workflow table)

Context

ADR-0134 accepted Woodpecker CI (Candidate B) as the full CI control plane replacement for GitHub Actions. Phases 1 (Terraform baseline) and 2 (3 cron pilot pipelines: security-scan, alembic-version-check, maxmind-asn-refresh) are complete and green. This ADR governs the execution design for Phases 3–5: the concrete decisions that every sub-card runs off.

The 86-workflow inventory breaks into four buckets (full per-workflow table in the linked migration table doc):

Three of the 79 MIGRATE workflows have partial Phase 2 pilot pipelines already running; they complete their port in Wave A/B.

Invariants

• No stored credentials between jobs. WP agent workspaces are ephemeral per pipeline; no credential state persists across jobs.
• Credentials into infra, not code. The 5 vault bootstrap credentials go into WP org-level secrets only. All application secrets continue to live in vault, fetched at pipeline runtime by the vault init step.
• Audit trail for every state change affecting money, permissions, or data access. Every deploy pipeline logs actor, ref SHA, target environment, and outcome. CloudWatch log groups retain 90 days.
• Paper-first gating. The prod-deploy authorization mechanism (tag-gating + WP secret restriction + Console RBAC) must be at least as strong as the current GHA Environment gate.

Decision

1. Vault access — reusable WP init step, not agent-image bake

Decision: Five WP org-level secrets carry the vault bootstrap credentials. Every pipeline that needs application secrets includes a vault init step that runs scripts/ci/wpc_load_vault_secrets.py — a direct port of the .github/actions/load-vault-secrets shell logic, adapted for Woodpecker's WOODPECKER_ENV_FILE output mechanism.

Rejected — bake into agent image: a custom agent AMI with the Infisical CLI couples the image build to vault credential rotation; a compromised AMI would carry vault access. The reusable step pattern fetches fresh credentials at runtime and keeps the AMI vendor-neutral.

WP org-level secrets to create (values ported from current GHA repo secrets; set via POST /api/orgsecrets or POST /api/repos/1/secrets):

Canonical vault init step (copy into every pipeline that needs vault secrets):

- name: load-vault-secrets
  image: python:3.12-slim
  environment:
    INFISICAL_CLIENT_ID:
      from_secret: INFISICAL_CLIENT_ID
    INFISICAL_CLIENT_SECRET:
      from_secret: INFISICAL_CLIENT_SECRET
    INFISICAL_PROJECT_ID:
      from_secret: INFISICAL_PROJECT_ID
    CF_ACCESS_CLIENT_ID:
      from_secret: CF_ACCESS_CLIENT_ID
    CF_ACCESS_CLIENT_SECRET:
      from_secret: CF_ACCESS_CLIENT_SECRET
    VAULT_SECRETS: "SECRET_A,EXPORT_NAME=VAULT_KEY"  # per-pipeline
    VAULT_PATH: "/MooseQuest/"                        # per-pipeline
    VAULT_ENV: "prod"                                 # or "staging"
  commands:
    - pip install --quiet requests
    - python3 scripts/ci/wpc_load_vault_secrets.py

wpc_load_vault_secrets.py performs the same Infisical Universal Auth REST flow as the existing bash action. Key differences from the GHA version: - Writes KEY=VALUE\n to $WOODPECKER_ENV_FILE (WP v3 inter-step env passing) instead of $GITHUB_ENV. - Multi-line values (PEM keys) use WP's heredoc env-file syntax: KEY<<WDPK_DELIM\n<value>\nWDPK_DELIM. - Omits ::add-mask:: — WP secrets injected via from_secret are auto-redacted by the WP server in build logs.

2. Composite action ports

The five GHA composite actions become inline scripts. Woodpecker has no uses: ./.github/actions/... equivalent; each calling pipeline includes the step by value. Wave A sub-card A1 ports all five:

3. Deploy/release pipeline — tag-gated, Woodpecker-native

The ADR-0115 tag-gated model is preserved entirely in Woodpecker:

release-* tag pushed → develop
  .woodpecker/gatekeeper-develop-to-release.yaml  [event: tag, tag: release-*]
    1. Assert tag commit is on develop
    2. Poll Woodpecker API for CI pipeline on tagged SHA → must show status=success
       GET /api/repos/1/builds?event=push&branch=develop&limit=10, filter by commit SHA
    3. Idempotency check (already in release?)
    4. git merge --no-ff develop→release, push (raxx-ops-bot token from vault)
  → push to release branch auto-fires staging deploy pipelines

vX.Y.Z semver tag pushed → release
  .woodpecker/promote-release-to-main.yaml  [event: tag, tag: v*.*.*]
    1. Assert tag commit is on release
    2. Heavy scan suite (gitleaks, PII, SAST, dep audits) — same tools as GHA
    3. Idempotency check
    4. git merge --no-ff release→main, push (raxx-ops-bot token from vault)
    5. gh release create vX.Y.Z --generate-notes  [replaces release-please]
  → push to main auto-fires production deploy pipelines

How gatekeeper verifies CI: replaces the GHA API poll (GET /actions/workflows/ci.yml/runs?head_sha=...) with the Woodpecker API: GET /api/repos/1/builds?event=push&branch=develop&limit=20, filtered by commit == tagged_sha, checking status == "success". Same bounded retry loop (10 × 30 s = 5 min max) as the GHA version.

Prod deploy authorization: Production secrets (HEROKU_API_KEY prod, CLOUDFLARE_PAGES_TOKEN prod, etc.) are stored in WP with the event restriction event: ["tag", "manual"]. They are unavailable to push- or PR-triggered pipelines at the WP server level — not just by YAML convention. This enforces that production deploys can only occur via: - A vX.Y.Z semver tag push (triggering promote-release-to-main.yaml), or - A Console Ops Dispatch call to the WP API with event=manual (existing Console RBAC gate; changed from GHA workflow_dispatch URL to WP API endpoint POST /api/repos/1/pipelines).

GHA Environments currently have no required reviewers configured per the ADR-0020 reviewer-gate waiver. The WP tag + secret event restriction model provides equivalent authorization strength.

Console Ops Dispatch cutover (Wave C dependency): The Console deploy API changes from POST /repos/{repo}/actions/workflows/{file}/dispatches to POST /api/repos/1/pipelines with body {"event":"manual","branch":"<ref>","variables":{"environment":"production"}}. A Console sub-card (C-console) is a Wave C gate dependency.

4. workflow_call and workflow_run retirement

workflow_call (GHA reusable workflows) and workflow_run (GHA cross-workflow trigger) have no WP equivalents. Two workflows use them:

• synthetic-gate.yml (workflow_call + schedule): The synthetic gate steps are inlined directly into .woodpecker/deploy-heroku.yaml as a conditional step block that runs only on branch: release (staging deploys). The standalone schedule run becomes a separate cron WP pipeline .woodpecker/synthetic-gate-cron.yaml.
• slack-notify.yml (workflow_run for "CI — main", which no longer exists post-ADR-0115): RETIRED. Slack failure notifications are added as on_failure step blocks inline in each WP pipeline where alerting is required.

5. release-please retirement

release.yml uses google-github-actions/release-please-action@v4 — a GHA action with no WP equivalent. RETIRED. Replacement: a final step in .woodpecker/promote-release-to-main.yaml runs gh release create ${TAG_NAME} --generate-notes --notes-start-tag <prev-semver-tag> --target ${MAIN_SHA} using the raxx-ops-bot token from vault. Automated release PRs and a structured CHANGELOG are dropped; GitHub auto-generated release notes from PR titles replace the rich CHANGELOG. Acceptable trade-off — the operator confirmed the ADR-0115 explicit tag model supersedes the release-please flow.

6. Branch protection swap — add-alongside, then remove

Woodpecker posts commit status checks via the GitHub Checks API. Status context format in WP v3: ci/woodpecker/<pipeline-filename>/<step-name>.

Required check name mapping (develop branch):

Wave A sub-card A7 audits the full required check list against the actual WP job/step names before any changes are made (exact names depend on the WP pipeline YAML written in A3–A6; audit AC is required before the swap).

Swap sequence: 1. Wave A — add WP check names as additional required checks alongside GHA. PRs must pass both during the parity window. Branch protection API: PATCH /repos/raxx-app/TradeMasterAPI/branches/develop/protection. 2. Wave B gate — after 7-day soak and 20+ successful WP backend-postgres runs: remove GHA check names; WP checks become the sole gate. 3. release/main branches — same pattern; swap happens in Wave C (deploy pipeline parity confirmed on staging before touching main protection).

7. Agent capacity — scheduled scale + heavy-fleet scale arbiter

Standard fleet (c6a.xlarge spot, size:standard label):

Weekend and off-hours PRs accept the ~90-second EC2 cold start + WP agent registration latency.

Heavy fleet (c6a.2xlarge on-demand, size:heavy label — backend-tests-postgres):

Default: min=0, desired=0 (scale to zero). A scale arbiter Lambda (infra/ci/scale_heavy_fleet.py, ~50 lines) runs every 5 minutes via EventBridge. It polls GET /api/repos/1/builds?status=pending for builds with environment labels containing size:heavy, then calls autoscaling:SetDesiredCapacity on the heavy fleet ASG to 1 (builds pending) or 0 (queue clear). IAM: the existing ci-runner-fleet-role gains autoscaling:SetDesiredCapacity scoped to the heavy ASG ARN. Cold start is ~90 seconds — acceptable for a 78-minute job.

Cost projection (full migration):

Within the operator-accepted ~$107/mo envelope. Savings vs $186/mo GHA bill: ~$92/month after full migration.

8. Decommission sequence (Wave D)

After Wave D 14-day prod soak:

1. gh workflow disable for all remaining active GHA workflows (bulk script in Wave D sub-card D4; disables via API, does not delete files).
2. .github/workflows/ stays in place for 30 days as an incident reference.
3. After 30-day soak: git mv .github/workflows .github/workflows-retired in a single cleanup PR.
4. Set GHA spending limit to $20/month (iOS macOS buffer).
5. Close GH issue #728 (Ubicloud migration superseded); update #726 (billing).
6. ios-ci.yml remains in .github/workflows-retired/ with a header comment documenting the defer decision; do not delete.

Language choice rationale

No new services are introduced by this ADR. All ported scripts are Tier 2.

scripts/ci/wpc_*.py / wpc_*.sh — Tier 2 (Python/shell). These are ephemeral CI helper scripts: no latency budget, no auth credential hot path, no memory-safety requirement. No Tier 1 criteria met. Portability risk to Rust: none (CI scripts, not runtime services).

Scale arbiter Lambda — Tier 2 (Python). Runs every 5 minutes; latency irrelevant; no customer PII; no auth credential handling. No Tier 1 criteria met.

Consequences

Positive

• After Wave D: single CI control plane (Woodpecker). GHA is fully decommissioned except the disabled ios-ci.yml and the $20/mo billing buffer.
• Prod deploy authorization preserved via WP secret event restriction (tag-only + manual-only for prod secrets). Enforced at WP server level, not just by convention.
• Vault access is simpler in WP (5 org-level secrets, one inline init step) vs the current GHA composite action + 5 repo secrets pattern.
• release-please retirement removes a flaky external dependency (it has caused GH token permission failures on several runs).
• Scale-arbiter Lambda for heavy fleet avoids always-on c6a.2xlarge cost (~$78/mo saved vs a kept-warm heavy agent during business hours).

Negative / risks

• workflow_call inlining increases YAML duplication. Synthetic gate steps must be duplicated in every pipeline that calls them. YAML anchors partially mitigate this in a future cleanup pass.
• Branch protection swap window doubles CI cost. During Wave A/B parity, PRs run on both GHA and WP. ~7–14 day window; operator-accepted cost.
• WP server is a stateful dependency. Server downtime stops all CI. Mitigated by: on-demand instance (not spot); RDS automated backups; CloudWatch alarm on server health; documented kill-switch (30 seconds).
• Console Ops Dispatch is a hard Wave C dependency. If the Console card slips, Wave C cannot close prod-deploy cutover. Console card must be filed as a Wave C gate before Wave C begins.
• release-please changelog richness is lost. --generate-notes produces PR-title-based release notes. Operator-accepted trade-off.

Neutral

• .github/workflows/ is retained (disabled, not deleted) for 30 days post-Wave D.
• GitHub Pro plan stays active during migration; spending limit drops at Wave D only.

Alternatives considered

Bake vault credentials into agent AMI

Rejected: couples AMI build to vault credential rotation; a compromised AMI carries vault access. The inline init step has identical security properties with zero image-build overhead.

Keep release-please running in GHA (hybrid forever)

Rejected: leaves a permanently non-zero GHA workflow count. The gh release create replacement covers the functional need with no GHA dependency.

Lambda webhook autoscaling for standard fleet

A Lambda reacting to WP webhook events to scale the standard fleet was considered. Rejected in favor of simpler scheduled scale for the standard fleet (predictable business-hours pattern; fewer moving parts). Scale arbiter Lambda is scoped to the heavy fleet only, where on-demand scaling is economically necessary.

Security / GDPR checklist

• PII collected: Build logs may contain developer git identities. No Raxx customer PII in CI pipelines. CloudWatch and WP build logs are IAM/RBAC-gated (ops@ only).
• Retention period: CloudWatch 90 days; WP pipeline history 30 days (server config). No long-term CI log archival required.
• Deletion on DSR: CI logs contain no customer PII. Developer DSR: delete relevant CloudWatch log streams + WP build records. No automated path needed.
• Audit trail: Every deploy pipeline writes structured log: WP trigger source, ref SHA, target environment, deploy outcome. Retained in CloudWatch 90 days.
• Stored credentials: 5 vault bootstrap secrets in WP RDS-backed secret store. No secret is written to agent disk between pipeline runs. WP agent workspaces deleted after each pipeline.
• Breach notification: Exposed WP org secret → rotate in vault + re-set via WP API; no customer PII at risk; ops@ within 1 hour. Exposed Heroku API key → rotate via Heroku + vault; ops@ within 1 hour.
• Secrets location + rotation: All application secrets remain in Infisical vault (unchanged). The 5 WP org-level bootstrap secrets are rotatable: update in vault + PATCH /api/orgsecrets/{name} in WP API; no redeploy needed.
• Kill-switch: Scale standard and heavy ASGs to 0 (< 2 minutes) or disable GitHub webhook in WP admin UI (30 seconds). All pipelines stop; Console Ops Dispatch returns unavailable. Staging/prod deploys halt immediately.

Revisit when

• Monthly EC2 cost exceeds $150/month: tune ASG schedule or instance types.
• Woodpecker project abandoned upstream: fork Apache 2.0 codebase or evaluate Gitea Actions / Forgejo.
• iOS CI re-evaluation triggered post-migration (separate decision).
• Second MooseQuest repo onboards: migrate infra/ci/ to raxx-infra repo (ADR-0134 consequence).
• Team size exceeds 5: revisit audit log access model per ADR-0134 §GDPR.

Wave D execution record — 2026-07-07 (#4019)

Wave D decommission executed via PR chore(ci): Wave D — retire GitHub Actions (#4019). GHA workflows retired: 45. GHA workflows kept active (WP gap): 41.

Retirement method

Each retired workflow was: 1. Modified in place: on: trigger block replaced with on:\n workflow_dispatch: (manual-only, never auto-fires). 2. A top-of-file # RETIRED 2026-07-07 comment added with the WP replacement reference and issue number. 3. Moved to .github/workflows-retired/ via git mv (rename tracked; content preserved for reference). 4. Every modified file validated with yaml.safe_load before commit.

Script: scripts/ci/retire_gha_workflows.py.

What was retired (45 workflows)

What is still active in .github/workflows/ (41 WP gaps)

These workflows have no Woodpecker equivalent confirmed in .woodpecker/ on develop as of 2026-07-07. They remain active to avoid dropping coverage.

Wave B cron gap (19 workflows — WP pipelines not yet in .woodpecker/): account-merge-tombstone-cron.yml, audit-integrity-cron.yml, audit-reconciler-cron.yml, bcp-smoke-monthly.yml, billing-collector-cron.yml, billing-retention-cron.yml, e2e-smoke.yml, freescout-backup.yml, heroku-config-health-nightly.yml, historical-bars-1min-warm-nightly.yml, mbt-assignment-at-expiry-cron.yml, mbt-dividend-split-cron.yml, mbt-drift-daily.yml, mbt-drift-per-symbol-weekly.yml, mbt-resting-orders-cron.yml, queue-docker-smoke.yml, security-zap.yml, tickets-e2e-smoke.yml, trace-integrity-cron.yml

Wave C deploy gap (22 workflows — services not yet deployed via WP): cut-release-candidate.yml, deploy-antlers-cutover.yml, deploy-antlers-next-prod.yml, deploy-antlers-next-staging.yml, deploy-console-shim.yml, deploy-customer-docs.yml, deploy-failure-streak-alert.yml, deploy-flag-docs.yml, deploy-getraxx.yml, deploy-internal-docs.yml, deploy-mockups.yml, deploy-queue-failure-monitor.yml, deploy-queue.yml, deploy-status-page.yml, deploy-status-worker.yml, deploy-support.yml, deploy-velvet.yml, deploy-worker-waf-log-shipper.yml, freescout-apply.yml, synthetic-gate.yml, terraform-email-delivery-stack.yml, waf-synthetic-probe.yml

Known side-effect: deploy-failure-streak-alert.yml and deploy-queue-failure-monitor.yml use workflow_run triggers that listen for now-retired GHA workflow names ("Deploy to Heroku", "Deploy console", etc.). Those workflow_run events will never fire now that the source workflows are retired. Both files retain workflow_dispatch for manual invocation. Port both to WP as part of the Wave B/C backlog.

Reversibility

Any retired workflow can be re-enabled in under 5 minutes:

# Restore original triggers from git history
git show <pre-wave-d-sha>:.github/workflows/<file>.yml > .github/workflows/<file>.yml
git add .github/workflows/<file>.yml
git commit -m "fix: re-enable <file>"

After 30 days (2026-08-06), if no rollback was required, the .github/workflows-retired/ directory can be removed and this ADR amended to reflect stable decommission status.

Operator actions required (not automated)

1. Drop GHA spending limit — set to $20/month (iOS macOS buffer per ADR-0134). GitHub Settings → Billing → Actions → Spending limit.
2. Close GH issue #728 — Ubicloud migration superseded by WP.
3. Update GH issue #726 — billing posture: WP is primary; GHA $20/mo buffer only.
4. Port Wave B + Wave C remaining 41 workflows to WP — file sub-cards as follow-on to #4019.