Raxx · internal docs

internal · gated

ADR-0136 — GHA deliberate-hold workflows: cutover decisions

Status: Accepted Date: 2026-07-09 UTC Deciders: Kristerpher (operator); software-architect (design) Scope: Eight deliberate GHA hold workflows in the Wave D migration (epic #4066, cutover epic #4106) Refs: ADR-0135 (Woodpecker full migration plan); docs/architecture/gha-deliberate-holds-cutover.md (full design) Implementation: Executed 2026-07-09 to 2026-09-28. As-built status, deviations and resolved open questions are in section 10 of the design doc; the "As built" notes below summarise the ones that touch a decision.

Numbering note. Several infrastructure files cite "ADR-0136 Option D" or "ADR-0136 (WP HA)" (for example infra/ci/server.tf, infra/ci/cloudwatch.tf, infra/ci/ssm.tf, infra/ci/iam.tf, infra/ci/outputs.tf, terraform/ci-monitoring/* and ADR-0143). That is the Woodpecker high-availability decision, which is [ADR-0142](https://internal-docs.raxx.app/architecture/adr/0142-woodpecker-ha-option-d.html). This ADR (0136) covers only the deliberate-hold cutover; infra/ci/ecr-cache.tf, infra/ci/ci-artifacts.tf and the W3 to W8 pipelines cite it correctly.

Context

ADR-0135 classified eight GitHub Actions workflows as "deliberate GHA hold" after Wave D retired 51 other workflows. Each of the eight has a specific technical blocker preventing a straightforward Wave A–C port. This ADR records the key decisions made during the design of the cutover for those eight.

Decision 1 — woodpecker-healthcheck.yml stays on GHA permanently

Decision: woodpecker-healthcheck.yml is classified as GENUINELY-STAYS-ON-GHA. No Woodpecker port will be created.

Rationale: A monitoring pipeline for Woodpecker must run on infrastructure independent from Woodpecker. If Woodpecker is down, a WP-hosted watchdog cannot fire. This is the precise failure mode the monitor detects. The external vantage requirement cannot be satisfied by any WP pipeline, a same-region Prometheus stack, or a WP cron.

Alternatives rejected: - WP cron: Fails when WP is down — exactly the wrong behaviour. - Prometheus blackbox exporter (#1022): Same AWS region and AZ as ci.moosequest.net; insufficient independence. - SaaS uptime monitor: Valid secondary layer but does not replace the GH anchor-issue dedup or Postmark edge-triggered alerting without custom wiring.

Consequence: GHA is retained for one workflow indefinitely. GHA spending limit remains constrained by this one cron (approximately 15 runs/hour × ~15 seconds each = negligible minutes).

Decision 2 — OIDC short-lived tokens replaced by vault-loaded long-lived IAM keys

Decision: terraform-email-delivery-stack.yml and freescout-apply.yml both use AWS credentials. The GHA OIDC federation mechanism (short-lived tokens, repo/branch bound) cannot be replicated in Woodpecker. The WP replacement uses long-lived IAM access keys stored in Infisical vault.

This is a security posture downgrade. Recorded here explicitly.

OIDC model (current GHA): - Tokens valid for 15-30 minutes per workflow run. - Cryptographically bound to sub: repo:raxx-app/TradeMasterAPI:ref:refs/heads/main. - No key rotation required — tokens are ephemeral. - A compromised token has a maximum 30-minute blast radius.

Long-lived key model (WP replacement): - Keys valid until rotated (90-day rotation schedule via velvet). - Not bound to a specific repo or branch — any process with the key can call AWS. - If a key leaks, blast radius is bounded only by the IAM policy until rotation.

Mitigations accepted as adequate for current team size (single operator): 1. Separate plan (read-only) and apply (read-write) IAM users. A leaked plan key cannot modify or destroy infrastructure. 2. Automated 90-day rotation via velvet handler. No manual rotation dependency. 3. CloudTrail alerts on unexpected API calls from unexpected IPs or time windows. 4. Least-privilege per-resource policies (not AdministratorAccess). 5. Keys stored in Infisical vault at /MooseQuest/aws/terraform/ and /MooseQuest/aws/freescout/ — never in repo files or WP YAML.

As built (2026-10-30): mitigations 1, 4 and 5 are in place (infra/ci/ci-artifacts.tf, scripts/ops/secrets/mint_ci_aws_keys.sh). Mitigations 2 and 3 are not delivered as written: - No velvet handler exists. Rotation is the operator-run mint_ci_aws_keys.sh --rotate (see docs/ops/runbooks/ci-aws-credentials.md), so the 90-day cadence depends on the operator. - infra/ci/ defines no CloudTrail alarm on these users. Only the runbook's manual aws cloudtrail lookup-events check exists.

Until both are built, treat the "90-day automated rotation" and "CloudTrail alerting" claims in .woodpecker/terraform-email-delivery-stack.yaml as aspirational.

Revisit trigger: If the team grows beyond 2 people OR if a public OIDC provider for self-hosted Woodpecker is available and audited, revisit this decision and migrate back to short-lived tokens.

Alternatives rejected: - Vault as OIDC proxy: Vault can issue OIDC tokens, but Woodpecker is not configured to call Vault's OIDC endpoint at pipeline start. Requires Vault Enterprise features or significant custom scripting — too complex for current team size. - AWS IAM Roles Anywhere (certificate-based): Requires a private CA and certificate management. Adds operational complexity that exceeds the benefit at current scale.

Decision 3 — workflow_run: trigger replaced by polling cron

Decision: deploy-failure-streak-alert.yml uses workflow_run: (no WP equivalent). The replacement is a 30-minute WP cron that polls the WP build API for consecutive failures, replacing the GH Actions API calls.

Rationale: workflow_run: is a GHA-native push event (fires when a named workflow run completes). ADR-0135 §4 already established that workflow_run and workflow_call workflows are RETIRE-GHA-NATIVE. The polling cron approach is the established pattern for this class of problem in WP (the existing deploy-queue-failure-monitor.yml used an independent cron for the same reason when the 30-minute window of the general monitor proved inadequate).

Consequence: deploy-queue-failure-monitor.yml is absorbed into the WP streak pipeline (its queue-specific unbounded-window check is added as an additional check block) and retired as a standalone GHA workflow.

Latency increase: Reactive (0-30 s lag after workflow completion) → polling (0-30 min lag). This is acceptable for a streak-detection alert — a 30-minute polling window matches the existing streak time window.

As built: .woodpecker/deploy-failure-monitor.yaml polls every 6 hours (0 */6 * * *), not every 30 minutes, so worst-case detection lag is up to 6 hours. It alerts by Postmark email to ops@raxx.app, not Slack. The Queue profile in that file keys on tag prefix refs/tags/queue-, which deploy-queue.yaml never produces, so it cannot fire until it is repointed at v* tags or release pushes.

Decision 4 — ZAP runs as a WP step image, not Docker-in-Docker

Decision: security-zap.yml uses zaproxy/action-baseline@v0.12.0, a GHA composite action that wraps Docker internally. The WP replacement uses image: zaproxy/zap-stable as the step image — no Docker-in-Docker or WOODPECKER_ESCALATE required.

Rationale: GHA actions that wrap Docker internally do so because GHA steps cannot natively specify a container image. Woodpecker steps natively specify a container image per step. Running ZAP as the step image is the natural WP translation of a Docker-wrapping GHA action. The ZAP container is executed by the WP agent's Docker runtime, not by a nested Docker daemon inside the step.

Consequence: ZAP port is classified as effort S (low), not L. No privileged agent modes are required. ZAP port is Wave B (can proceed before the Docker-socket infrastructure for deploy-queue is in place).

As built: .woodpecker/security-zap.yaml pins zaproxy/zap-stable to 2.17.0 (bumped from 2.14.0 on 2026-08-19) and a pin-check step fails the pipeline if the tag becomes :latest.

Decision 5 — Docker socket mount preferred over DinD for queue builds

Decision: deploy-queue.yml and queue-docker-smoke.yml require Docker for building the C++ container image and running smoke containers. WP will use Docker socket mount (/var/run/docker.sock:/var/run/docker.sock in WOODPECKER_VOLUMES) rather than Docker-in-Docker (WOODPECKER_PRIVILEGED=true).

Rationale: DinD starts a nested Docker daemon inside each pipeline step. On spot EC2 instances, the daemon startup adds 5-15 seconds per step and increases OOM risk. Socket mount reuses the host daemon — faster and lower overhead for a workload (25-30 min C++ builds) where step startup time matters.

Security trade-off: Socket mount gives pipeline code access to the host Docker daemon. A malicious pipeline step could read/write any container on the agent. Mitigated by: (1) only raxx-app/TradeMasterAPI pipelines run on this WP instance (single-repo, operator-controlled); (2) pipeline YAML is in the operator's own repo (not a public fork); (3) secret event restrictions prevent public PR pipelines from accessing prod secrets.

Open question (OQ-1): Operator must confirm socket mount is acceptable over DinD before the P1 card is claimed.

As built: OQ-1 resolved in favour of the socket mount. The host socket is bind-mounted into the agent container (infra/ci/templates/user_data_agent.sh.tpl) and trusted.volumes=true on the WP repo lets steps inherit it (infra/ci/ecr-cache.tf). The layer cache is ECR repo raxx-queue-vcpkg-cache in us-east-2.

Decision 6 — Approval gates use WP confirm-variable pattern (not GH Environments)

Decision: freescout-apply.yml and the destroy-action gate in terraform-email-delivery-stack.yml used GH Environments for manual approval. WP has no equivalent. The replacement pattern is: a CONFIRM_VAR=magic-string pipeline variable checked at step 1.

Rationale: The GH Environment reviewer gate in these workflows currently has no required reviewers configured (ADR-0020 reviewer-gate waiver). The Environment gate is effectively a "confirm by running the workflow yourself" check — which the WP confirm-variable pattern replicates. The pattern is already in use in the migrated deploy-prod.yaml pipeline and is the established WP authorization mechanism per ADR-0135 §3.

Consequence: The GH deployment audit trail (visible in the Environments UI) is replaced by the WP build log as the audit trail. Acceptable for a single-operator team. Revisit if team grows or if a regulator requires environment-level audit trails.

As built: the confirm variables are APPLY_CONFIRM=APPLY-FREESCOUT-NOW (freescout-apply.yaml), APPLY_CONFIRM=yes plus DESTROY_CONFIRM=ALLOW-DESTROY-EMAIL-STACK (terraform-email-delivery-stack.yaml, one pipeline, resolving OQ-2) and DEPLOY_PROD_CONFIRM=deploy-prod-now (deploy-queue.yaml, break-glass from main).

Language choice rationale (no new services)

This ADR does not introduce any new services or processes. The 8 workflows are CI/CD pipelines that run on existing WP infrastructure. No Tier 1 / Tier 2 language classification is required.

Consequences

Area Before After
GHA workflows remaining 8 deliberate holds + ios-ci.yml 1 (woodpecker-healthcheck.yml); as built, ios-ci.yml was retired in Wave D (#4064) and the paused billing-collector-cron.yml is a fallback, not a hold
AWS auth model OIDC short-lived IAM long-lived keys (90-day rotation, operator-run as built)
ZAP DAST GHA composite action (Docker wrapper) WP step image (zaproxy/zap-stable)
Failure-streak alerting Reactive (workflow_run:) Polling cron (designed 30 min; as built every 6 hours, Postmark email)
Approval gates GH Environments (no reviewers) WP confirm-variable pattern
vcpkg cache GHA actions/cache@v4 ECR layer cache (type=registry)

Open Questions (all resolved)

Raised at design time; each is now closed. Detail is in section 10.3 of the design doc.