Raxx · internal docs

internal · gated

Woodpecker CI runbook

System: ci-woodpecker (Woodpecker CI on EC2 — ci.moosequest.net) Owner: operator / sre-agent ADR: ADR-0134 (Candidate B, operator-authorized 2026-07-03) Card: #4012 (Phase 1 baseline IaC) Last incident: 2026-07-05 (WAF SizeRestrictions_BODY blocked all GitHub webhooks + WP v3 YAML parser rejected ${{ in comments — see docs/incidents/2026-07-04-waf-webhook-block-batch1-parse-error.md) Last reviewed: 2026-07-08

Architecture summary

Internet
  └── Oracle Dyn CNAME: ci.moosequest.net → ALB
        └── ALB (public, us-east-2a/b)
              ├── AWS WAF WebACL (managed rules + rate limit)
              ├── :80 → redirect :443
              └── :443 → listener rules
                    ├── /hook*         → Woodpecker server :8000 (no OIDC — priority 10)
                    ├── /api/badge*    → Woodpecker server :8000 (no OIDC — priority 20)
                    ├── /api/* /authorize /login /logout → Woodpecker :8000 (no OIDC — priority 25)
                    └── (default)      → Google Workspace OIDC → Woodpecker :8000
                          └── Woodpecker server (t4g.small, private subnet us-east-2a)
                                ├── docker-compose: woodpecker-server container
                                ├── gRPC :9000 → build agents (spot ASG, c7a/c6a.xlarge)
                                └── PostgreSQL → RDS db.t4g.micro (private subnet)

All instances: IMDSv2 required, no SSH keys, SSM Session Manager for break-glass access
Secrets: SSM Parameter Store SecureString (/ci/* path)

How to tell it's broken

Apply order (Phase 1 first-time deploy)

Prerequisites

Before running terraform apply:

  1. AWS credentials configured for the Terraform executor role (see §IAM executor role)
  2. State bootstrap completed (see §State bootstrap below)
  3. SSM parameter placeholders created by bootstrap apply
  4. Google OAuth client created (see §Google OAuth client setup)
  5. GitHub OAuth App registered (see §GitHub OAuth App registration)
  6. Real SSM parameter values filled (see §Secrets fill-in checklist)

State bootstrap (one-time)

cd infra/ci/bootstrap
terraform init
terraform plan
terraform apply
# Note the state_bucket_name and lock_table_name outputs.
# These match the values hardcoded in infra/ci/main.tf backend block.

Bootstrap state lives at infra/ci/bootstrap/terraform.tfstate — do NOT commit this file (it is gitignored). Keep a copy in a secure location (Google Drive ops folder).

Main apply

cd infra/ci
terraform init   # connects to S3 backend created by bootstrap
terraform plan -out=ci-phase1.plan -var-file=terraform.tfvars
terraform apply ci-phase1.plan

After first apply

  1. Read outputs: bash terraform output -json

  2. Add two CNAME records in Oracle Dyn (see §DNS setup below).

  3. Confirm ACM certificate validates (5-30 min after DNS propagation): bash aws acm describe-certificate \ --certificate-arn "$(terraform output -raw acm_certificate_arn)" \ --region us-east-2 \ --query "Certificate.Status" # Expected: "ISSUED"

  4. Fill SSM parameter values (see §Secrets fill-in checklist).

  5. Restart Woodpecker server to pick up real secrets: bash INSTANCE_ID="$(terraform output -raw woodpecker_server_instance_id)" aws ssm start-session --target "$INSTANCE_ID" --region us-east-2 # In the session: sudo systemctl restart woodpecker sudo docker compose -f /opt/woodpecker/docker-compose.yml logs -f

  6. Confirm SNS subscription for ops@raxx.app (check email, click confirmation link).

  7. Verify Woodpecker UI at https://ci.moosequest.net (requires Dyn CNAME and ACM ISSUED).

DNS setup (Oracle Dyn)

Add two CNAME records in Oracle Dyn after first terraform apply:

Record 1: ACM validation CNAME

Type:  CNAME
Name:  <terraform output acm_validation_cname_name>
       (looks like: _acm-challenge.ci.moosequest.net. or _abc123.ci.moosequest.net.)
Value: <terraform output acm_validation_cname_value>
       (looks like: _xyz789.acm-validations.aws.)
TTL:   300

Purpose: Allows AWS Certificate Manager to validate ownership of ci.moosequest.net and issue the TLS certificate. This record can remain indefinitely (ACM uses it for renewals too).

Record 2: ci.moosequest.net service CNAME

Type:  CNAME
Name:  ci.moosequest.net
Value: <terraform output alb_dns_name>
       (looks like: ci-woodpecker-alb-1234567890.us-east-2.elb.amazonaws.com)
TTL:   300 (lower during initial setup for faster propagation)

Purpose: Points ci.moosequest.net to the ALB. Required for the Woodpecker UI, webhook delivery, and the OIDC redirect from Google.

After adding both records: Wait for DNS propagation (typically 2-5 minutes for Dyn's authoritative nameservers; up to 5 minutes for resolvers to pick up TTL=300 records).

Google OAuth client setup (operator browser steps)

The ALB OIDC action uses Google Workspace OAuth 2.0. Create a client once:

  1. Open Google Cloud Console: https://console.cloud.google.com/
  2. Select the MooseQuest project (or create one named "moosequest-ops").
  3. Navigate to APIs & Services → Credentials.
  4. Click Create Credentials → OAuth client ID.
  5. Application type: Web application.
  6. Name: Woodpecker CI ALB OIDC
  7. Authorized redirect URIs — add: https://ci.moosequest.net/oauth2/idpresponse (This is the AWS ALB OIDC callback endpoint — do NOT change the path.)
  8. Click Create.
  9. Copy the Client ID and Client Secret.

  10. Fill SSM parameters (see §Secrets fill-in checklist): ```bash aws ssm put-parameter \ --name "/ci/oidc/google-client-id" \ --value "PASTE_CLIENT_ID_HERE" \ --type SecureString \ --overwrite \ --region us-east-2

    aws ssm put-parameter \ --name "/ci/oidc/google-client-secret" \ --value "PASTE_CLIENT_SECRET_HERE" \ --type SecureString \ --overwrite \ --region us-east-2 ```

Important: The redirect URI must be https://ci.moosequest.net/oauth2/idpresponse exactly. The ALB uses this path for the OIDC code exchange. If ci.moosequest.net is not yet resolving, the Google OAuth client can be created first (the URI is validated by Google only at runtime, not at creation time).

Status (2026-07-03): Google OIDC credentials are already staged in Infisical vault at raxx-secrets /Raxx/CI (prod) as CI_GOOGLE_OIDC_CLIENT_ID and CI_GOOGLE_OIDC_CLIENT_SECRET. Skip steps 1–9 above. Proceed directly to the SSM fill step, reading from vault (requires a session with CF Access service token):

GOOGLE_CLIENT_ID="$(infisical secrets get CI_GOOGLE_OIDC_CLIENT_ID \
  --projectId raxx-secrets --path /Raxx/CI --env prod --plain)"
GOOGLE_CLIENT_SECRET="$(infisical secrets get CI_GOOGLE_OIDC_CLIENT_SECRET \
  --projectId raxx-secrets --path /Raxx/CI --env prod --plain)"
aws ssm put-parameter --name "/ci/oidc/google-client-id" \
  --value "$GOOGLE_CLIENT_ID" --type SecureString --overwrite --region us-east-2
aws ssm put-parameter --name "/ci/oidc/google-client-secret" \
  --value "$GOOGLE_CLIENT_SECRET" --type SecureString --overwrite --region us-east-2
unset GOOGLE_CLIENT_ID GOOGLE_CLIENT_SECRET

GitHub OAuth App registration (operator browser steps)

Woodpecker uses a GitHub OAuth App for SCM integration (pipeline triggers, status checks, user authentication).

  1. Go to GitHub → Settings → Developer settings → OAuth Apps → New OAuth App: - Application name: Woodpecker CI — moosequest.net - Homepage URL: https://ci.moosequest.net - Authorization callback URL: https://ci.moosequest.net/authorize
  2. Click Register application.
  3. On the app page, click Generate a new client secret.
  4. Copy Client ID and Client Secret.
  5. Fill SSM parameters: ```bash aws ssm put-parameter \ --name "/ci/woodpecker/github-client-id" \ --value "PASTE_CLIENT_ID_HERE" \ --type SecureString \ --overwrite \ --region us-east-2

aws ssm put-parameter \ --name "/ci/woodpecker/github-client-secret" \ --value "PASTE_CLIENT_SECRET_HERE" \ --type SecureString \ --overwrite \ --region us-east-2 ```

Secrets fill-in checklist

After terraform apply, all SSM parameters exist as PLACEHOLDER values. Fill them in this order (prerequisite steps noted):

# 1. Woodpecker admin GitHub username (your GitHub login)
aws ssm put-parameter \
  --name "/ci/woodpecker/admin-user" \
  --value "GITHUB_USERNAME" \
  --type SecureString \
  --overwrite \
  --region us-east-2

# 2. Woodpecker agent↔server RPC secret (must be identical on server and agents)
AGENT_SECRET="$(openssl rand -hex 32)"
aws ssm put-parameter \
  --name "/ci/woodpecker/agent-secret" \
  --value "$AGENT_SECRET" \
  --type SecureString \
  --overwrite \
  --region us-east-2
# Save $AGENT_SECRET in 1Password / secure notes — needed if agents are
# rebuilt without Terraform re-reading SSM.

# 3. PostgreSQL password
DB_PASSWORD="$(openssl rand -base64 32 | tr -d /=+ | head -c 32)"
aws ssm put-parameter \
  --name "/ci/woodpecker/db-password" \
  --value "$DB_PASSWORD" \
  --type SecureString \
  --overwrite \
  --region us-east-2
# ALSO update the RDS master password to match:
aws rds modify-db-instance \
  --db-instance-identifier ci-woodpecker \
  --master-user-password "$DB_PASSWORD" \
  --apply-immediately \
  --region us-east-2

# 4. GitHub OAuth App credentials (from §GitHub OAuth App registration above)
aws ssm put-parameter \
  --name "/ci/woodpecker/github-client-id" \
  --value "GITHUB_CLIENT_ID" \
  --type SecureString \
  --overwrite \
  --region us-east-2

aws ssm put-parameter \
  --name "/ci/woodpecker/github-client-secret" \
  --value "GITHUB_CLIENT_SECRET" \
  --type SecureString \
  --overwrite \
  --region us-east-2

# 5. Google Workspace OIDC credentials (from §Google OAuth client setup above)
aws ssm put-parameter \
  --name "/ci/oidc/google-client-id" \
  --value "GOOGLE_CLIENT_ID" \
  --type SecureString \
  --overwrite \
  --region us-east-2

aws ssm put-parameter \
  --name "/ci/oidc/google-client-secret" \
  --value "GOOGLE_CLIENT_SECRET" \
  --type SecureString \
  --overwrite \
  --region us-east-2

After filling all 7 parameters, restart the Woodpecker server (see §Apply order step 5).

IAM executor role

The identity running terraform apply needs the ci-terraform-executor-policy attached (provisioned by the IaC itself on first apply — bootstrapping caveat below).

Bootstrapping caveat: On first apply, the executor policy does not yet exist (Terraform creates it). The operator's identity must have sufficient permissions to create it. Simplest path for first apply: use an existing role with broad permissions, then attach the scoped policy and switch to it for subsequent applies.

Do NOT use claude-infisical-bootstrap — that user must not be widened per ADR-0134.

Recommended first-apply path: 1. Create a new IAM role ci-terraform-executor with a trust policy for the operator's IAM user or SSO identity. 2. Attach AdministratorAccess temporarily for the first apply only. 3. After first apply, detach AdministratorAccess and attach ci-terraform-executor-policy (ARN from terraform output terraform_executor_policy_arn). 4. All subsequent applies use the scoped policy only.

Smoke checklist (after first apply + secrets filled)

Run in order; each step should pass before proceeding:

Current versions

Bumping server + agent versions

Server and agent versions are coupled: agent LT user-data embeds the agent image tag, so a version bump requires terraform apply to publish a new LT version. Agents then relaunch from the new LT on next scale-up.

Normal upgrade path (immutable re-provision): 1. Update woodpecker_server_version and woodpecker_agent_version in infra/ci/variables.tf. 2. Remove the lifecycle { ignore_changes = [user_data] } block from server.tf (if present — see below). 3. terraform plan — confirm server is marked for replace (not destroy). 4. terraform apply — server is replaced, new LT version published. 5. Re-add the lifecycle block if needed (see below).

In-place upgrade path (SSM patch, no terraform replace): If a version was applied via SSM (e.g., systemctl restart with a new image tag in docker-compose), the TF state user_data hash will drift from the computed hash, causing user_data_replace_on_change = true to schedule a server replacement even though the server is already healthy at the new version.

Fix: add lifecycle { ignore_changes = [user_data] } to aws_instance.woodpecker_server in infra/ci/server.tf BEFORE running terraform plan. This makes the plan read 0 destroy instead of scheduling a replacement. The lifecycle block is already present on develop as of 2026-07-04 (incident: docs/incidents/2026-07-04-woodpecker-v3-upgrade.md).

When you next want to intentionally re-provision the server (e.g., major version upgrade via immutable path), remove the lifecycle { ignore_changes = [user_data] } block from server.tf, confirm the plan proposes a replace (not a destroy), then apply.

Woodpecker clone plugin allow list

Woodpecker v3 maintains a trusted-clone plugin list. The default list (per --plugins-trusted-clone) is:

docker.io/woodpeckerci/plugin-git:2.9.2
docker.io/woodpeckerci/plugin-git
quay.io/woodpeckerci/plugin-git

Pipeline YAML must use woodpeckerci/plugin-git (Docker Hub org, no hyphen). The common mistake is woodpecker-ci/plugin-git (GHCR-style, with hyphen) which is NOT in the list — Woodpecker will refuse to inject netrc credentials, and private repo clones fail.

v3.x API changes (vs v2.x)

v3.x env vars — confirmed unchanged from v2.x

WOODPECKER_GITHUB_CLIENT, WOODPECKER_GITHUB_SECRET, WOODPECKER_ADMIN, WOODPECKER_AGENT_SECRET, WOODPECKER_DATABASE_DATASOURCE, WOODPECKER_DATABASE_DRIVER, WOODPECKER_HOST, WOODPECKER_OPEN, WOODPECKER_GRPC_ADDR, WOODPECKER_GRPC_SECURE, WOODPECKER_SERVER_ADDR, WOODPECKER_LOG_LEVEL. No renames required for current config.

v3.x pipeline YAML breaking changes

Known failure modes

Failure mode A: ACM certificate stuck in PENDING_VALIDATION

Symptom: aws acm describe-certificate returns Status: PENDING_VALIDATION for more than 30 minutes. Cause: ACM validation CNAME not present in Dyn, or DNS propagation not yet complete. Fix:

# Verify the validation record is expected:
terraform output acm_validation_cname_name
terraform output acm_validation_cname_value
# Check Dyn DNS console — confirm CNAME exists with the exact name and value.
# Test DNS propagation:
dig CNAME "$(terraform output -raw acm_validation_cname_name)" @8.8.8.8
# Expected: non-empty CNAME answer matching the validation value.

Verification: aws acm describe-certificate --certificate-arn ... --query "Certificate.Status" returns "ISSUED".

Failure mode B: Woodpecker server not starting

Symptom: ALB health checks fail; /healthz returns 502/503. Cause: Docker container failed to start (bad SSM secrets, RDS unreachable, image pull failure). Fix:

# Get instance ID
INSTANCE_ID="$(terraform -chdir=infra/ci output -raw woodpecker_server_instance_id)"
# Open SSM session
aws ssm start-session --target "$INSTANCE_ID" --region us-east-2
# In session:
sudo systemctl status woodpecker
sudo docker compose -f /opt/woodpecker/docker-compose.yml ps
sudo docker compose -f /opt/woodpecker/docker-compose.yml logs woodpecker-server
# Common root causes:
#   - "invalid database" / connection refused: DB password wrong or RDS SG blocks
#   - "cannot reach github.com": NAT Gateway issue; check route tables
#   - "placeholder" in env: SSM parameters not filled; fill and restart
sudo systemctl restart woodpecker

Verification: curl -sf http://localhost:8000/healthz from SSM session returns 200.

Failure mode C: Agents not connecting to server

Symptom: Pipelines queue forever; Woodpecker Admin → Agents shows 0 connected. Cause: Agent-secret mismatch, SG blocking port 9000, or agents not running. Fix:

# Check ASG desired capacity:
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names ci-woodpecker-agents-asg \
  --region us-east-2 \
  --query "AutoScalingGroups[0].{desired:DesiredCapacity,inService:Instances[?LifecycleState=='InService'].InstanceId}" \
  --output json

# If desired=0, scale up:
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 1 \
  --region us-east-2

# If agents are running but not connected, SSH into an agent via SSM:
# (find instance ID from ASG describe output above)
aws ssm start-session --target i-XXXXXXXXXX --region us-east-2
# Check agent logs:
sudo docker compose -f /opt/woodpecker-agent/docker-compose.yml logs woodpecker-agent
# Common error: "rpc error: code = Unauthenticated" → agent-secret mismatch
# Verify server and agent SSM parameter have same value:
aws ssm get-parameter --name /ci/woodpecker/agent-secret --with-decryption --region us-east-2

Verification: Woodpecker Admin → Agents shows agent with status "connected".

Failure mode D: WAF blocking legitimate webhook traffic

Symptom: GitHub webhook delivery shows 403; WAF metrics show blocked requests on /hook*. Cause: GitHub IP range hit the WAF rate limit or a managed rule matched the payload. Fix:

# Check WAF logs in CloudWatch:
aws logs filter-log-events \
  --log-group-name aws-waf-logs-ci-woodpecker \
  --region us-east-2 \
  --filter-pattern '{ $.action = "BLOCK" }' \
  --start-time "$(date -d '-1 hour' +%s000)" \
  | jq '.events[].message | fromjson | {action,httpRequest,terminatingRuleId}'
# Identify which rule blocked the request.
# If rate limit: increase var.waf_rate_limit_webhook_per_5min in terraform.tfvars + apply.
# If managed rule: add an override in waf.tf for that specific rule (count instead of block).

Verification: GitHub webhook delivery test returns 200 in repository settings.

Failure mode D-2: WAF SizeRestrictions_BODY blocks all GitHub webhook deliveries

Symptom: Every GitHub webhook delivery to ci.moosequest.net/api/hook returns HTTP 403. No Woodpecker pipelines trigger on PR or push events. WAF CloudWatch logs show terminatingRuleId: SizeRestrictions_BODY from AWSManagedRulesCommonRuleSet. Cause: The AWSManagedRulesCommonRuleSet SizeRestrictions_BODY rule enforces an 8192-byte body inspection cap. Real GitHub PR/push webhook payloads are 12–36 KB and exceed this cap. The managed rule blocks the request before Woodpecker ever sees it. This is distinct from rate-limit blocks (Failure mode D) — it fires on the very first delivery regardless of volume. Fix:

# Confirm the root cause: look for SizeRestrictions_BODY in the terminatingRuleId field
aws logs filter-log-events \
  --log-group-name aws-waf-logs-ci-woodpecker \
  --region us-east-2 \
  --filter-pattern '{ $.action = "BLOCK" }' \
  --start-time "$(date -d '-1 hour' +%s000)" \
  | python3 -c "
import sys, json
for line in sys.stdin:
    try:
        obj = json.loads(line)
        for ev in obj.get('events', []):
            msg = json.loads(ev['message'])
            print(msg.get('terminatingRuleId'), msg.get('httpRequest', {}).get('uri'))
    except Exception:
        pass
"

# If you see SizeRestrictions_BODY, verify the AllowGitHubWebhook rule exists in waf.tf:
grep -A 20 'AllowGitHubWebhook' infra/ci/waf.tf

# If the rule is missing, apply via Terraform to restore it:
cd infra/ci
terraform plan -out ci-waf-fix.plan
# Verify plan: 0 to destroy, no server/RDS/ALB/ASG replacement
terraform apply ci-waf-fix.plan

Verification: Push a commit to any branch. In GitHub → repository Settings → Webhooks → Recent Deliveries, the new delivery returns HTTP 204. A Woodpecker pipeline appears in GET /api/repos/1/pipelines.

RCA: docs/incidents/2026-07-04-waf-webhook-block-batch1-parse-error.md

Failure mode E-2: Nil-pointer panic at repo activation (server/api/repo.go)

Symptom: POST /api/repos?forge_remote_id=ID returns a 500 panic trace referencing server/api/repo.go:149. Repo activation fails. Cause: Pre-v3.0 bug in v2.8.x — nil forge OAuth token on user record. Fixed in v3.x. Fix: This should not recur on v3.16.0. If it does, the admin user's forge OAuth token may be missing (user never completed OAuth flow). Operator must log out of Woodpecker UI and log back in via the GitHub OAuth flow to mint a fresh token. Verification: POST /api/repos?forge_remote_id=ID returns HTTP 200 JSON with "active":true.

Failure mode E-3: Pipeline compile errors after v3.x migration

Symptom: Manual pipeline trigger returns {"status":"error","errors":[{"type":"compiler","message":"go-yaml load error..."}]}. No GitHub commit status posted. Cause: .woodpecker/*.yaml pipeline definitions use v2.x syntax. v3.x YAML parser is stricter. Common causes: group: field (must be depends_on:), multi-line command continuation syntax, pipeline: keyword. Fix: Update .woodpecker/*.yaml files for v3.x syntax. See §v3.x pipeline YAML breaking changes above. Common v3 compile errors to check: - secret "NAME" not found — the secret named in from_secret: is not registered in Woodpecker. Either register the secret via POST /api/repos/{owner}/{name}/secrets or remove event: manual from the pipeline's when: block until the secret is available (cron triggers are unaffected). - Specified clone image does not match allow list, netrc is not injected — clone block uses woodpecker-ci/plugin-git (with hyphen) instead of woodpeckerci/plugin-git (without hyphen). See §Woodpecker clone plugin allow list. Verification: Manual pipeline trigger completes; status field is pending or running, not error. GitHub commit status appears on the HEAD commit.

Failure mode E-4: Shell-defined variables silently empty in pipeline commands

Symptom: Pipeline runs but produces wrong output or fails with empty-argument errors: - git checkout "" — git fails with "bad revision" - --base-sha "" — script fails with invalid SHA - MaxMind URL missing license_key parameter (download returns 401) - echo prints "Expected head revision: " with nothing after the colon All are caused by the same root issue.

Cause: WP v3 performs ${VAR} variable substitution at YAML-parse time on ALL text in the pipeline definition. Only WP built-in variables (CI_*, WOODPECKER_*) and secrets registered via from_secret: are available at parse time. Shell-defined variables (assigned by earlier shell commands: BASELINE=$(...), BASE_SHA=$(...), FILESIZE=$(...)) do NOT exist at parse time. WP substitutes them to empty string silently — no parse error, no warning, commands just receive empty arguments.

Known broken patterns (all fixed 2026-07-05): - git checkout "${BASELINE}" → git checkout "" — empty baseline SHA in vcpkg-manifest-check.yaml - --base-sha "${BASE_SHA}" → --base-sha "" — empty diff base in migration-collision-check.yaml - URL ...&license_key=${MAXMIND_LICENSE_KEY}&... → URL with missing key param in maxmind-asn-refresh.yaml - echo "${HEAD}" where HEAD is set by a prior shell command in alembic-version-check.yaml

Fix: Use $VAR (no braces) for shell-defined variables. WP only substitutes ${VAR} (brace form). Bare $VAR is passed through to the shell unchanged.

# WRONG — WP substitutes ${BASELINE} to empty at parse time
git -C /opt/vcpkg checkout "${BASELINE}"

# CORRECT — $BASELINE is passed to the shell which resolves it at runtime
git -C /opt/vcpkg checkout "$BASELINE"

Exception: WP built-in variables (${CI_COMMIT_BRANCH}, ${CI_REPO}, etc.) MUST use the ${VAR} brace form in YAML environment: blocks because they need WP substitution. In shell command strings, both $CI_COMMIT_BRANCH and ${CI_COMMIT_BRANCH} work since WP substitutes them either way; but prefer $VAR in commands for consistency.

Lint check: grep -rn '\${[A-Z][A-Z_0-9]*}' .woodpecker/ then manually verify each match is a WP built-in variable (CI_, WOODPECKER_) or a step environment: key. If it's a shell-defined variable, replace ${VAR} with $VAR.

Verification: Re-run the pipeline; commands receive the correct non-empty values.

Failure mode E-5: WP v3 YAML pre-processor rejects ${{ or ${VAR:-} — ALL pipelines break

Symptom: Every pipeline for every event (PR, push, cron, manual) fails immediately at compile time with "unable to parse variable name". No steps execute. GitHub commit status checks are absent (Woodpecker never posts them). The error is global — it affects ALL .woodpecker/*.yaml files in the repo, not just the file containing the offending pattern.

Cause: Woodpecker v3's substitution pre-processor scans ALL YAML text — including YAML comments — before handing the file to the YAML parser. Two patterns trigger the error: 1. ${{ — WP parses ${...} substitutions; the second { is not a valid identifier character and causes "unable to parse variable name". This is the GitHub Actions expression syntax. It is invalid in WP YAML even inside a comment line. 2. ${VAR:-default} — the :- inside {} is not a valid identifier character. Use the WP escape $${VAR:-default} which WP treats as a literal $ in the shell command.

Fix:

# Scan for both patterns:
grep -rn '\${{' .woodpecker/
grep -rn '\${[A-Z_]*:-' .woodpecker/

# For ${{ occurrences: remove or reword even in comments.
# BAD:  # GH_TOKEN: ${{ steps.mint.outputs.token || github.token }}
# GOOD: # GH_TOKEN: (mint_bot_token.outputs.token || github.token) from GHA workflow

# For ${VAR:-default}: use $$ prefix.
# BAD:  if [ -z "${GH_TOKEN:-}" ]; then
# GOOD: if [ -z "$${GH_TOKEN:-}" ]; then

Verification: curl -s -X POST https://ci.moosequest.net/api/repos/1/pipelines \ -H "Authorization: Bearer $WP_TOKEN" \ -H "Content-Type: application/json" \ -d '{"branch":"develop"}' | jq '{id:.id,status:.status,errors:.errors}' status should be pending or running, not error.

RCA: docs/incidents/2026-07-04-waf-webhook-block-batch1-parse-error.md

Failure mode F: RDS storage alarm firing

Symptom: ci-rds-storage-low CloudWatch alarm in ALARM state. Cause: Woodpecker pipeline history accumulating; database approaching 16 GB. Fix:

# Option 1: Purge old build logs in Woodpecker UI (Admin → Repositories)
# Option 2: Increase RDS allocated storage via Terraform:
# In terraform.tfvars, allocated_storage is managed by auto-scaling.
# The RDS resource has max_allocated_storage=100, so RDS auto-scales
# up to 100 GB automatically. If already at 100 GB, increase max_allocated_storage
# in rds.tf and apply.

# Check current storage:
aws rds describe-db-instances \
  --db-instance-identifier ci-woodpecker \
  --region us-east-2 \
  --query "DBInstances[0].AllocatedStorage"

Verification: ci-rds-storage-low alarm returns to OK state.

Failure mode F: Login buttons dead — "Login to CI" / "Abort" do nothing

Symptom: Browser passes the Google OIDC gate and reaches the Woodpecker login screen, but clicking "Login to CI" or "Abort" has no effect. No GitHub OAuth redirect occurs.

Cause: The ALB default OIDC rule is intercepting Woodpecker's own auth paths (/api/*, /authorize, /login, /logout). The Woodpecker SPA makes XHR calls to /api/* to initiate the GitHub OAuth flow; if those calls get a Google 302 redirect instead of a Woodpecker JSON response, the JavaScript login handler silently fails. Additionally, the GitHub OAuth callback at /authorize?code=...&state=... must reach Woodpecker directly — an OIDC interception at that path loses the GitHub code and state parameters.

Diagnose:

# Each of these should NOT redirect to accounts.google.com.
# Expected: 401 JSON (api/user), 303 to github.com (authorize/login), 303 to / (logout)
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/api/user
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/authorize
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/login

# Confirm ALB rule at priority 25 exists:
aws elbv2 describe-rules \
  --listener-arn arn:aws:elasticloadbalancing:us-east-2:521228113048:listener/app/ci-woodpecker-alb/2bf78b6ff4a0780f/2b0157eb96f64e49 \
  --region us-east-2 \
  --query 'Rules[?Priority==`25`]'

Fix (if priority-25 rule is missing — e.g., after a destructive terraform apply that lost the rule):

LISTENER_ARN="arn:aws:elasticloadbalancing:us-east-2:521228113048:listener/app/ci-woodpecker-alb/2bf78b6ff4a0780f/2b0157eb96f64e49"
TG_ARN="arn:aws:elasticloadbalancing:us-east-2:521228113048:targetgroup/ci-woodpecker-server-tg/89042c9f57a2d3ad"
aws elbv2 create-rule \
  --listener-arn "$LISTENER_ARN" \
  --priority 25 \
  --conditions '[{"Field":"path-pattern","PathPatternConfig":{"Values":["/api/*","/authorize","/login","/logout"]}}]' \
  --actions "[{\"Type\":\"forward\",\"TargetGroupArn\":\"$TG_ARN\"}]" \
  --tags '[{"Key":"Name","Value":"ci-woodpecker-auth-bypass-oidc"},{"Key":"Purpose","Value":"woodpecker-own-auth-endpoints"}]' \
  --region us-east-2

Verification:

# /api/user must return 401 (not a Google redirect)
curl -sS https://ci.moosequest.net/api/user
# Expected: "User not authorized" with HTTP 401

# /authorize must redirect to github.com
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/authorize
# Expected: 303 https://github.com/login/oauth/authorize?...

Then retry the login in a fresh browser tab.

Security note: Removing the OIDC gate from /api/*, /authorize, /login, and /logout does NOT expose these paths — Woodpecker enforces its own GitHub-OAuth + session auth on every /api/* endpoint (returns 401 for unauthenticated requests) and validates a signed JWT state on the OAuth callback. The WAF rate-limit rule still applies.

Emergency stop

To immediately halt all CI builds (no deploy risk):

# Option 1 (fastest — 30 seconds): Scale agents to 0
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 0 \
  --region us-east-2

# Option 2: Stop Woodpecker server entirely (queues all new jobs)
INSTANCE_ID="$(terraform -chdir=infra/ci output -raw woodpecker_server_instance_id)"
aws ssm start-session --target "$INSTANCE_ID" --region us-east-2
# In session:
sudo systemctl stop woodpecker

# Option 3: Quarantine at ALB level (blocks all traffic including webhooks)
# Update ALB listener to return 503 maintenance response:
# (do via AWS Console or CLI; do NOT `terraform apply` the change as it
#  will create drift — this is a break-glass manual action)

To resume: restart the service (sudo systemctl start woodpecker) and/or scale agents back up.

Teardown

# IMPORTANT: Remove the prevent_destroy lifecycle guard in rds.tf first
# by setting db_deletion_protection = false and applying, then destroy.
cd infra/ci
terraform apply -var="db_deletion_protection=false"
terraform destroy

# Bootstrap teardown (ONLY if you want to delete the state bucket too —
# this destroys the state of all other Terraform roots using this bucket):
# cd infra/ci/bootstrap
# Edit bootstrap/main.tf: remove lifecycle { prevent_destroy = true } from both resources
# terraform destroy
# (This also deletes moosequest-tf-state-us-east-2 and all state within it)

Security considerations

No SSH keys

All instances use SSM Session Manager for break-glass access. Key-pair based SSH is explicitly disabled (key_name = null). If SSM is unreachable, the VPC endpoints (ssm, ssmmessages, ec2messages) must be functioning. Check VPC endpoint health first.

IMDSv2

All EC2 instances have http_tokens = required. The instance metadata service is protected against SSRF. Containers cannot reach IMDS (hop limit 1 on server; limit 2 on agents to allow builds that need IAM credentials from the instance role).

Docker socket on agents

The Woodpecker agent container bind-mounts /var/run/docker.sock. This grants build containers root-equivalent access to the agent host. Acceptable for Phase 1 with internal-only, trusted pipeline definitions. Action item before adding external contributors: evaluate rootless Docker or Kaniko for image builds.

State encryption

Terraform state in S3 is encrypted (AES-256 via aws/s3 KMS). The ALB OIDC client secret appears in state because the aws_lb_listener resource stores it inline. Mitigations: state bucket access restricted to terraform-executor role, TLS enforced on all S3 access.

Secret rotation

All secrets in SSM are versioned (20 versions retained). Zero-downtime rotation: 1. Update SSM parameter value (--overwrite). 2. Restart Woodpecker server (reads SSM at boot via user-data). 3. For agent-secret: scale ASG to 0, update SSM, scale back up (agents read SSM at boot). 4. For OIDC client secret: update SSM → terraform apply (ALB listener rule update propagates within seconds, no downtime).

Phase 2 — cron pilot soak (active, started 2026-07-04)

Pilot crons

Three low-risk cron workflows run on Woodpecker agents IN PARALLEL with their GH Actions counterparts during the 48-hour soak window. GH Actions versions remain authoritative (issue filing, PR creation, notifications) until Phase 5 decommission per ADR-0134.

Woodpecker pipeline GH Actions counterpart Schedule (UTC) Secrets needed
.woodpecker/security-scan-nightly.yaml nightly-security-scan.yml 7 8 * * * none
.woodpecker/alembic-version-check.yaml alembic-version-cron.yml (head job only) 0 6 * * * none
.woodpecker/maxmind-asn-refresh.yaml maxmind-asn-refresh.yml (download+validate only) 0 3 1 * * MAXMIND_LICENSE_KEY (optional)

Cron registration — COMPLETED 2026-07-04 via API (cron IDs 1, 2, 3):

All 3 crons registered and enabled via POST /api/repos/1/cron + PATCH /api/repos/1/cron/{id} (enabled=true). Repo ID = 1, pipeline definitions live in .woodpecker/ directory.

To re-register if needed (repo ID = 1, TOKEN = admin Woodpecker JWT):

curl -sf -X POST -H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"security-scan-nightly","schedule":"7 8 * * *","branch":"develop"}' \
  https://ci.moosequest.net/api/repos/1/cron
# PATCH to enable: curl -X PATCH ... -d '{"enabled":true}' .../cron/{id}

Repo activation — COMPLETED 2026-07-04 (repo ID = 1):

Repo raxx-app/TradeMasterAPI activated via POST /api/repos?forge_remote_id=1172174585. GitHub webhook ID: 649361254 (active, events: push, pull_request, pull_request_review, deployment). Enabling the repo installs the Woodpecker webhook on the GitHub repo and lets Woodpecker post commit statuses back to GitHub.

Trigger manual pilot runs (after repo enabled and PR merged to develop):

# Option A: Woodpecker UI — TradeMasterAPI → run pipeline → select each .woodpecker/ file
# Option B: Woodpecker API (replace TOKEN and ORG):
curl -X POST "https://ci.moosequest.net/api/repos/raxx-app/TradeMasterAPI/pipelines" \
  -H "Authorization: Bearer $WOODPECKER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"branch": "develop"}'

Scale up 1 agent before manual trigger (if desired_capacity is 0):

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 1 \
  --region us-east-2

Scale back when pilot builds finish:

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 0 \
  --region us-east-2

48-hour soak plan

Window: 2026-07-04 through 2026-07-06 (48 hours from pilot registration)

What is being watched:

Signal Green Red
Build success parity Woodpecker security-scan + alembic-check exit 0 within 5 min of GH Actions counterpart Either exits non-zero where GH is green
Agent scale-up/down ASG launches spot instance, agent connects, job runs, agent terminates after scale-in Agent stuck in "launching" or build never picked up
GitHub commit status Woodpecker posts green status check on HEAD commit after each successful run Status missing or red when build was green
Cost Agent hours during soak < $1 total (2 daily short jobs, 1 manual) Runaway agent billing (stuck job, desired stays >0)
Flake rate Zero unexplained failures in 2 daily runs per pilot Any failure not explained by a known GH Actions failure in the same window

Success criteria for go/no-go on Phase 3: 1. All 3 pilot pipelines run successfully on real agents (build numbers captured) 2. GitHub commit statuses appear on the triggered commit within 5 minutes of build completion 3. No agent stuck-running cost event (desired > 0 more than 60 min after a job completes) 4. security-scan and alembic-check run successfully at their scheduled times on at least one natural cron trigger during the soak window 5. Zero data integrity events (no writes to prod, no emails sent, no GH issues created by the Woodpecker versions)

Go/no-go decision: operator reviews soak results on 2026-07-06. Green = proceed to Phase 3 (PR-gate workflow migration). Red = investigate before advancing.

Phase 2 enhancements (deferred until after soak)

Secrets — Wave A bootstrap set

Created: 2026-07-04 UTC Card: #4037 (Wave A PRE-REQ) ADR: ADR-0135 §1 Scope: Org-level (org_id=2, org=raxx-app) — inherited by all repos in the org.

These 5 secrets are the ONLY credentials stored in Woodpecker. All application secrets continue to live in Infisical vault and are fetched at pipeline runtime by the wpc_load_vault_secrets.py init step.

Bootstrap secret list

WP secret name What it holds Vault source path
INFISICAL_CLIENT_ID Infisical Universal Auth machine identity client ID Injected from GHA env at Wave A PRE-REQ; rotate via vault path raxx-secrets /Raxx/CI
INFISICAL_CLIENT_SECRET Infisical Universal Auth machine identity client secret Same as above
INFISICAL_PROJECT_ID Infisical project ID (MooseQuest workspace) Same as above
CF_ACCESS_CLIENT_ID CF Access service token ID for vault.raxx.app Same as above
CF_ACCESS_CLIENT_SECRET CF Access service token secret for vault.raxx.app Same as above

Event restrictions (bootstrap secrets): push, pull_request, tag, cron, manual. These are intentionally broad — they enable vault access for ALL pipeline types. Prod-only secrets (e.g. HEROKU_API_KEY prod) will be created with restricted events (tag, manual only) in Wave C.

How to verify secrets are present

WP_TOKEN="<admin JWT>"
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/orgs/2/secrets" \
  | python3 -c "
import json, sys
secrets = json.load(sys.stdin)
names = {s['name'] for s in secrets}
required = {'INFISICAL_CLIENT_ID','INFISICAL_CLIENT_SECRET','INFISICAL_PROJECT_ID',
            'CF_ACCESS_CLIENT_ID','CF_ACCESS_CLIENT_SECRET'}
missing = required - names
if missing:
    print('MISSING:', missing); sys.exit(1)
print(f'All 5 bootstrap secrets present. Total org secrets: {len(secrets)}')
"

How to rotate a bootstrap secret

Zero-downtime rotation. No deploy or agent restart needed — WP injects secrets at pipeline start, not agent boot.

# 1. Update the value in Infisical vault (via vault UI or REST API)
# 2. Read the new value from vault (never paste inline):
NEW_VALUE="$(infisical secrets get <SECRET_NAME> --path /Raxx/CI --env prod --plain)"

# 3. PATCH the Woodpecker org secret (replaces value, preserves event restrictions):
WP_TOKEN="<admin JWT>"
curl -sf -X PATCH \
  -H "Authorization: Bearer $WP_TOKEN" \
  -H "User-Agent: sre-agent/1.0" \
  -H "Content-Type: application/json" \
  -d "{\"name\":\"<SECRET_NAME>\",\"value\":\"${NEW_VALUE}\"}" \
  "https://ci.moosequest.net/api/orgs/2/secrets/<SECRET_NAME>" \
  && echo "rotated"
unset NEW_VALUE

# 4. Verify the next pipeline run using the secret completes successfully.
# No pipeline restart needed — rotation takes effect at next pipeline start.

WOODPECKER_ENV_FILE gap — FIXED (Wave A A1)

Finding (WP build #6, 2026-07-04): vault auth from a real WP agent succeeded but WOODPECKER_ENV_FILE was not set in the step environment. The inter-step env propagation write was silently skipped; the smoke-verify step failed with exit 1.

Root cause: Woodpecker v3 runs each step in a separate Docker container. Unlike GHA, there is no automatic inter-step env propagation. WOODPECKER_ENV_FILE is not set in step context on v3.16.0 — the GHA analogy was incorrect.

Fix (ADR-0135 Wave A A1): wpc_load_vault_secrets.py now writes export KEY=… lines (base64-encoded) to a shell-sourceable file (/tmp/vault-env.sh by default), which the SAME step sources before running its real work:

- pip install --quiet requests
- python3 scripts/ci/wpc_load_vault_secrets.py
- . /tmp/vault-env.sh && rm -f /tmp/vault-env.sh
- # real work that uses the vault-fetched env vars

Live proof: WP pipeline .woodpecker/vault-secret-smoke.yaml, captured in PR for ADR-0135 A1. All 30 Wave A vault-dependent pipelines must use the same pattern. See docs/architecture/woodpecker-wpc-step-usage.md §1 for the canonical snippet.

Wave B (ADR-0135) — heavy CI + scale arbiter

Status: active soak (2026-07-05 +). Go/no-go gate: §Gate-B2 in ADR-0135.

Scale arbiter (Lambda ci-scale-arbiter)

What it does: An EventBridge rate(2 minutes) rule invokes Lambda ci-scale-arbiter. The Lambda calls GET /api/queue/info on the WP server (via internal ALB, Bearer token from SSM /ci/woodpecker/admin-api-token), then: - pending > 0 AND running == 0 AND desired == 0 → SetDesiredCapacity 1 (burst scale-up) - pending == 0 AND running == 0 AND desired > 0 → SetDesiredCapacity 0 (idle scale-down) - Otherwise → no action (work in flight or already scaling)

IaC: infra/ci/lambda.tf — Lambda + EventBridge rule + EventBridge target + permission. Logs: /aws/lambda/ci-scale-arbiter (90-day retention).

Verify scale arbiter is active:

# Check EventBridge rule is enabled:
aws events describe-rule --name ci-scale-arbiter-trigger --region us-east-2 \
  --query "State"
# Expected: "ENABLED"

# Check last 5 invocations:
aws logs filter-log-events \
  --log-group-name /aws/lambda/ci-scale-arbiter \
  --region us-east-2 \
  --start-time $(python3 -c "import time; print(int((time.time()-3600)*1000))") \
  | python3 -c "
import sys, json
for ev in json.load(sys.stdin).get('events', []):
    print(ev['timestamp'], ev['message'][:120])
"

Manual scale-up (emergency — bypasses arbiter):

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 1 \
  --region us-east-2

Manual scale-down (after incident resolved):

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 0 \
  --region us-east-2

Failure mode G: Scale arbiter not triggering scale-up

Symptom: Push event triggered a pipeline, but no agent picks it up within 5 minutes. ASG desired stays 0.

Diagnose:

# Check EventBridge rule is enabled:
aws events describe-rule --name ci-scale-arbiter-trigger --region us-east-2 --query "State"

# Check Lambda error rate in last 30 min:
aws logs filter-log-events \
  --log-group-name /aws/lambda/ci-scale-arbiter \
  --region us-east-2 \
  --start-time $(python3 -c "import time; print(int((time.time()-1800)*1000))") \
  --filter-pattern "ERROR"

# Manually invoke the arbiter to test it:
aws lambda invoke \
  --function-name ci-scale-arbiter \
  --region us-east-2 \
  --payload '{}' \
  /tmp/arbiter-result.json \
  --log-type Tail \
  --query "LogResult" --output text | base64 -d
cat /tmp/arbiter-result.json

Common causes: - SSM /ci/woodpecker/admin-api-token expired or wrong → Lambda logs WP API error 401 - WP server unhealthy (500 from /api/queue/info) → Lambda logs WP API error 500; fix WP server first - Lambda execution role lost SSM permission → Lambda logs permission error

Fix: If Lambda is healthy but ASG is stuck, scale up manually (see above). Then investigate the root cause from Lambda logs before trusting automatic scale-down.

Verification: aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ci-woodpecker-agents-asg --region us-east-2 --query "AutoScalingGroups[0].DesiredCapacity" returns 1.

Postgres heavy CI pipeline (.woodpecker/backend-tests-postgres.yaml)

Check context posted to GitHub: ci/woodpecker/backend-tests-postgres

What it runs: 1. path-guard — exit 0 if no backend_v2/** changes (informational, never blocks) 2. wait-for-postgres — polls pg_isready -h postgres -p 5432 until the postgres:15 service container is accepting connections 3. backend-tests-postgres: - apt-get: postgresql-client (psql for create_ci_postgres_roles.sh) + postgresql (pg_ctl for pytest-postgresql) - pip: root requirements.txt + backend_v2/requirements.txt (pytest-postgresql≥5.0) + pytest/cov/timeout - PATH extended to /usr/lib/postgresql/15/bin for pg_ctl binary - vault secrets loaded via wpc_load_vault_secrets.py (VAULT_ALLOW_MISSING=true — vault failure is non-fatal) - create_ci_postgres_roles.sh against the postgres service container - (cd backend_v2 && alembic upgrade head) — subshell keeps CWD at repo root - pytest run (first pass: collect failures, --maxfail=5; second pass: coverage) - python -m coverage report --fail-under=30 — gate

Service container behavior: The postgres:15 service container is on the same Docker bridge network as build steps. Connect via hostname postgres (= service name), port 5432. No port-mapping needed. pg_isready -h postgres is the correct readiness check.

conftest.py postgres path: When DATABASE_URL starts with postgresql://, conftest.py does NOT connect to the service container directly. It imports pytest-postgresql and calls postgresql_proc to spawn an EPHEMERAL pg_ctl-managed postgres instance on a random port. The service container is used only for the explicit alembic upgrade head step. This design (from ADR-0070) provides test isolation — each test session gets a fresh, isolated DB. Consequence: postgresql server package + pytest-postgresql>=5.0 are both required in the pipeline, not just postgresql-client.

Wave B coexistence: The GHA backend-tests-postgres job (in .github/workflows/ci.yml) continues to run during the Wave B soak. Remove the GHA job only after 2-week green parity per ADR-0135 §Gate-B2.

Failure mode H: backend-tests-postgres step exits 1 with no coverage data

Symptom: backend-tests-postgres step exits with exit_code=1. Coverage report logs Coverage.py warning: No data was collected. or similar.

Causes (in order of probability): 1. cd persisting — a command runs cd backend_v2 && ... without a subshell, so subsequent pytest commands look in backend_v2/backend_v2/tests which does not exist. Pytest exits 4 (path not found), swallowed by || echo; coverage has no data. Fix: use (cd backend_v2 && ...). 2. pg_ctl not on PATH — pytest-postgresql cannot spawn ephemeral postgres. Fix: install the full postgresql server package (not just postgresql-client) and export /usr/lib/postgresql/15/bin to PATH. 3. pytest-postgresql not installed — conftest.py falls through to pytest.skip() on all DB fixtures; tests are collected but skipped; coverage may be below 30%. Fix: install -r backend_v2/requirements.txt in the pipeline step. 4. Alembic migration fails — a new migration has a syntax error or references a missing role. Check: (cd backend_v2 && python3 -m alembic upgrade head) logs.

Diagnose:

# Check the pipeline step output — look for these signals:
# "No such file or directory: backend_v2/tests"  → cause 1 (cd persistence)
# "FileNotFoundError: [Errno 2]" or "pg_ctl: command not found" → cause 2
# "pytest-postgresql is not installed" → cause 3
# "alembic.exc.CommandError" or migration error → cause 4

Verification: Pipeline step exits 0; coverage report logs TOTAL ... XX% where XX ≥ 30.

Branch protection — Wave B dual-gate coexistence

Principle (ADR-0135 §4): During the Wave B soak, develop branch protection requires BOTH GHA and WP status checks. WP checks are added ALONGSIDE GHA — GHA checks are NOT removed until §Gate-B2 parity is confirmed (2-week green run).

Currently required checks on develop:

GHA (app_id: null):
  smoke_suite
  base_branch_lint
  stale-branch-guard
  pii-scan
  flag_promotion_check
  migration-gate

WP (added by Wave B, context prefix: ci/woodpecker/):
  ci/woodpecker/backend-tests-postgres
  ci/woodpecker/ci-pr
  ci/woodpecker/migration-collision-check

To add a new WP check to branch protection (operator or sre-agent):

# Read current protection config:
gh api repos/raxx-app/TradeMasterAPI/branches/develop/protection \
  --jq '.required_status_checks'

# Add a check (PATCH is additive; include all existing + new contexts):
gh api repos/raxx-app/TradeMasterAPI/branches/develop/protection \
  -X PUT \
  --input - <<'EOF'
{
  "required_status_checks": {
    "strict": false,
    "contexts": [
      "smoke_suite",
      "base_branch_lint",
      "stale-branch-guard",
      "pii-scan",
      "flag_promotion_check",
      "migration-gate",
      "ci/woodpecker/backend-tests-postgres",
      "ci/woodpecker/ci-pr",
      "ci/woodpecker/migration-collision-check"
    ]
  },
  "enforce_admins": false,
  "required_pull_request_reviews": null,
  "restrictions": null
}
EOF
# Verify after update:
gh api repos/raxx-app/TradeMasterAPI/branches/develop/protection \
  --jq '.required_status_checks.contexts'

To remove GHA checks (§Gate-B2 — only after 2-week green parity): Repeat the PUT above with only the WP contexts in the contexts array, omitting the GHA app_id: null entries. The GHA workflow files in .github/workflows/ remain for reference but their check contexts are no longer blocking.

WP status context name format (confirmed 2026-07-05): WP v3 posts GitHub status checks using ci/woodpecker/{event}/{pipeline-name} format, where {event} is push for push events and pr for pull_request events. Example: - ci/woodpecker/pr/backend-tests-postgres (PR trigger) - ci/woodpecker/push/backend-tests-postgres (push trigger) - ci/woodpecker/pr/ci-pr (ci-pr pipeline, PR trigger)

Note: some older YAML pipeline comments use ci/woodpecker/{pipeline-name} without the event prefix — this was a WP v2 / early v3 format. The {event} prefix has been confirmed in production on WP v3.16.0.

WP built-in variables in environment: blocks use the brace form ${CI_COMMIT_BRANCH}, ${CI_REPO}, ${CI_FORGE_TOKEN} — WP substitutes these at YAML parse time. Using bare $CI_COMMIT_BRANCH in an environment: value passes the literal dollar-sign string to the container, NOT the variable value. See Failure mode E-4 exception. Fixed in Wave B for ci-pr.yaml, console-degraded-auto-file.yaml, migration-collision-check.yaml.

ci/woodpecker/pr/ci-pr known failure (Wave A issue): The ci-pr.yaml pipeline reports failure on every PR since its introduction in Wave A batch 2 (PR #4051, 2026-07-05). Now that the ALB /api/* OIDC bypass rule is deployed and in Terraform state (priority 25: /api/*, /authorize, /login, /logout — imported 2026-07-08), the WP API is accessible from external clients without SSM. The ci-pr failure can now be diagnosed directly via the WP API. Do NOT add ci/woodpecker/pr/ci-pr to develop branch protection until this failure is diagnosed and resolved.

WP API access from external clients: The ALB now has a priority-25 OIDC-bypass rule for /api/*, /authorize, /login, and /logout (deployed 2026-07-08, managed as aws_lb_listener_rule.woodpecker_own_auth in infra/ci/alb.tf). Direct WP API calls from outside the VPC now work without authentication at the ALB layer — Woodpecker enforces its own token auth on all /api/* requests. Example:

curl -s -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  https://ci.moosequest.net/api/queue/info | python3 -m json.tool

Escalation

Wake the operator when: - RDS data integrity suspected (unclean shutdown, storage corruption) - GitHub OAuth App or Google OIDC credentials compromised - ACM certificate cannot be renewed (Dyn CNAME missing after 60-day renewal attempt) - Woodpecker server repeatedly crashes and restart loop does not recover within 15 minutes - WAF blocking appears to be a false positive on production pipeline webhooks

Monitoring — external watchdog (2026-07-07)

Woodpecker health is monitored by scripts/ops/woodpecker_healthcheck.py, run every 15 min by .github/workflows/woodpecker-healthcheck.yml — intentionally on GitHub Actions, because a monitor hosted on Woodpecker can't fire when Woodpecker is down. Do not retire that workflow.

Checks: (1) server reachable (/api/version 200), (2) TLS cert expiry (< 21d fails), (3) pipeline health (stuck > 90 min, or ≥4 of last 5 finished failed) — tier 3 only runs if a WOODPECKER_API_TOKEN repo secret is set.

Alerting is edge-triggered via a woodpecker-health GitHub issue anchor: one email to kris@moosequest.net on healthy→unhealthy, one on recovery — no per-run spam during a sustained outage. Email goes via Postmark (POSTMARK_OPS_ALERT_TOKEN + OPS_ALERT_FROM secrets).

Verify the alert path any time: Actions → "Woodpecker healthcheck" → Run workflow → tick test_email (sends one confirmation email, then exits). Or locally: POSTMARK_SERVER_TOKEN=... python3 scripts/ops/woodpecker_healthcheck.py --test-email.

To enable the richer pipeline checks, add a WOODPECKER_API_TOKEN repo secret (a WP user API token from the ci.moosequest.net UI).