Raxx · internal docs

internal · gated

Woodpecker CI runbook

System: ci-woodpecker (Woodpecker CI on EC2 — ci.moosequest.net) Owner: operator / sre-agent ADR: ADR-0134 (Candidate B, operator-authorized 2026-07-03) Card: #4012 (Phase 1 baseline IaC) Last incident: 2026-08-12 (three sibling WP crons re-verified via scoped trigger found real, previously-unvalidated drift — npm ci lockfile gap and a missing init_engine() bootstrap — plus a cron-trigger workflow-scoping anomaly; see docs/incidents/2026-08-12-cron-reverify-batch-e2e-warm-waf.md) Last reviewed: 2026-08-12

Architecture summary

Internet
  └── Oracle Dyn CNAME: ci.moosequest.net → ALB
        └── ALB (public, us-east-2a/b)
              ├── AWS WAF WebACL (managed rules + rate limit)
              ├── :80 → redirect :443
              └── :443 → listener rules
                    ├── /hook*         → Woodpecker server :8000 (no OIDC — priority 10)
                    ├── /api/badge*    → Woodpecker server :8000 (no OIDC — priority 20)
                    ├── /api/* /authorize /login /logout → Woodpecker :8000 (no OIDC — priority 25)
                    └── (default)      → Google Workspace OIDC → Woodpecker :8000
                          └── Woodpecker server (t4g.small, private subnet us-east-2a)
                                ├── docker-compose: woodpecker-server container
                                ├── gRPC :9000 → build agents (spot ASG, c7a/c6a.xlarge)
                                └── PostgreSQL → RDS db.t4g.micro (private subnet)

All instances: IMDSv2 required, no SSH keys, SSM Session Manager for break-glass access
Secrets: SSM Parameter Store SecureString (/ci/* path)

Build cache (2026-08-05, card #4393): the agent fleet's Docker daemon now points at a pull-through registry mirror + PyPI caching proxy on the ci-monitoring box, collapsing NAT data-processing cost from ~1.49 TB/mo of repeat Docker Hub/PyPI traffic. See docs/ops/runbooks/ci-build-cache.md for the full system runbook — do not debug agent build-cache issues here.

VPC endpoints single-AZ (2026-08-05, card #4394): the four interface endpoints (ssm/ssmmessages/ec2messages/logs, infra/ci/vpc.tf) are now scoped to us-east-2a only (previously both AZs) — halves their hourly cost from ~$53.76/mo to ~$26.88/mo. The agent fleet's AZ placement is NOT pinned — an agent that lands in us-east-2b (still in the ASG's failover subnet set) falls back to routing SSM/CloudWatch-Logs traffic via NAT instead of the local endpoint. This is expected and accepted (see vpc.tf header comment for the full risk writeup) — it does not break SSM access, it just routes it differently. An AZ-A outage would take VPC-endpoint-based SSM access down fleet-wide until endpoints are restored or re-added to AZ-B (rollback: re-add aws_subnet.private[1].id to each endpoint's subnet_ids and re-apply).

How to tell it's broken

Apply order (Phase 1 first-time deploy)

Prerequisites

Before running terraform apply:

  1. AWS credentials configured for the Terraform executor role (see §IAM executor role)
  2. State bootstrap completed (see §State bootstrap below)
  3. SSM parameter placeholders created by bootstrap apply
  4. Google OAuth client created (see §Google OAuth client setup)
  5. GitHub OAuth App registered (see §GitHub OAuth App registration)
  6. Real SSM parameter values filled (see §Secrets fill-in checklist)

State bootstrap (one-time)

cd infra/ci/bootstrap
terraform init
terraform plan
terraform apply
# Note the state_bucket_name and lock_table_name outputs.
# These match the values hardcoded in infra/ci/main.tf backend block.

Bootstrap state lives at infra/ci/bootstrap/terraform.tfstate — do NOT commit this file (it is gitignored). Keep a copy in a secure location (Google Drive ops folder).

Main apply

cd infra/ci
terraform init   # connects to S3 backend created by bootstrap
terraform plan -out=ci-phase1.plan -var-file=terraform.tfvars
terraform apply ci-phase1.plan

After first apply

  1. Read outputs: bash terraform output -json

  2. Add two CNAME records in Oracle Dyn (see §DNS setup below).

  3. Confirm ACM certificate validates (5-30 min after DNS propagation): bash aws acm describe-certificate \ --certificate-arn "$(terraform output -raw acm_certificate_arn)" \ --region us-east-2 \ --query "Certificate.Status" # Expected: "ISSUED"

  4. Fill SSM parameter values (see §Secrets fill-in checklist).

  5. Restart Woodpecker server to pick up real secrets: bash INSTANCE_ID="$(terraform output -raw woodpecker_server_instance_id)" aws ssm start-session --target "$INSTANCE_ID" --region us-east-2 # In the session: sudo systemctl restart woodpecker sudo docker compose -f /opt/woodpecker/docker-compose.yml logs -f

  6. Confirm SNS subscription for ops@raxx.app (check email, click confirmation link).

  7. Verify Woodpecker UI at https://ci.moosequest.net (requires Dyn CNAME and ACM ISSUED).

DNS setup (Oracle Dyn)

Add two CNAME records in Oracle Dyn after first terraform apply:

Record 1: ACM validation CNAME

Type:  CNAME
Name:  <terraform output acm_validation_cname_name>
       (looks like: _acm-challenge.ci.moosequest.net. or _abc123.ci.moosequest.net.)
Value: <terraform output acm_validation_cname_value>
       (looks like: _xyz789.acm-validations.aws.)
TTL:   300

Purpose: Allows AWS Certificate Manager to validate ownership of ci.moosequest.net and issue the TLS certificate. This record can remain indefinitely (ACM uses it for renewals too).

Record 2: ci.moosequest.net service CNAME

Type:  CNAME
Name:  ci.moosequest.net
Value: <terraform output alb_dns_name>
       (looks like: ci-woodpecker-alb-1234567890.us-east-2.elb.amazonaws.com)
TTL:   300 (lower during initial setup for faster propagation)

Purpose: Points ci.moosequest.net to the ALB. Required for the Woodpecker UI, webhook delivery, and the OIDC redirect from Google.

After adding both records: Wait for DNS propagation (typically 2-5 minutes for Dyn's authoritative nameservers; up to 5 minutes for resolvers to pick up TTL=300 records).

Google OAuth client setup (operator browser steps)

The ALB OIDC action uses Google Workspace OAuth 2.0. Create a client once:

  1. Open Google Cloud Console: https://console.cloud.google.com/
  2. Select the MooseQuest project (or create one named "moosequest-ops").
  3. Navigate to APIs & Services → Credentials.
  4. Click Create Credentials → OAuth client ID.
  5. Application type: Web application.
  6. Name: Woodpecker CI ALB OIDC
  7. Authorized redirect URIs — add: https://ci.moosequest.net/oauth2/idpresponse (This is the AWS ALB OIDC callback endpoint — do NOT change the path.)
  8. Click Create.
  9. Copy the Client ID and Client Secret.

  10. Fill SSM parameters (see §Secrets fill-in checklist): ```bash aws ssm put-parameter \ --name "/ci/oidc/google-client-id" \ --value "PASTE_CLIENT_ID_HERE" \ --type SecureString \ --overwrite \ --region us-east-2

    aws ssm put-parameter \ --name "/ci/oidc/google-client-secret" \ --value "PASTE_CLIENT_SECRET_HERE" \ --type SecureString \ --overwrite \ --region us-east-2 ```

Important: The redirect URI must be https://ci.moosequest.net/oauth2/idpresponse exactly. The ALB uses this path for the OIDC code exchange. If ci.moosequest.net is not yet resolving, the Google OAuth client can be created first (the URI is validated by Google only at runtime, not at creation time).

Status (2026-07-03): Google OIDC credentials are already staged in Infisical vault at raxx-secrets /Raxx/CI (prod) as CI_GOOGLE_OIDC_CLIENT_ID and CI_GOOGLE_OIDC_CLIENT_SECRET. Skip steps 1–9 above. Proceed directly to the SSM fill step, reading from vault (requires a session with CF Access service token):

GOOGLE_CLIENT_ID="$(infisical secrets get CI_GOOGLE_OIDC_CLIENT_ID \
  --projectId raxx-secrets --path /Raxx/CI --env prod --plain)"
GOOGLE_CLIENT_SECRET="$(infisical secrets get CI_GOOGLE_OIDC_CLIENT_SECRET \
  --projectId raxx-secrets --path /Raxx/CI --env prod --plain)"
aws ssm put-parameter --name "/ci/oidc/google-client-id" \
  --value "$GOOGLE_CLIENT_ID" --type SecureString --overwrite --region us-east-2
aws ssm put-parameter --name "/ci/oidc/google-client-secret" \
  --value "$GOOGLE_CLIENT_SECRET" --type SecureString --overwrite --region us-east-2
unset GOOGLE_CLIENT_ID GOOGLE_CLIENT_SECRET

GitHub OAuth App registration (operator browser steps)

Woodpecker uses a GitHub OAuth App for SCM integration (pipeline triggers, status checks, user authentication).

  1. Go to GitHub → Settings → Developer settings → OAuth Apps → New OAuth App: - Application name: Woodpecker CI — moosequest.net - Homepage URL: https://ci.moosequest.net - Authorization callback URL: https://ci.moosequest.net/authorize
  2. Click Register application.
  3. On the app page, click Generate a new client secret.
  4. Copy Client ID and Client Secret.
  5. Fill SSM parameters: ```bash aws ssm put-parameter \ --name "/ci/woodpecker/github-client-id" \ --value "PASTE_CLIENT_ID_HERE" \ --type SecureString \ --overwrite \ --region us-east-2

aws ssm put-parameter \ --name "/ci/woodpecker/github-client-secret" \ --value "PASTE_CLIENT_SECRET_HERE" \ --type SecureString \ --overwrite \ --region us-east-2 ```

Secrets fill-in checklist

After terraform apply, all SSM parameters exist as PLACEHOLDER values. Fill them in this order (prerequisite steps noted):

# 1. Woodpecker admin GitHub username (your GitHub login)
aws ssm put-parameter \
  --name "/ci/woodpecker/admin-user" \
  --value "GITHUB_USERNAME" \
  --type SecureString \
  --overwrite \
  --region us-east-2

# 2. Woodpecker agent↔server RPC secret (must be identical on server and agents)
AGENT_SECRET="$(openssl rand -hex 32)"
aws ssm put-parameter \
  --name "/ci/woodpecker/agent-secret" \
  --value "$AGENT_SECRET" \
  --type SecureString \
  --overwrite \
  --region us-east-2
# Save $AGENT_SECRET in 1Password / secure notes — needed if agents are
# rebuilt without Terraform re-reading SSM.

# 3. PostgreSQL password
DB_PASSWORD="$(openssl rand -base64 32 | tr -d /=+ | head -c 32)"
aws ssm put-parameter \
  --name "/ci/woodpecker/db-password" \
  --value "$DB_PASSWORD" \
  --type SecureString \
  --overwrite \
  --region us-east-2
# ALSO update the RDS master password to match:
aws rds modify-db-instance \
  --db-instance-identifier ci-woodpecker \
  --master-user-password "$DB_PASSWORD" \
  --apply-immediately \
  --region us-east-2

# 4. GitHub OAuth App credentials (from §GitHub OAuth App registration above)
aws ssm put-parameter \
  --name "/ci/woodpecker/github-client-id" \
  --value "GITHUB_CLIENT_ID" \
  --type SecureString \
  --overwrite \
  --region us-east-2

aws ssm put-parameter \
  --name "/ci/woodpecker/github-client-secret" \
  --value "GITHUB_CLIENT_SECRET" \
  --type SecureString \
  --overwrite \
  --region us-east-2

# 5. Google Workspace OIDC credentials (from §Google OAuth client setup above)
aws ssm put-parameter \
  --name "/ci/oidc/google-client-id" \
  --value "GOOGLE_CLIENT_ID" \
  --type SecureString \
  --overwrite \
  --region us-east-2

aws ssm put-parameter \
  --name "/ci/oidc/google-client-secret" \
  --value "GOOGLE_CLIENT_SECRET" \
  --type SecureString \
  --overwrite \
  --region us-east-2

After filling all 7 parameters, restart the Woodpecker server (see §Apply order step 5).

IAM executor role

The identity running terraform apply needs the ci-terraform-executor-policy attached (provisioned by the IaC itself on first apply — bootstrapping caveat below).

Bootstrapping caveat: On first apply, the executor policy does not yet exist (Terraform creates it). The operator's identity must have sufficient permissions to create it. Simplest path for first apply: use an existing role with broad permissions, then attach the scoped policy and switch to it for subsequent applies.

Do NOT use claude-infisical-bootstrap — that user must not be widened per ADR-0134.

Recommended first-apply path: 1. Create a new IAM role ci-terraform-executor with a trust policy for the operator's IAM user or SSO identity. 2. Attach AdministratorAccess temporarily for the first apply only. 3. After first apply, detach AdministratorAccess and attach ci-terraform-executor-policy (ARN from terraform output terraform_executor_policy_arn). 4. All subsequent applies use the scoped policy only.

Smoke checklist (after first apply + secrets filled)

Run in order; each step should pass before proceeding:

Current versions

Bumping server + agent versions

Server and agent versions are coupled: agent LT user-data embeds the agent image tag, so a version bump requires terraform apply to publish a new LT version. Agents then relaunch from the new LT on next scale-up.

Normal upgrade path (immutable re-provision): 1. Update woodpecker_server_version and woodpecker_agent_version in infra/ci/variables.tf. 2. Remove the lifecycle { ignore_changes = [user_data] } block from server.tf (if present — see below). 3. terraform plan — confirm server is marked for replace (not destroy). 4. terraform apply — server is replaced, new LT version published. 5. Re-add the lifecycle block if needed (see below).

In-place upgrade path (SSM patch, no terraform replace): If a version was applied via SSM (e.g., systemctl restart with a new image tag in docker-compose), the TF state user_data hash will drift from the computed hash, causing user_data_replace_on_change = true to schedule a server replacement even though the server is already healthy at the new version.

Fix: add lifecycle { ignore_changes = [user_data] } to aws_instance.woodpecker_server in infra/ci/server.tf BEFORE running terraform plan. This makes the plan read 0 destroy instead of scheduling a replacement. The lifecycle block is already present on develop as of 2026-07-04 (incident: docs/incidents/2026-07-04-woodpecker-v3-upgrade.md).

When you next want to intentionally re-provision the server (e.g., major version upgrade via immutable path), remove the lifecycle { ignore_changes = [user_data] } block from server.tf, confirm the plan proposes a replace (not a destroy), then apply.

Woodpecker clone plugin allow list

Woodpecker v3 maintains a trusted-clone plugin list. The default list (per --plugins-trusted-clone) is:

docker.io/woodpeckerci/plugin-git:2.9.2
docker.io/woodpeckerci/plugin-git
quay.io/woodpeckerci/plugin-git

Pipeline YAML must use woodpeckerci/plugin-git (Docker Hub org, no hyphen). The common mistake is woodpecker-ci/plugin-git (GHCR-style, with hyphen) which is NOT in the list — Woodpecker will refuse to inject netrc credentials, and private repo clones fail.

v3.x API changes (vs v2.x)

v3.x env vars — confirmed unchanged from v2.x

WOODPECKER_GITHUB_CLIENT, WOODPECKER_GITHUB_SECRET, WOODPECKER_ADMIN, WOODPECKER_AGENT_SECRET, WOODPECKER_DATABASE_DATASOURCE, WOODPECKER_DATABASE_DRIVER, WOODPECKER_HOST, WOODPECKER_OPEN, WOODPECKER_GRPC_ADDR, WOODPECKER_GRPC_SECURE, WOODPECKER_SERVER_ADDR, WOODPECKER_LOG_LEVEL. No renames required for current config.

v3.x pipeline YAML breaking changes

Known failure modes

Failure mode A: ACM certificate stuck in PENDING_VALIDATION

Symptom: aws acm describe-certificate returns Status: PENDING_VALIDATION for more than 30 minutes. Cause: ACM validation CNAME not present in Dyn, or DNS propagation not yet complete. Fix:

# Verify the validation record is expected:
terraform output acm_validation_cname_name
terraform output acm_validation_cname_value
# Check Dyn DNS console — confirm CNAME exists with the exact name and value.
# Test DNS propagation:
dig CNAME "$(terraform output -raw acm_validation_cname_name)" @8.8.8.8
# Expected: non-empty CNAME answer matching the validation value.

Verification: aws acm describe-certificate --certificate-arn ... --query "Certificate.Status" returns "ISSUED".

Failure mode B: Woodpecker server not starting

Symptom: ALB health checks fail; /healthz returns 502/503. Cause: Docker container failed to start (bad SSM secrets, RDS unreachable, image pull failure). Fix:

# Get instance ID
INSTANCE_ID="$(terraform -chdir=infra/ci output -raw woodpecker_server_instance_id)"
# Open SSM session
aws ssm start-session --target "$INSTANCE_ID" --region us-east-2
# In session:
sudo systemctl status woodpecker
sudo docker compose -f /opt/woodpecker/docker-compose.yml ps
sudo docker compose -f /opt/woodpecker/docker-compose.yml logs woodpecker-server
# Common root causes:
#   - "invalid database" / connection refused: DB password wrong or RDS SG blocks
#   - "cannot reach github.com": NAT Gateway issue; check route tables
#   - "placeholder" in env: SSM parameters not filled; fill and restart
sudo systemctl restart woodpecker

Verification: curl -sf http://localhost:8000/healthz from SSM session returns 200.

Failure mode B-2: Forge connectivity timeout — container-level (host network healthy)

Symptom: No new pipelines are created after webhook events. Woodpecker server logs show repeated context deadline exceeded errors:

"could not get folder from forge: context deadline exceeded"
"error while fetching config '' in 'refs/tags/v1.10.3' with user: 'MooseQuest', and did not get any config"

ALB /healthz returns 200. EC2 host can reach api.github.com (200 in <100ms). Container network appears normal.

Cause: Go HTTP client connection pool in the server process accumulates stale/half-open TCP connections after extended uptime (typically 3–7 days with high webhook volume). The pool reuses dead sockets; requests time out before falling back to fresh connections. Distinct from Failure mode B (container down) and WAF blocks (GitHub sees 403/200 but pipeline not created).

Diagnose:

# 1. Confirm host network is healthy (should be <100ms, 200):
aws ssm send-command --region us-east-2 --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["curl -sf --max-time 5 https://api.github.com/zen; echo exit:$?"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"
# retrieve with: aws ssm get-command-invocation --command-id <ID> --instance-id i-082ee835595d90ae0

# 2. Confirm container error pattern (look for 'context deadline exceeded' in last 5 min):
aws ssm send-command --region us-east-2 --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["docker logs woodpecker-woodpecker-server-1 --since 5m 2>&1 | grep deadline"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

# 3. Confirm conntrack is not saturated (should be << /proc/sys/net/netfilter/nf_conntrack_max):
aws ssm send-command --region us-east-2 --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["wc -l /proc/net/nf_conntrack && cat /proc/sys/net/netfilter/nf_conntrack_max"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

# 4. Confirm queue is empty before restarting:
WP_TOKEN=$(aws ssm get-parameter --name /ci/woodpecker/admin-api-token --with-decryption \
  --region us-east-2 --query "Parameter.Value" --output text)
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  https://ci.moosequest.net/api/queue/info | python3 -c "
import sys,json; d=json.load(sys.stdin)
print('pending:', len(d.get('pending',[])), 'running:', len(d.get('running',[])))
"

Fix (requires operator authorization for service restart):

# Option A — SSM send-command (no interactive session needed, ~15s):
CMD_ID=$(aws ssm send-command \
  --region us-east-2 \
  --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["sudo systemctl restart woodpecker && echo RESTART_OK"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])")
sleep 8
aws ssm get-command-invocation \
  --command-id "$CMD_ID" --instance-id i-082ee835595d90ae0 \
  --query "StandardOutputContent" --output text

# Option B — SSM interactive session:
aws ssm start-session --target i-082ee835595d90ae0 --region us-east-2
# In session:
# sudo systemctl restart woodpecker
# sudo docker compose -f /opt/woodpecker/docker-compose.yml logs -f --tail=20

After restart: Wait 30s, then redeliver the blocked webhook OR manually trigger the pipeline:

# Redeliver the specific delivery (get ID from GitHub repo Settings → Webhooks → Recent Deliveries):
gh api "repos/raxx-app/TradeMasterAPI/hooks/649427232/deliveries/<DELIVERY_ID>/attempts" --method POST

# OR manual pipeline trigger for deploy-prod (break-glass):
WP_TOKEN=$(aws ssm get-parameter --name /ci/woodpecker/admin-api-token --with-decryption \
  --region us-east-2 --query "Parameter.Value" --output text)
curl -sf -X POST \
  -H "Authorization: Bearer $WP_TOKEN" \
  -H "Content-Type: application/json" \
  -H "User-Agent: sre-agent/1.0" \
  -d '{"branch":"main","variables":{"DEPLOY_PROD_CONFIRM":"yes","CI_COMMIT_BRANCH":"main"}}' \
  https://ci.moosequest.net/api/repos/raxx-app/TradeMasterAPI/pipelines

Note on create-github-release: The deploy-prod pipeline's create-github-release step swallows errors as non-fatal when CI_COMMIT_TAG is empty (manual trigger). If the pipeline was triggered manually (not via a tag event), the GitHub Release will NOT be created automatically. Create it manually after the pipeline succeeds:

gh api repos/raxx-app/TradeMasterAPI/releases -X POST \
  --field tag_name="vX.Y.Z" \
  --field target_commitish="<main HEAD SHA>" \
  --field name="Release vX.Y.Z" \
  --field generate_release_notes=true

Verification:

# Forge connectivity restored: new pipeline created after webhook redeliver
WP_TOKEN=$(aws ssm get-parameter --name /ci/woodpecker/admin-api-token --with-decryption \
  --region us-east-2 --query "Parameter.Value" --output text)
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/pipelines?page=1&perPage=3" | \
  python3 -c "import sys,json; [print(f\"#{p['number']} {p['status']}\") for p in json.load(sys.stdin)]"
# Expected: most recent pipeline is running or success, not error/pending indefinitely

Prevention (action items): #1 (forge connectivity probe in healthcheck) and #2 (weekly restart schedule) in docs/incidents/2026-07-09-wp-forge-context-deadline.md.

RCA: docs/incidents/2026-07-09-wp-forge-context-deadline.md

Failure mode B-3: ForgeConnectivityDown alert — false positive from monitoring-stack restart

Symptom: ForgeConnectivityDown Prometheus alert fires (email to ops@raxx.app). WP server logs show NO auth errors. GitHub webhooks are being received and processed normally.

Cause: The alert fires on probe_success{job="blackbox-forge"} == 0 for: 10m. This probe runs inside the blackbox-exporter container on the ci-monitoring EC2 (i-0a656c6631ddc45b3). If the container is restarted by an SSM command (e.g., injecting config changes for an unrelated probe), consecutive Prometheus scrapes at the 5m interval can find the probe failing long enough to cross the for: 10m threshold. The Woodpecker server's actual forge connectivity to GitHub is unaffected.

Distinguish from genuine forge failure (Failure mode B-2):

# 1. Check WP server logs — genuine failure shows 'context deadline exceeded' or auth errors:
aws ssm send-command --region us-east-2 --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["docker logs woodpecker-woodpecker-server-1 --since=15m 2>&1 | grep -E \"deadline|401|403|bad credential|forge\" | tail -20"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

# 2. Check if blackbox-exporter recently restarted:
aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["docker inspect blackbox-exporter --format \"Started: {{.State.StartedAt}} Status: {{.State.Status}}\""]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

# 3. Directly probe the forge target from the monitoring box (should return probe_success 1):
aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["curl -s \"http://localhost:9115/probe?target=https://api.github.com/zen&module=github_api_2xx\" | grep probe_success"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

If false positive (blackbox restarted, probe now succeeds): No fix needed. Alert clears automatically on next Prometheus scrape (max 5m wait). Check Alertmanager current state:

aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["curl -s http://localhost:9093/api/v2/alerts | python3 -m json.tool | head -10"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

Expected: [] (empty).

If genuine forge connectivity failure: See Failure mode B-2 (stale TCP pool → restart woodpecker-server container) or check GitHub status at https://githubstatus.com + review security group outbound rules on i-082ee835595d90ae0.

Prevention: The hardened alert expression guards against blackbox-exporter restarts:

probe_success{job="blackbox-forge"} == 0 and on() (up{job="blackbox-http"} == 1)

Fires only when the blackbox-exporter is reachable but the forge probe fails. This is the expression baked into the IaC template (terraform/ci-monitoring/templates/cloud-init.sh.tpl) as of the 2026-07-14 drift consolidation.

RCA: docs/incidents/2026-07-14-forge-connectivity-down.md

Failure mode C: Agents not connecting to server

Symptom: Pipelines queue forever; Woodpecker Admin → Agents shows 0 connected. Cause: Agent-secret mismatch, SG blocking port 9000, or agents not running. Fix:

# Check ASG desired capacity:
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names ci-woodpecker-agents-asg \
  --region us-east-2 \
  --query "AutoScalingGroups[0].{desired:DesiredCapacity,inService:Instances[?LifecycleState=='InService'].InstanceId}" \
  --output json

# If desired=0, scale up:
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 1 \
  --region us-east-2

# If agents are running but not connected, SSH into an agent via SSM:
# (find instance ID from ASG describe output above)
aws ssm start-session --target i-XXXXXXXXXX --region us-east-2
# Check agent logs:
sudo docker compose -f /opt/woodpecker-agent/docker-compose.yml logs woodpecker-agent
# Common error: "rpc error: code = Unauthenticated" → agent-secret mismatch
# Verify server and agent SSM parameter have same value:
aws ssm get-parameter --name /ci/woodpecker/agent-secret --with-decryption --region us-east-2

Verification: Woodpecker Admin → Agents shows agent with status "connected".

Failure mode D: WAF blocking legitimate webhook traffic

Symptom: GitHub webhook delivery shows 403; WAF metrics show blocked requests on /hook*. Cause: GitHub IP range hit the WAF rate limit or a managed rule matched the payload. Fix:

# Check WAF logs in CloudWatch:
aws logs filter-log-events \
  --log-group-name aws-waf-logs-ci-woodpecker \
  --region us-east-2 \
  --filter-pattern '{ $.action = "BLOCK" }' \
  --start-time "$(date -d '-1 hour' +%s000)" \
  | jq '.events[].message | fromjson | {action,httpRequest,terminatingRuleId}'
# Identify which rule blocked the request.
# If rate limit: increase var.waf_rate_limit_webhook_per_5min in terraform.tfvars + apply.
# If managed rule: add an override in waf.tf for that specific rule (count instead of block).

Verification: GitHub webhook delivery test returns 200 in repository settings.

Failure mode D-2: WAF SizeRestrictions_BODY blocks all GitHub webhook deliveries

Symptom: Every GitHub webhook delivery to ci.moosequest.net/api/hook returns HTTP 403. No Woodpecker pipelines trigger on PR or push events. WAF CloudWatch logs show terminatingRuleId: SizeRestrictions_BODY from AWSManagedRulesCommonRuleSet. Cause: The AWSManagedRulesCommonRuleSet SizeRestrictions_BODY rule enforces an 8192-byte body inspection cap. Real GitHub PR/push webhook payloads are 12–36 KB and exceed this cap. The managed rule blocks the request before Woodpecker ever sees it. This is distinct from rate-limit blocks (Failure mode D) — it fires on the very first delivery regardless of volume. Fix:

# Confirm the root cause: look for SizeRestrictions_BODY in the terminatingRuleId field
aws logs filter-log-events \
  --log-group-name aws-waf-logs-ci-woodpecker \
  --region us-east-2 \
  --filter-pattern '{ $.action = "BLOCK" }' \
  --start-time "$(date -d '-1 hour' +%s000)" \
  | python3 -c "
import sys, json
for line in sys.stdin:
    try:
        obj = json.loads(line)
        for ev in obj.get('events', []):
            msg = json.loads(ev['message'])
            print(msg.get('terminatingRuleId'), msg.get('httpRequest', {}).get('uri'))
    except Exception:
        pass
"

# If you see SizeRestrictions_BODY, verify the AllowGitHubWebhook rule exists in waf.tf:
grep -A 20 'AllowGitHubWebhook' infra/ci/waf.tf

# If the rule is missing, apply via Terraform to restore it:
cd infra/ci
terraform plan -out ci-waf-fix.plan
# Verify plan: 0 to destroy, no server/RDS/ALB/ASG replacement
terraform apply ci-waf-fix.plan

Verification: Push a commit to any branch. In GitHub → repository Settings → Webhooks → Recent Deliveries, the new delivery returns HTTP 204. A Woodpecker pipeline appears in GET /api/repos/1/pipelines.

RCA: docs/incidents/2026-07-04-waf-webhook-block-batch1-parse-error.md

Failure mode E-2: Nil-pointer panic at repo activation (server/api/repo.go)

Symptom: POST /api/repos?forge_remote_id=ID returns a 500 panic trace referencing server/api/repo.go:149. Repo activation fails. Cause: Pre-v3.0 bug in v2.8.x — nil forge OAuth token on user record. Fixed in v3.x. Fix: This should not recur on v3.16.0. If it does, the admin user's forge OAuth token may be missing (user never completed OAuth flow). Operator must log out of Woodpecker UI and log back in via the GitHub OAuth flow to mint a fresh token. Verification: POST /api/repos?forge_remote_id=ID returns HTTP 200 JSON with "active":true.

Failure mode E-3: Pipeline compile errors after v3.x migration

Symptom: Manual pipeline trigger returns {"status":"error","errors":[{"type":"compiler","message":"go-yaml load error..."}]}. No GitHub commit status posted. Cause: .woodpecker/*.yaml pipeline definitions use v2.x syntax. v3.x YAML parser is stricter. Common causes: group: field (must be depends_on:), multi-line command continuation syntax, pipeline: keyword. Fix: Update .woodpecker/*.yaml files for v3.x syntax. See §v3.x pipeline YAML breaking changes above. Common v3 compile errors to check: - secret "NAME" not found — the secret named in from_secret: is not registered in Woodpecker. Either register the secret via POST /api/repos/{owner}/{name}/secrets or remove event: manual from the pipeline's when: block until the secret is available (cron triggers are unaffected). - Specified clone image does not match allow list, netrc is not injected — clone block uses woodpecker-ci/plugin-git (with hyphen) instead of woodpeckerci/plugin-git (without hyphen). See §Woodpecker clone plugin allow list. Verification: Manual pipeline trigger completes; status field is pending or running, not error. GitHub commit status appears on the HEAD commit.

Failure mode E-4: Shell-defined variables silently empty in pipeline commands

Symptom: Pipeline runs but produces wrong output or fails with empty-argument errors: - git checkout "" — git fails with "bad revision" - --base-sha "" — script fails with invalid SHA - MaxMind URL missing license_key parameter (download returns 401) - echo prints "Expected head revision: " with nothing after the colon All are caused by the same root issue.

Cause: WP v3 performs ${VAR} variable substitution at YAML-parse time on ALL text in the pipeline definition. Only WP built-in variables (CI_*, WOODPECKER_*) and secrets registered via from_secret: are available at parse time. Shell-defined variables (assigned by earlier shell commands: BASELINE=$(...), BASE_SHA=$(...), FILESIZE=$(...)) do NOT exist at parse time. WP substitutes them to empty string silently — no parse error, no warning, commands just receive empty arguments.

Known broken patterns (all fixed 2026-07-05, except the 2026-08-19 recurrence below): - git checkout "${BASELINE}" → git checkout "" — empty baseline SHA in vcpkg-manifest-check.yaml - --base-sha "${BASE_SHA}" → --base-sha "" — empty diff base in migration-collision-check.yaml - URL ...&license_key=${MAXMIND_LICENSE_KEY}&... → URL with missing key param in maxmind-asn-refresh.yaml - echo "${HEAD}" where HEAD is set by a prior shell command in alembic-version-check.yaml - Recurrence, 2026-08-19: git push --force heroku "${SUBTREE_SHA}:refs/heads/main" → git push --force heroku ":refs/heads/main" (empty-source refspec) in deploy-velvet.yaml's deploy step. Introduced 2026-07-08 (6bf74fefab), three days after this class was first fixed elsewhere — the fix was never generalized into an enforced check, so a new file reintroduced the identical anti-pattern. Latent/undetected for ~6 weeks because the step's path filter (velvet/** or the workflow file itself) never matched a release/main push until PR #4474 incidentally touched deploy-velvet.yaml and tripped it. Heroku's pre-receive hook refused the empty-refspec push ("cannot delete main branch") — loud failure, but the 6-week latency before that first trigger is the real gap. Discovered during the release-2026.08.19 staging shepherd run. See docs/incidents/2026-08-19-ci-boundary-velvet-deploy-false-positive-refspec.md and issue #4513 (fix filed, not yet applied).

Fix: Use $VAR (no braces) for shell-defined variables. WP only substitutes ${VAR} (brace form). Bare $VAR is passed through to the shell unchanged.

# WRONG — WP substitutes ${BASELINE} to empty at parse time
git -C /opt/vcpkg checkout "${BASELINE}"

# CORRECT — $BASELINE is passed to the shell which resolves it at runtime
git -C /opt/vcpkg checkout "$BASELINE"

Exception: WP built-in variables (${CI_COMMIT_BRANCH}, ${CI_REPO}, etc.) MUST use the ${VAR} brace form in YAML environment: blocks because they need WP substitution. In shell command strings, both $CI_COMMIT_BRANCH and ${CI_COMMIT_BRANCH} work since WP substitutes them either way; but prefer $VAR in commands for consistency.

Lint check: grep -rn '\${[A-Z][A-Z_0-9]*}' .woodpecker/ then manually verify each match is a WP built-in variable (CI_, WOODPECKER_) or a step environment: key. If it's a shell-defined variable, replace ${VAR} with $VAR.

Verification: Re-run the pipeline; commands receive the correct non-empty values.

Failure mode E-5: WP v3 YAML pre-processor rejects ${{ — ALL pipelines break

Symptom: Every pipeline for every event (PR, push, cron, manual) fails immediately at compile time with "unable to parse variable name". No steps execute. GitHub commit status checks are absent (Woodpecker never posts them). The error is global — it affects ALL .woodpecker/*.yaml files in the repo, not just the file containing the offending pattern.

Cause: Woodpecker v3's substitution pre-processor scans ALL YAML text — including YAML comments — before handing the file to the YAML parser. ${{ — WP parses ${...} substitutions; the second { is not a valid identifier character and causes "unable to parse variable name". This is the GitHub Actions expression syntax. It is invalid in WP YAML even inside a comment line.

Fix:

# Scan for the pattern:
grep -rn '\${{' .woodpecker/

# BAD:  # GH_TOKEN: ${{ steps.mint.outputs.token || github.token }}
# GOOD: # GH_TOKEN: (mint_bot_token.outputs.token || github.token) from GHA workflow

Verification: curl -s -X POST https://ci.moosequest.net/api/repos/1/pipelines \ -H "Authorization: Bearer $WP_TOKEN" \ -H "Content-Type: application/json" \ -d '{"branch":"develop"}' | jq '{id:.id,status:.status,errors:.errors}' status should be pending or running, not error.

RCA: docs/incidents/2026-07-04-waf-webhook-block-batch1-parse-error.md

Correction (2026-08-10, #4452): The original version of this entry also listed ${VAR:-default} (the :- default-expansion form) as a second global-breakage pattern, recommending the $${VAR:-default} escape for all cases. That was an overbroad reading of the 2026-07-04 incident — the confirmed root cause there was ${{ alone (see the RCA's own "What didn't go well": "Hypothesis 1 [${VAR:-}] was plausible but incomplete"). In practice, ${VAR:-default} on a genuine WP built-in (CI_COMMIT_BEFORE, CI_COMMIT_TARGET_BRANCH, etc.) compiles and runs cleanly — WP substitutes the brace form at parse time same as bare ${VAR}, and a present-but-unset built-in still resolves through the default expansion. Confirmed working precedents already live on develop before this correction: .woodpecker/ci-pr.yaml (BASE="${CI_COMMIT_BEFORE:-}", FALLBACK="${CI_COMMIT_TARGET_BRANCH:-develop}", merged 2026-07-10, passing as a required check daily since) and this same runbook's own §Failure mode E-6 fix (${CI_COMMIT_TARGET_BRANCH:-develop}, #4347, merged 2026-07-25). Re-verified live via a scoped draft-PR pipeline run for .woodpecker/queue-docker-smoke.yaml (BEFORE="${CI_COMMIT_BEFORE:-}") — see docs/incidents/2026-08-10-queue-docker-smoke-ci-commit-before-unset.md.

The $${VAR:-default} escape is still the correct choice for two distinct, narrower cases — neither of which applies to WP built-ins: 1. A secret registered via from_secret: that you want resolved by the shell at runtime from the step's environment: block, not baked into the compiled pipeline JSON at parse time (secret-leak-into-pipeline-definition risk). See .woodpecker/console-degraded-auto-file.yaml's $${CI_FORGE_TOKEN} usage. 2. A variable that is not a WP built-in and not registered as a step environment: key at all (WP has nothing to substitute it with) — bare ${VAR} in that case silently resolves to empty string at parse time (see Failure mode E-4) rather than raising a compile error, so $${VAR:-default} is the only way to defer resolution to the shell.

Updated fix/lint guidance:

# Always invalid — scan and remove/reword even in comments:
grep -rn '\${{' .woodpecker/

# ${VAR:-default} on a WP built-in (CI_*, WOODPECKER_*): fine as-is, no escape needed.
# ${VAR:-default} on a secret or step-local shell var: escape with $$ so the shell
# resolves it at runtime instead of WP substituting (or silently blanking) it at parse time.

Failure mode E-6: stale-branch-guard red repo-wide — two independent dimensions

Symptom: ci/woodpecker/pr/ci-pr fails on the stale-branch-guard step across MULTIPLE unrelated open PRs simultaneously (not just one branch) — a strong signal this is a guard-logic bug, not a genuinely-stale individual branch.

Cause: stale-branch-guard (.woodpecker/ci-pr.yaml + scripts/ci/check_stale_branch.sh) has two independent bugs found so far, each of which can false-fail every open PR at once. Diagnose which one (or both) is active before touching thresholds:

  1. Base-ref hardcoded (fixed 2026-07-25, #4347): the guard always compared against origin/develop regardless of the PR's real base. PRs based on main or release (ADR-0115 emergency-hotfix path) always failed. Fixed by deriving GITHUB_BASE_REF from ${CI_COMMIT_TARGET_BRANCH:-develop}.
  2. Absolute fork-point age (fixed 2026-07-30): the age check computed now - <fork-point commit's own timestamp>. For a branch cut fresh off the base branch's current tip, the fork point IS the base tip — so whenever the base branch (develop) went quiet for longer than STALE_BRANCH_MAX_AGE_HOURS (48h), EVERY PR branch failed the age check with 0 commits behind. Fixed by measuring age from the OLDEST commit the PR branch is missing from the base branch, and skipping the age check entirely when COMMITS_BEHIND == 0.

Diagnose which dimension is active:

git fetch origin develop --quiet
git log -1 --format='%H %ci %s' origin/develop   # is develop's tip itself old (>48h)?

# For the specific failing PR branch:
git fetch origin <pr-branch> --quiet
FORK_SHA=$(git merge-base origin/develop <pr-branch>)
git rev-list --count "${FORK_SHA}..origin/develop"   # commits behind — should be low for a fresh branch
git log -1 --format='%ci' "${FORK_SHA}"              # fork-point date — if old AND commits-behind is 0,
                                                       # this is dimension 2 (absolute-age bug)

If commits-behind is 0 (or small) but the fork-point date is old, this is dimension 2 — the base branch has gone quiet, not the PR branch falling behind. Confirm scripts/ci/check_stale_branch.sh measures age from the oldest MISSING commit (not the fork-point's own date) — if it still uses FORK_TS=$(git log -1 --format="%ct" "${FORK_SHA}"), the 2026-07-30 fix has regressed or was reverted.

Fix: See scripts/ci/check_stale_branch.sh for the corrected logic (relative-to-base-progression age, skipped when 0 commits behind). Do NOT raise STALE_BRANCH_MAX_AGE_HOURS as a workaround — that weakens the gate for genuinely-stale branches without fixing the underlying measurement bug.

Verification: Cut a fresh test branch off origin/develop HEAD and run the guard locally:

git fetch origin develop --quiet
git checkout -b /tmp-stale-guard-check origin/develop
GITHUB_BASE_REF=develop bash scripts/ci/check_stale_branch.sh
# Expected: "Commits behind origin/develop: 0" and exit 0, regardless of how old
# develop's tip commit is.
git checkout - && git branch -D /tmp-stale-guard-check

RCA: docs/incidents/2026-07-25-ci-pr-stale-branch-guard-hardcoded-base.md (dimension 1), docs/incidents/2026-07-30-ci-pr-stale-branch-guard-absolute-age.md (dimension 2)

Failure mode F: RDS storage alarm firing

Symptom: ci-rds-storage-low CloudWatch alarm in ALARM state. Cause: Woodpecker pipeline history accumulating; database approaching 16 GB. Fix:

# Option 1: Purge old build logs in Woodpecker UI (Admin → Repositories)
# Option 2: Increase RDS allocated storage via Terraform:
# In terraform.tfvars, allocated_storage is managed by auto-scaling.
# The RDS resource has max_allocated_storage=100, so RDS auto-scales
# up to 100 GB automatically. If already at 100 GB, increase max_allocated_storage
# in rds.tf and apply.

# Check current storage:
aws rds describe-db-instances \
  --db-instance-identifier ci-woodpecker \
  --region us-east-2 \
  --query "DBInstances[0].AllocatedStorage"

Verification: ci-rds-storage-low alarm returns to OK state.

Failure mode F: Login buttons dead — "Login to CI" / "Abort" do nothing

Symptom: Browser passes the Google OIDC gate and reaches the Woodpecker login screen, but clicking "Login to CI" or "Abort" has no effect. No GitHub OAuth redirect occurs.

Cause: The ALB default OIDC rule is intercepting Woodpecker's own auth paths (/api/*, /authorize, /login, /logout). The Woodpecker SPA makes XHR calls to /api/* to initiate the GitHub OAuth flow; if those calls get a Google 302 redirect instead of a Woodpecker JSON response, the JavaScript login handler silently fails. Additionally, the GitHub OAuth callback at /authorize?code=...&state=... must reach Woodpecker directly — an OIDC interception at that path loses the GitHub code and state parameters.

Diagnose:

# Each of these should NOT redirect to accounts.google.com.
# Expected: 401 JSON (api/user), 303 to github.com (authorize/login), 303 to / (logout)
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/api/user
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/authorize
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/login

# Confirm ALB rule at priority 25 exists:
aws elbv2 describe-rules \
  --listener-arn arn:aws:elasticloadbalancing:us-east-2:521228113048:listener/app/ci-woodpecker-alb/2bf78b6ff4a0780f/2b0157eb96f64e49 \
  --region us-east-2 \
  --query 'Rules[?Priority==`25`]'

Fix (if priority-25 rule is missing — e.g., after a destructive terraform apply that lost the rule):

LISTENER_ARN="arn:aws:elasticloadbalancing:us-east-2:521228113048:listener/app/ci-woodpecker-alb/2bf78b6ff4a0780f/2b0157eb96f64e49"
TG_ARN="arn:aws:elasticloadbalancing:us-east-2:521228113048:targetgroup/ci-woodpecker-server-tg/89042c9f57a2d3ad"
aws elbv2 create-rule \
  --listener-arn "$LISTENER_ARN" \
  --priority 25 \
  --conditions '[{"Field":"path-pattern","PathPatternConfig":{"Values":["/api/*","/authorize","/login","/logout"]}}]' \
  --actions "[{\"Type\":\"forward\",\"TargetGroupArn\":\"$TG_ARN\"}]" \
  --tags '[{"Key":"Name","Value":"ci-woodpecker-auth-bypass-oidc"},{"Key":"Purpose","Value":"woodpecker-own-auth-endpoints"}]' \
  --region us-east-2

Verification:

# /api/user must return 401 (not a Google redirect)
curl -sS https://ci.moosequest.net/api/user
# Expected: "User not authorized" with HTTP 401

# /authorize must redirect to github.com
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/authorize
# Expected: 303 https://github.com/login/oauth/authorize?...

Then retry the login in a fresh browser tab.

Security note: Removing the OIDC gate from /api/*, /authorize, /login, and /logout does NOT expose these paths — Woodpecker enforces its own GitHub-OAuth + session auth on every /api/* endpoint (returns 401 for unauthenticated requests) and validates a signed JWT state on the OAuth callback. The WAF rate-limit rule still applies.

Emergency stop

To immediately halt all CI builds (no deploy risk):

# Option 1 (fastest — 30 seconds): Scale agents to 0
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 0 \
  --region us-east-2

# Option 2: Stop Woodpecker server entirely (queues all new jobs)
INSTANCE_ID="$(terraform -chdir=infra/ci output -raw woodpecker_server_instance_id)"
aws ssm start-session --target "$INSTANCE_ID" --region us-east-2
# In session:
sudo systemctl stop woodpecker

# Option 3: Quarantine at ALB level (blocks all traffic including webhooks)
# Update ALB listener to return 503 maintenance response:
# (do via AWS Console or CLI; do NOT `terraform apply` the change as it
#  will create drift — this is a break-glass manual action)

To resume: restart the service (sudo systemctl start woodpecker) and/or scale agents back up.

Teardown

# IMPORTANT: Remove the prevent_destroy lifecycle guard in rds.tf first
# by setting db_deletion_protection = false and applying, then destroy.
cd infra/ci
terraform apply -var="db_deletion_protection=false"
terraform destroy

# Bootstrap teardown (ONLY if you want to delete the state bucket too —
# this destroys the state of all other Terraform roots using this bucket):
# cd infra/ci/bootstrap
# Edit bootstrap/main.tf: remove lifecycle { prevent_destroy = true } from both resources
# terraform destroy
# (This also deletes moosequest-tf-state-us-east-2 and all state within it)

Security considerations

No SSH keys

All instances use SSM Session Manager for break-glass access. Key-pair based SSH is explicitly disabled (key_name = null). If SSM is unreachable, the VPC endpoints (ssm, ssmmessages, ec2messages) must be functioning. Check VPC endpoint health first.

IMDSv2

All EC2 instances have http_tokens = required. The instance metadata service is protected against SSRF. Containers cannot reach IMDS (hop limit 1 on server; limit 2 on agents to allow builds that need IAM credentials from the instance role).

Docker socket on agents

The Woodpecker agent container bind-mounts /var/run/docker.sock. This grants build containers root-equivalent access to the agent host. Acceptable for Phase 1 with internal-only, trusted pipeline definitions. Action item before adding external contributors: evaluate rootless Docker or Kaniko for image builds.

Trusted-repo posture (WP Docker volume mounts) — enabled 2026-07-09 (#4107)

What "trusted" means in Woodpecker v3: Woodpecker v3 exposes three trust dimensions per repo: - trusted.network — pipeline steps can use host networking - trusted.volumes — pipeline steps can mount host paths via volumes: in step YAML - trusted.security — pipeline steps can request privileged containers

Current state (raxx-app/TradeMasterAPI, repo ID = 1):

Flag Value Rationale
trusted.network true Set at activation; required for gRPC connectivity in build steps
trusted.volumes true Set 2026-07-09 (#4107) — enables pipeline steps to mount /var/run/docker.sock from the host (already bind-mounted into the agent container by user_data_agent.sh.tpl)
trusted.security false Not needed; DinD is NOT the accepted posture (OQ-1 resolved = socket-mount)

Why trusted.volumes = true is required: The host Docker socket is bind-mounted into the WP agent container at startup (/var/run/docker.sock:/var/run/docker.sock in agent docker-compose.yml). For a PIPELINE STEP to reach the same socket, the pipeline YAML must declare a volumes: block that mounts the socket into the step container. Woodpecker only permits this when the repo has trusted.volumes = true. Without it, the step container cannot reach the host daemon and docker buildx build fails with "Cannot connect to the Docker daemon".

Security trade-off: Any pipeline step can mount arbitrary host paths and interact with the host Docker daemon, giving it root-equivalent access to the agent host. This is accepted for Phase 1 because: 1. Single-repo WP instance — pipeline YAML is controlled by the operator alone 2. No external contributors — no untrusted code executes in pipelines 3. Prod secrets are restricted by event type on WP secrets (push, pull_request only, not manual for prod-credential secrets) 4. Agent instances are spot/ephemeral — compromised agent is terminated on scale-in

Action items before adding external contributors: - Evaluate rootless Docker (no root socket exposure) or Kaniko (no Docker daemon) - OR implement fork isolation (WP require_approval: all for non-org members) - See #4106 tracking note

How to view/change the trusted state:

WP_TOKEN="<admin JWT from ci.moosequest.net>"
# View:
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: raxx-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['trusted'])"

# Set (trusted.volumes=true, trusted.security=false):
curl -sf -X PATCH \
  -H "Authorization: Bearer $WP_TOKEN" \
  -H "User-Agent: raxx-agent/1.0" \
  -H "Content-Type: application/json" \
  -d '{"trusted":{"network":true,"volumes":true,"security":false}}' \
  "https://ci.moosequest.net/api/repos/1"

This setting persists in the Woodpecker database — it is not managed by Terraform (WP API-only). If the WP server is rebuilt from scratch, re-apply this setting before W7/W8 pipelines run.

State encryption

Terraform state in S3 is encrypted (AES-256 via aws/s3 KMS). The ALB OIDC client secret appears in state because the aws_lb_listener resource stores it inline. Mitigations: state bucket access restricted to terraform-executor role, TLS enforced on all S3 access.

Secret rotation

All secrets in SSM are versioned (20 versions retained). Zero-downtime rotation: 1. Update SSM parameter value (--overwrite). 2. Restart Woodpecker server (reads SSM at boot via user-data). 3. For agent-secret: scale ASG to 0, update SSM, scale back up (agents read SSM at boot). 4. For OIDC client secret: update SSM → terraform apply (ALB listener rule update propagates within seconds, no downtime).

Phase 2 — cron pilot soak (active, started 2026-07-04)

Pilot crons

Three low-risk cron workflows run on Woodpecker agents IN PARALLEL with their GH Actions counterparts during the 48-hour soak window. GH Actions versions remain authoritative (issue filing, PR creation, notifications) until Phase 5 decommission per ADR-0134.

Woodpecker pipeline GH Actions counterpart Schedule (UTC) Secrets needed
.woodpecker/security-scan-nightly.yaml nightly-security-scan.yml 7 8 * * * none
.woodpecker/alembic-version-check.yaml alembic-version-cron.yml (head job only) 0 6 * * * none
.woodpecker/maxmind-asn-refresh.yaml maxmind-asn-refresh.yml (download+validate only) 0 3 1 * * MAXMIND_LICENSE_KEY (optional)

Cron registration — COMPLETED 2026-07-04 via API (cron IDs 1, 2, 3):

All 3 crons registered and enabled via POST /api/repos/1/cron + PATCH /api/repos/1/cron/{id} (enabled=true). Repo ID = 1, pipeline definitions live in .woodpecker/ directory.

To re-register if needed (repo ID = 1, TOKEN = admin Woodpecker JWT):

curl -sf -X POST -H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"security-scan-nightly","schedule":"7 8 * * *","branch":"develop"}' \
  https://ci.moosequest.net/api/repos/1/cron
# PATCH to enable: curl -X PATCH ... -d '{"enabled":true}' .../cron/{id}

Repo activation — COMPLETED 2026-07-04 (repo ID = 1):

Repo raxx-app/TradeMasterAPI activated via POST /api/repos?forge_remote_id=1172174585. GitHub webhook ID: 649361254 (active, events: push, pull_request, pull_request_review, deployment). Enabling the repo installs the Woodpecker webhook on the GitHub repo and lets Woodpecker post commit statuses back to GitHub.

Trigger manual pilot runs (after repo enabled and PR merged to develop):

# Option A: Woodpecker UI — TradeMasterAPI → run pipeline → select each .woodpecker/ file
# Option B: Woodpecker API (replace TOKEN and ORG):
curl -X POST "https://ci.moosequest.net/api/repos/raxx-app/TradeMasterAPI/pipelines" \
  -H "Authorization: Bearer $WOODPECKER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"branch": "develop"}'

Scale up 1 agent before manual trigger (if desired_capacity is 0):

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 1 \
  --region us-east-2

Scale back when pilot builds finish:

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 0 \
  --region us-east-2

48-hour soak plan

Window: 2026-07-04 through 2026-07-06 (48 hours from pilot registration)

What is being watched:

Signal Green Red
Build success parity Woodpecker security-scan + alembic-check exit 0 within 5 min of GH Actions counterpart Either exits non-zero where GH is green
Agent scale-up/down ASG launches spot instance, agent connects, job runs, agent terminates after scale-in Agent stuck in "launching" or build never picked up
GitHub commit status Woodpecker posts green status check on HEAD commit after each successful run Status missing or red when build was green
Cost Agent hours during soak < $1 total (2 daily short jobs, 1 manual) Runaway agent billing (stuck job, desired stays >0)
Flake rate Zero unexplained failures in 2 daily runs per pilot Any failure not explained by a known GH Actions failure in the same window

Success criteria for go/no-go on Phase 3: 1. All 3 pilot pipelines run successfully on real agents (build numbers captured) 2. GitHub commit statuses appear on the triggered commit within 5 minutes of build completion 3. No agent stuck-running cost event (desired > 0 more than 60 min after a job completes) 4. security-scan and alembic-check run successfully at their scheduled times on at least one natural cron trigger during the soak window 5. Zero data integrity events (no writes to prod, no emails sent, no GH issues created by the Woodpecker versions)

Go/no-go decision: operator reviews soak results on 2026-07-06. Green = proceed to Phase 3 (PR-gate workflow migration). Red = investigate before advancing.

Phase 2 enhancements (deferred until after soak)

Secrets — Wave A bootstrap set

Created: 2026-07-04 UTC Card: #4037 (Wave A PRE-REQ) ADR: ADR-0135 §1 Scope: Org-level (org_id=2, org=raxx-app) — inherited by all repos in the org.

These 5 secrets are the ONLY credentials stored in Woodpecker. All application secrets continue to live in Infisical vault and are fetched at pipeline runtime by the wpc_load_vault_secrets.py init step.

Bootstrap secret list

WP secret name What it holds Vault source path
INFISICAL_CLIENT_ID Infisical Universal Auth machine identity client ID Injected from GHA env at Wave A PRE-REQ; rotate via vault path raxx-secrets /Raxx/CI
INFISICAL_CLIENT_SECRET Infisical Universal Auth machine identity client secret Same as above
INFISICAL_PROJECT_ID Infisical project ID (MooseQuest workspace) Same as above
CF_ACCESS_CLIENT_ID CF Access service token ID for vault.raxx.app Same as above
CF_ACCESS_CLIENT_SECRET CF Access service token secret for vault.raxx.app Same as above

Event restrictions (bootstrap secrets): push, pull_request, tag, cron, manual. These are intentionally broad — they enable vault access for ALL pipeline types. Prod-only secrets (e.g. HEROKU_API_KEY prod) will be created with restricted events (tag, manual only) in Wave C.

How to verify secrets are present

WP_TOKEN="<admin JWT>"
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/orgs/2/secrets" \
  | python3 -c "
import json, sys
secrets = json.load(sys.stdin)
names = {s['name'] for s in secrets}
required = {'INFISICAL_CLIENT_ID','INFISICAL_CLIENT_SECRET','INFISICAL_PROJECT_ID',
            'CF_ACCESS_CLIENT_ID','CF_ACCESS_CLIENT_SECRET'}
missing = required - names
if missing:
    print('MISSING:', missing); sys.exit(1)
print(f'All 5 bootstrap secrets present. Total org secrets: {len(secrets)}')
"

How to rotate a bootstrap secret

Zero-downtime rotation. No deploy or agent restart needed — WP injects secrets at pipeline start, not agent boot.

# 1. Update the value in Infisical vault (via vault UI or REST API)
# 2. Read the new value from vault (never paste inline):
NEW_VALUE="$(infisical secrets get <SECRET_NAME> --path /Raxx/CI --env prod --plain)"

# 3. PATCH the Woodpecker org secret (replaces value, preserves event restrictions):
WP_TOKEN="<admin JWT>"
curl -sf -X PATCH \
  -H "Authorization: Bearer $WP_TOKEN" \
  -H "User-Agent: sre-agent/1.0" \
  -H "Content-Type: application/json" \
  -d "{\"name\":\"<SECRET_NAME>\",\"value\":\"${NEW_VALUE}\"}" \
  "https://ci.moosequest.net/api/orgs/2/secrets/<SECRET_NAME>" \
  && echo "rotated"
unset NEW_VALUE

# 4. Verify the next pipeline run using the secret completes successfully.
# No pipeline restart needed — rotation takes effect at next pipeline start.

WOODPECKER_ENV_FILE gap — FIXED (Wave A A1)

Finding (WP build #6, 2026-07-04): vault auth from a real WP agent succeeded but WOODPECKER_ENV_FILE was not set in the step environment. The inter-step env propagation write was silently skipped; the smoke-verify step failed with exit 1.

Root cause: Woodpecker v3 runs each step in a separate Docker container. Unlike GHA, there is no automatic inter-step env propagation. WOODPECKER_ENV_FILE is not set in step context on v3.16.0 — the GHA analogy was incorrect.

Fix (ADR-0135 Wave A A1): wpc_load_vault_secrets.py now writes export KEY=… lines (base64-encoded) to a shell-sourceable file (/tmp/vault-env.sh by default), which the SAME step sources before running its real work:

- pip install --quiet requests
- python3 scripts/ci/wpc_load_vault_secrets.py
- . /tmp/vault-env.sh && rm -f /tmp/vault-env.sh
- # real work that uses the vault-fetched env vars

Live proof: WP pipeline .woodpecker/vault-secret-smoke.yaml, captured in PR for ADR-0135 A1. All 30 Wave A vault-dependent pipelines must use the same pattern. See docs/architecture/woodpecker-wpc-step-usage.md §1 for the canonical snippet.

Wave B (ADR-0135) — heavy CI + scale arbiter

Status: active soak (2026-07-05 +). Go/no-go gate: §Gate-B2 in ADR-0135.

Scale arbiter (Lambda ci-scale-arbiter)

What it does: An EventBridge rate(2 minutes) rule invokes Lambda ci-scale-arbiter. The Lambda calls GET /api/queue/info on the WP server (via internal ALB, Bearer token from SSM /ci/woodpecker/admin-api-token), then: - pending > 0 AND running == 0 AND desired == 0 → SetDesiredCapacity 1 (burst scale-up) - pending == 0 AND running == 0 AND desired > 0 → SetDesiredCapacity 0 (idle scale-down) - Otherwise → no action (work in flight or already scaling)

IaC: infra/ci/lambda.tf — Lambda + EventBridge rule + EventBridge target + permission. Logs: /aws/lambda/ci-scale-arbiter (90-day retention).

Verify scale arbiter is active:

# Check EventBridge rule is enabled:
aws events describe-rule --name ci-scale-arbiter-trigger --region us-east-2 \
  --query "State"
# Expected: "ENABLED"

# Check last 5 invocations:
aws logs filter-log-events \
  --log-group-name /aws/lambda/ci-scale-arbiter \
  --region us-east-2 \
  --start-time $(python3 -c "import time; print(int((time.time()-3600)*1000))") \
  | python3 -c "
import sys, json
for ev in json.load(sys.stdin).get('events', []):
    print(ev['timestamp'], ev['message'][:120])
"

Manual scale-up (emergency — bypasses arbiter):

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 1 \
  --region us-east-2

Manual scale-down (after incident resolved):

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 0 \
  --region us-east-2

Failure mode J: Scale arbiter sets desired=0 while jobs are executing

Symptom: Pipelines fail with received sigterm termination signal part-way through execution. Agent CloudWatch logs show "grpc error: done(): code: Unknown desc = sql: no rows in result set" immediately after SIGTERM. Scale arbiter logs show SCALE DOWN: queue idle — setting desired=0 (or desired=min_size) at the same timestamp as the job failures.

Cause: WP's /api/queue/info running_count transitions to 0 before agents have finished executing all steps. This is a WP v3 queue-tracking edge case: once a task is dispatched to an agent and moves from "dispatched" to "executing", the server may clear it from the running queue, causing running_count=0 while workers are still actively processing. The arbiter's scale-down condition (pending==0 AND running==0) fires on the stale count.

Observed (2026-07-13 18:12:35 UTC): pending=0, running=0, workers=4 — four workers actively running pipeline 1067 (workflows 2962-2964), but arbiter scaled desired→0.

Fix: The arbiter now targets min_size on scale-down (desired=1 with the on-demand floor) instead of desired=0. This reduces blast radius: only spot burst instances are terminated, the on-demand floor survives and CI keeps running. Full protection requires the floor (see §On-demand floor).

Verify:

# Check scale-down target in arbiter logs — should say "desired X→1 (floor)", not "desired X→0"
aws logs filter-log-events \
  --log-group-name /aws/lambda/ci-scale-arbiter \
  --region us-east-2 \
  --filter-pattern "SCALE DOWN" \
  --start-time $(python3 -c "import time; print(int((time.time()-3600)*1000))") \
  | python3 -c "import json,sys; [print(e['message']) for e in json.load(sys.stdin).get('events',[])]"

Prevention: On-demand floor (min_size=1) means even if the arbiter fires prematurely, the floor instance stays alive and absorbs the next queue entry without a cold-start delay.

Failure mode G: Scale arbiter not triggering scale-up

Symptom: Push event triggered a pipeline, but no agent picks it up within 5 minutes. ASG desired stays 0.

Diagnose:

# Check EventBridge rule is enabled:
aws events describe-rule --name ci-scale-arbiter-trigger --region us-east-2 --query "State"

# Check Lambda error rate in last 30 min:
aws logs filter-log-events \
  --log-group-name /aws/lambda/ci-scale-arbiter \
  --region us-east-2 \
  --start-time $(python3 -c "import time; print(int((time.time()-1800)*1000))") \
  --filter-pattern "ERROR"

# Manually invoke the arbiter to test it:
aws lambda invoke \
  --function-name ci-scale-arbiter \
  --region us-east-2 \
  --payload '{}' \
  /tmp/arbiter-result.json \
  --log-type Tail \
  --query "LogResult" --output text | base64 -d
cat /tmp/arbiter-result.json

Common causes: - SSM /ci/woodpecker/admin-api-token expired or wrong → Lambda logs WP API error 401 - WP server unhealthy (500 from /api/queue/info) → Lambda logs WP API error 500; fix WP server first - Lambda execution role lost SSM permission → Lambda logs permission error

Fix: If Lambda is healthy but ASG is stuck, scale up manually (see above). Then investigate the root cause from Lambda logs before trusting automatic scale-down.

Verification: aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ci-woodpecker-agents-asg --region us-east-2 --query "AutoScalingGroups[0].DesiredCapacity" returns 1.

Postgres heavy CI pipeline (.woodpecker/backend-tests-postgres.yaml)

Check context posted to GitHub: ci/woodpecker/backend-tests-postgres

What it runs: 1. path-guard — exit 0 if no backend_v2/** changes (informational, never blocks) 2. wait-for-postgres — polls pg_isready -h postgres -p 5432 until the postgres:15 service container is accepting connections 3. backend-tests-postgres: - apt-get: postgresql-client (psql for create_ci_postgres_roles.sh) + postgresql (pg_ctl for pytest-postgresql) - pip: root requirements.txt + backend_v2/requirements.txt (pytest-postgresql≥5.0) + pytest/cov/timeout - PATH extended to /usr/lib/postgresql/15/bin for pg_ctl binary - vault secrets loaded via wpc_load_vault_secrets.py (VAULT_ALLOW_MISSING=true — vault failure is non-fatal) - create_ci_postgres_roles.sh against the postgres service container - (cd backend_v2 && alembic upgrade head) — subshell keeps CWD at repo root - pytest run (first pass: collect failures, --maxfail=5; second pass: coverage) - python -m coverage report --fail-under=30 — gate

Service container behavior: The postgres:15 service container is on the same Docker bridge network as build steps. Connect via hostname postgres (= service name), port 5432. No port-mapping needed. pg_isready -h postgres is the correct readiness check.

conftest.py postgres path: When DATABASE_URL starts with postgresql://, conftest.py does NOT connect to the service container directly. It imports pytest-postgresql and calls postgresql_proc to spawn an EPHEMERAL pg_ctl-managed postgres instance on a random port. The service container is used only for the explicit alembic upgrade head step. This design (from ADR-0070) provides test isolation — each test session gets a fresh, isolated DB. Consequence: postgresql server package + pytest-postgresql>=5.0 are both required in the pipeline, not just postgresql-client.

Wave B coexistence: The GHA backend-tests-postgres job (in .github/workflows/ci.yml) continues to run during the Wave B soak. Remove the GHA job only after 2-week green parity per ADR-0135 §Gate-B2.

Failure mode H: backend-tests-postgres step exits 1 with no coverage data

Symptom: backend-tests-postgres step exits with exit_code=1. Coverage report logs Coverage.py warning: No data was collected. or similar.

Causes (in order of probability): 1. cd persisting — a command runs cd backend_v2 && ... without a subshell, so subsequent pytest commands look in backend_v2/backend_v2/tests which does not exist. Pytest exits 4 (path not found), swallowed by || echo; coverage has no data. Fix: use (cd backend_v2 && ...). 2. pg_ctl not on PATH — pytest-postgresql cannot spawn ephemeral postgres. Fix: install the full postgresql server package (not just postgresql-client) and export /usr/lib/postgresql/15/bin to PATH. 3. pytest-postgresql not installed — conftest.py falls through to pytest.skip() on all DB fixtures; tests are collected but skipped; coverage may be below 30%. Fix: install -r backend_v2/requirements.txt in the pipeline step. 4. Alembic migration fails — a new migration has a syntax error or references a missing role. Check: (cd backend_v2 && python3 -m alembic upgrade head) logs.

Diagnose:

# Check the pipeline step output — look for these signals:
# "No such file or directory: backend_v2/tests"  → cause 1 (cd persistence)
# "FileNotFoundError: [Errno 2]" or "pg_ctl: command not found" → cause 2
# "pytest-postgresql is not installed" → cause 3
# "alembic.exc.CommandError" or migration error → cause 4

Verification: Pipeline step exits 0; coverage report logs TOTAL ... XX% where XX ≥ 30.

Branch protection — Wave B dual-gate coexistence

Principle (ADR-0135 §4): During the Wave B soak, develop branch protection requires BOTH GHA and WP status checks. WP checks are added ALONGSIDE GHA — GHA checks are NOT removed until §Gate-B2 parity is confirmed (2-week green run).

Currently required checks on develop:

GHA (app_id: null):
  smoke_suite
  base_branch_lint
  stale-branch-guard
  pii-scan
  flag_promotion_check
  migration-gate

WP (added by Wave B, context prefix: ci/woodpecker/):
  ci/woodpecker/backend-tests-postgres
  ci/woodpecker/ci-pr
  ci/woodpecker/migration-collision-check

To add a new WP check to branch protection (operator or sre-agent):

# Read current protection config:
gh api repos/raxx-app/TradeMasterAPI/branches/develop/protection \
  --jq '.required_status_checks'

# Add a check (PATCH is additive; include all existing + new contexts):
gh api repos/raxx-app/TradeMasterAPI/branches/develop/protection \
  -X PUT \
  --input - <<'EOF'
{
  "required_status_checks": {
    "strict": false,
    "contexts": [
      "smoke_suite",
      "base_branch_lint",
      "stale-branch-guard",
      "pii-scan",
      "flag_promotion_check",
      "migration-gate",
      "ci/woodpecker/backend-tests-postgres",
      "ci/woodpecker/ci-pr",
      "ci/woodpecker/migration-collision-check"
    ]
  },
  "enforce_admins": false,
  "required_pull_request_reviews": null,
  "restrictions": null
}
EOF
# Verify after update:
gh api repos/raxx-app/TradeMasterAPI/branches/develop/protection \
  --jq '.required_status_checks.contexts'

To remove GHA checks (§Gate-B2 — only after 2-week green parity): Repeat the PUT above with only the WP contexts in the contexts array, omitting the GHA app_id: null entries. The GHA workflow files in .github/workflows/ remain for reference but their check contexts are no longer blocking.

WP status context name format (confirmed 2026-07-05): WP v3 posts GitHub status checks using ci/woodpecker/{event}/{pipeline-name} format, where {event} is push for push events and pr for pull_request events. Example: - ci/woodpecker/pr/backend-tests-postgres (PR trigger) - ci/woodpecker/push/backend-tests-postgres (push trigger) - ci/woodpecker/pr/ci-pr (ci-pr pipeline, PR trigger)

Note: some older YAML pipeline comments use ci/woodpecker/{pipeline-name} without the event prefix — this was a WP v2 / early v3 format. The {event} prefix has been confirmed in production on WP v3.16.0.

WP built-in variables in environment: blocks use the brace form ${CI_COMMIT_BRANCH}, ${CI_REPO}, ${CI_FORGE_TOKEN} — WP substitutes these at YAML parse time. Using bare $CI_COMMIT_BRANCH in an environment: value passes the literal dollar-sign string to the container, NOT the variable value. See Failure mode E-4 exception. Fixed in Wave B for ci-pr.yaml, console-degraded-auto-file.yaml, migration-collision-check.yaml.

ci/woodpecker/pr/ci-pr known failure (Wave A issue): The ci-pr.yaml pipeline reports failure on every PR since its introduction in Wave A batch 2 (PR #4051, 2026-07-05). Now that the ALB /api/* OIDC bypass rule is deployed and in Terraform state (priority 25: /api/*, /authorize, /login, /logout — imported 2026-07-08), the WP API is accessible from external clients without SSM. The ci-pr failure can now be diagnosed directly via the WP API. Do NOT add ci/woodpecker/pr/ci-pr to develop branch protection until this failure is diagnosed and resolved.

WP API access from external clients: The ALB now has a priority-25 OIDC-bypass rule for /api/*, /authorize, /login, and /logout (deployed 2026-07-08, managed as aws_lb_listener_rule.woodpecker_own_auth in infra/ci/alb.tf). Direct WP API calls from outside the VPC now work without authentication at the ALB layer — Woodpecker enforces its own token auth on all /api/* requests. Example:

curl -s -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  https://ci.moosequest.net/api/queue/info | python3 -m json.tool

Wave C — Console Ops Dispatch cutover (#4038, ADR-0135)

Status: implementation landed behind FLAG_OPS_DISPATCH_USE_WOODPECKER (default OFF). Flip at Wave C prod cutover per the operator decision below.

What changed: console/app/blueprints/ops.py (POST /ops/dispatch/<action_id>) can now dispatch via either the GitHub Actions workflow_dispatch API (unchanged, flag off) or the Woodpecker API (flag on). See console/app/services/ops_dispatch.py module docstring for the full design.

Safety constraint (mandatory — do not relax without a new incident review): the Woodpecker dispatch path NEVER issues a bare POST /api/repos/1/pipelines (event:manual) call. That endpoint fans out to every .woodpecker/*.yaml file whose when: block matches event: manual — roughly 65 pipeline files as of 2026-08, most unrelated to the action an operator clicked (see the "Warning — manual trigger fires all N workflows" note earlier in this doc, and .woodpecker/deploy-prod.yaml's TRIGGERS comment re: incident #110). Instead, the Console only ever calls the documented per-cron endpoint, POST /api/repos/1/cron/{cron_id}, and only for WorkflowSpec entries with an explicit wp_cron_id allowlist mapping (an operator-curated list, not free-form input).

Current allowlist (as of #4038):

Console action wp_cron_id Notes
nightly_security_scan 1 Maps to the security-scan-nightly pilot cron registered in "Phase 2 — cron pilot soak" above. Re-verify the ID via GET /api/repos/1/cron before flipping the flag on in prod — cron IDs are not guaranteed stable across a server rebuild.
deploy_heroku_prod (none) No safe scoped WP trigger exists yet — .woodpecker/deploy-prod.yaml is tag + break-glass-manual with an unscoped when: event: manual block. Hidden from the ops menu whenever the flag is on (hide-don't-gray, #1961); stays on the GHA dispatcher until a dedicated scoped trigger is built.

Operator decision (2026-07-05, issue #4038): Option C — a project-scoped Woodpecker API token if v3.16 supports it, stored at vault path /MooseQuest/console/WP_API_TOKEN and exposed to the Console app as the WP_API_TOKEN Heroku config var (same pattern as FREESCOUT_API_KEY / HEROKU_API_KEY — read via os.environ, never inlined). Falls back to a dedicated raxx-ci-bot WP user with minimal org access if v3.16 lacks project-scoped tokens.

Follow-up before prod flip: provision WP_API_TOKEN per the operator decision above, confirm cron ID 1 still maps to security-scan-nightly via GET /api/repos/1/cron, and register a scoped WP trigger for deploy_heroku_prod (or an equivalent staging-deploy action) before that action can be exposed over Woodpecker.

ECR layer cache — raxx-queue-vcpkg-cache (#4107, 2026-07-09)

Purpose: Stores Docker build-cache layers for the queue C++ vcpkg dependency tree. Consumed by W7 (deploy-queue) and W8 (queue-docker-smoke) pipelines. A warm cache reduces vcpkg build time from ~25-30 min (cold) to ~2-3 min.

Repository:

Name: raxx-queue-vcpkg-cache
Region: us-east-2
URI: 521228113048.dkr.ecr.us-east-2.amazonaws.com/raxx-queue-vcpkg-cache
ARN: arn:aws:ecr:us-east-2:521228113048:repository/raxx-queue-vcpkg-cache

Lifecycle policy: Untagged images are expired after 14 days. Tagged layers (e.g., vcpkg-cache-bookworm-<sha>) are retained until explicitly deleted.

IaC: infra/ci/ecr-cache.tf — ECR repo + lifecycle policy + agent IAM inline policy. Do NOT edit iam.tf for ECR-related changes — ecr-cache.tf owns the agent ECR policy.

Agent IAM policy ci-woodpecker-agent-ecr (inline on ci-woodpecker-agent-role): - ecr:GetAuthorizationToken on * (IAM limitation — cannot scope to one repo) - Pull actions (BatchCheckLayerAvailability, GetDownloadUrlForLayer, BatchGetImage) on repo ARN only - Push actions (PutImage, InitiateLayerUpload, UploadLayerPart, CompleteLayerUpload) on repo ARN only

Authenticate agent to ECR (used in W7/W8 pipeline steps):

# WP pipeline step YAML (agent fetches token via instance role — no credential needed):
aws ecr get-login-password --region us-east-2 \
  | docker login --username AWS --password-stdin \
    521228113048.dkr.ecr.us-east-2.amazonaws.com

Bootstrap cache (one-time before first W7 run — avoids 25-30 min cold build): Run from a machine or GHA runner with ECR push rights to raxx-queue-vcpkg-cache:

# Build the vcpkg base layer and push to ECR:
cd queue/
docker buildx build \
  --cache-to type=registry,ref=521228113048.dkr.ecr.us-east-2.amazonaws.com/raxx-queue-vcpkg-cache:vcpkg-cache-bookworm-bootstrap,mode=max \
  --push \
  -f Dockerfile.vcpkg \
  .
# Verify the tag exists:
aws ecr describe-images \
  --repository-name raxx-queue-vcpkg-cache \
  --region us-east-2 \
  --query "imageDetails[*].{tags:imageTags,pushed:imagePushedAt,size:imageSizeInBytes}" \
  --output table

Failure mode — ECR push/pull fails from agent with "no credentials": - Confirm the ci-woodpecker-agent-ecr inline policy is attached: aws iam get-role-policy --role-name ci-woodpecker-agent-role --policy-name ci-woodpecker-agent-ecr - Confirm IMDSv2 hop limit is 2 on the agent LT (allows containers to reach IMDS for STS): aws ec2 describe-launch-template-versions --launch-template-name ci-woodpecker-agent-lt --query "LaunchTemplateVersions[-1].LaunchTemplateData.MetadataOptions" - The agent must call aws ecr get-login-password BEFORE docker login in the same step — the token expires after 12h

Failure mode I: Stale Go HTTP connection pool — context deadline exceeded on GitHub API calls

Symptom: A pipeline triggered by a tag (or push) event is created immediately with status=error and zero steps executed. Server logs show context deadline exceeded on a pipeline-fetch path. The WP server is otherwise healthy (/healthz 200, queue not paused, agents connected). Typically surfaces after 4–7 days of continuous server uptime.

Cause: Go's net/http default transport pools idle keep-alive connections. After multi-day uptime, connections to api.github.com held in the pool have been silently closed server-side (TCP RST or keepalive timeout), but the WP process has not detected the close. On the next GitHub API call (fetching .woodpecker/*.yaml), WP reuses a dead connection; the TCP write fails; the request-scoped deadline expires before a successful retry completes.

Fix:

# 1. Confirm the symptom in WP server logs:
aws ssm send-command \
  --region us-east-2 \
  --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["sudo docker compose -f /opt/woodpecker/docker-compose.yml logs --tail=30 woodpecker-server 2>&1 | grep -E \"context deadline|unexpected status\" | tail -10"]}' \
  --output json | python3 -c "import json,sys; print(json.load(sys.stdin)['Command']['CommandId'])"
# Poll get-command-invocation until Status=Success, then read StandardOutputContent.

# 2. Restart to clear the pool (~30s downtime; ensure WP queue is EMPTY first):
aws ssm send-command \
  --region us-east-2 \
  --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["sudo systemctl restart woodpecker && echo RESTART_OK"]}' \
  --output json | python3 -c "import json,sys; print(json.load(sys.stdin)['Command']['CommandId'])"
# Poll; expect Status=Success + StandardOutputContent=RESTART_OK

# 3. Confirm healthy after ~30s:
curl -sS -o /dev/null -w "%{http_code}\n" -H "User-Agent: raxx-agent/1.0" https://ci.moosequest.net/api/version
# Expected: 200

Re-fire the failed pipeline after restart:

Option A — Redeliver the tag webhook via GitHub API (requires admin:repo_hook scope — preferred, preserves tag event context):

# WP webhook ID: 649361254
gh api repos/raxx-app/TradeMasterAPI/hooks/649361254/deliveries \
  --jq '.[] | select(.event == "create") | {id,delivered_at}' | head -4
gh api -X POST repos/raxx-app/TradeMasterAPI/hooks/649361254/deliveries/<delivery_id>/attempts

Option B — Break-glass manual trigger (use when admin:repo_hook unavailable):

WP_TOKEN="<admin WP JWT from ci.moosequest.net>"
# Note: ALB may return 504 even if WP accepted the request (60s idle timeout).
# Always confirm creation via GET after a 504.
curl -s -X POST \
  -H "Authorization: Bearer $WP_TOKEN" \
  -H "User-Agent: raxx-agent/1.0" \
  -H "Content-Type: application/json" \
  -d '{"branch":"main","variables":{"DEPLOY_PROD_CONFIRM":"yes"}}' \
  "https://ci.moosequest.net/api/repos/1/pipelines"
curl -s -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: raxx-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/pipelines?page=1&perPage=3" \
  | python3 -c "import json,sys; [print(f'#{p[\"number\"]} {p[\"event\"]} {p[\"branch\"]} {p[\"status\"]}') for p in json.load(sys.stdin)]"

Warning — manual trigger fires all 59 workflows: A manual trigger on main dispatches every .woodpecker/*.yaml file that matches event: manual, not just deploy-prod. Most are harmless (cron workflows that error on non-scheduled contexts), but deploy-staging, cut-release-candidate, and deploy-velvet may also run. Prefer Option A (webhook redeliver) for a clean tag-only trigger. As of #4474, deploy-velvet's manual-triggered prod path now fails closed rather than silently deploying: it requires DEPLOY_PROD_CONFIRM=deploy-prod-now (not deploy-prod.yaml's yes — different pipeline, different confirm value) on branch main, so the Option B DEPLOY_PROD_CONFIRM=yes example above will make deploy-velvet fail its own guard (harmless, no Heroku push) rather than deploy Velvet to prod as an unintended side effect.

Verification: deploy-prod workflow in the new pipeline reaches state=success for all 9 steps. Confirm api.raxx.app/health → 200.

Long-term mitigations: 1. Weekly WP server restart cron (Sunday 04:00 UTC) via AWS EventBridge + SSM Run Command 2. Tune Go HTTP transport via WP env var WOODPECKER_FORGE_TIMEOUT (or similar) if supported in v3.x 3. Cache git subtree split to avoid 17+ min cold splits on ephemeral agents (first run = 1073s on 6-month-history repo)

RCA: docs/incidents/2026-07-09-wp-stale-http-pool-prod-deploy.md

Escalation

Wake the operator when: - RDS data integrity suspected (unclean shutdown, storage corruption) - GitHub OAuth App or Google OIDC credentials compromised - ACM certificate cannot be renewed (Dyn CNAME missing after 60-day renewal attempt) - Woodpecker server repeatedly crashes and restart loop does not recover within 15 minutes - WAF blocking appears to be a false positive on production pipeline webhooks

Monitoring — external watchdog (2026-07-07)

Permanent GHA stay (ADR-0136 Decision 1; #4109): .github/workflows/woodpecker-healthcheck.yml is classified GENUINELY-STAYS-ON-GHA. It is not a pending migration item. Do not port it to Woodpecker and do not retire it. See migration table §2b for the formal record.

Woodpecker health is monitored by scripts/ops/woodpecker_healthcheck.py, run every 15 min by .github/workflows/woodpecker-healthcheck.yml — intentionally on GitHub Actions, because a monitor hosted on Woodpecker can't fire when Woodpecker is down.

Checks: (1) server reachable (/api/version 200), (2) TLS cert expiry (< 21d fails), (3) pipeline health (stuck > 90 min, or ≥4 of last 5 finished failed) — tier 3 only runs if a WOODPECKER_API_TOKEN repo secret is set.

Alerting is edge-triggered via a woodpecker-health GitHub issue anchor: one email to kris@moosequest.net on healthy→unhealthy, one on recovery — no per-run spam during a sustained outage. Email goes via Postmark (POSTMARK_OPS_ALERT_TOKEN + OPS_ALERT_FROM secrets).

Verify the alert path any time: Actions → "Woodpecker healthcheck" → Run workflow → tick test_email (sends one confirmation email, then exits). Or locally: POSTMARK_SERVER_TOKEN=... python3 scripts/ops/woodpecker_healthcheck.py --test-email.

To enable the richer pipeline checks, add a WOODPECKER_API_TOKEN repo secret (a WP user API token from the ci.moosequest.net UI).

Failure mode J: woodpecker-restart.service crash loop — 502 since 08:00 UTC

Symptom: ALB returns 502 for all requests. systemctl status woodpecker shows the service cycling through "Starting → Finished → Stopping → Stopped → Starting" every ~2 seconds. journalctl -u woodpecker -u woodpecker-restart.service shows this alternating pattern repeating continuously:

systemd: Finished woodpecker.service
systemd: Starting woodpecker-restart.service...
systemd: woodpecker-restart.service: Main process exited, code=killed, status=15/TERM
systemd: woodpecker-restart.service: Failed with result 'signal'.
systemd: Stopping woodpecker.service...
systemd: woodpecker.service: Deactivated successfully.
systemd: Starting woodpecker.service...

Cause: woodpecker-restart.service has Requires=woodpecker.service. When the timer fires and woodpecker-restart.service runs systemctl restart woodpecker, that command internally stops woodpecker.service. The moment woodpecker.service goes inactive, the Requires= dependency causes systemd to SIGTERM woodpecker-restart.service before the restart process can complete. Systemd then restores woodpecker.service (queued start from the partial restart), and also attempts to re-activate woodpecker-restart.service (systemd heals collaterally stopped units). The cycle repeats with no exit condition: ~2 seconds per loop, indefinitely. The WP container never stays up long enough for the ALB target group health check to pass. The loop starts the moment the timer fires (08:00 UTC daily) and the outage persists until the loop is manually broken.

Fix (break the loop + patch the unit file):

# 1. Break the loop — stop both units
sudo systemctl stop woodpecker-restart.timer woodpecker-restart.service woodpecker

# 2. Verify stopped (should say "failed" or "inactive", never "active")
sudo systemctl is-active woodpecker && echo STILL_UP || echo STOPPED_OK
sudo docker ps | grep woodpecker && echo CONTAINER_STILL_UP || echo DOWN_OK

# 3. Remove the faulty Requires= line
sudo sed -i '/^Requires=woodpecker.service/d' /etc/systemd/system/woodpecker-restart.service

# 4. Confirm the patch (should show After= but NO Requires=)
sudo cat /etc/systemd/system/woodpecker-restart.service

# 5. Reload daemon and restart cleanly
sudo systemctl daemon-reload
sudo systemctl start woodpecker

# 6. Confirm container is up and healthz returns 200
sudo docker ps | grep woodpecker
curl -sf http://localhost:8000/healthz && echo OK

# 7. Re-enable the timer
sudo systemctl start woodpecker-restart.timer
sudo systemctl status woodpecker-restart.timer --no-pager

# 8. Verify no restart.service start fires immediately
sleep 5
sudo journalctl -u woodpecker-restart.service --since "1 minute ago" --no-pager
# Expected: no new "Starting woodpecker-restart.service" entries

Verification:

# External endpoint must return 200
curl -s -o /dev/null -w "%{http_code}" https://ci.moosequest.net/api/version
# Expected: 200

# Service must be stable (active, not cycling)
sudo systemctl status woodpecker --no-pager
# Expected: Active: active (exited) — no repeated Starting/Stopping

Permanent fix — IaC: The woodpecker-restart.service unit must be created WITHOUT Requires=woodpecker.service. After=woodpecker.service alone is sufficient. This is now codified in infra/ci/templates/user_data_server.sh.tpl (section 9) so a server rebuild will not re-introduce the bug.

Correct woodpecker-restart.service:

[Unit]
Description=Woodpecker CI daily restart -- clears Go HTTP connection-pool staleness (RCA 2026-07-09)
After=woodpecker.service

[Service]
Type=oneshot
ExecStart=/bin/systemctl restart woodpecker
StandardOutput=journal
StandardError=journal

Why Requires= is wrong here: systemctl restart woodpecker = stop + start. The stop phase puts woodpecker.service into inactive state. Requires=woodpecker.service propagates that stop to woodpecker-restart.service (killing the restart process mid-execution). After=woodpecker.service gives the same boot-ordering guarantee without the stop-propagation side effect.

RCA: docs/incidents/2026-07-11-woodpecker-restart-loop.md Incident log: 2026-07-11 — loop started 08:00 UTC, detected 13:36 UTC, resolved 13:47 UTC

Failure mode K: "functioning correctly" cron claims go stale — always re-verify via a real scoped trigger before enabled:true

Symptom: A cron created and validated in the same PR (e.g. "fixed + confirmed working" in the PR description) but never issued the follow-up PATCH .../cron/{id} {"enabled":true} sits dark (next_exec frozen at its creation-time computed value) for weeks or months. When someone finally goes to enable it, blindly trusting the original PR's claim risks silently arming a job that either always fails, or worse, always false-succeeds without doing its actual job.

Cause: A pipeline step exiting 0 is not proof the step did anything useful — only that nothing inside it raised past whatever error handling it has. Two concrete cases found on the same day (2026-08-12, cards

4477/#4478, following the #4429/#4450 precedent that first established

this pattern for tickets-e2e-smoke-daily):

  1. e2e-smoke-nightly (cron 18): npm ci in frontend/raxx-next fails outright under the CI image's npm version — this one fails loudly, but had never actually been exercised in that exact image/npm-version combination since the lockfile drift was introduced (PR #2917, 2026-05-27, ~2.5 months before this cron was even created).
  2. historical-bars-warm-nightly (cron 20): the underlying command (warm_historical_bars_1min.py) never bootstraps its DB engine (init_engine()) — every real invocation raises RuntimeError: get_engine() called before init_engine(), which is caught by a broad except Exception and silently downgraded to "no symbols to warm," so the step reports success while doing nothing. This is the more dangerous case: a naive re-enable based on the step's green exit code alone would have armed a cron that runs "successfully" forever without ever warming a single row.

Diagnose (do this before every enabled:true PATCH on a cron that has never actually fired, regardless of how recently or confidently a prior PR claimed it works):

# 1. Baseline — confirm current enabled/next_exec state
WP_TOKEN="<admin JWT, from SSM /ci/woodpecker/admin-api-token>"
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/cron/<CRON_ID>"
# next_exec frozen at creation time == zero fires since creation.

# 2. Check WP queue is idle before triggering (avoid stacking on live PR load)
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/queue/info"

# 3. Trigger via the SCOPED single-workflow endpoint — never a bare manual
#    dispatch (fans out to every event:manual workflow in the repo, ~65+
#    unrelated pipelines as of 2026-08).
curl -sf -X POST -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/cron/<CRON_ID>"
# Note: as of 2026-08-12 this endpoint did NOT reliably scope to exactly
# one workflow in this repo — `ci-boundary` rode along on both cron 18
# and cron 20 triggers despite its own `when:` not matching a cron event.
# See #4490 (open investigation). Do not be alarmed by an unrelated
# second workflow in the response's `workflows` array; just evaluate the
# TARGET workflow's own state independently of the pipeline's aggregate
# status.

# 4. Read the FULL step log (not just the summary exit code) for the
#    target workflow's real-work step. The v3 log endpoint returns
#    base64-encoded lines:
curl -sS -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/logs/<PIPELINE_NUMBER>/<STEP_ID>" \
  | python3 -c "
import json,sys,base64
for l in json.load(sys.stdin):
    data = l.get('data')
    if data:
        print(base64.b64decode(data).decode('utf-8', errors='replace'), end='')
"
# Read for the ACTUAL side effect described in the step's own log lines
# (rows loaded, files uploaded, bars warmed, etc.) — not just "exit 0" /
# a generic "PASS" echo. A step can print PASS and exit 0 while its own
# earlier log lines show an ERROR that was silently caught.

Fix: There is no generic fix — each case is its own root cause (see

4488, #4489 for the two concrete examples above). The fix IS the

process: never PATCH enabled:true on the strength of a stale PR claim alone. Always re-trigger via the scoped endpoint and read the real step output for evidence of actual work done, not just a clean exit code.

Verification: Re-trigger via the scoped endpoint after the underlying fix lands; confirm the step's own log shows the real expected side effect (not a masked "nothing to do" path); only then PATCH .../cron/{id} {"enabled":true} and confirm next_exec moves forward.

RCA: docs/incidents/2026-08-12-cron-reverify-batch-e2e-warm-waf.md, docs/incidents/2026-08-09-tickets-e2e-smoke-mailto-drift.md (originating precedent)

Incident log

Date Duration Cause Fix applied
2026-07-04 ~2h WAF SizeRestrictions_BODY + WP v3 YAML ${{ parse error WAF override + pipeline YAML fixes (see RCA)
2026-07-11 ~5h 47m (08:00–13:47 UTC) woodpecker-restart.service Requires=woodpecker.service crash loop triggered by 08:00 timer — SEV-2 Removed Requires= line via SSM sed; service stable; IaC template patched
2026-08-12 N/A (verification pass, not an outage) e2e-smoke-nightly/historical-bars-warm-nightly crons re-verified per #4477/#4478; both found real pre-existing bugs (lockfile gap, missing init_engine()) rather than being safe to enable — SEV-3 Filed #4488/#4489 for feature-developer; left both crons disabled; filed #4490 for the cron-trigger co-scoping anomaly observed during the same pass