Raxx · internal docs

internal · gated

Woodpecker CI runbook

System: ci-woodpecker (Woodpecker CI on EC2 — ci.moosequest.net) Owner: operator / sre-agent ADR: ADR-0134 (Candidate B, operator-authorized 2026-07-03) Card: #4012 (Phase 1 baseline IaC) Last incident: 2026-11-03 22:00Z (the ci-agents-scale-in-evening scheduled action terminated 4 busy agents and orphaned their pipelines; fixed by the ci-agent-drain lifecycle hook, §Scale-in drain hook, Failure mode T, #4576). Before that 2026-11-03 (promote #4915, v1.22.0: deploy-antlers-next-prod silently skipped because the main push changed_files list was capped at 300 files; Failure mode S). Before that 2026-10-30 (base-branch-lint false-positive on every release-promotion PR + migration-collision-check red on every PR because of a token-dependent base-SHA lookup; see docs/incidents/2026-10-30-ci-gates-promotion-fp-and-migration-collision.md, Failure modes E-7 and E-8) Last reviewed: 2026-11-04

Architecture summary

Internet
  └── Oracle Dyn CNAME: ci.moosequest.net → ALB
        └── ALB (public, us-east-2a/b)
              ├── AWS WAF WebACL (managed rules + rate limit)
              ├── :80 → redirect :443
              └── :443 → listener rules
                    ├── /hook*         → Woodpecker server :8000 (no OIDC — priority 10)
                    ├── /api/badge*    → Woodpecker server :8000 (no OIDC — priority 20)
                    ├── /api/* /authorize /login /logout → Woodpecker :8000 (no OIDC — priority 25)
                    └── (default)      → Google Workspace OIDC → Woodpecker :8000
                          └── Woodpecker server (t4g.small, private subnet us-east-2a)
                                ├── docker-compose: woodpecker-server container
                                ├── gRPC :9000 → build agents (spot ASG, c7a/c6a.xlarge)
                                └── PostgreSQL → RDS db.t4g.micro (private subnet)

All instances: IMDSv2 required, no SSH keys, SSM Session Manager for break-glass access
Secrets: SSM Parameter Store SecureString (/ci/* path)

Build cache (2026-08-05, card #4393): the agent fleet's Docker daemon now points at a pull-through registry mirror + PyPI caching proxy on the ci-monitoring box, collapsing NAT data-processing cost from ~1.49 TB/mo of repeat Docker Hub/PyPI traffic. See docs/ops/runbooks/ci-build-cache.md for the full system runbook — do not debug agent build-cache issues here.

VPC endpoints single-AZ (2026-08-05, card #4394): the interface endpoints (ssm/ssmmessages/ec2messages, infra/ci/vpc.tf) are scoped to us-east-2a only (previously both AZs). The agent fleet's AZ placement is NOT pinned — an agent that lands in us-east-2b (still in the ASG's failover subnet set) falls back to routing SSM traffic via NAT instead of the local endpoint. This is expected and accepted (see vpc.tf header comment for the full risk writeup) — it does not break SSM access, it just routes it differently. An AZ-A outage would take VPC-endpoint-based SSM access down fleet-wide until endpoints are restored or re-added to AZ-B (rollback: re-add aws_subnet.private[1].id to each endpoint's subnet_ids and re-apply).

CloudWatch Logs endpoint removed (2026-11-04, #4576): the fourth interface endpoint (com.amazonaws.us-east-2.logs, ci-logs-endpoint, ~$7.30/mo) was dropped; the SSM trio stays (~$21.90/mo). Agent container logs (awslogs driver) now reach CloudWatch through NAT. If NAT is degraded, container logs lag or drop, but builds are unaffected. Rollback: re-add an aws_vpc_endpoint for the logs service (Interface, aws_subnet.private[0].id, aws_security_group.vpc_endpoints, private DNS on) and re-apply.

How to tell it's broken

Apply order (Phase 1 first-time deploy)

Prerequisites

Before running terraform apply:

  1. AWS credentials configured for the Terraform executor role (see §IAM executor role)
  2. State bootstrap completed (see §State bootstrap below)
  3. SSM parameter placeholders created by bootstrap apply
  4. Google OAuth client created (see §Google OAuth client setup)
  5. GitHub OAuth App registered (see §GitHub OAuth App registration)
  6. Real SSM parameter values filled (see §Secrets fill-in checklist)

State bootstrap (one-time)

cd infra/ci/bootstrap
terraform init
terraform plan
terraform apply
# Note the state_bucket_name and lock_table_name outputs.
# These match the values hardcoded in infra/ci/main.tf backend block.

Bootstrap state lives at infra/ci/bootstrap/terraform.tfstate — do NOT commit this file (it is gitignored). Keep a copy in a secure location (Google Drive ops folder).

Main apply

cd infra/ci
terraform init   # connects to S3 backend created by bootstrap
terraform plan -out=ci-phase1.plan -var-file=terraform.tfvars
terraform apply ci-phase1.plan

After first apply

  1. Read outputs: bash terraform output -json

  2. Add two CNAME records in Oracle Dyn (see §DNS setup below).

  3. Confirm ACM certificate validates (5-30 min after DNS propagation): bash aws acm describe-certificate \ --certificate-arn "$(terraform output -raw acm_certificate_arn)" \ --region us-east-2 \ --query "Certificate.Status" # Expected: "ISSUED"

  4. Fill SSM parameter values (see §Secrets fill-in checklist).

  5. Restart Woodpecker server to pick up real secrets: bash INSTANCE_ID="$(terraform output -raw woodpecker_server_instance_id)" aws ssm start-session --target "$INSTANCE_ID" --region us-east-2 # In the session: sudo systemctl restart woodpecker sudo docker compose -f /opt/woodpecker/docker-compose.yml logs -f

  6. Confirm SNS subscription for ops@raxx.app (check email, click confirmation link).

  7. Verify Woodpecker UI at https://ci.moosequest.net (requires Dyn CNAME and ACM ISSUED).

DNS setup (Oracle Dyn)

Add two CNAME records in Oracle Dyn after first terraform apply:

Record 1: ACM validation CNAME

Type:  CNAME
Name:  <terraform output acm_validation_cname_name>
       (looks like: _acm-challenge.ci.moosequest.net. or _abc123.ci.moosequest.net.)
Value: <terraform output acm_validation_cname_value>
       (looks like: _xyz789.acm-validations.aws.)
TTL:   300

Purpose: Allows AWS Certificate Manager to validate ownership of ci.moosequest.net and issue the TLS certificate. This record can remain indefinitely (ACM uses it for renewals too).

Record 2: ci.moosequest.net service CNAME

Type:  CNAME
Name:  ci.moosequest.net
Value: <terraform output alb_dns_name>
       (looks like: ci-woodpecker-alb-1234567890.us-east-2.elb.amazonaws.com)
TTL:   300 (lower during initial setup for faster propagation)

Purpose: Points ci.moosequest.net to the ALB. Required for the Woodpecker UI, webhook delivery, and the OIDC redirect from Google.

After adding both records: Wait for DNS propagation (typically 2-5 minutes for Dyn's authoritative nameservers; up to 5 minutes for resolvers to pick up TTL=300 records).

Manual DNS inventory (monitored, #4547)

These are the only two moosequest.net DNS records that gate the CI control plane and are hand-typed in Oracle Dyn rather than Terraform-managed (the zone itself lives at Oracle Dyn — standing decision: never migrate). Both are now actively monitored (see below); this section is the canonical inventory of what they should currently point to.

Record Type Expected value (source of truth) Purpose
ci.moosequest.net CNAME terraform output alb_dns_name (infra/ci) — looks like ci-woodpecker-alb-1991574766.us-east-2.elb.amazonaws.com Points the CI UI/webhook/OIDC-redirect hostname at the ALB.
<terraform output acm_validation_cname_name> (infra/ci) — looks like _acm-challenge.ci.moosequest.net. CNAME terraform output acm_validation_cname_value (infra/ci) ACM DNS validation for the ci.moosequest.net certificate; re-checked by ACM on every renewal, not just first issuance.

Blast radius if either goes missing/wrong: all CI for raxx-app/TradeMasterAPI (82 pipelines) and MooseQuest-LLC/project-avenue (4 pipelines), raxx-console-prod's Ops Dispatch feature (console/app/services/ops_dispatch.py:277), the GHA watchdog, the scale arbiter, and ACM renewal. A silently deleted _acm-challenge record is the most dangerous case: ci.moosequest.net keeps resolving fine day-to-day and nothing looks wrong until ACM's ~60-day renewal re-validation fails — see Escalation below.

Monitoring: terraform/ci-monitoring runs a blackbox_exporter dns probe against both records every 60s (dns_ci_cname / dns_acm_challenge modules, blackbox-dns Prometheus job) and pages on the DynRecordMissing alert (critical, 10m) if either stops resolving to its expected value. ACMCertificateExpiringSoon (warning, 45 days — longer lead time than the fleet-wide 30-day TLSCertExpirySoon, since this cert needs the extra manual Dyn step) backstops the _acm-challenge case specifically. Both expected values are wired into terraform/ci-monitoring/main.tf via a terraform_remote_state read of infra/ci's outputs (same cross-root pattern as wp_server_private_ip), so they never drift out of sync with the real Terraform outputs — no separate value to keep updated by hand here beyond this table's descriptive text.

Post-apply checklist (first deploy or ALB/cert replace): 1. terraform apply in infra/ci. 2. Read terraform output alb_dns_name, acm_validation_cname_name, acm_validation_cname_value. 3. Add/update both CNAMEs in Oracle Dyn per §DNS setup above. 4. terraform apply in terraform/ci-monitoring (picks up the new values via remote state) — note the lifecycle { ignore_changes = [user_data_base64] } caveat in docs/ops/runbooks/ci-monitoring.md §Cross-root remote state: a fresh apply does not push to an already-running monitoring box; see that section's fix-forward step. 5. Confirm the Grafana/Prometheus blackbox-dns target is up == 1 for both records before considering the change complete.

Google OAuth client setup (operator browser steps)

The ALB OIDC action uses Google Workspace OAuth 2.0. Create a client once:

  1. Open Google Cloud Console: https://console.cloud.google.com/
  2. Select the MooseQuest project (or create one named "moosequest-ops").
  3. Navigate to APIs & Services → Credentials.
  4. Click Create Credentials → OAuth client ID.
  5. Application type: Web application.
  6. Name: Woodpecker CI ALB OIDC
  7. Authorized redirect URIs — add: https://ci.moosequest.net/oauth2/idpresponse (This is the AWS ALB OIDC callback endpoint — do NOT change the path.)
  8. Click Create.
  9. Copy the Client ID and Client Secret.

  10. Fill SSM parameters (see §Secrets fill-in checklist): ```bash aws ssm put-parameter \ --name "/ci/oidc/google-client-id" \ --value "PASTE_CLIENT_ID_HERE" \ --type SecureString \ --overwrite \ --region us-east-2

    aws ssm put-parameter \ --name "/ci/oidc/google-client-secret" \ --value "PASTE_CLIENT_SECRET_HERE" \ --type SecureString \ --overwrite \ --region us-east-2 ```

Important: The redirect URI must be https://ci.moosequest.net/oauth2/idpresponse exactly. The ALB uses this path for the OIDC code exchange. If ci.moosequest.net is not yet resolving, the Google OAuth client can be created first (the URI is validated by Google only at runtime, not at creation time).

Status (2026-07-03): Google OIDC credentials are already staged in Infisical vault at raxx-secrets /Raxx/CI (prod) as CI_GOOGLE_OIDC_CLIENT_ID and CI_GOOGLE_OIDC_CLIENT_SECRET. Skip steps 1–9 above. Proceed directly to the SSM fill step, reading from vault (requires a session with CF Access service token):

GOOGLE_CLIENT_ID="$(infisical secrets get CI_GOOGLE_OIDC_CLIENT_ID \
  --projectId raxx-secrets --path /Raxx/CI --env prod --plain)"
GOOGLE_CLIENT_SECRET="$(infisical secrets get CI_GOOGLE_OIDC_CLIENT_SECRET \
  --projectId raxx-secrets --path /Raxx/CI --env prod --plain)"
aws ssm put-parameter --name "/ci/oidc/google-client-id" \
  --value "$GOOGLE_CLIENT_ID" --type SecureString --overwrite --region us-east-2
aws ssm put-parameter --name "/ci/oidc/google-client-secret" \
  --value "$GOOGLE_CLIENT_SECRET" --type SecureString --overwrite --region us-east-2
unset GOOGLE_CLIENT_ID GOOGLE_CLIENT_SECRET

GitHub OAuth App registration (operator browser steps)

Woodpecker uses a GitHub OAuth App for SCM integration (pipeline triggers, status checks, user authentication).

  1. Go to GitHub → Settings → Developer settings → OAuth Apps → New OAuth App: - Application name: Woodpecker CI — moosequest.net - Homepage URL: https://ci.moosequest.net - Authorization callback URL: https://ci.moosequest.net/authorize
  2. Click Register application.
  3. On the app page, click Generate a new client secret.
  4. Copy Client ID and Client Secret.
  5. Fill SSM parameters: ```bash aws ssm put-parameter \ --name "/ci/woodpecker/github-client-id" \ --value "PASTE_CLIENT_ID_HERE" \ --type SecureString \ --overwrite \ --region us-east-2

aws ssm put-parameter \ --name "/ci/woodpecker/github-client-secret" \ --value "PASTE_CLIENT_SECRET_HERE" \ --type SecureString \ --overwrite \ --region us-east-2 ```

Secrets fill-in checklist

After terraform apply, all SSM parameters exist as PLACEHOLDER values. Fill them in this order (prerequisite steps noted):

# 1. Woodpecker admin GitHub username (your GitHub login)
aws ssm put-parameter \
  --name "/ci/woodpecker/admin-user" \
  --value "GITHUB_USERNAME" \
  --type SecureString \
  --overwrite \
  --region us-east-2

# 2. Woodpecker agent↔server RPC secret (must be identical on server and agents)
AGENT_SECRET="$(openssl rand -hex 32)"
aws ssm put-parameter \
  --name "/ci/woodpecker/agent-secret" \
  --value "$AGENT_SECRET" \
  --type SecureString \
  --overwrite \
  --region us-east-2
# Save $AGENT_SECRET in 1Password / secure notes — needed if agents are
# rebuilt without Terraform re-reading SSM.

# 3. PostgreSQL password
DB_PASSWORD="$(openssl rand -base64 32 | tr -d /=+ | head -c 32)"
aws ssm put-parameter \
  --name "/ci/woodpecker/db-password" \
  --value "$DB_PASSWORD" \
  --type SecureString \
  --overwrite \
  --region us-east-2
# ALSO update the RDS master password to match:
aws rds modify-db-instance \
  --db-instance-identifier ci-woodpecker \
  --master-user-password "$DB_PASSWORD" \
  --apply-immediately \
  --region us-east-2

# 4. GitHub OAuth App credentials (from §GitHub OAuth App registration above)
aws ssm put-parameter \
  --name "/ci/woodpecker/github-client-id" \
  --value "GITHUB_CLIENT_ID" \
  --type SecureString \
  --overwrite \
  --region us-east-2

aws ssm put-parameter \
  --name "/ci/woodpecker/github-client-secret" \
  --value "GITHUB_CLIENT_SECRET" \
  --type SecureString \
  --overwrite \
  --region us-east-2

# 5. Google Workspace OIDC credentials (from §Google OAuth client setup above)
aws ssm put-parameter \
  --name "/ci/oidc/google-client-id" \
  --value "GOOGLE_CLIENT_ID" \
  --type SecureString \
  --overwrite \
  --region us-east-2

aws ssm put-parameter \
  --name "/ci/oidc/google-client-secret" \
  --value "GOOGLE_CLIENT_SECRET" \
  --type SecureString \
  --overwrite \
  --region us-east-2

After filling all 7 parameters, restart the Woodpecker server (see §Apply order step 5).

IAM executor role

The identity running terraform apply needs the ci-terraform-executor-policy attached (provisioned by the IaC itself on first apply — bootstrapping caveat below).

Bootstrapping caveat: On first apply, the executor policy does not yet exist (Terraform creates it). The operator's identity must have sufficient permissions to create it. Simplest path for first apply: use an existing role with broad permissions, then attach the scoped policy and switch to it for subsequent applies.

Do NOT use claude-infisical-bootstrap — that user must not be widened per ADR-0134.

Recommended first-apply path: 1. Create a new IAM role ci-terraform-executor with a trust policy for the operator's IAM user or SSO identity. 2. Attach AdministratorAccess temporarily for the first apply only. 3. After first apply, detach AdministratorAccess and attach ci-terraform-executor-policy (ARN from terraform output terraform_executor_policy_arn). 4. All subsequent applies use the scoped policy only.

Smoke checklist (after first apply + secrets filled)

Run in order; each step should pass before proceeding:

Current versions

Bumping server + agent versions

Server and agent versions are coupled: agent LT user-data embeds the agent image tag, so a version bump requires terraform apply to publish a new LT version. Agents then relaunch from the new LT on next scale-up.

Normal upgrade path (immutable re-provision): 1. Update woodpecker_server_version and woodpecker_agent_version in infra/ci/variables.tf. 2. Remove the lifecycle { ignore_changes = [user_data] } block from server.tf (if present — see below). 3. terraform plan — confirm server is marked for replace (not destroy). 4. terraform apply — server is replaced, new LT version published. 5. Re-add the lifecycle block if needed (see below).

In-place upgrade path (SSM patch, no terraform replace): If a version was applied via SSM (e.g., systemctl restart with a new image tag in docker-compose), the TF state user_data hash will drift from the computed hash, causing user_data_replace_on_change = true to schedule a server replacement even though the server is already healthy at the new version.

Fix: add lifecycle { ignore_changes = [user_data] } to aws_instance.woodpecker_server in infra/ci/server.tf BEFORE running terraform plan. This makes the plan read 0 destroy instead of scheduling a replacement. The lifecycle block is already present on develop as of 2026-07-04 (incident: docs/incidents/2026-07-04-woodpecker-v3-upgrade.md).

When you next want to intentionally re-provision the server (e.g., major version upgrade via immutable path), remove the lifecycle { ignore_changes = [user_data] } block from server.tf, confirm the plan proposes a replace (not a destroy), then apply.

Woodpecker clone plugin allow list

Woodpecker v3 maintains a trusted-clone plugin list. The default list (per --plugins-trusted-clone) is:

docker.io/woodpeckerci/plugin-git:2.9.2
docker.io/woodpeckerci/plugin-git
quay.io/woodpeckerci/plugin-git

Pipeline YAML must use woodpeckerci/plugin-git (Docker Hub org, no hyphen). The common mistake is woodpecker-ci/plugin-git (GHCR-style, with hyphen) which is NOT in the list — Woodpecker will refuse to inject netrc credentials, and private repo clones fail.

v3.x API changes (vs v2.x)

v3.x env vars — confirmed unchanged from v2.x

WOODPECKER_GITHUB_CLIENT, WOODPECKER_GITHUB_SECRET, WOODPECKER_ADMIN, WOODPECKER_AGENT_SECRET, WOODPECKER_DATABASE_DATASOURCE, WOODPECKER_DATABASE_DRIVER, WOODPECKER_HOST, WOODPECKER_OPEN, WOODPECKER_GRPC_ADDR, WOODPECKER_GRPC_SECURE, WOODPECKER_SERVER_ADDR, WOODPECKER_LOG_LEVEL. No renames required for current config.

v3.x pipeline YAML breaking changes

Known failure modes

Failure mode A: ACM certificate stuck in PENDING_VALIDATION

Symptom: aws acm describe-certificate returns Status: PENDING_VALIDATION for more than 30 minutes. Cause: ACM validation CNAME not present in Dyn, or DNS propagation not yet complete. Fix:

# Verify the validation record is expected:
terraform output acm_validation_cname_name
terraform output acm_validation_cname_value
# Check Dyn DNS console — confirm CNAME exists with the exact name and value.
# Test DNS propagation:
dig CNAME "$(terraform output -raw acm_validation_cname_name)" @8.8.8.8
# Expected: non-empty CNAME answer matching the validation value.

Verification: aws acm describe-certificate --certificate-arn ... --query "Certificate.Status" returns "ISSUED".

Failure mode B: Woodpecker server not starting

Symptom: ALB health checks fail; /healthz returns 502/503. Cause: Docker container failed to start (bad SSM secrets, RDS unreachable, image pull failure). Fix:

# Get instance ID
INSTANCE_ID="$(terraform -chdir=infra/ci output -raw woodpecker_server_instance_id)"
# Open SSM session
aws ssm start-session --target "$INSTANCE_ID" --region us-east-2
# In session:
sudo systemctl status woodpecker
sudo docker compose -f /opt/woodpecker/docker-compose.yml ps
sudo docker compose -f /opt/woodpecker/docker-compose.yml logs woodpecker-server
# Common root causes:
#   - "invalid database" / connection refused: DB password wrong or RDS SG blocks
#   - "cannot reach github.com": NAT Gateway issue; check route tables
#   - "placeholder" in env: SSM parameters not filled; fill and restart
sudo systemctl restart woodpecker

Verification: curl -sf http://localhost:8000/healthz from SSM session returns 200.

Failure mode B-2: Forge connectivity timeout — container-level (host network healthy)

Symptom: No new pipelines are created after webhook events. Woodpecker server logs show repeated context deadline exceeded errors:

"could not get folder from forge: context deadline exceeded"
"error while fetching config '' in 'refs/tags/v1.10.3' with user: 'MooseQuest', and did not get any config"

ALB /healthz returns 200. EC2 host can reach api.github.com (200 in <100ms). Container network appears normal.

Cause: Go HTTP client connection pool in the server process accumulates stale/half-open TCP connections after extended uptime (typically 3–7 days with high webhook volume). The pool reuses dead sockets; requests time out before falling back to fresh connections. Distinct from Failure mode B (container down) and WAF blocks (GitHub sees 403/200 but pipeline not created).

Diagnose:

# 1. Confirm host network is healthy (should be <100ms, 200):
aws ssm send-command --region us-east-2 --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["curl -sf --max-time 5 https://api.github.com/zen; echo exit:$?"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"
# retrieve with: aws ssm get-command-invocation --command-id <ID> --instance-id i-082ee835595d90ae0

# 2. Confirm container error pattern (look for 'context deadline exceeded' in last 5 min):
aws ssm send-command --region us-east-2 --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["docker logs woodpecker-woodpecker-server-1 --since 5m 2>&1 | grep deadline"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

# 3. Confirm conntrack is not saturated (should be << /proc/sys/net/netfilter/nf_conntrack_max):
aws ssm send-command --region us-east-2 --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["wc -l /proc/net/nf_conntrack && cat /proc/sys/net/netfilter/nf_conntrack_max"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

# 4. Confirm queue is empty before restarting:
WP_TOKEN=$(aws ssm get-parameter --name /ci/woodpecker/admin-api-token --with-decryption \
  --region us-east-2 --query "Parameter.Value" --output text)
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  https://ci.moosequest.net/api/queue/info | python3 -c "
import sys,json; d=json.load(sys.stdin)
print('pending:', len(d.get('pending',[])), 'running:', len(d.get('running',[])))
"

Fix (requires operator authorization for service restart):

# Option A — SSM send-command (no interactive session needed, ~15s):
CMD_ID=$(aws ssm send-command \
  --region us-east-2 \
  --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["sudo systemctl restart woodpecker && echo RESTART_OK"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])")
sleep 8
aws ssm get-command-invocation \
  --command-id "$CMD_ID" --instance-id i-082ee835595d90ae0 \
  --query "StandardOutputContent" --output text

# Option B — SSM interactive session:
aws ssm start-session --target i-082ee835595d90ae0 --region us-east-2
# In session:
# sudo systemctl restart woodpecker
# sudo docker compose -f /opt/woodpecker/docker-compose.yml logs -f --tail=20

After restart: Wait 30s, then redeliver the blocked webhook OR manually trigger the pipeline:

# Redeliver the specific delivery (get ID from GitHub repo Settings → Webhooks → Recent Deliveries):
gh api "repos/raxx-app/TradeMasterAPI/hooks/649427232/deliveries/<DELIVERY_ID>/attempts" --method POST

# OR manual pipeline trigger for deploy-prod (break-glass):
WP_TOKEN=$(aws ssm get-parameter --name /ci/woodpecker/admin-api-token --with-decryption \
  --region us-east-2 --query "Parameter.Value" --output text)
curl -sf -X POST \
  -H "Authorization: Bearer $WP_TOKEN" \
  -H "Content-Type: application/json" \
  -H "User-Agent: sre-agent/1.0" \
  -d '{"branch":"main","variables":{"DEPLOY_PROD_CONFIRM":"yes","CI_COMMIT_BRANCH":"main"}}' \
  https://ci.moosequest.net/api/repos/raxx-app/TradeMasterAPI/pipelines

Note on create-github-release: The deploy-prod pipeline's create-github-release step swallows errors as non-fatal when CI_COMMIT_TAG is empty (manual trigger). If the pipeline was triggered manually (not via a tag event), the GitHub Release will NOT be created automatically. Create it manually after the pipeline succeeds:

gh api repos/raxx-app/TradeMasterAPI/releases -X POST \
  --field tag_name="vX.Y.Z" \
  --field target_commitish="<main HEAD SHA>" \
  --field name="Release vX.Y.Z" \
  --field generate_release_notes=true

Verification:

# Forge connectivity restored: new pipeline created after webhook redeliver
WP_TOKEN=$(aws ssm get-parameter --name /ci/woodpecker/admin-api-token --with-decryption \
  --region us-east-2 --query "Parameter.Value" --output text)
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/pipelines?page=1&perPage=3" | \
  python3 -c "import sys,json; [print(f\"#{p['number']} {p['status']}\") for p in json.load(sys.stdin)]"
# Expected: most recent pipeline is running or success, not error/pending indefinitely

Prevention (action items): #1 (forge connectivity probe in healthcheck) and #2 (weekly restart schedule) in docs/incidents/2026-07-09-wp-forge-context-deadline.md.

RCA: docs/incidents/2026-07-09-wp-forge-context-deadline.md

Failure mode B-3: ForgeConnectivityDown alert — false positive from monitoring-stack restart

Symptom: ForgeConnectivityDown Prometheus alert fires (email to ops@raxx.app). WP server logs show NO auth errors. GitHub webhooks are being received and processed normally.

Cause: The alert fires on probe_success{job="blackbox-forge"} == 0 for: 10m. This probe runs inside the blackbox-exporter container on the ci-monitoring EC2 (i-0a656c6631ddc45b3). If the container is restarted by an SSM command (e.g., injecting config changes for an unrelated probe), consecutive Prometheus scrapes at the 5m interval can find the probe failing long enough to cross the for: 10m threshold. The Woodpecker server's actual forge connectivity to GitHub is unaffected.

Distinguish from genuine forge failure (Failure mode B-2):

# 1. Check WP server logs — genuine failure shows 'context deadline exceeded' or auth errors:
aws ssm send-command --region us-east-2 --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["docker logs woodpecker-woodpecker-server-1 --since=15m 2>&1 | grep -E \"deadline|401|403|bad credential|forge\" | tail -20"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

# 2. Check if blackbox-exporter recently restarted:
aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["docker inspect blackbox-exporter --format \"Started: {{.State.StartedAt}} Status: {{.State.Status}}\""]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

# 3. Directly probe the forge target from the monitoring box (should return probe_success 1):
aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["curl -s \"http://localhost:9115/probe?target=https://api.github.com/zen&module=github_api_2xx\" | grep probe_success"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

If false positive (blackbox restarted, probe now succeeds): No fix needed. Alert clears automatically on next Prometheus scrape (max 5m wait). Check Alertmanager current state:

aws ssm send-command --region us-east-2 --instance-ids i-0a656c6631ddc45b3 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["curl -s http://localhost:9093/api/v2/alerts | python3 -m json.tool | head -10"]}' \
  --output json | python3 -c "import sys,json; print(json.load(sys.stdin)['Command']['CommandId'])"

Expected: [] (empty).

If genuine forge connectivity failure: See Failure mode B-2 (stale TCP pool → restart woodpecker-server container) or check GitHub status at https://githubstatus.com + review security group outbound rules on i-082ee835595d90ae0.

Prevention: The hardened alert expression guards against blackbox-exporter restarts:

probe_success{job="blackbox-forge"} == 0 and on() (up{job="blackbox-http"} == 1)

Fires only when the blackbox-exporter is reachable but the forge probe fails. This is the expression baked into the IaC template (terraform/ci-monitoring/templates/cloud-init.sh.tpl) as of the 2026-07-14 drift consolidation.

RCA: docs/incidents/2026-07-14-forge-connectivity-down.md

Failure mode C: Agents not connecting to server

Symptom: Pipelines queue forever; Woodpecker Admin → Agents shows 0 connected. Cause: Agent-secret mismatch, SG blocking port 9000, or agents not running. Fix:

# Check ASG desired capacity:
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names ci-woodpecker-agents-asg \
  --region us-east-2 \
  --query "AutoScalingGroups[0].{desired:DesiredCapacity,inService:Instances[?LifecycleState=='InService'].InstanceId}" \
  --output json

# If desired=0, scale up:
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 1 \
  --region us-east-2

# If agents are running but not connected, SSH into an agent via SSM:
# (find instance ID from ASG describe output above)
aws ssm start-session --target i-XXXXXXXXXX --region us-east-2
# Check agent logs:
sudo docker compose -f /opt/woodpecker-agent/docker-compose.yml logs woodpecker-agent
# Common error: "rpc error: code = Unauthenticated" → agent-secret mismatch
# Verify server and agent SSM parameter have same value:
aws ssm get-parameter --name /ci/woodpecker/agent-secret --with-decryption --region us-east-2

Verification: Woodpecker Admin → Agents shows agent with status "connected".

Failure mode D: WAF blocking legitimate webhook traffic

Symptom: GitHub webhook delivery shows 403; WAF metrics show blocked requests on /hook*. Cause: GitHub IP range hit the WAF rate limit or a managed rule matched the payload. Fix:

# Check WAF logs in CloudWatch:
aws logs filter-log-events \
  --log-group-name aws-waf-logs-ci-woodpecker \
  --region us-east-2 \
  --filter-pattern '{ $.action = "BLOCK" }' \
  --start-time "$(date -d '-1 hour' +%s000)" \
  | jq '.events[].message | fromjson | {action,httpRequest,terminatingRuleId}'
# Identify which rule blocked the request.
# If rate limit: increase var.waf_rate_limit_webhook_per_5min in terraform.tfvars + apply.
# If managed rule: add an override in waf.tf for that specific rule (count instead of block).

Verification: GitHub webhook delivery test returns 200 in repository settings.

Failure mode D-2: WAF SizeRestrictions_BODY blocks all GitHub webhook deliveries

Symptom: Every GitHub webhook delivery to ci.moosequest.net/api/hook returns HTTP 403. No Woodpecker pipelines trigger on PR or push events. WAF CloudWatch logs show terminatingRuleId: SizeRestrictions_BODY from AWSManagedRulesCommonRuleSet. Cause: The AWSManagedRulesCommonRuleSet SizeRestrictions_BODY rule enforces an 8192-byte body inspection cap. Real GitHub PR/push webhook payloads are 12–36 KB and exceed this cap. The managed rule blocks the request before Woodpecker ever sees it. This is distinct from rate-limit blocks (Failure mode D) — it fires on the very first delivery regardless of volume. Fix:

# Confirm the root cause: look for SizeRestrictions_BODY in the terminatingRuleId field
aws logs filter-log-events \
  --log-group-name aws-waf-logs-ci-woodpecker \
  --region us-east-2 \
  --filter-pattern '{ $.action = "BLOCK" }' \
  --start-time "$(date -d '-1 hour' +%s000)" \
  | python3 -c "
import sys, json
for line in sys.stdin:
    try:
        obj = json.loads(line)
        for ev in obj.get('events', []):
            msg = json.loads(ev['message'])
            print(msg.get('terminatingRuleId'), msg.get('httpRequest', {}).get('uri'))
    except Exception:
        pass
"

# If you see SizeRestrictions_BODY, verify the AllowGitHubWebhook rule exists in waf.tf:
grep -A 20 'AllowGitHubWebhook' infra/ci/waf.tf

# If the rule is missing, apply via Terraform to restore it:
cd infra/ci
terraform plan -out ci-waf-fix.plan
# Verify plan: 0 to destroy, no server/RDS/ALB/ASG replacement
terraform apply ci-waf-fix.plan

Verification: Push a commit to any branch. In GitHub → repository Settings → Webhooks → Recent Deliveries, the new delivery returns HTTP 204. A Woodpecker pipeline appears in GET /api/repos/1/pipelines.

RCA: docs/incidents/2026-07-04-waf-webhook-block-batch1-parse-error.md

Failure mode E-2: Nil-pointer panic at repo activation (server/api/repo.go)

Symptom: POST /api/repos?forge_remote_id=ID returns a 500 panic trace referencing server/api/repo.go:149. Repo activation fails. Cause: Pre-v3.0 bug in v2.8.x — nil forge OAuth token on user record. Fixed in v3.x. Fix: This should not recur on v3.16.0. If it does, the admin user's forge OAuth token may be missing (user never completed OAuth flow). Operator must log out of Woodpecker UI and log back in via the GitHub OAuth flow to mint a fresh token. Verification: POST /api/repos?forge_remote_id=ID returns HTTP 200 JSON with "active":true.

Failure mode E-3: Pipeline compile errors after v3.x migration

Symptom: Manual pipeline trigger returns {"status":"error","errors":[{"type":"compiler","message":"go-yaml load error..."}]}. No GitHub commit status posted. Cause: .woodpecker/*.yaml pipeline definitions use v2.x syntax. v3.x YAML parser is stricter. Common causes: group: field (must be depends_on:), multi-line command continuation syntax, pipeline: keyword. Fix: Update .woodpecker/*.yaml files for v3.x syntax. See §v3.x pipeline YAML breaking changes above. Common v3 compile errors to check: - secret "NAME" not found — the secret named in from_secret: is not registered in Woodpecker. Either register the secret via POST /api/repos/{owner}/{name}/secrets or remove event: manual from the pipeline's when: block until the secret is available (cron triggers are unaffected). - Specified clone image does not match allow list, netrc is not injected — clone block uses woodpecker-ci/plugin-git (with hyphen) instead of woodpeckerci/plugin-git (without hyphen). See §Woodpecker clone plugin allow list. Verification: Manual pipeline trigger completes; status field is pending or running, not error. GitHub commit status appears on the HEAD commit.

Failure mode E-4: Shell-defined variables silently empty in pipeline commands

Symptom: Pipeline runs but produces wrong output or fails with empty-argument errors: - git checkout "" — git fails with "bad revision" - --base-sha "" — script fails with invalid SHA - MaxMind URL missing license_key parameter (download returns 401) - echo prints "Expected head revision: " with nothing after the colon All are caused by the same root issue.

Cause: WP v3 performs ${VAR} variable substitution at YAML-parse time on ALL text in the pipeline definition. Only WP built-in variables (CI_*, WOODPECKER_*) and secrets registered via from_secret: are available at parse time. Shell-defined variables (assigned by earlier shell commands: BASELINE=$(...), BASE_SHA=$(...), FILESIZE=$(...)) do NOT exist at parse time. WP substitutes them to empty string silently — no parse error, no warning, commands just receive empty arguments.

Known broken patterns (all fixed 2026-07-05, except the 2026-08-19 recurrence below): - git checkout "${BASELINE}" → git checkout "" — empty baseline SHA in vcpkg-manifest-check.yaml - --base-sha "${BASE_SHA}" → --base-sha "" — empty diff base in migration-collision-check.yaml - URL ...&license_key=${MAXMIND_LICENSE_KEY}&... → URL with missing key param in maxmind-asn-refresh.yaml - echo "${HEAD}" where HEAD is set by a prior shell command in alembic-version-check.yaml - Recurrence, 2026-08-19: git push --force heroku "${SUBTREE_SHA}:refs/heads/main" → git push --force heroku ":refs/heads/main" (empty-source refspec) in deploy-velvet.yaml's deploy step. Introduced 2026-07-08 (6bf74fefab), three days after this class was first fixed elsewhere — the fix was never generalized into an enforced check, so a new file reintroduced the identical anti-pattern. Latent/undetected for ~6 weeks because the step's path filter (velvet/** or the workflow file itself) never matched a release/main push until PR #4474 incidentally touched deploy-velvet.yaml and tripped it. Heroku's pre-receive hook refused the empty-refspec push ("cannot delete main branch") — loud failure, but the 6-week latency before that first trigger is the real gap. Discovered during the release-2026.08.19 staging shepherd run. See docs/incidents/2026-08-19-ci-boundary-velvet-deploy-false-positive-refspec.md and issue #4513 (fix filed, not yet applied). - Second recurrence, same file, 2026-09-07 (ci-pipelines sweep): migration-collision-check.yaml's check step declares GITHUB_REPOSITORY: ${CI_REPO} in its own environment: block (correct — that is a genuine WP built-in on the right-hand side of the declaration), but then re-referenced it as ${GITHUB_REPOSITORY} inside commands: (gh api "repos/${GITHUB_REPOSITORY}/pulls/${CI_COMMIT_PULL_REQUEST}"). GITHUB_REPOSITORY itself is a step environment: key, NOT a WP built-in — it does not exist at YAML-parse time, so WP blanked it to "", producing repos//pulls/N and killing this REQUIRED status check (gh api non-zero exit under set -eu). This is the same file, and the same class of bug, that was "fixed" here in 2026-07-05 for a different variable (BASE_SHA) — the fix was never generalized, so the file relapsed with a new variable. Fixed by dropping the braces in commands: (repos/$GITHUB_REPOSITORY/pulls/$CI_COMMIT_PULL_REQUEST); the environment: declaration itself is unchanged (brace form there is correct — see Fix below).

Fix: Use $VAR (no braces) for shell-defined variables. WP only substitutes ${VAR} (brace form). Bare $VAR is passed through to the shell unchanged.

# WRONG — WP substitutes ${BASELINE} to empty at parse time
git -C /opt/vcpkg checkout "${BASELINE}"

# CORRECT — $BASELINE is passed to the shell which resolves it at runtime
git -C /opt/vcpkg checkout "$BASELINE"

Exception: WP built-in variables (${CI_COMMIT_BRANCH}, ${CI_REPO}, etc.) MUST use the ${VAR} brace form in YAML environment: blocks because they need WP substitution. In shell command strings, both $CI_COMMIT_BRANCH and ${CI_COMMIT_BRANCH} work since WP substitutes them either way; but prefer $VAR in commands for consistency.

A step's own environment: key (e.g. GITHUB_REPOSITORY: ${CI_REPO}) is a RUNTIME-only variable once declared — it is populated into the shell's environment when the container starts, not at YAML-parse time. Brace form belongs ONLY on the right-hand side of its environment: declaration (where a genuine WP built-in is being assigned to it); every use of that key inside commands: MUST be bare $VAR. Treating a step environment: key as eligible for brace form in commands: is exactly what caused the 2026-09-07 recurrence above — do not repeat it.

Lint check: grep -rn '\${[A-Z][A-Z_0-9]*}' .woodpecker/ then manually verify each match is a WP built-in variable (CI_, WOODPECKER_) used correctly — i.e. either in an environment: block's right-hand side, or (for a genuine built-in) inside commands:. A match on a step environment: key's name inside commands:, or on any shell-assigned variable (BASE=$(...), FILESIZE=$(...), etc.), is a bug — replace ${VAR} with $VAR.

Verification: Re-run the pipeline; commands receive the correct non-empty values.

Failure mode E-5: WP v3 YAML pre-processor rejects ${{ — ALL pipelines break

Symptom: Every pipeline for every event (PR, push, cron, manual) fails immediately at compile time with "unable to parse variable name". No steps execute. GitHub commit status checks are absent (Woodpecker never posts them). The error is global — it affects ALL .woodpecker/*.yaml files in the repo, not just the file containing the offending pattern.

Cause: Woodpecker v3's substitution pre-processor scans ALL YAML text — including YAML comments — before handing the file to the YAML parser. ${{ — WP parses ${...} substitutions; the second { is not a valid identifier character and causes "unable to parse variable name". This is the GitHub Actions expression syntax. It is invalid in WP YAML even inside a comment line.

Fix:

# Scan for the pattern:
grep -rn '\${{' .woodpecker/

# BAD:  # GH_TOKEN: ${{ steps.mint.outputs.token || github.token }}
# GOOD: # GH_TOKEN: (mint_bot_token.outputs.token || github.token) from GHA workflow

Verification: curl -s -X POST https://ci.moosequest.net/api/repos/1/pipelines \ -H "Authorization: Bearer $WP_TOKEN" \ -H "Content-Type: application/json" \ -d '{"branch":"develop"}' | jq '{id:.id,status:.status,errors:.errors}' status should be pending or running, not error.

RCA: docs/incidents/2026-07-04-waf-webhook-block-batch1-parse-error.md

Correction (2026-08-10, #4452): The original version of this entry also listed ${VAR:-default} (the :- default-expansion form) as a second global-breakage pattern, recommending the $${VAR:-default} escape for all cases. That was an overbroad reading of the 2026-07-04 incident — the confirmed root cause there was ${{ alone (see the RCA's own "What didn't go well": "Hypothesis 1 [${VAR:-}] was plausible but incomplete"). In practice, ${VAR:-default} on a genuine WP built-in (CI_COMMIT_BEFORE, CI_COMMIT_TARGET_BRANCH, etc.) compiles and runs cleanly — WP substitutes the brace form at parse time same as bare ${VAR}, and a present-but-unset built-in still resolves through the default expansion. Confirmed working precedents already live on develop before this correction: .woodpecker/ci-pr.yaml (BASE="${CI_COMMIT_BEFORE:-}", FALLBACK="${CI_COMMIT_TARGET_BRANCH:-develop}", merged 2026-07-10, passing as a required check daily since) and this same runbook's own §Failure mode E-6 fix (${CI_COMMIT_TARGET_BRANCH:-develop}, #4347, merged 2026-07-25). Re-verified live via a scoped draft-PR pipeline run for .woodpecker/queue-docker-smoke.yaml (BEFORE="${CI_COMMIT_BEFORE:-}") — see docs/incidents/2026-08-10-queue-docker-smoke-ci-commit-before-unset.md.

The $${VAR:-default} escape is still the correct choice for two distinct, narrower cases — neither of which applies to WP built-ins: 1. A secret registered via from_secret: that you want resolved by the shell at runtime from the step's environment: block, not baked into the compiled pipeline JSON at parse time (secret-leak-into-pipeline-definition risk). See .woodpecker/console-degraded-auto-file.yaml's $${CI_FORGE_TOKEN} usage. 2. A variable that is not a WP built-in and not registered as a step environment: key at all (WP has nothing to substitute it with) — bare ${VAR} in that case silently resolves to empty string at parse time (see Failure mode E-4) rather than raising a compile error, so $${VAR:-default} is the only way to defer resolution to the shell.

Updated fix/lint guidance:

# Always invalid — scan and remove/reword even in comments:
grep -rn '\${{' .woodpecker/

# ${VAR:-default} on a WP built-in (CI_*, WOODPECKER_*): fine as-is, no escape needed.
# ${VAR:-default} on a secret or step-local shell var: escape with $$ so the shell
# resolves it at runtime instead of WP substituting (or silently blanking) it at parse time.

Failure mode E-6: stale-branch-guard red on PRs it should pass — three independent dimensions

Symptom: ci/woodpecker/pr/ci-pr fails on the stale-branch-guard step across MULTIPLE unrelated open PRs simultaneously (dimensions 1 and 2), or on every promote/* PR and on any PR whose branch has the base merged into it (dimension 3) — a strong signal this is a guard-logic bug, not a genuinely-stale individual branch.

Cause: stale-branch-guard (.woodpecker/ci-pr.yaml + scripts/ci/check_stale_branch.sh + scripts/ci/resolve_pr_base_sha.sh) has three independent bugs found so far, each of which can false-fail PRs that are not stale. Diagnose which one (or more) is active before touching thresholds:

  1. Base-ref hardcoded (fixed 2026-07-25, #4347): the guard always compared against origin/develop regardless of the PR's real base. PRs based on main or release (ADR-0115 emergency-hotfix path) always failed. Fixed by deriving GITHUB_BASE_REF from ${CI_COMMIT_TARGET_BRANCH:-develop}.
  2. Absolute fork-point age (fixed 2026-07-30): the age check computed now - <fork-point commit's own timestamp>. For a branch cut fresh off the base branch's current tip, the fork point IS the base tip — so whenever the base branch (develop) went quiet for longer than STALE_BRANCH_MAX_AGE_HOURS (48h), EVERY PR branch failed the age check with 0 commits behind. Fixed by measuring age from the OLDEST commit the PR branch is missing from the base branch, and skipping the age check entirely when COMMITS_BEHIND == 0.
  3. PR head that is itself a merge commit read as a merge ref (fixed 2026-10-31, #4855 and #4885): Woodpecker's clone step does git fetch --no-tags origin +<CI_COMMIT_SHA>: then git reset --hard <CI_COMMIT_SHA>, so HEAD is the PR head commit, not GitHub's merge ref. The resolver treated "pull_request event and HEAD has 2+ parents" as "HEAD is the merge ref" and returned HEAD^1 as the base and HEAD^2 as the head. For a promote/* branch (origin/release with origin/main merged in) the parents are (release tip, main tip), so the guard printed Base: ... (rev: <release tip>), Head: <main tip> and Commits behind origin/main: 24 on a branch that fully contains main. The same misread hits any PR whose author merged the base into the branch. Fixed by classifying the checkout by identity: HEAD == CI_COMMIT_SHA is the PR head (base = origin/<target>, never HEAD^1); HEAD^2 == CI_COMMIT_SHA is a real merge ref (base = HEAD^1); anything else, or an ambiguous merge-shaped HEAD with CI_COMMIT_SHA unset, exits 1. The other gates that call the resolver (b1-migration-path-lint, flag-promotion-check, route-migration-parity, smoke-surface-drift-check, migration-collision-check) now diff a promotion PR against origin/main instead of the release tip, so they examine the promoted work for the first time on those PRs.

Diagnose which dimension is active:

git fetch origin develop --quiet
git log -1 --format='%H %ci %s' origin/develop   # is develop's tip itself old (>48h)?

# For the specific failing PR branch:
git fetch origin <pr-branch> --quiet
FORK_SHA=$(git merge-base origin/develop <pr-branch>)
git rev-list --count "${FORK_SHA}..origin/develop"   # commits behind — should be low for a fresh branch
git log -1 --format='%ci' "${FORK_SHA}"              # fork-point date — if old AND commits-behind is 0,
                                                       # this is dimension 2 (absolute-age bug)

If commits-behind is 0 (or small) but the fork-point date is old, this is dimension 2 — the base branch has gone quiet, not the PR branch falling behind. Confirm scripts/ci/check_stale_branch.sh measures age from the oldest MISSING commit (not the fork-point's own date) — if it still uses FORK_TS=$(git log -1 --format="%ct" "${FORK_SHA}"), the 2026-07-30 fix has regressed or was reverted.

Dimension 3 fingerprint, in the step log: the Base: rev is not origin/<target>'s tip, the Head: rev is not the PR head, and the resolver's own lines say pull_request merge ref — base is HEAD^1 although the clone log shows git reset --hard <sha>. Confirm with git rev-list --parents -n 1 <CI_COMMIT_SHA>: two parents and git merge-base --is-ancestor origin/<target> <CI_COMMIT_SHA> true means the branch already contains the base.

Fix: See scripts/ci/check_stale_branch.sh for the corrected logic (relative-to-base-progression age, skipped when 0 commits behind) and scripts/ci/resolve_pr_base_sha.sh for the checkout classification. Do NOT raise STALE_BRANCH_MAX_AGE_HOURS or STALE_BRANCH_MAX_COMMITS as a workaround — that weakens the gate for genuinely-stale branches without fixing the underlying measurement bug. A promotion branch cut before the fix reached release still carries the old scripts; on such a PR dimension 3 is expected, and git merge-base --is-ancestor origin/<target> <head> being true is what shows the red is a false positive.

Verification: Cut a fresh test branch off origin/develop HEAD and run the guard locally:

git fetch origin develop --quiet
git checkout -b /tmp-stale-guard-check origin/develop
GITHUB_BASE_REF=develop bash scripts/ci/check_stale_branch.sh
# Expected: "Commits behind origin/develop: 0" and exit 0, regardless of how old
# develop's tip commit is.
git checkout - && git branch -D /tmp-stale-guard-check

RCA: docs/incidents/2026-07-25-ci-pr-stale-branch-guard-hardcoded-base.md (dimension 1), docs/incidents/2026-07-30-ci-pr-stale-branch-guard-absolute-age.md (dimension 2), docs/incidents/2026-10-31-ci-pr-stale-branch-guard-promote-head-read-as-merge-ref.md (dimension 3)

Failure mode E-7: base-branch-lint red on every release-promotion PR (CONTAMINATED COMMIT)

Symptom: ci/woodpecker/pr/ci-pr fails at the base-branch-lint step on a promote/release-*-to-main PR (base main, or release) with CONTAMINATED COMMIT <sha> naming a develop commit, usually the develop tip. Both release PRs #4809 and #4817 needed an admin merge because of it.

Cause: A promotion PR carries every develop commit since the last promotion by design (ADR-0115, ADR-0148). Every open feature branch cut from develop contains those same commits, so scripts/ci/check_pr_base_clean.sh ("commit also on another remote branch") fires on every promotion. A second defect hid behind it: the step passed only GITHUB_HEAD_REF, so GITHUB_BASE_REF was unset and the script compared every PR, including PRs into main, against origin/develop.

Fix (2026-10-30): The script exits 0 with an EXEMPT: line when the head matches promote/* AND the base is main or release. The step now exports GITHUB_BASE_REF from CI_COMMIT_TARGET_BRANCH, then CI_COMMIT_BRANCH (the target on pull_request events), then develop, and the script falls back to CI_COMMIT_TARGET_BRANCH itself. Both conditions are required: a promote/* head into develop, or any other head into main/release, is still checked in full. Pinned by scripts/ci/tests/test_check_pr_base_clean.py.

Verification: In the step log of a promotion PR look for EXEMPT: 'promote/...' -> 'main' is a release-promotion PR. Locally:

git fetch --all --quiet
git switch --detach origin/promote/release-<date>-to-main
GITHUB_BASE_REF=main GITHUB_HEAD_REF=promote/release-<date>-to-main bash scripts/ci/check_pr_base_clean.sh
# Expected: EXEMPT line, exit 0

If it recurs: the promotion branch was probably not named promote/*. Do not widen the glob; rename the branch or fix whatever creates it.

RCA: docs/incidents/2026-10-30-ci-gates-promotion-fp-and-migration-collision.md

Failure mode E-8: migration-collision-check red in about 20 seconds on every PR

Symptom: ci/woodpecker/pr/migration-collision-check fails fast (about 20 to 30 s) on every PR that touches console/migrations/versions/**, backend_v2/alembic/versions/** or the sqitch plan. About 35 consecutive PRs (#4294 to #4812), none passed. scripts/ci/check_migration_collision.py passes locally on the same merge ref under Python 3.11 and 3.14.

Read the cause first. Since 2026-10-30 a failing run posts its own output as a sticky PR comment titled migration-collision-check failed (raxx-ops-bot, via the report-failure step). Woodpecker step logs need a login; that comment does not. If it is absent, the failure happened before check produced output, or the vault/bot credentials were unavailable to report-failure; use the UI log.

Cause, defect 1 (established): the old check step resolved the PR base SHA with gh api repos/$GITHUB_REPOSITORY/pulls/$CI_COMMIT_PULL_REQUEST under set -eu, which needs a working GitHub token. A probe workflow on PR #4846 showed that gh api and gh pr list using the CI_FORGE_TOKEN secret FAIL on pull_request events (token empty or rejected; injecting it via from_secret in #4613 did not change that). Same class as the five ci-pr gates fixed in #4595.

Cause, defect 2 (established from the failure-report comment on PR #4846): with the lookup removed the check was still red. The step output showed HEAD is the single-parent PR head commit (not the merge ref), CI_COMMIT_BEFORE is unset and origin/develop is not present, so resolve_pr_base_sha.sh exited with FATAL: could not determine a PR base SHA. Its branch (a), the merge-ref first parent, never applies in this Woodpecker; only branch (c), origin/<target>, can work. The ci-pr gates that use the resolver succeed only because their earlier base-branch-lint step runs git fetch --all with a vault token. CI_COMMIT_PULL_REQUEST and CI_COMMIT_TARGET_BRANCH were populated, so empty PR-event variables were not the cause. Rule: any workflow that calls resolve_pr_base_sha.sh must populate origin/<target> first, or it fails closed. There is one way to do that: vault read token, throwaway netrc, then scripts/ci/fetch_pr_base_ref.sh (added in #4863 for the fetch-pr-base step of ci-pr; migration-collision-check calls the same script). Do not write a second fetch.

Fix (2026-10-30): the check step fetches the target branch first, with the same mechanism as ci-pr's fetch-pr-base step (vault GITHUB_API_READONLY_TOKEN from /MooseQuest/github/, a throwaway ~/.netrc deleted right after the fetch, bash scripts/ci/fetch_pr_base_ref.sh, which is best effort and always exits 0), resolves the tip with the shared resolver (fails closed; CI_COMMIT_BEFORE is ignored because on a PR it can be the previous head of the branch), and diffs against git merge-base <tip> HEAD so files develop deleted or renamed since the fork do not look newly added. For the cross-PR comparison it uses the first of CI_FORGE_TOKEN and the vault read token that can actually run gh pr list; the gh install and tokens are best-effort, and a failure prints a WARNING saying the comparison is SKIPPED without failing the gate. Output is teed to .mcc-check.log and report-failure posts it on failure. The pipeline self-triggers on its own YAML, check_migration_collision.py, resolve_pr_base_sha.sh and post_ci_failure_comment.py. Pinned by scripts/ci/tests/test_migration_collision_check_step.py (CI-shaped fixture: single-parent HEAD, no origin/* refs, moved-on base) and test_post_ci_failure_comment.py, which run the real commands: blocks through a reproduction of Woodpecker's script wrapper.

What a green run does NOT prove: if the output contains WARNING: no usable GitHub token or gh CLI is not installed, the cross-PR collision comparison did not run and the check only confirmed the PR is well-formed. To enable it, enable the pull_request event on the CI_FORGE_TOKEN secret in the Woodpecker UI, or give the vault read token pull-request read access. The success line is cross-PR comparison is enabled (token from <name>).

Verification: the output starts with [mcc] event=pull_request pr=<n> ref=refs/pull/<n>/merge, shows PR <n> base is <40-char sha> (target tip <40-char sha>), then either cross-PR comparison is enabled or a WARNING. Do not accept a green run without reading which one. If the fetch warns (no GitHub read token from the vault or fetch of origin/<target> failed) the step fails with the resolver's FATAL message: that is the gate refusing to run blind, not a defect in the PR.

RCA: docs/incidents/2026-10-30-ci-gates-promotion-fp-and-migration-collision.md

Failure mode F: RDS storage alarm firing

Symptom: ci-rds-storage-low CloudWatch alarm in ALARM state. Cause: Woodpecker pipeline history accumulating; database approaching 16 GB. Fix:

# Option 1: Purge old build logs in Woodpecker UI (Admin → Repositories)
# Option 2: Increase RDS allocated storage via Terraform:
# In terraform.tfvars, allocated_storage is managed by auto-scaling.
# The RDS resource has max_allocated_storage=100, so RDS auto-scales
# up to 100 GB automatically. If already at 100 GB, increase max_allocated_storage
# in rds.tf and apply.

# Check current storage:
aws rds describe-db-instances \
  --db-instance-identifier ci-woodpecker \
  --region us-east-2 \
  --query "DBInstances[0].AllocatedStorage"

Verification: ci-rds-storage-low alarm returns to OK state.

Failure mode F: Login buttons dead — "Login to CI" / "Abort" do nothing

Symptom: Browser passes the Google OIDC gate and reaches the Woodpecker login screen, but clicking "Login to CI" or "Abort" has no effect. No GitHub OAuth redirect occurs.

Cause: The ALB default OIDC rule is intercepting Woodpecker's own auth paths (/api/*, /authorize, /login, /logout). The Woodpecker SPA makes XHR calls to /api/* to initiate the GitHub OAuth flow; if those calls get a Google 302 redirect instead of a Woodpecker JSON response, the JavaScript login handler silently fails. Additionally, the GitHub OAuth callback at /authorize?code=...&state=... must reach Woodpecker directly — an OIDC interception at that path loses the GitHub code and state parameters.

Diagnose:

# Each of these should NOT redirect to accounts.google.com.
# Expected: 401 JSON (api/user), 303 to github.com (authorize/login), 303 to / (logout)
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/api/user
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/authorize
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/login

# Confirm ALB rule at priority 25 exists:
aws elbv2 describe-rules \
  --listener-arn arn:aws:elasticloadbalancing:us-east-2:521228113048:listener/app/ci-woodpecker-alb/2bf78b6ff4a0780f/2b0157eb96f64e49 \
  --region us-east-2 \
  --query 'Rules[?Priority==`25`]'

Fix (if priority-25 rule is missing — e.g., after a destructive terraform apply that lost the rule):

LISTENER_ARN="arn:aws:elasticloadbalancing:us-east-2:521228113048:listener/app/ci-woodpecker-alb/2bf78b6ff4a0780f/2b0157eb96f64e49"
TG_ARN="arn:aws:elasticloadbalancing:us-east-2:521228113048:targetgroup/ci-woodpecker-server-tg/89042c9f57a2d3ad"
aws elbv2 create-rule \
  --listener-arn "$LISTENER_ARN" \
  --priority 25 \
  --conditions '[{"Field":"path-pattern","PathPatternConfig":{"Values":["/api/*","/authorize","/login","/logout"]}}]' \
  --actions "[{\"Type\":\"forward\",\"TargetGroupArn\":\"$TG_ARN\"}]" \
  --tags '[{"Key":"Name","Value":"ci-woodpecker-auth-bypass-oidc"},{"Key":"Purpose","Value":"woodpecker-own-auth-endpoints"}]' \
  --region us-east-2

Verification:

# /api/user must return 401 (not a Google redirect)
curl -sS https://ci.moosequest.net/api/user
# Expected: "User not authorized" with HTTP 401

# /authorize must redirect to github.com
curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" --max-redirs 0 https://ci.moosequest.net/authorize
# Expected: 303 https://github.com/login/oauth/authorize?...

Then retry the login in a fresh browser tab.

Security note: Removing the OIDC gate from /api/*, /authorize, /login, and /logout does NOT expose these paths — Woodpecker enforces its own GitHub-OAuth + session auth on every /api/* endpoint (returns 401 for unauthenticated requests) and validates a signed JWT state on the OAuth callback. The WAF rate-limit rule still applies.

Bounded ops wrappers (agent sessions — no raw admin token/AWS access needed)

Two scripts exist so an operator or an agent session can perform the handful of mutating ops that came up repeatedly during the 2026-08-30/31 CI fleet incident, without needing (or being handed) the WP admin API token or broad AWS credentials directly. Each script holds its own scoped credential internally and is hardened as the actual security boundary — not just the permission rule that lets a session invoke it (see each script's header for the full threat model and an adversarial review's findings/fixes, 2026-08-31).

# Cancel or restart ONE specific pipeline by its number (never a bare
# trigger — that fans out every workflow matching event:manual, ~65 on
# this repo).
scripts/ops/wp-pipeline.sh cancel  <repo-id> <pipeline-number>
scripts/ops/wp-pipeline.sh restart <repo-id> <pipeline-number>

# Terminate-and-replace ONE specific wedged agent instance (verified to be
# a member of ci-woodpecker-agents-asg before anything is touched — cannot
# be pointed at the Woodpecker server or any other ASG).
scripts/ops/ci-agent-recycle.sh <instance-id> [--force-unprotect]

Neither script is configurable via env or flags beyond the arguments shown above — host, SSM parameter name, target ASG, and region are fixed in the script source. If you need this tooling to do something it doesn't currently do, that's a reviewed code change to the script, not an environment variable at call time.

For a fleet-wide launch-template rollout (not a single dead instance), use ci-asg-refresh.sh instead (see Wave B section below). For the Woodpecker SERVER itself, neither script applies — follow the server-replace procedure documented earlier in this runbook.

Failure mode: queue stalls with one agent in business hours

Symptom: ci-woodpecker-queue-waiting alarm email at ops@raxx.app (PendingCount >= 1 for ~10 min), or jobs pending in the WP UI while worker_count is 1 and busy. Cause: ASG ran 1 agent (2026-10-27). The scale arbiter drains desired to ASG min_size when the queue is idle, so a min of 1 lets the fleet fall back to one agent mid-day. Policy (operator decision 2026-10-30, "Floor 2 plus alarm"; supersedes the weekday-floor-of-1 decision in #4803): business hours are min=desired=2 (ci-agents-scale-out-morning, 09:00 UTC Mon-Fri); overnight and weekends are min=desired=1 (ci-agents-scale-in-evening, 22:00 UTC Mon-Fri). The ASG ci-woodpecker-agents-asg (us-east-2) is shared with Project Avenue; the second agent is a spot instance (small added spot cost). Terraform does not manage desired_capacity or min_size on the ASG (ignore_changes), so an apply at any hour does not revert the schedule-set floor or a hand-set/arbiter-set desired count; the two scheduled actions re-assert min_size every weekday. max_size and the scheduled actions themselves remain Terraform-managed. Fix: Change the policy via Terraform (infra/ci/agents.tf) only. For an immediate stopgap: aws autoscaling update-auto-scaling-group --auto-scaling-group-name ci-woodpecker-agents-asg --min-size 2 --desired-capacity 2 --region us-east-2, then follow with the PR. One-time hand-made scheduled actions (for example ci-agents-temp-hold-4-2026-10-30, min=desired=4 at 09:05 UTC) are not in Terraform and expire on their own; do not import or delete them from a Terraform change. Verification: aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ci-woodpecker-agents-asg --region us-east-2 shows 2 InService during business hours; aws autoscaling describe-scheduled-actions --auto-scaling-group-name ci-woodpecker-agents-asg --region us-east-2 shows ci-agents-scale-out-morning MinSize=2 with no DesiredCapacity (the ASG lifts desired to the new min) and ci-agents-scale-in-evening MinSize=1 DesiredCapacity=1; aws cloudwatch describe-alarms --alarm-names ci-woodpecker-queue-waiting --region us-east-2 shows the alarm with the ci-woodpecker-alerts SNS topic as alarm action; WP /api/queue/info pending drains.

Emergency stop

To immediately halt all CI builds (no deploy risk):

# Option 1 (fastest — 30 seconds): Scale agents to 0
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 0 \
  --region us-east-2

# Option 2: Stop Woodpecker server entirely (queues all new jobs)
INSTANCE_ID="$(terraform -chdir=infra/ci output -raw woodpecker_server_instance_id)"
aws ssm start-session --target "$INSTANCE_ID" --region us-east-2
# In session:
sudo systemctl stop woodpecker

# Option 3: Quarantine at ALB level (blocks all traffic including webhooks)
# Update ALB listener to return 503 maintenance response:
# (do via AWS Console or CLI; do NOT `terraform apply` the change as it
#  will create drift — this is a break-glass manual action)

To resume: restart the service (sudo systemctl start woodpecker) and/or scale agents back up.

Teardown

# IMPORTANT: Remove the prevent_destroy lifecycle guard in rds.tf first
# by setting db_deletion_protection = false and applying, then destroy.
cd infra/ci
terraform apply -var="db_deletion_protection=false"
terraform destroy

# Bootstrap teardown (ONLY if you want to delete the state bucket too —
# this destroys the state of all other Terraform roots using this bucket):
# cd infra/ci/bootstrap
# Edit bootstrap/main.tf: remove lifecycle { prevent_destroy = true } from both resources
# terraform destroy
# (This also deletes moosequest-tf-state-us-east-2 and all state within it)

Security considerations

No SSH keys

All instances use SSM Session Manager for break-glass access. Key-pair based SSH is explicitly disabled (key_name = null). If SSM is unreachable, the VPC endpoints (ssm, ssmmessages, ec2messages) must be functioning. Check VPC endpoint health first.

IMDSv2

All EC2 instances have http_tokens = required. The instance metadata service is protected against SSRF. Containers cannot reach IMDS (hop limit 1 on server; limit 2 on agents to allow builds that need IAM credentials from the instance role).

Docker socket on agents

The Woodpecker agent container bind-mounts /var/run/docker.sock. This grants build containers root-equivalent access to the agent host. Acceptable for Phase 1 with internal-only, trusted pipeline definitions. Action item before adding external contributors: evaluate rootless Docker or Kaniko for image builds.

Trusted-repo posture (WP Docker volume mounts) — enabled 2026-07-09 (#4107)

What "trusted" means in Woodpecker v3: Woodpecker v3 exposes three trust dimensions per repo: - trusted.network — pipeline steps can use host networking - trusted.volumes — pipeline steps can mount host paths via volumes: in step YAML - trusted.security — pipeline steps can request privileged containers

Current state (raxx-app/TradeMasterAPI, repo ID = 1):

Flag Value Rationale
trusted.network true Set at activation; required for gRPC connectivity in build steps
trusted.volumes true Set 2026-07-09 (#4107) — enables pipeline steps to mount /var/run/docker.sock from the host (already bind-mounted into the agent container by user_data_agent.sh.tpl)
trusted.security false Not needed; DinD is NOT the accepted posture (OQ-1 resolved = socket-mount)

Why trusted.volumes = true is required: The host Docker socket is bind-mounted into the WP agent container at startup (/var/run/docker.sock:/var/run/docker.sock in agent docker-compose.yml). For a PIPELINE STEP to reach the same socket, the pipeline YAML must declare a volumes: block that mounts the socket into the step container. Woodpecker only permits this when the repo has trusted.volumes = true. Without it, the step container cannot reach the host daemon and docker buildx build fails with "Cannot connect to the Docker daemon".

Security trade-off: Any pipeline step can mount arbitrary host paths and interact with the host Docker daemon, giving it root-equivalent access to the agent host. This is accepted for Phase 1 because: 1. Single-repo WP instance — pipeline YAML is controlled by the operator alone 2. No external contributors — no untrusted code executes in pipelines 3. Prod secrets are restricted by event type on WP secrets (push, pull_request only, not manual for prod-credential secrets) 4. Agent instances are spot/ephemeral — compromised agent is terminated on scale-in

Action items before adding external contributors: - Evaluate rootless Docker (no root socket exposure) or Kaniko (no Docker daemon) - OR implement fork isolation (WP require_approval: all for non-org members) - See #4106 tracking note

How to view/change the trusted state:

WP_TOKEN="<admin JWT from ci.moosequest.net>"
# View:
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: raxx-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['trusted'])"

# Set (trusted.volumes=true, trusted.security=false):
curl -sf -X PATCH \
  -H "Authorization: Bearer $WP_TOKEN" \
  -H "User-Agent: raxx-agent/1.0" \
  -H "Content-Type: application/json" \
  -d '{"trusted":{"network":true,"volumes":true,"security":false}}' \
  "https://ci.moosequest.net/api/repos/1"

This setting persists in the Woodpecker database — it is not managed by Terraform (WP API-only). If the WP server is rebuilt from scratch, re-apply this setting before W7/W8 pipelines run.

State encryption

Terraform state in S3 is encrypted (AES-256 via aws/s3 KMS). The ALB OIDC client secret appears in state because the aws_lb_listener resource stores it inline. Mitigations: state bucket access restricted to terraform-executor role, TLS enforced on all S3 access.

Secret rotation

All secrets in SSM are versioned (20 versions retained). Zero-downtime rotation: 1. Update SSM parameter value (--overwrite). 2. Restart Woodpecker server (reads SSM at boot via user-data). 3. For agent-secret: scale ASG to 0, update SSM, scale back up (agents read SSM at boot). 4. For OIDC client secret: update SSM → terraform apply (ALB listener rule update propagates within seconds, no downtime).

Phase 2 — cron pilot soak (active, started 2026-07-04)

Pilot crons

Three low-risk cron workflows run on Woodpecker agents IN PARALLEL with their GH Actions counterparts during the 48-hour soak window. GH Actions versions remain authoritative (issue filing, PR creation, notifications) until Phase 5 decommission per ADR-0134.

Woodpecker pipeline GH Actions counterpart Schedule (UTC) Secrets needed
.woodpecker/security-scan-nightly.yaml nightly-security-scan.yml — retired 2026-07-07 (moved to .github/workflows-retired/ by commit 4f6281c224, [PR #4064](https://github.com/raxx-app/TradeMasterAPI/pull/4064), merged 2026-07-07T22:51:27Z); WP has been the SOLE nightly scan since, not a dual-run pair. See alerting posture in "Cron inventory" row below. 7 8 * * * POSTMARK_OPS_ALERT_TOKEN, OPS_ALERT_EMAIL, OPS_ALERT_FROM, SLACK_WEBHOOK_URL (all optional/non-fatal — notify-on-failure step)
.woodpecker/alembic-version-check.yaml alembic-version-cron.yml (head job only) 0 6 * * * none
.woodpecker/maxmind-asn-refresh.yaml maxmind-asn-refresh.yml (download+validate only) 0 3 1 * * MAXMIND_LICENSE_KEY (optional)

Cron registration — COMPLETED 2026-07-04 via API (cron IDs 1, 2, 3):

All 3 crons registered and enabled via POST /api/repos/1/cron + PATCH /api/repos/1/cron/{id} (enabled=true). Repo ID = 1, pipeline definitions live in .woodpecker/ directory.

To re-register if needed (repo ID = 1, TOKEN = admin Woodpecker JWT):

curl -sf -X POST -H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"security-scan-nightly","schedule":"7 8 * * *","branch":"develop"}' \
  https://ci.moosequest.net/api/repos/1/cron
# PATCH to enable: curl -X PATCH ... -d '{"enabled":true}' .../cron/{id}

Repo activation — COMPLETED 2026-07-04 (repo ID = 1):

Repo raxx-app/TradeMasterAPI activated via POST /api/repos?forge_remote_id=1172174585. GitHub webhook ID: 649361254 (active, events: push, pull_request, pull_request_review, deployment). Enabling the repo installs the Woodpecker webhook on the GitHub repo and lets Woodpecker post commit statuses back to GitHub.

Trigger manual pilot runs (after repo enabled and PR merged to develop):

# Option A: Woodpecker UI — TradeMasterAPI → run pipeline → select each .woodpecker/ file
# Option B: Woodpecker API (replace TOKEN and ORG):
curl -X POST "https://ci.moosequest.net/api/repos/raxx-app/TradeMasterAPI/pipelines" \
  -H "Authorization: Bearer $WOODPECKER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"branch": "develop"}'

Scale up 1 agent before manual trigger (if desired_capacity is 0):

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 1 \
  --region us-east-2

Scale back when pilot builds finish:

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 0 \
  --region us-east-2

48-hour soak plan

Window: 2026-07-04 through 2026-07-06 (48 hours from pilot registration)

What is being watched:

Signal Green Red
Build success parity Woodpecker security-scan + alembic-check exit 0 within 5 min of GH Actions counterpart Either exits non-zero where GH is green
Agent scale-up/down ASG launches spot instance, agent connects, job runs, agent terminates after scale-in Agent stuck in "launching" or build never picked up
GitHub commit status Woodpecker posts green status check on HEAD commit after each successful run Status missing or red when build was green
Cost Agent hours during soak < $1 total (2 daily short jobs, 1 manual) Runaway agent billing (stuck job, desired stays >0)
Flake rate Zero unexplained failures in 2 daily runs per pilot Any failure not explained by a known GH Actions failure in the same window

Success criteria for go/no-go on Phase 3: 1. All 3 pilot pipelines run successfully on real agents (build numbers captured) 2. GitHub commit statuses appear on the triggered commit within 5 minutes of build completion 3. No agent stuck-running cost event (desired > 0 more than 60 min after a job completes) 4. security-scan and alembic-check run successfully at their scheduled times on at least one natural cron trigger during the soak window 5. Zero data integrity events (no writes to prod, no emails sent, no GH issues created by the Woodpecker versions)

Go/no-go decision: operator reviews soak results on 2026-07-06. Green = proceed to Phase 3 (PR-gate workflow migration). Red = investigate before advancing.

Phase 2 enhancements (deferred until after soak)

Secrets — Wave A bootstrap set

Created: 2026-07-04 UTC Card: #4037 (Wave A PRE-REQ) ADR: ADR-0135 §1 Scope: Org-level (org_id=2, org=raxx-app) — inherited by all repos in the org.

These 5 secrets are the ONLY credentials stored in Woodpecker. All application secrets continue to live in Infisical vault and are fetched at pipeline runtime by the wpc_load_vault_secrets.py init step.

Bootstrap secret list

WP secret name What it holds Vault source path
INFISICAL_CLIENT_ID Infisical Universal Auth machine identity client ID Injected from GHA env at Wave A PRE-REQ; rotate via vault path raxx-secrets /Raxx/CI
INFISICAL_CLIENT_SECRET Infisical Universal Auth machine identity client secret Same as above
INFISICAL_PROJECT_ID Infisical project ID (MooseQuest workspace) Same as above
CF_ACCESS_CLIENT_ID CF Access service token ID for vault.raxx.app Same as above
CF_ACCESS_CLIENT_SECRET CF Access service token secret for vault.raxx.app Same as above

Event restrictions (bootstrap secrets): push, pull_request, tag, cron, manual. These are intentionally broad — they enable vault access for ALL pipeline types.

Amendment (2026-08-29, #4544): Wave C originally created a separate, event-restricted (tag, manual only) WP org secret for Heroku prod (HEROKU_PROD_API_KEY) as described in the line this replaces. That secret was a static copy of the vault value and went stale after the 2026-07-21 rotation, breaking prod deploys for 5 weeks (see docs/ops/runbooks/heroku-api-key-rotation.md rotation log, 2026-08-29 row; RCA docs/incidents/2026-08-29-deploy-prod-stale-wp-secret.md). #4544 retired it — prod Heroku auth now uses the same live vault read as every other environment (HEROKU_API_KEY via VAULT_SECRETS="HEROKU_API_KEY"), gated by the pipeline when: block + in-step guards instead of by secret event-restriction. No prod-only static-copy secrets remain in this repo's WP org.

How to verify secrets are present

WP_TOKEN="<admin JWT>"
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/orgs/2/secrets" \
  | python3 -c "
import json, sys
secrets = json.load(sys.stdin)
names = {s['name'] for s in secrets}
required = {'INFISICAL_CLIENT_ID','INFISICAL_CLIENT_SECRET','INFISICAL_PROJECT_ID',
            'CF_ACCESS_CLIENT_ID','CF_ACCESS_CLIENT_SECRET'}
missing = required - names
if missing:
    print('MISSING:', missing); sys.exit(1)
print(f'All 5 bootstrap secrets present. Total org secrets: {len(secrets)}')
"

How to rotate a bootstrap secret

Zero-downtime rotation. No deploy or agent restart needed — WP injects secrets at pipeline start, not agent boot.

# 1. Update the value in Infisical vault (via vault UI or REST API)
# 2. Read the new value from vault (never paste inline):
NEW_VALUE="$(infisical secrets get <SECRET_NAME> --path /Raxx/CI --env prod --plain)"

# 3. PATCH the Woodpecker org secret (replaces value, preserves event restrictions):
WP_TOKEN="<admin JWT>"
curl -sf -X PATCH \
  -H "Authorization: Bearer $WP_TOKEN" \
  -H "User-Agent: sre-agent/1.0" \
  -H "Content-Type: application/json" \
  -d "{\"name\":\"<SECRET_NAME>\",\"value\":\"${NEW_VALUE}\"}" \
  "https://ci.moosequest.net/api/orgs/2/secrets/<SECRET_NAME>" \
  && echo "rotated"
unset NEW_VALUE

# 4. Verify the next pipeline run using the secret completes successfully.
# No pipeline restart needed — rotation takes effect at next pipeline start.

WOODPECKER_ENV_FILE gap — FIXED (Wave A A1)

Finding (WP build #6, 2026-07-04): vault auth from a real WP agent succeeded but WOODPECKER_ENV_FILE was not set in the step environment. The inter-step env propagation write was silently skipped; the smoke-verify step failed with exit 1.

Root cause: Woodpecker v3 runs each step in a separate Docker container. Unlike GHA, there is no automatic inter-step env propagation. WOODPECKER_ENV_FILE is not set in step context on v3.16.0 — the GHA analogy was incorrect.

Fix (ADR-0135 Wave A A1): wpc_load_vault_secrets.py now writes export KEY=… lines (base64-encoded) to a shell-sourceable file (/tmp/vault-env.sh by default), which the SAME step sources before running its real work:

- pip install --quiet requests
- python3 scripts/ci/wpc_load_vault_secrets.py
- . /tmp/vault-env.sh && rm -f /tmp/vault-env.sh
- # real work that uses the vault-fetched env vars

Live proof: WP pipeline .woodpecker/vault-secret-smoke.yaml, captured in PR for ADR-0135 A1. All 30 Wave A vault-dependent pipelines must use the same pattern. See docs/architecture/woodpecker-wpc-step-usage.md §1 for the canonical snippet.

Wave B (ADR-0135) — heavy CI + scale arbiter

Status: active soak (2026-07-05 +). Go/no-go gate: §Gate-B2 in ADR-0135.

Scale arbiter (Lambda ci-scale-arbiter)

What it does: An EventBridge rate(2 minutes) rule invokes Lambda ci-scale-arbiter. The Lambda calls GET /api/queue/info on the WP server (via internal ALB, Bearer token from SSM /ci/woodpecker/admin-api-token), then (current logic, infra/ci/lambda/scale_arbiter.py): - Drain first: every instance in Terminating:Wait is cordoned and released when idle (§Scale-in drain hook below). - pending > 0 AND workers == 0 AND desired == 0 → cold start: desired = max(1, floor, backlog target) - pending > 0 AND workers == 0 AND desired >= 1 with no InService or launching instance → wedge ladder (bump desired to 3, then instance refresh; #4039) - pending == 0 AND running == 0 AND desired > floor → SetDesiredCapacity <floor> (idle scale-down, floor = live ASG MinSize) - Burst on backlog (#4576): pending > 0 and nothing above acted → desired = max(floor, ceil(pending / 3)), capped at 6 and at the ASG max, applied only when it is higher than the current desired - Otherwise → no action (work in flight or already scaling)

Burst rule notes (#4576). floor is the live MinSize (2 in business hours, 1 overnight), so the schedules keep owning it. The rule only ever raises desired; it never lowers it, so it cannot fight the two scheduled actions and cannot terminate an agent. worker_count in /api/queue/info is the number of idle polling workers: a fully busy fleet reads workers == 0 with jobs pending, which is exactly when the burst rule is needed. Example: 2 busy agents and 9 pending jobs gives max(2, ceil(9/3)) = 3, one more agent. JOBS_PER_AGENT and BURST_CAP are constants in scale_arbiter.py; size_for_backlog() is unit-tested (infra/ci/lambda/test_scale_arbiter.py). After 22:00Z the evening action sets desired = 1 and the next tick re-raises it if jobs are still pending; that is the burst rule working, not a loop (the schedule fires once a day).

IaC: infra/ci/lambda.tf — Lambda + EventBridge rule + EventBridge target + permission. Logs: /aws/lambda/ci-scale-arbiter (90-day retention).

Verify scale arbiter is active:

# Check EventBridge rule is enabled:
aws events describe-rule --name ci-scale-arbiter-trigger --region us-east-2 \
  --query "State"
# Expected: "ENABLED"

# Check last 5 invocations:
aws logs filter-log-events \
  --log-group-name /aws/lambda/ci-scale-arbiter \
  --region us-east-2 \
  --start-time $(python3 -c "import time; print(int((time.time()-3600)*1000))") \
  | python3 -c "
import sys, json
for ev in json.load(sys.stdin).get('events', []):
    print(ev['timestamp'], ev['message'][:120])
"

Manual scale-up (emergency — bypasses arbiter):

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 1 \
  --region us-east-2

Manual scale-down (after incident resolved):

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --desired-capacity 0 \
  --region us-east-2

Scale-in drain hook (ci-agent-drain, #4576)

Why it exists. 2026-11-03 22:00Z: the ci-agents-scale-in-evening scheduled action (min 1, desired 1) terminated 4 busy agents. The pipelines they held were orphaned (Failure mode P). A terminated agent cannot post a terminal state and Woodpecker has no reaper.

What it does. aws_autoscaling_lifecycle_hook.agent_drain (infra/ci/agents.tf) parks every ASG-initiated termination in Terminating:Wait (scheduled scale-in, arbiter scale-down, OldestInstance selection, health replacement, instance refresh). heartbeat_timeout 2700 s (45 min), default_result CONTINUE. The scale arbiter releases the instance (no notification target, no hook role; it polls the ASG every 2 min): 1. Match the instance to its Woodpecker agent by name. New agents register as their EC2 instance ID (WOODPECKER_HOSTNAME in user_data_agent.sh.tpl). 2. Cordon it: PATCH /api/agents/<id> with no_schedule: true (the body also echoes name; Woodpecker copies both fields). The server stops handing it work immediately and the job in flight keeps running. 3. Call CompleteLifecycleAction (CONTINUE) once /api/queue/info shows no running task with that agent_id AND the agent's last_work (refreshed by Woodpecker on log lines, task extensions and Done, at most once a minute) is older than 3 minutes. The second signal guards against the running list under-reporting a task (the suspicion behind Failure mode J). It adds up to 3 minutes of wait after a job ends.

Why server-side and not a poller on the instance: SIGTERM to a Woodpecker v3 agent cancels the running workflow (received sigterm termination signal), and the agent has no flag to finish its job and stop taking more. A poller can only wait for a gap between jobs, which a backlog never produces; no_schedule closes that gap on the server. The arbiter also needs no new IAM on the agent role, and works for instances that booted before the change.

Agents that predate #4576 registered under their container hostname, so the arbiter cannot match them to an instance. Rule for such an instance: hold while any running task belongs to an agent whose name is not an i-... instance ID, or such an agent worked within the last 3 minutes (cannot prove the instance is idle); release when neither is true. Rows of long-gone agents linger in Woodpecker with an old last_work and do not hold anything. The 45-minute hook timeout is the upper bound either way. The old fleet turns over by itself within a day or two (OldestInstance goes first); to speed it up, start an instance refresh in a quiet window.

Limits to know. The drain only protects ASG-initiated terminations. A spot reclaim by EC2 or an instance crash still kills the job (Failure mode P, M, N). A draining instance costs up to 45 extra minutes of instance time per scale-in. A job longer than 45 min after the instance is chosen for scale-in is still cut off at the timeout; raise heartbeat_timeout (hook max per heartbeat is 7200 s) if jobs routinely run that long.

Skip a drain by hand (instance is wedged, or you need it gone now):

aws autoscaling complete-lifecycle-action --lifecycle-hook-name ci-agent-drain \
  --auto-scaling-group-name ci-woodpecker-agents-asg --instance-id <instance-id> \
  --lifecycle-action-result CONTINUE --region us-east-2

Verify the hook and watch a drain (read-only):

aws autoscaling describe-lifecycle-hooks --auto-scaling-group-name ci-woodpecker-agents-asg \
  --region us-east-2 --query 'LifecycleHooks[].[LifecycleHookName,LifecycleTransition,HeartbeatTimeout,DefaultResult]' --output text
# Expected: ci-agent-drain  autoscaling:EC2_INSTANCE_TERMINATING  2700  CONTINUE

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ci-woodpecker-agents-asg \
  --region us-east-2 --query 'AutoScalingGroups[0].Instances[].[InstanceId,LifecycleState]' --output text
# A draining instance shows Terminating:Wait; it should clear within one job length plus ~2 min.

aws logs filter-log-events --log-group-name /aws/lambda/ci-scale-arbiter --region us-east-2 \
  --filter-pattern DRAIN --start-time $(python3 -c "import time; print(int((time.time()-3600)*1000))") \
  --query 'events[].message' --output text
# Lines: "DRAIN i-...: agent busy|agent idle cordoned_now=... released=..."

Post-apply checks for #4576 (run after terraform apply of infra/ci)

Expected plan shape, from the 2026-11-04 read-only plan: 27 to add, 82 to change, 1 to destroy; the only destroy is aws_vpc_endpoint.logs. Stop if the apply would replace the ASG, the launch template, the server, RDS or the ALB.

  1. Confirm the SNS email subscription at ops@raxx.app. The topic ci-woodpecker-alerts had zero subscriptions live before this apply, so every CI alarm (queue-waiting, floor-missing, ALB 5xx, RDS storage, server CPU) notified nobody. After apply, open the "AWS Notification - Subscription Confirmation" mail in ops@raxx.app and click Confirm, then: bash aws sns list-subscriptions-by-topic --region us-east-2 \ --topic-arn "$(aws sns list-topics --region us-east-2 --query "Topics[?contains(TopicArn,'ci-woodpecker-alerts')].TopicArn|[0]" --output text)" \ --query 'Subscriptions[].[Protocol,Endpoint,SubscriptionArn]' --output text # Expected: email ops@raxx.app arn:aws:sns:... (NOT "PendingConfirmation")
  2. Hook present: the describe-lifecycle-hooks command above.
  3. Arbiter deployed and healthy. The live function was last modified 2026-07-05, so this apply ships the wedge ladder (#4039), queue metrics, the burst rule and the drain in one step. bash aws lambda get-function-configuration --function-name ci-scale-arbiter --region us-east-2 \ --query '[LastModified,Timeout,Environment.Variables.DRAIN_HOOK_NAME]' --output text # Expected: a 2026-11 timestamp, 60, ci-agent-drain aws lambda invoke --function-name ci-scale-arbiter --region us-east-2 --payload '{}' \ --cli-binary-format raw-in-base64-out /tmp/arbiter-result.json >/dev/null && cat /tmp/arbiter-result.json # Expected: "action", "drain" keys and no "error". An "error": "wp_api_401" means the admin token in SSM is stale (Failure mode G). aws cloudwatch get-metric-statistics --namespace CI/Woodpecker --metric-name PendingCount --region us-east-2 \ --start-time "$(date -u -v-15M +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '15 min ago' +%Y-%m-%dT%H:%M:%SZ)" \ --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --period 300 --statistics Maximum --query 'Datapoints[].Maximum' # Expected: datapoints (the namespace was empty before this apply)
  4. No fleet cycling. The launch template gets a new version (user-data adds WOODPECKER_HOSTNAME, and the AMI float picks up the current AL2023 image) but the ASG is only tag-updated: bash aws autoscaling describe-instance-refreshes --auto-scaling-group-name ci-woodpecker-agents-asg --region us-east-2 \ --query 'InstanceRefreshes[0].[Status,StartTime]' --output text # Expected: no refresh newer than the apply. Running agents keep their old launch template version until replaced.
  5. Agent naming (first instance launched after the apply): Woodpecker UI, Admin, Agents: the new agent's name is its i-... instance ID. If it is a 12-hex container id, WOODPECKER_HOSTNAME did not reach the agent (/opt/woodpecker-agent/.env on the instance) and the drain will hold that instance until the hook timeout.
  6. Logs endpoint gone, SSM trio kept, logs still flow: bash aws ec2 describe-vpc-endpoints --region us-east-2 --filters Name=vpc-endpoint-state,Values=available \ --query 'VpcEndpoints[].ServiceName' --output text # Expected: ssm, ssmmessages, ec2messages, s3 only. No ".logs". aws logs describe-log-streams --log-group-name /ci/woodpecker-agents --region us-east-2 --order-by LastEventTime \ --descending --max-items 1 --query 'logStreams[0].lastEventTimestamp' # Expected: within the last few minutes while any pipeline is running.
  7. Gitleaks upload without a key: the next push to release or main runs ci-boundary; gitleaks-report-upload goes green and bash aws s3 ls s3://raxx-ci-artifacts/gitleaks-reports/ --recursive --region us-east-2 # Expected: <pipeline-number>/gitleaks.json If the step logs Unable to locate credentials, the step container cannot reach IMDS (hop limit) or the role policy is not attached: aws iam list-attached-role-policies --role-name ci-woodpecker-agent-role must list ci-woodpecker-agent-gitleaks-reports-policy.
  8. First scheduled scale-in after the apply (22:00Z Mon-Fri): watch Failure mode T's commands; an instance that was mid-job must sit in Terminating:Wait until its job ends, then clear.

Failure mode T: instance stuck in Terminating:Wait, or a job still lost at scale-in

Symptom: an agent instance shows Terminating:Wait for much longer than a job (up to the 45-minute hook timeout), or a pipeline still dies with received sigterm termination signal right after a scale-in event.

Diagnose (read-only):

# 1. Which instances are held, and why (arbiter DRAIN lines)?
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ci-woodpecker-agents-asg \
  --region us-east-2 --query 'AutoScalingGroups[0].Instances[?starts_with(LifecycleState,`Terminating`)].[InstanceId,LifecycleState]' --output text
aws logs filter-log-events --log-group-name /aws/lambda/ci-scale-arbiter --region us-east-2 \
  --filter-pattern '?DRAIN ?ERROR' --start-time $(python3 -c "import time; print(int((time.time()-3600)*1000))") \
  --query 'events[].message' --output text
# 2. Why was the instance terminated (scheduled action, arbiter, health, refresh)?
aws autoscaling describe-scaling-activities --auto-scaling-group-name ci-woodpecker-agents-asg --region us-east-2 \
  --max-items 10 --query 'Activities[].[StartTime,Description,Cause]' --output text

Causes: - agent busy (expected): the job is still running. Wait, up to 45 min. - no agent matches; legacy-named agents are running work: the instance predates #4576 and some other pre-#4576 agent is busy, so the arbiter cannot prove this one is idle. Wait, or skip the drain by hand if you know the instance is idle. - cannot list agents or cordon failed: the Woodpecker API is unreachable or rejected the call (admin token expired or the server is down). Fix per Failure mode G, then the next tick (2 min) retries. The hook timeout still releases the instance. - The job is an orphan (running in Woodpecker, agent already dead; Failure modes M, N, P): the agent is "busy" forever from the server's view. Cancel the orphaned pipeline in the Woodpecker UI, or skip the drain by hand. - A job lost with no Terminating:Wait at all: the termination was not ASG-initiated (spot reclaim, host failure, manual ec2 terminate-instances). The hook does not cover those; see Failure mode P.

Fix: none needed for the first two (they resolve); otherwise the "Skip a drain by hand" command above, then follow up on the root cause.

Verification: the instance leaves Terminating:Wait and describe-auto-scaling-groups shows the expected DesiredCapacity; the pipeline that was running at scale-in finished with a real status, not received sigterm.

Failure mode J: Scale arbiter sets desired=0 while jobs are executing

Symptom: Pipelines fail with received sigterm termination signal part-way through execution. Agent CloudWatch logs show "grpc error: done(): code: Unknown desc = sql: no rows in result set" immediately after SIGTERM. Scale arbiter logs show SCALE DOWN: queue idle — setting desired=0 (or desired=min_size) at the same timestamp as the job failures.

Cause: WP's /api/queue/info running_count transitions to 0 before agents have finished executing all steps. This is a WP v3 queue-tracking edge case: once a task is dispatched to an agent and moves from "dispatched" to "executing", the server may clear it from the running queue, causing running_count=0 while workers are still actively processing. The arbiter's scale-down condition (pending==0 AND running==0) fires on the stale count.

Observed (2026-07-13 18:12:35 UTC): pending=0, running=0, workers=4 — four workers actively running pipeline 1067 (workflows 2962-2964), but arbiter scaled desired→0.

Fix: The arbiter now targets min_size on scale-down (desired=1 with the on-demand floor) instead of desired=0. This reduces blast radius: only spot burst instances are terminated, the on-demand floor survives and CI keeps running. Full protection requires the floor (see §On-demand floor).

Verify:

# Check scale-down target in arbiter logs — should say "desired X→1 (floor)", not "desired X→0"
aws logs filter-log-events \
  --log-group-name /aws/lambda/ci-scale-arbiter \
  --region us-east-2 \
  --filter-pattern "SCALE DOWN" \
  --start-time $(python3 -c "import time; print(int((time.time()-3600)*1000))") \
  | python3 -c "import json,sys; [print(e['message']) for e in json.load(sys.stdin).get('events',[])]"

Prevention: On-demand floor (min_size=1 overnight; min_size=2 in business hours, see "Failure mode: queue stalls with one agent in business hours") means even if the arbiter fires prematurely, the floor instance stays alive and absorbs the next queue entry without a cold-start delay.

Failure mode G: Scale arbiter not triggering scale-up

Symptom: Push event triggered a pipeline, but no agent picks it up within 5 minutes. ASG desired stays 0.

Diagnose:

# Check EventBridge rule is enabled:
aws events describe-rule --name ci-scale-arbiter-trigger --region us-east-2 --query "State"

# Check Lambda error rate in last 30 min:
aws logs filter-log-events \
  --log-group-name /aws/lambda/ci-scale-arbiter \
  --region us-east-2 \
  --start-time $(python3 -c "import time; print(int((time.time()-1800)*1000))") \
  --filter-pattern "ERROR"

# Manually invoke the arbiter to test it:
aws lambda invoke \
  --function-name ci-scale-arbiter \
  --region us-east-2 \
  --payload '{}' \
  /tmp/arbiter-result.json \
  --log-type Tail \
  --query "LogResult" --output text | base64 -d
cat /tmp/arbiter-result.json

Common causes: - SSM /ci/woodpecker/admin-api-token expired or wrong → Lambda logs WP API error 401 - WP server unhealthy (500 from /api/queue/info) → Lambda logs WP API error 500; fix WP server first - Lambda execution role lost SSM permission → Lambda logs permission error

Fix: If Lambda is healthy but ASG is stuck, scale up manually (see above). Then investigate the root cause from Lambda logs before trusting automatic scale-down.

Verification: aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ci-woodpecker-agents-asg --region us-east-2 --query "AutoScalingGroups[0].DesiredCapacity" returns 1.

Postgres heavy CI pipeline (.woodpecker/backend-tests-postgres.yaml)

Check context posted to GitHub: ci/woodpecker/backend-tests-postgres

What it runs: 1. path-guard — exit 0 if no backend_v2/** changes (informational, never blocks) 2. wait-for-postgres — polls pg_isready -h postgres -p 5432 until the postgres:15 service container is accepting connections 3. apply-migrations — roles script + (cd backend_v2 && python3 -m alembic upgrade head) against the postgres service. No failure: ignore, so a broken migration goes red. Needs DATABASE_URL in the STEP's own environment: (see Failure mode H cause 5) 4. backend-tests-postgres: - apt-get: postgresql-client (psql for create_ci_postgres_roles.sh) + postgresql (pg_ctl for pytest-postgresql) - pip: root requirements.txt + backend_v2/requirements.txt (pytest-postgresql≥5.0) + pytest/cov/timeout - PATH extended to /usr/lib/postgresql/15/bin for pg_ctl binary - vault secrets loaded via wpc_load_vault_secrets.py (VAULT_ALLOW_MISSING=true — vault failure is non-fatal) - create_ci_postgres_roles.sh against the postgres service container - (cd backend_v2 && alembic upgrade head) — subshell keeps CWD at repo root - pytest run (first pass: collect failures, --maxfail=5; second pass: coverage) - python -m coverage report --fail-under=30 — gate

Service container behavior: The postgres:15 service container is on the same Docker bridge network as build steps. Connect via hostname postgres (= service name), port 5432. No port-mapping needed. pg_isready -h postgres is the correct readiness check.

conftest.py postgres path: When DATABASE_URL starts with postgresql://, conftest.py does NOT connect to the service container directly. It imports pytest-postgresql and calls postgresql_proc to spawn an EPHEMERAL pg_ctl-managed postgres instance on a random port. The service container is used only for the explicit alembic upgrade head step. This design (from ADR-0070) provides test isolation — each test session gets a fresh, isolated DB. Consequence: postgresql server package + pytest-postgresql>=5.0 are both required in the pipeline, not just postgresql-client.

Wave B coexistence: The GHA backend-tests-postgres job (in .github/workflows/ci.yml) continues to run during the Wave B soak. Remove the GHA job only after 2-week green parity per ADR-0135 §Gate-B2.

Failure mode H: backend-tests-postgres step exits 1 with no coverage data

Symptom: backend-tests-postgres step exits with exit_code=1. Coverage report logs Coverage.py warning: No data was collected. or similar.

Causes (in order of probability): 1. cd persisting — a command runs cd backend_v2 && ... without a subshell, so subsequent pytest commands look in backend_v2/backend_v2/tests which does not exist. Pytest exits 4 (path not found), swallowed by || echo; coverage has no data. Fix: use (cd backend_v2 && ...). 2. pg_ctl not on PATH — pytest-postgresql cannot spawn ephemeral postgres. Fix: install the full postgresql server package (not just postgresql-client) and export /usr/lib/postgresql/15/bin to PATH. 3. pytest-postgresql not installed — conftest.py falls through to pytest.skip() on all DB fixtures; tests are collected but skipped; coverage may be below 30%. Fix: install -r backend_v2/requirements.txt in the pipeline step. 4. Alembic migration fails — a new migration has a syntax error or references a missing role. Check: (cd backend_v2 && python3 -m alembic upgrade head) logs. 5. apply-migrations red with NotImplementedError: No support for ALTER of constraints in SQLite dialect (migration 0037 create_foreign_key) and Context impl SQLiteImpl in the log — Alembic is running against SQLite, not the postgres service. backend_v2/alembic/env.py reads DATABASE_URL from the environment and silently falls back to sqlite+pysqlite:///./dev.db when it is empty; environment: is per-step in Woodpecker, so a DATABASE_URL on the pytest step does not reach apply-migrations. Fix: set DATABASE_URL: postgresql://raptor:raptor_test@postgres:5432/raptor_test under apply-migrations environment: (the step now also opens with test -n "$DATABASE_URL"). Healthy log shows Context impl PostgresqlImpl. Introduced by the step split in #4606 (2f6572439); seen on develop pushes f39fc2b8c and 6d26c8506 (pipeline 19970).

Diagnose:

# Check the pipeline step output — look for these signals:
# "No such file or directory: backend_v2/tests"  → cause 1 (cd persistence)
# "FileNotFoundError: [Errno 2]" or "pg_ctl: command not found" → cause 2
# "pytest-postgresql is not installed" → cause 3
# "alembic.exc.CommandError" or migration error → cause 4
# "Context impl SQLiteImpl" + "No support for ALTER of constraints in SQLite" → cause 5

Verification: Pipeline step exits 0; coverage report logs TOTAL ... XX% where XX ≥ 30.

Branch protection — Wave B dual-gate coexistence

Principle (ADR-0135 §4): During the Wave B soak, develop branch protection requires BOTH GHA and WP status checks. WP checks are added ALONGSIDE GHA — GHA checks are NOT removed until §Gate-B2 parity is confirmed (2-week green run).

Currently required checks on develop:

GHA (app_id: null):
  smoke_suite
  base_branch_lint
  stale-branch-guard
  pii-scan
  flag_promotion_check
  migration-gate

WP (added by Wave B, context prefix: ci/woodpecker/):
  ci/woodpecker/backend-tests-postgres
  ci/woodpecker/ci-pr
  ci/woodpecker/migration-collision-check

To add a new WP check to branch protection (operator or sre-agent):

# Read current protection config:
gh api repos/raxx-app/TradeMasterAPI/branches/develop/protection \
  --jq '.required_status_checks'

# Add a check (PATCH is additive; include all existing + new contexts):
gh api repos/raxx-app/TradeMasterAPI/branches/develop/protection \
  -X PUT \
  --input - <<'EOF'
{
  "required_status_checks": {
    "strict": false,
    "contexts": [
      "smoke_suite",
      "base_branch_lint",
      "stale-branch-guard",
      "pii-scan",
      "flag_promotion_check",
      "migration-gate",
      "ci/woodpecker/backend-tests-postgres",
      "ci/woodpecker/ci-pr",
      "ci/woodpecker/migration-collision-check"
    ]
  },
  "enforce_admins": false,
  "required_pull_request_reviews": null,
  "restrictions": null
}
EOF
# Verify after update:
gh api repos/raxx-app/TradeMasterAPI/branches/develop/protection \
  --jq '.required_status_checks.contexts'

To remove GHA checks (§Gate-B2 — only after 2-week green parity): Repeat the PUT above with only the WP contexts in the contexts array, omitting the GHA app_id: null entries. The GHA workflow files in .github/workflows/ remain for reference but their check contexts are no longer blocking.

WP status context name format (confirmed 2026-07-05): WP v3 posts GitHub status checks using ci/woodpecker/{event}/{pipeline-name} format, where {event} is push for push events and pr for pull_request events. Example: - ci/woodpecker/pr/backend-tests-postgres (PR trigger) - ci/woodpecker/push/backend-tests-postgres (push trigger) - ci/woodpecker/pr/ci-pr (ci-pr pipeline, PR trigger)

Note: some older YAML pipeline comments use ci/woodpecker/{pipeline-name} without the event prefix — this was a WP v2 / early v3 format. The {event} prefix has been confirmed in production on WP v3.16.0.

WP built-in variables in environment: blocks use the brace form ${CI_COMMIT_BRANCH}, ${CI_REPO}, ${CI_FORGE_TOKEN} — WP substitutes these at YAML parse time. Using bare $CI_COMMIT_BRANCH in an environment: value passes the literal dollar-sign string to the container, NOT the variable value. See Failure mode E-4 exception. Fixed in Wave B for ci-pr.yaml, console-degraded-auto-file.yaml, migration-collision-check.yaml.

ci/woodpecker/pr/ci-pr known failure (Wave A issue): The ci-pr.yaml pipeline reports failure on every PR since its introduction in Wave A batch 2 (PR #4051, 2026-07-05). Now that the ALB /api/* OIDC bypass rule is deployed and in Terraform state (priority 25: /api/*, /authorize, /login, /logout — imported 2026-07-08), the WP API is accessible from external clients without SSM. The ci-pr failure can now be diagnosed directly via the WP API. Do NOT add ci/woodpecker/pr/ci-pr to develop branch protection until this failure is diagnosed and resolved.

WP API access from external clients: The ALB now has a priority-25 OIDC-bypass rule for /api/*, /authorize, /login, and /logout (deployed 2026-07-08, managed as aws_lb_listener_rule.woodpecker_own_auth in infra/ci/alb.tf). Direct WP API calls from outside the VPC now work without authentication at the ALB layer — Woodpecker enforces its own token auth on all /api/* requests. Example:

curl -s -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  https://ci.moosequest.net/api/queue/info | python3 -m json.tool

Wave C — Console Ops Dispatch cutover (#4038, ADR-0135)

Status: implementation landed behind FLAG_OPS_DISPATCH_USE_WOODPECKER (default OFF). Flip at Wave C prod cutover per the operator decision below.

What changed: console/app/blueprints/ops.py (POST /ops/dispatch/<action_id>) can now dispatch via either the GitHub Actions workflow_dispatch API (unchanged, flag off) or the Woodpecker API (flag on). See console/app/services/ops_dispatch.py module docstring for the full design.

Safety constraint (mandatory — do not relax without a new incident review): the Woodpecker dispatch path NEVER issues a bare POST /api/repos/1/pipelines (event:manual) call. That endpoint fans out to every .woodpecker/*.yaml file whose when: block matches event: manual — roughly 65 pipeline files as of 2026-08, most unrelated to the action an operator clicked (see the "Warning — manual trigger fires all N workflows" note earlier in this doc, and .woodpecker/deploy-prod.yaml's TRIGGERS comment re: incident #110). Instead, the Console only ever calls the documented per-cron endpoint, POST /api/repos/1/cron/{cron_id}, and only for WorkflowSpec entries with an explicit wp_cron_id allowlist mapping (an operator-curated list, not free-form input).

Current allowlist (as of #4038):

Console action wp_cron_id Notes
nightly_security_scan 1 Maps to the security-scan-nightly pilot cron registered in "Phase 2 — cron pilot soak" above. Re-verify the ID via GET /api/repos/1/cron before flipping the flag on in prod — cron IDs are not guaranteed stable across a server rebuild.
deploy_heroku_prod (none) No safe scoped WP trigger exists yet — .woodpecker/deploy-prod.yaml is tag + break-glass-manual with an unscoped when: event: manual block. Hidden from the ops menu whenever the flag is on (hide-don't-gray, #1961); stays on the GHA dispatcher until a dedicated scoped trigger is built.

Operator decision (2026-07-05, issue #4038): Option C — a project-scoped Woodpecker API token if v3.16 supports it, stored at vault path /MooseQuest/console/WP_API_TOKEN and exposed to the Console app as the WP_API_TOKEN Heroku config var (same pattern as FREESCOUT_API_KEY / HEROKU_API_KEY — read via os.environ, never inlined). Falls back to a dedicated raxx-ci-bot WP user with minimal org access if v3.16 lacks project-scoped tokens.

Follow-up before prod flip: provision WP_API_TOKEN per the operator decision above, confirm cron ID 1 still maps to security-scan-nightly via GET /api/repos/1/cron, and register a scoped WP trigger for deploy_heroku_prod (or an equivalent staging-deploy action) before that action can be exposed over Woodpecker.

ECR layer cache — raxx-queue-vcpkg-cache (#4107, 2026-07-09)

Purpose: Stores Docker build-cache layers for the queue C++ vcpkg dependency tree. Consumed by W7 (deploy-queue) and W8 (queue-docker-smoke) pipelines. A warm cache reduces vcpkg build time from ~25-30 min (cold) to ~2-3 min.

Repository:

Name: raxx-queue-vcpkg-cache
Region: us-east-2
URI: 521228113048.dkr.ecr.us-east-2.amazonaws.com/raxx-queue-vcpkg-cache
ARN: arn:aws:ecr:us-east-2:521228113048:repository/raxx-queue-vcpkg-cache

Lifecycle policy: Untagged images are expired after 14 days. Tagged layers (e.g., vcpkg-cache-bookworm-<sha>) are retained until explicitly deleted.

IaC: infra/ci/ecr-cache.tf — ECR repo + lifecycle policy + agent IAM inline policy. Do NOT edit iam.tf for ECR-related changes — ecr-cache.tf owns the agent ECR policy.

Agent IAM policy ci-woodpecker-agent-ecr (inline on ci-woodpecker-agent-role): - ecr:GetAuthorizationToken on * (IAM limitation — cannot scope to one repo) - Pull actions (BatchCheckLayerAvailability, GetDownloadUrlForLayer, BatchGetImage) on repo ARN only - Push actions (PutImage, InitiateLayerUpload, UploadLayerPart, CompleteLayerUpload) on repo ARN only

Authenticate agent to ECR (used in W7/W8 pipeline steps):

# WP pipeline step YAML (agent fetches token via instance role — no credential needed):
aws ecr get-login-password --region us-east-2 \
  | docker login --username AWS --password-stdin \
    521228113048.dkr.ecr.us-east-2.amazonaws.com

Bootstrap cache (one-time before first W7 run — avoids 25-30 min cold build): Run from a machine or GHA runner with ECR push rights to raxx-queue-vcpkg-cache:

# Build the vcpkg base layer and push to ECR:
cd queue/
docker buildx build \
  --cache-to type=registry,ref=521228113048.dkr.ecr.us-east-2.amazonaws.com/raxx-queue-vcpkg-cache:vcpkg-cache-bookworm-bootstrap,mode=max \
  --push \
  -f Dockerfile.vcpkg \
  .
# Verify the tag exists:
aws ecr describe-images \
  --repository-name raxx-queue-vcpkg-cache \
  --region us-east-2 \
  --query "imageDetails[*].{tags:imageTags,pushed:imagePushedAt,size:imageSizeInBytes}" \
  --output table

Failure mode — ECR push/pull fails from agent with "no credentials": - Confirm the ci-woodpecker-agent-ecr inline policy is attached: aws iam get-role-policy --role-name ci-woodpecker-agent-role --policy-name ci-woodpecker-agent-ecr - Confirm IMDSv2 hop limit is 2 on the agent LT (allows containers to reach IMDS for STS): aws ec2 describe-launch-template-versions --launch-template-name ci-woodpecker-agent-lt --query "LaunchTemplateVersions[-1].LaunchTemplateData.MetadataOptions" - The agent must call aws ecr get-login-password BEFORE docker login in the same step — the token expires after 12h

Failure mode I: Stale Go HTTP connection pool — context deadline exceeded on GitHub API calls

Symptom: A pipeline triggered by a tag (or push) event is created immediately with status=error and zero steps executed. Server logs show context deadline exceeded on a pipeline-fetch path. The WP server is otherwise healthy (/healthz 200, queue not paused, agents connected). Typically surfaces after 4–7 days of continuous server uptime.

Cause: Go's net/http default transport pools idle keep-alive connections. After multi-day uptime, connections to api.github.com held in the pool have been silently closed server-side (TCP RST or keepalive timeout), but the WP process has not detected the close. On the next GitHub API call (fetching .woodpecker/*.yaml), WP reuses a dead connection; the TCP write fails; the request-scoped deadline expires before a successful retry completes.

Fix:

# 1. Confirm the symptom in WP server logs:
aws ssm send-command \
  --region us-east-2 \
  --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["sudo docker compose -f /opt/woodpecker/docker-compose.yml logs --tail=30 woodpecker-server 2>&1 | grep -E \"context deadline|unexpected status\" | tail -10"]}' \
  --output json | python3 -c "import json,sys; print(json.load(sys.stdin)['Command']['CommandId'])"
# Poll get-command-invocation until Status=Success, then read StandardOutputContent.

# 2. Restart to clear the pool (~30s downtime; ensure WP queue is EMPTY first):
aws ssm send-command \
  --region us-east-2 \
  --instance-ids i-082ee835595d90ae0 \
  --document-name AWS-RunShellScript \
  --parameters '{"commands":["sudo systemctl restart woodpecker && echo RESTART_OK"]}' \
  --output json | python3 -c "import json,sys; print(json.load(sys.stdin)['Command']['CommandId'])"
# Poll; expect Status=Success + StandardOutputContent=RESTART_OK

# 3. Confirm healthy after ~30s:
curl -sS -o /dev/null -w "%{http_code}\n" -H "User-Agent: raxx-agent/1.0" https://ci.moosequest.net/api/version
# Expected: 200

Re-fire the failed pipeline after restart:

Option A — Redeliver the tag webhook via GitHub API (requires admin:repo_hook scope — preferred, preserves tag event context):

# WP webhook ID: 649361254
gh api repos/raxx-app/TradeMasterAPI/hooks/649361254/deliveries \
  --jq '.[] | select(.event == "create") | {id,delivered_at}' | head -4
gh api -X POST repos/raxx-app/TradeMasterAPI/hooks/649361254/deliveries/<delivery_id>/attempts

Option B — Break-glass manual trigger (use when admin:repo_hook unavailable):

WP_TOKEN="<admin WP JWT from ci.moosequest.net>"
# Note: ALB may return 504 even if WP accepted the request (60s idle timeout).
# Always confirm creation via GET after a 504.
curl -s -X POST \
  -H "Authorization: Bearer $WP_TOKEN" \
  -H "User-Agent: raxx-agent/1.0" \
  -H "Content-Type: application/json" \
  -d '{"branch":"main","variables":{"DEPLOY_PROD_CONFIRM":"yes"}}' \
  "https://ci.moosequest.net/api/repos/1/pipelines"
curl -s -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: raxx-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/pipelines?page=1&perPage=3" \
  | python3 -c "import json,sys; [print(f'#{p[\"number\"]} {p[\"event\"]} {p[\"branch\"]} {p[\"status\"]}') for p in json.load(sys.stdin)]"

Warning — manual trigger fires all 59 workflows: A manual trigger on main dispatches every .woodpecker/*.yaml file that matches event: manual, not just deploy-prod. Most are harmless (cron workflows that error on non-scheduled contexts), but deploy-staging, cut-release-candidate, and deploy-velvet may also run. Prefer Option A (webhook redeliver) for a clean tag-only trigger. As of #4474, deploy-velvet's manual-triggered prod path now fails closed rather than silently deploying: it requires DEPLOY_PROD_CONFIRM=deploy-prod-now (not deploy-prod.yaml's yes — different pipeline, different confirm value) on branch main, so the Option B DEPLOY_PROD_CONFIRM=yes example above will make deploy-velvet fail its own guard (harmless, no Heroku push) rather than deploy Velvet to prod as an unintended side effect.

Never use the bare manual trigger (POST /api/repos/1/pipelines {"branch":"<any-branch>"}) to pre-merge-test a feature branch's cron fixes. It fans out to all ~55 event: manual workflows in the repo, not just the one(s) under test — including reminders-second-factor-nightly (real customer 2FA emails, no dry-run path, explicitly operator-gated in #4536), deploy-staging, heroku-admin, freescout-apply, and terraform-email-delivery-stack. The only safe way to verify a cron fix pre-enable is the scoped per-cron endpoint (POST /api/repos/1/cron/{cron_id}, Failure Mode K below) — and that endpoint always runs against the cron's registered branch (normally develop), so a code-level fix to a .woodpecker/*.yaml file cannot be verified this way until it is merged. Sequence: merge first, then scoped-trigger, then read the real step log, then enabled:true.

Real incident, not just a near-miss (2026-08-23, verifying #4531-#4533): a first curl -sf ... -X POST .../pipelines > file.json call against a feature branch appeared to fail silently (empty output, empty file on a later check) and was assumed to have not fired. It had: the POST had already succeeded server-side and created a real pipeline (billing-retention-cron ran for real — POST /api/internal/jobs/billing-retention?dry_run=false, HTTP 200, eligible_count:0 so zero rows actually touched; deploy-staging's deploy-raptor step ran a real heroku config:set RELEASE_VERSION=... before being caught, creating a real Heroku release — cosmetic only, no code push landed, and immediately superseded by an unrelated legitimate tag-triggered deploy already in flight; reminders-second-factor-nightly failed at heroku: not found before reaching the actual send command — zero customer emails). A second, deliberate trigger attempt (this time correctly observed) is what actually got caught and cancelled before executing (see above). Lesson: an empty/failed-looking response from a WP trigger POST is not proof nothing fired — always follow any trigger POST with a GET /api/repos/1/pipelines?page=1 (or the returned pipeline id/number if you have it) to positively confirm pipeline state before concluding "nothing happened." Full evidence trail: docs/incidents/2026-08-23-cron-fix-verification-manual-trigger-fanout.md.

Verification: deploy-prod workflow in the new pipeline reaches state=success for all 9 steps. Confirm api.raxx.app/health → 200.

Long-term mitigations: 1. Weekly WP server restart cron (Sunday 04:00 UTC) via AWS EventBridge + SSM Run Command 2. Tune Go HTTP transport via WP env var WOODPECKER_FORGE_TIMEOUT (or similar) if supported in v3.x 3. Cache git subtree split to avoid 17+ min cold splits on ephemeral agents (first run = 1073s on 6-month-history repo)

RCA: docs/incidents/2026-07-09-wp-stale-http-pool-prod-deploy.md

Failure mode O: cross-step state via /tmp/<file> silently doesn't work across different-image steps

Symptom: A step reads a value that an earlier step in the same workflow supposedly wrote to /tmp/<name>, and gets an empty/missing result even though the earlier step reported success and logged the write. Concretely: ERROR: /tmp/ecr-password is missing or empty. ecr-auth step may have failed. immediately after a preceding ecr-auth step logged ECR auth token written to /tmp/ecr-password.

Cause: Each WP step runs in a fresh container. Only the git-checked-out workspace is a persistent volume shared across steps in the same workflow — /tmp is per-container and does NOT persist across a step boundary when the two steps use different image: values (confirmed empirically, not documented anywhere in WP's own docs as of v3.16). This pattern was originally adopted as a workaround for WOODPECKER_ENV_FILE being unavailable in this WP version (see Failure mode E-4 for the sibling ${VAR}-at-parse-time gotcha) — the workaround solves the "no env-file" problem but silently reintroduces a different one.

Trap for the unwary: this bug can look like it "works" if the value being passed has a safe-default fallback (e.g. SKIP=$(cat /tmp/queue-smoke-skip 2>/dev/null || echo "false")) — a missing file and a real false read produce the identical downstream behavior, so the broken handoff is invisible until a value with NO safe default (like a password) needs to cross the same step boundary.

Fix: Do not pass state via /tmp/<file> across steps with different image: values. Options, in order of preference: 1. Do the work in a single step (same container) — e.g. apk add --no-cache aws-cli inside the build step itself, then auth + build in one script block. This is the proven pattern in .woodpecker/deploy-queue.yaml's build-container step. 2. If genuinely different images are required across steps, pass small values via environment: + from_secret: (WP-managed secrets), not disk files. 3. If two steps in a row use the same image:, workspace-relative paths (not /tmp) are safe to use for handoff, since the workspace volume is genuinely shared.

Verification: Re-run the pipeline; the consuming step's read of the passed value succeeds with the real (non-empty, non-default-fallback) content, not just a plausible-looking default.

Real incident: .woodpecker/queue-docker-smoke.yaml's ecr-auth → docker-build-smoke handoff, found live on pipeline PR #4614 while verifying the fix for a separate bug ($CI_BUILD_NUMBER not being a real WP built-in — should be $CI_PIPELINE_NUMBER) in the same file. Full RCA: docs/incidents/2026-09-28-queue-docker-smoke-wp-pipeline-broken.md.

Failure mode Q: static-site deploy pipeline warns [cf-cache-purge-failed] (HTTP 401) after a successful Pages deploy

Symptom: deploy-customer-docs, deploy-status-page or deploy-internal-docs logs WARNING [cf-cache-purge-failed] edge-cache purge for <host> did not complete (HTTP 401) right after Deployment complete!. Before the fix this was curl: (22) ... returned error: 401 and a red pipeline (main push 20027, v1.21.0; same on 6998, 2026-08-22).

Cause: CF_CACHE_PURGE_TOKEN (vault /MooseQuest/cloudflare/, scope Zone:Cache Purge:Edit on raxx.app) is present in the vault but Cloudflare no longer accepts it. Its recorded expiry was 2026-08-17 (docs/ops/health-sweep-2026-05-23.md), and the first 401 in the Woodpecker history is 2026-08-22. A vault secret being present says nothing about whether it is still valid.

Behavior: the purge is best-effort (scripts/ci/wpc_purge_cf_cache.sh). The purge runs after wrangler pages deploy, so the deploy has already landed; a failed purge only leaves old edge copies until their own TTL. The helper exits 0 and prints the HTTP status, the Cloudflare error body and, for 401/403, the secret to rotate. A token that is MISSING from the vault still fails the step at the vault load (VAULT_ALLOW_MISSING="false").

Fix: rotate CF_CACHE_PURGE_TOKEN (operator or the token-rotation service; agents do not mint credentials). Then re-run a deploy and confirm the log line CF edge cache purged for <host>. Grep a deploy log for cf-cache-purge-failed to find every host still unpurged.

Not covered: deploy-support. It still carries the inline curl -f purge, and it is deliberately not on the helper yet. deploy-support lists its own pipeline file in path.include, so editing it would trigger a run on the next main push, and that run fails at vite build (Could not resolve "./lib/auth.js" from "src/App.jsx", pipeline 6998 step build-and-deploy). frontend/support/src/lib/auth.js is not in git: the root .gitignore rule lib/ (line 16) ignores it. Fixing that is a product-source change (negate the ignore for frontend/support/src/lib/ and commit the file); once vite build passes, replace the inline purge with sh scripts/ci/wpc_purge_cf_cache.sh support.raxx.app in the same PR and add deploy-support.yaml back to PURGE_HOSTS in scripts/ci/tests/test_static_site_deploy_main_push_fixes.py. In the same PR drop its path filter on the main push and remove its EXEMPT entry in scripts/ci/check_main_push_path_filters.py (failure mode S).

Failure mode R: deploy-getraxx smoke fails with HTTP 403 against getraxx.com

Symptom: smoke step: Navigating to https://getraxx.com/waitlist, HTTP status: 403, SMOKE FAIL. curl -I https://getraxx.com/waitlist shows cf-mitigated: challenge. Present on main push 20027 (v1.21.0) and on 6998 and 5136 before it.

Cause: Bot Fight Mode is ON for the getraxx.com zone by design (public marketing surface, no headless or service-token clients; see the BFM-per-zone note in docs/incidents/2026-06-19-bfm-restored.md). The CI agent's headless Chromium comes from an AWS address range and always receives the managed challenge. The smoke can never pass against the zoned domain, and the challenge page has no email input, which used to surface as the misleading "input not found".

Fix (applied): smoke targets https://getraxx.pages.dev (env SMOKE_BASE_URL), which serves the artifact wrangler pages deploy just published and is not behind the zone's bot rules. scripts/ci/getraxx_waitlist_smoke.js fails by name on any non-200. Do NOT turn Bot Fight Mode off on the getraxx zone to make CI pass.

Verification: SMOKE_BASE_URL=https://getraxx.pages.dev node scripts/ci/getraxx_waitlist_smoke.js (with playwright installed) prints Smoke passed.; the same run with SMOKE_BASE_URL=https://getraxx.com fails with HTTP 403.

Failure mode S: a deploy workflow silently does not run after a large promote (path filter vs the 300-file cap)

Symptom: a promote (or release cut) merges, the main or release push pipeline is green, but a deploy workflow that should have fired is absent from the pipeline's workflow list, and the site still serves the old build. Promote #4915 (v1.22.0): main push 20894 listed ci-boundary, ci-console, deploy-customer-docs, deploy-flag-docs, deploy-getraxx, deploy-internal-docs, deploy-mockups, deploy-status-page, but not deploy-antlers-next-prod; /tools and /start returned 404 on raxx.app. Release push 20429 (821 files changed, 325 under frontend/raxx-next/, 300 recorded, none of them under it) had the same shape for deploy-antlers-next-staging; the next release push, 20490, ran it and caught staging up.

How to confirm: read the pipeline with the admin token (never print it): curl -H "Authorization: Bearer $T" https://ci.moosequest.net/api/repos/1/pipelines/<N> | jq '.changed_files | length'. A value of exactly 300 on a diff known to be larger is this failure mode. Then jq '[.changed_files[] | select(startswith("frontend/raxx-next/"))] | length' shows how many of the files the workflow's filter was looking for made it into the list.

Cause: a when.path filter on a push event matches against changed_files. For GitHub, Woodpecker v3.16.0 builds that list in loadChangedFilesFromCommits (server/forge/github/github.go) by paging CompareCommits(prev, curr) and keeping the files it returns. GitHub's compare endpoint returns at most 300 files for the whole comparison ("it includes up to 300 changed files for the entire comparison", GitHub REST docs, Compare two commits), and Woodpecker does not record that the list was cut short. path.on_empty cannot help: Path.Match (pipeline/frontend/yaml/constraint/path.go) consults on_empty only when len(files) == 0, and a capped list is not empty. The docs for path (docs/usage/workflow-syntax, v3.16.0) say nothing about a file limit. path.ignore_message bypasses the filter only if the commit message carries a marker, which is not something to rely on for a merge commit.

Fix (applied): the deploy workflows that run on a push to main or release carry no path filter, so they run on every push to those branches: deploy-antlers-next-prod, deploy-antlers-next-staging, deploy-console-shim, deploy-customer-docs, deploy-getraxx, deploy-internal-docs, deploy-mockups, deploy-queue, deploy-status-page, deploy-status-worker, deploy-velvet. Each deploy is idempotent on an unchanged tree. pull_request path filters are unchanged (a PR file list is paged to completion). scripts/ci/check_main_push_path_filters.py, run by lint-ci-config on every PR, fails if a path filter on a main or release push comes back. A deliberate exception goes in EXEMPT in that script with a reason; the check also fails on an exemption that is no longer needed. Current exemptions: deploy-support (its vite build fails because frontend/support/src/lib/auth.js is not in git, failure mode Q; remove the exemption and the filter in the PR that commits it), terraform-email-delivery-stack (terraform validate fails on the module and the workflow auto-applies; the nightly plan-only nightly-tf-email-drift cron is the path-independent check), and four validation-only workflows that deploy nothing (deploy-flag-docs, ci-console, infra-ci-validate, terraform-validate).

Recovery for a deploy that was skipped before this fix: re-run the workflow on the branch head. deploy-antlers-next-prod takes event: manual with the pipeline variable DEPLOY_ANTLERS_PROD=yes; deploy-antlers-next-staging takes event: manual with no variable. The next promote also triggers every unfiltered workflow by itself.

Verification: after the next main or release push, the pipeline's workflow list contains every deploy workflow in the list above (curl ... /pipelines/<N> | jq '[.workflows[].name]'), and curl -sI https://raxx.app/tools and https://raxx.app/start return 200 once deploy-antlers-next-prod has run.

Escalation

Wake the operator when: - RDS data integrity suspected (unclean shutdown, storage corruption) - GitHub OAuth App or Google OIDC credentials compromised - ACM certificate cannot be renewed (Dyn CNAME missing after 60-day renewal attempt) — monitored since #4547 via DynRecordMissing/ACMCertificateExpiringSoon, see §Manual DNS inventory above; this condition should now page well before the 60-day window closes rather than being discovered here - Woodpecker server repeatedly crashes and restart loop does not recover within 15 minutes - WAF blocking appears to be a false positive on production pipeline webhooks

Monitoring — external watchdog (2026-07-07)

Permanent GHA stay (ADR-0136 Decision 1; #4109): .github/workflows/woodpecker-healthcheck.yml is classified GENUINELY-STAYS-ON-GHA. It is not a pending migration item. Do not port it to Woodpecker and do not retire it. See migration table §2b for the formal record.

Woodpecker health is monitored by scripts/ops/woodpecker_healthcheck.py, run every 15 min by .github/workflows/woodpecker-healthcheck.yml — intentionally on GitHub Actions, because a monitor hosted on Woodpecker can't fire when Woodpecker is down.

Checks: (1) server reachable (/api/version 200), (2) TLS cert expiry (< 21d fails), (3) pipeline health (stuck > 90 min, or ≥4 of last 5 finished failed) — tier 3 only runs if a WOODPECKER_API_TOKEN repo secret is set.

Alerting is edge-triggered via a woodpecker-health GitHub issue anchor: one email to kris@moosequest.net on healthy→unhealthy, one on recovery — no per-run spam during a sustained outage. Email goes via Postmark (POSTMARK_OPS_ALERT_TOKEN + OPS_ALERT_FROM secrets).

Verify the alert path any time: Actions → "Woodpecker healthcheck" → Run workflow → tick test_email (sends one confirmation email, then exits). Or locally: POSTMARK_SERVER_TOKEN=... python3 scripts/ops/woodpecker_healthcheck.py --test-email.

To enable the richer pipeline checks, add a WOODPECKER_API_TOKEN repo secret (a WP user API token from the ci.moosequest.net UI).

Failure mode L: woodpecker-restart.service crash loop — 502 since 08:00 UTC

Symptom: ALB returns 502 for all requests. systemctl status woodpecker shows the service cycling through "Starting → Finished → Stopping → Stopped → Starting" every ~2 seconds. journalctl -u woodpecker -u woodpecker-restart.service shows this alternating pattern repeating continuously:

systemd: Finished woodpecker.service
systemd: Starting woodpecker-restart.service...
systemd: woodpecker-restart.service: Main process exited, code=killed, status=15/TERM
systemd: woodpecker-restart.service: Failed with result 'signal'.
systemd: Stopping woodpecker.service...
systemd: woodpecker.service: Deactivated successfully.
systemd: Starting woodpecker.service...

Cause: woodpecker-restart.service has Requires=woodpecker.service. When the timer fires and woodpecker-restart.service runs systemctl restart woodpecker, that command internally stops woodpecker.service. The moment woodpecker.service goes inactive, the Requires= dependency causes systemd to SIGTERM woodpecker-restart.service before the restart process can complete. Systemd then restores woodpecker.service (queued start from the partial restart), and also attempts to re-activate woodpecker-restart.service (systemd heals collaterally stopped units). The cycle repeats with no exit condition: ~2 seconds per loop, indefinitely. The WP container never stays up long enough for the ALB target group health check to pass. The loop starts the moment the timer fires (08:00 UTC daily) and the outage persists until the loop is manually broken.

Fix (break the loop + patch the unit file):

# 1. Break the loop — stop both units
sudo systemctl stop woodpecker-restart.timer woodpecker-restart.service woodpecker

# 2. Verify stopped (should say "failed" or "inactive", never "active")
sudo systemctl is-active woodpecker && echo STILL_UP || echo STOPPED_OK
sudo docker ps | grep woodpecker && echo CONTAINER_STILL_UP || echo DOWN_OK

# 3. Remove the faulty Requires= line
sudo sed -i '/^Requires=woodpecker.service/d' /etc/systemd/system/woodpecker-restart.service

# 4. Confirm the patch (should show After= but NO Requires=)
sudo cat /etc/systemd/system/woodpecker-restart.service

# 5. Reload daemon and restart cleanly
sudo systemctl daemon-reload
sudo systemctl start woodpecker

# 6. Confirm container is up and healthz returns 200
sudo docker ps | grep woodpecker
curl -sf http://localhost:8000/healthz && echo OK

# 7. Re-enable the timer
sudo systemctl start woodpecker-restart.timer
sudo systemctl status woodpecker-restart.timer --no-pager

# 8. Verify no restart.service start fires immediately
sleep 5
sudo journalctl -u woodpecker-restart.service --since "1 minute ago" --no-pager
# Expected: no new "Starting woodpecker-restart.service" entries

Verification:

# External endpoint must return 200
curl -s -o /dev/null -w "%{http_code}" https://ci.moosequest.net/api/version
# Expected: 200

# Service must be stable (active, not cycling)
sudo systemctl status woodpecker --no-pager
# Expected: Active: active (exited) — no repeated Starting/Stopping

Permanent fix — IaC: The woodpecker-restart.service unit must be created WITHOUT Requires=woodpecker.service. After=woodpecker.service alone is sufficient. This is now codified in infra/ci/templates/user_data_server.sh.tpl (section 9) so a server rebuild will not re-introduce the bug.

Correct woodpecker-restart.service:

[Unit]
Description=Woodpecker CI daily restart -- clears Go HTTP connection-pool staleness (RCA 2026-07-09)
After=woodpecker.service

[Service]
Type=oneshot
ExecStart=/bin/systemctl restart woodpecker
StandardOutput=journal
StandardError=journal

Why Requires= is wrong here: systemctl restart woodpecker = stop + start. The stop phase puts woodpecker.service into inactive state. Requires=woodpecker.service propagates that stop to woodpecker-restart.service (killing the restart process mid-execution). After=woodpecker.service gives the same boot-ordering guarantee without the stop-propagation side effect.

RCA: docs/incidents/2026-07-11-woodpecker-restart-loop.md Incident log: 2026-07-11 — loop started 08:00 UTC, detected 13:36 UTC, resolved 13:47 UTC

Failure mode M: Agent process dead / step stuck running while the EC2 instance itself stays InService (OOM)

Symptom: A pipeline step shows as "running" in the Woodpecker UI indefinitely — no log output progress, no completion, no failure — well past the step's normal duration. aws ec2 describe-instance-status and the ASG console both report the agent instance as InService / Ok / Ok (EC2 status checks pass). The instance looks completely healthy from the AWS-infrastructure layer. This is what makes this failure mode distinct from Failure mode C (agents not connecting to server) and from a wedged spot fleet (#4039): the instance is fine, but the agent process on it is not.

Cause: The woodpecker-agent container's process (or a build container it spawned) was OOM-killed by the Linux kernel. WOODPECKER_MAX_WORKFLOWS allowed more than one workflow to run concurrently on a single agent instance; when two memory-heavy builds (observed: a C++ compile and a cargo build, both large peak-RSS jobs) land on the same agent at the same time, their combined memory demand can exceed the instance's physical RAM. The kernel OOM-killer takes out the agent (or its running build) to protect the box, but: - the underlying EC2 instance keeps passing status checks (the OS itself is fine — only one process died), - docker-compose.yml has restart: unless-stopped for the agent, so the container typically restarts on its own within seconds to a few minutes — but the step it was running has no supervisor: Woodpecker's server does not get a clean failure signal for a step whose executing agent vanished mid-run, so the step can be left showing "running" against a agent gRPC connection that no longer corresponds to a live execution. - the net effect: the pipeline is silently stuck, and nothing in EC2/ASG health checks reflects it.

Detection:

# 1. Confirm the instance itself is healthy (rules out Failure mode C / wedged fleet)
aws ec2 describe-instance-status --instance-ids <instance-id> --region us-east-2 \
  --query 'InstanceStatuses[0].[InstanceState.Name,InstanceStatus.Status,SystemStatus.Status]'
# Expected if this is Failure mode M: ["running", "ok", "ok"] — instance is fine.

# 2. Check dmesg / kernel log on the instance via SSM for an OOM kill
aws ssm start-session --target <instance-id> --region us-east-2
sudo dmesg -T | grep -i "killed process\|out of memory\|oom-kill"
# A line like "Out of memory: Killed process <pid> (woodpecker-agent)" or
# naming a build-step process (cc1plus, rustc, cargo) confirms this failure mode.

# 3. Confirm the agent container's actual uptime is short relative to the
#    stuck step's apparent start time (i.e. it already restarted once)
sudo docker ps --format '{{.Names}}\t{{.Status}}' | grep woodpecker-agent
# "Up 2 minutes" while the UI step claims to have been running for 45+ minutes
# is the signature of this failure mode: the agent came back, but Woodpecker's
# server-side step state didn't get reconciled.

# 4. Cross-check current WOODPECKER_MAX_WORKFLOWS and running container count
#    on the box to confirm co-scheduling was possible at the time
grep MAX_WORKFLOWS /opt/woodpecker-agent/.env
sudo docker ps | grep -v woodpecker-agent  # build containers spawned by the agent

Restore:

# 1. On the Woodpecker server: cancel the stuck pipeline/step from the UI or API
#    so the queue is not blocked waiting on a step that will never complete.
#    UI: Repo -> Pipeline -> "Cancel". API equivalent:
curl -sf -X POST "https://ci.moosequest.net/api/repos/<repo-id>/pipelines/<pipeline-number>/cancel" \
  -H "Authorization: Bearer $WOODPECKER_ADMIN_TOKEN"

# 2. Confirm the agent container on the affected instance is actually back up
#    and re-registered with the server (it usually already is, from
#    restart: unless-stopped — this step is a verification, not a fix):
sudo docker ps | grep woodpecker-agent
sudo docker compose -f /opt/woodpecker-agent/docker-compose.yml logs --tail=50 woodpecker-agent
# Expect a recent "agent registered" / gRPC-connected log line.

# 3. If the agent did NOT come back on its own (rare — restart: unless-stopped
#    should handle it), restart it explicitly via SSM Run Command (no SSH):
aws ssm send-command \
  --instance-ids <instance-id> \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["cd /opt/woodpecker-agent && docker compose up -d woodpecker-agent"]' \
  --region us-east-2

# 4. Re-fire the cancelled pipeline with a NEW tag on the same commit, rather
#    than a bare UI "Restart" or a manual /api/repos/.../pipelines POST — the
#    latter re-triggers the full ~65-workflow fan-out for the repo and can
#    overload an already-degraded fleet. A new tag on the existing commit
#    scopes the re-run to just the tag-triggered pipeline(s). Do this only
#    once at least one agent shows connected workers > 0 in Grafana /
#    woodpecker_workers metric.

Verification: The cancelled pipeline shows a terminal state (not stuck "running") in the UI; woodpecker_workers > 0 in Prometheus; the re-triggered pipeline completes normally; sudo dmesg -T | grep -i oom shows no new entries after the restore.

Prevention (IaC, #4545 fold-in): WOODPECKER_MAX_WORKFLOWS reduced from 4 to 1 in infra/ci/templates/user_data_agent.sh.tpl — this removes the co-scheduling condition that lets two heavy builds land on the same agent instance at once. The ASG absorbs concurrent pipeline load by adding agent instances, not by stacking workflows on one instance. If throughput demands raising this again, do so only alongside a real per-workflow memory ceiling (docker-compose mem_limit on the build containers, or a larger/dedicated instance class sized for the expected concurrent peak-RSS) — not as a bare config bump.

RCA: Live incident, 2026-08-29 — agent OOM twice during co-scheduled heavy C++/cargo builds, stalling prod deploy pipeline 8174 (tag v1.13.2). A dedicated incident doc was not available on this branch at fix time; this section documents the detection/restore procedure from the runbook's existing conventions. If a fuller incident writeup lands separately, cross-link it here.

Failure mode N: Agent host OOM: instance InService, SSM ConnectionLost, steps stuck running

Symptom signature: - woodpecker_workers / the agent's queue-info shows 0 connected workers, but the pipeline the dead agent was executing still shows running in the Woodpecker UI (server never got a clean failure signal — same server-side symptom as Failure mode M). - aws ssm describe-instance-information --filters Key=InstanceIds,Values=<id> reports PingStatus: ConnectionLost for the instance, while aws ec2 describe-instance-status / the ASG still call it InService / Ok / Ok — the EC2 layer never sees the failure. - The step's log output stops mid-command (no error, no exit code, no further lines) rather than ending with a shell prompt or an explicit failure. - CloudWatch CPUUtilization for the instance flatlines at the moment of death instead of trailing off — consistent with the kernel OOM-killer ending the process abruptly rather than the build finishing or erroring.

Root cause: The same underlying mechanism as Failure mode M (kernel OOM-killer triggered by a memory-heavy build), but this section covers the specific case where the OOM-killer also takes out amazon-ssm-agent, not just woodpecker-agent. That distinction matters operationally: Failure mode M's restore path assumes aws ssm send-command / aws ssm start-session still work against the box — they do not if SSM itself was killed. A ConnectionLost PingStatus with an otherwise-healthy-looking InService instance is the signature that this is the SSM-down variant, and the fix is an ASG-level instance action, not an in-box SSM restart.

2026-08-29 occurrence: - ~04:20Z — i-08007805850b8c88d — OOM/SSM-ConnectionLost signature confirmed (pipeline 8174, C++ Queue container build over the host Docker socket, co-scheduled with a heavy cargo build from MooseQuest-LLC/project-avenue — the co-scheduling condition WOODPECKER_MAX_WORKFLOWS=1 (#4554) now removes). Source: operator follow-up comment on #4545 (2026-08-29T05:41Z). - 06:00:26Z — i-0cb30ea38e52dd7f7 — second occurrence, same signature (PingStatus: ConnectionLost with last ping 06:00:26Z; the co-scheduled project-avenue cargo build's log stops at 06:05Z; queue-info workers=0 pending=9). #4554 had merged at 05:51Z but had not yet been applied to the fleet, so the co-scheduling condition was still live. Both wedged instances were terminated and replaced from launch template v7 at 09:40Z. Source: corroboration table on #4545 (2026-08-29, UTC timeline from describe-instance-information + ASG activity log).

Pipeline 8155 (03:02Z, i-0705ab5039be84b29) was orphaned in the same window but is not this mode: the ASG activity log shows the scale arbiter setting desired 1→0 and terminating that instance mid-clone — that is Failure mode J (scale-down while executing). Its clone step stayed running server-side for the same reason as here (no clean failure signal), which is why the two are easy to confuse from the Woodpecker UI alone; the PingStatus + ASG activity pair is what tells them apart.

Remediation: 1. Do not attempt aws ssm send-command / start-session against the affected instance — if PingStatus is ConnectionLost, SSM is the thing that's dead; those calls will just time out. 2. Terminate/replace the instance via the ASG (aws autoscaling terminate-instance-in-auto-scaling-group --instance-id <id> --no-should-decrement-desired-capacity --region us-east-2, or replace via the console/ASG instance refresh). Note --should-decrement-desired-capacity is a boolean flag pair, not a value-taking option — --should-decrement-desired-capacity false fails at argument-parse time with Unknown options: false; use --no-should-decrement-desired-capacity to keep the ASG at its current desired capacity while the replacement spins up. A plain aws ec2 reboot-instances is acceptable as a faster stopgap to restore a working agent, but note it does not pick up any user_data changes (including this PR's hardening) — user_data only runs on first boot of a new instance, not on reboot of an existing one. Prefer terminate/replace when the goal is to land new agent-template hardening on the fleet. 3. Cancel the orphaned pipeline/step via the Woodpecker UI or the /cancel API (same as Failure mode M step 1) so the queue isn't blocked on a step that will never complete. 4. Re-fire the cancelled pipeline with a new tag on the same commit rather than a bare UI "Restart" or a manual pipelines POST — the latter re-triggers the repo's full workflow fan-out and can overload an already-degraded fleet (same guidance as Failure mode M / #4555).

Prevention: This PR (#4545 follow-up) adds three layers so the kernel picks the runaway build over the agent/SSM processes: a per-step container memory ceiling (WOODPECKER_BACKEND_DOCKER_LIMIT_MEM / ..._LIMIT_MEM_SWAP, bounds work running inside a step container but not docker build over the host socket), oom_score_adj: -900 on the woodpecker-agent compose service, OOMScoreAdjust=-1000 on amazon-ssm-agent.service via a systemd drop-in, and OOMScoreAdjust=-500 on the woodpecker-agent.service wrapper unit (matching dockerd's own untouched default). Combined with WOODPECKER_MAX_WORKFLOWS=1 (#4554), this removes both the co-scheduling condition that caused the 2026-08-29 events and ensures that if a single heavy build still exhausts host memory, the kernel kills the build process, not the agent or SSM.

Detection gap: There is currently no Alertmanager rule that fires specifically on "ASG says InService but SSM PingStatus is ConnectionLost" — this incident class is only visible today via manual investigation (describe-instance-information + dmesg) after a pipeline is already noticed stuck. Tracked as an open alerting gap on #4545; a SSMConnectionLostWhileInService-style comparison alert (cross-referencing EC2 instance-status against SSM ping status) is the natural fix and is not yet implemented.

Failure mode P: a long workflow (deploy-queue) shows running for hours because its agent was SIGTERMed mid-run

Symptom: The GitHub status ci/woodpecker/tag/deploy-queue (or .../push/deploy-queue) reads "Pipeline is running" (state pending) and never gets a terminal status, while sibling workflows in the same pipeline finished (v1.20.4: deploy-prod succeeded at 04:15Z, deploy-queue still "running" 8+ hours later). It is not a never-scheduled run: it left pending and started.

Cause: deploy-queue is the one long job (cold C++ build with sanitizers, about 25-30 min). If the agent holding it is stopped mid-run, the agent cannot post the terminal state and Woodpecker has no reaper for the orphaned workflow, so it stays running. The agent log signature is received sigterm termination signal for that pipeline, then failed to update workflow status ... sql: no rows in result set (the same signature as Failure mode J). Ways an agent gets stopped mid-run: - Spot reclaim of a burst instance. 2026-10-30: the fleet was raised to min=desired=4 at 04:08:30Z, the v1.20.4 tag was pushed at 04:08:54Z, deploy-queue (workflow 19694/2) started on i-0c1e8b34d57e00a56 at 04:09:56Z and was in build-test when the agent got SIGTERM at 04:18:11Z; the ASG logged the instance as terminated at 04:20:30Z. A second burst instance went at 04:18:35Z. A third burst instance showed Server.SpotInstanceTermination at 12:18:26Z. - Manual scale-in during a run. 2026-10-07: v1.20.0 (pipeline 15762), desired set 3 to 1 at 21:32:14Z, two minutes into the run. - ASG health replacement. 2026-10-08: v1.20.2 (pipeline 15883), instance i-00a67cb035fd4d457 SIGTERM at 15:49:36Z.

Three of the last five tag-triggered deploy-queue runs ended this way.

Diagnose (read-only):

# 1. Is there ever a terminal status for the workflow? A stuck run has none.
gh api repos/raxx-app/TradeMasterAPI/commits/<tag-commit-sha>/statuses \
  --jq '.[] | select(.context|test("deploy-queue")) | [.state,.description,.created_at] | @tsv'

# 2. Agent log for that pipeline number: expect SIGTERM, then the failed status update.
#    Window in epoch milliseconds, e.g.
#    python3 -c "import datetime as d; print(int(d.datetime(2026,10,30,4,0,tzinfo=d.timezone.utc).timestamp()*1000))"
aws logs filter-log-events --log-group-name /ci/woodpecker-agents --region us-east-2 \
  --start-time <start-ms> --end-time <end-ms> --filter-pattern '<pipeline-number>' \
  --query 'events[].[logStreamName,message]' --output text

# 3. Why did that instance go away?
aws autoscaling describe-scaling-activities --auto-scaling-group-name ci-woodpecker-agents-asg \
  --region us-east-2 --max-items 40 --query 'Activities[].[StartTime,Description,Cause]' --output text
aws ec2 describe-instances --instance-ids <instance-id> --region us-east-2 \
  --query 'Reservations[].Instances[].[State.Name,StateReason.Message]' --output text

Fix: Cancel the orphaned pipeline from the Woodpecker UI (or POST /api/repos/1/pipelines/<number>/cancel with an admin token). Before re-firing anything, check whether the deploy had real content: git diff --stat <previous-tag> <tag> -- queue/ .woodpecker/deploy-queue.yaml. Empty means the tag deployed nothing new to Queue and no re-fire is needed (v1.20.3 to v1.20.4 was empty). If it did change queue/, re-fire with a NEW tag on the same commit, not a UI restart (same rule as Failure modes M and N).

Detection: deploy-failure-monitor.yaml now reports any deploy-queue workflow still pending or running after 2 hours (DEPLOY_STUCK_SECONDS, default 7200) in its 6-hourly email. Before this, pending and running were skipped, so a run like this was invisible.

Prevention: Since #4576 every ASG-initiated scale-in (the evening schedule, the arbiter, a manual set-desired-capacity, an instance refresh) parks the instance in Terminating:Wait until its agent is idle (the ci-agent-drain hook; §Scale-in drain hook, Failure mode T), so lowering the fleet no longer cuts a running job. It does not cover a spot reclaim by EC2, a crash, or a direct ec2 terminate-instances. Still check the Woodpecker queue (/api/queue/info, running count 0) before any manual EC2-level termination. The structural fix for spot reclaim is running deploy-queue on non-spot capacity; that is a cost decision for the operator and is not applied here.

Failure mode T: deploy-failure-monitor is DARK (WOODPECKER_API_TOKEN not provisioned) and mails ops@ on every cron fire

Symptom: The deploy-failure-monitor cron (cron 26, every 30 minutes) logs ERROR - deploy-failure-monitor DARK - WOODPECKER_API_TOKEN not provisioned (vault /MooseQuest/woodpecker/) and the pipeline goes red; ops@raxx.app gets [Raxx ops] deploy-failure-monitor DARK - WOODPECKER_API_TOKEN missing, up to 48 times a day.

Cause: WOODPECKER_API_TOKEN is not in the vault at /MooseQuest/woodpecker/. The monitor cannot read pipeline history, so deploy failure-streak detection is inactive. The earlier mail gate (hour < 6) assumed a 6-hour cron and let 12 mails a day through on the 30-minute registration.

Behaviour since 2026-11-04 (operator decision, pre-launch daily-digest rule): a missing token is a known, alerted-daily state. - The run created in the 13:00 UTC window (13:00 to 13:29, i.e. the :00 fire) logs the DARK line, sends one email to ops@raxx.app and exits 1 (red). A marker file stops notify-on-failure from sending a second email. - Every other run logs the DARK line plus no email, exiting 0 and is green. - A manual run always mails and fails red. - The gate reads CI_PIPELINE_CREATED (the cron fire time), not the step's wall clock, so agent queue lag cannot move the 13:00 run out of the window. Constants DARK_ALERT_UTC_HOUR and DARK_ALERT_WINDOW_MINUTES sit above the token check in the pipeline file. - The rule is stateless, so it is tied to the cron cadence. A cron slower than 30 minutes that skips the window (every 2 or 6 hours) never alerts; a cron faster than 30 minutes alerts more than once a day. If cron 26 is re-registered, re-check the window against the new schedule. The cron table is the source of truth for the schedule; it is not in the repo.

Fix: Provision WOODPECKER_API_TOKEN in Infisical at /MooseQuest/woodpecker/ (env prod): a Woodpecker personal/API token with read access to GET /api/repos/1/pipelines, per docs/ops/runbooks/cron-heartbeat-monitor.md (Known gap section). Do not paste it into chat or commit it.

Verification: Manually trigger the pipeline (event=manual) after the token is stored. The log must show Fetched N pipelines from WP. and no DARK line. The same token un-darks cron-heartbeat-monitor (cron 29).

Detection: The daily 13:00 UTC email is the signal. Silence from this alert while the token is still missing means the cron stopped firing or fell out of the window: check cron 26 is enabled and its schedule.

Failure mode K: "functioning correctly" cron claims go stale — always re-verify via a real scoped trigger before enabled:true

Symptom: A cron created and validated in the same PR (e.g. "fixed + confirmed working" in the PR description) but never issued the follow-up PATCH .../cron/{id} {"enabled":true} sits dark (next_exec frozen at its creation-time computed value) for weeks or months. When someone finally goes to enable it, blindly trusting the original PR's claim risks silently arming a job that either always fails, or worse, always false-succeeds without doing its actual job.

Cause: A pipeline step exiting 0 is not proof the step did anything useful — only that nothing inside it raised past whatever error handling it has. Two concrete cases found on the same day (2026-08-12, cards

4477/#4478, following the #4429/#4450 precedent that first established

this pattern for tickets-e2e-smoke-daily):

  1. e2e-smoke-nightly (cron 18): npm ci in frontend/raxx-next fails outright under the CI image's npm version — this one fails loudly, but had never actually been exercised in that exact image/npm-version combination since the lockfile drift was introduced (PR #2917, 2026-05-27, ~2.5 months before this cron was even created).

  2. historical-bars-warm-nightly (cron 20): the underlying command (warm_historical_bars_1min.py) never bootstraps its DB engine (init_engine()) — every real invocation raises RuntimeError: get_engine() called before init_engine(), which is caught by a broad except Exception and silently downgraded to "no symbols to warm," so the step reports success while doing nothing. This is the more dangerous case: a naive re-enable based on the step's green exit code alone would have armed a cron that runs "successfully" forever without ever warming a single row.

  3. flag-drift-check (cron 33, #4539): report.error was set on a total Heroku-fetch failure (missing HEROKU_API_TOKEN at /MooseQuest/console/) but only mismatch_count > 0 gated the exit code, so the step exited 0 with mismatch_count=0, identical to "checked, zero drift." The inline CLI now exits 3 whenever report.error is set (1 = real drift, 2 = FlagDriftReconcilerError); guarded by console/tests/test_flag_drift_check_exit_gate_4539.py. Cron 33 stays disabled until the dedicated read-only token exists at /MooseQuest/console/HEROKU_API_TOKEN and a scoped re-trigger shows a real comparison.

Diagnose (do this before every enabled:true PATCH on a cron that has never actually fired, regardless of how recently or confidently a prior PR claimed it works):

# 1. Baseline — confirm current enabled/next_exec state
WP_TOKEN="<admin JWT, from SSM /ci/woodpecker/admin-api-token>"
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/cron/<CRON_ID>"
# next_exec frozen at creation time == zero fires since creation.

# 2. Check WP queue is idle before triggering (avoid stacking on live PR load)
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/queue/info"

# 3. Trigger via the SCOPED single-workflow endpoint — never a bare manual
#    dispatch (fans out to every event:manual workflow in the repo, ~65+
#    unrelated pipelines as of 2026-08).
curl -sf -X POST -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/cron/<CRON_ID>"
# Note: as of 2026-08-12 this endpoint did NOT reliably scope to exactly
# one workflow in this repo — `ci-boundary` rode along on both cron 18
# and cron 20 triggers despite its own `when:` not matching a cron event.
# See #4490 (open investigation). Do not be alarmed by an unrelated
# second workflow in the response's `workflows` array; just evaluate the
# TARGET workflow's own state independently of the pipeline's aggregate
# status.

# 4. Read the FULL step log (not just the summary exit code) for the
#    target workflow's real-work step. The v3 log endpoint returns
#    base64-encoded lines:
curl -sS -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/logs/<PIPELINE_NUMBER>/<STEP_ID>" \
  | python3 -c "
import json,sys,base64
for l in json.load(sys.stdin):
    data = l.get('data')
    if data:
        print(base64.b64decode(data).decode('utf-8', errors='replace'), end='')
"
# Read for the ACTUAL side effect described in the step's own log lines
# (rows loaded, files uploaded, bars warmed, etc.) — not just "exit 0" /
# a generic "PASS" echo. A step can print PASS and exit 0 while its own
# earlier log lines show an ERROR that was silently caught.

Fix: There is no generic fix — each case is its own root cause (see

4488, #4489 for the two concrete examples above). The fix IS the

process: never PATCH enabled:true on the strength of a stale PR claim alone. Always re-trigger via the scoped endpoint and read the real step output for evidence of actual work done, not just a clean exit code.

Verification: Re-trigger via the scoped endpoint after the underlying fix lands; confirm the step's own log shows the real expected side effect (not a masked "nothing to do" path); only then PATCH .../cron/{id} {"enabled":true} and confirm next_exec moves forward.

RCA: docs/incidents/2026-08-12-cron-reverify-batch-e2e-warm-waf.md, docs/incidents/2026-08-09-tickets-e2e-smoke-mailto-drift.md (originating precedent)

Cron inventory (reconciled 2026-08-23)

Full GET /api/repos/1/cron state as of the reconciliation sweep below. This table is the source of truth for "is cron N supposed to be enabled right now" — update it after every enable/disable decision, not just after incidents.

ID Name Workflow file Enabled Notes
1 security-scan-nightly security-scan-nightly.yaml yes Sole nightly scan since GHA counterpart retired 2026-07-07 by commit 4f6281c224 ([PR #4064](https://github.com/raxx-app/TradeMasterAPI/pull/4064)) — was falsely documented as dual-run for ~2 months (fixed). Alerting posture (fixed): normalize-deduplicate-summary calls .github/scripts/scan_gate.py, which exits 1 on two independent gates — (1) finding severity: any CRITICAL group. HIGH does NOT fail the pipeline — scan_normalize.parse_pip_audit() hardcodes tool_sev=HIGH for every pip-audit finding (pip-audit exposes no CVSS severity) and scan_severity_map.yaml maps it 1:1, so gating on HIGH would redden the nightly on every routine un-triaged transitive-dep CVE and fire a per-event ops@ alert nightly, against the daily-digest posture. Promote genuinely dangerous rules to CRITICAL via .github/config/scan_severity_map.yaml instead of widening the gate. (2) scan integrity: a missing/empty/unparseable scanner report, or any detect-gap finding — always fatal, severity-independent; a dark scan stays loud per ADR-0101 SC-5. notify-on-failure sends Slack + Postmark ops@ naming which gate tripped. Unit tests: .github/tests/test_scan_gate.py. GH issue-filing (scan_file_issues.py) is still NOT ported — blocked on the raxx-ops-bot PEM being corrupt (#4530); triage from the Slack/email alert until that credential is fixed.
2 alembic-version-check alembic-version-check.yaml yes Pilot cron, stable
3 maxmind-asn-refresh maxmind-asn-refresh.yaml yes Pilot cron, stable
4 console-degraded-auto-file console-degraded-auto-file.yaml no Never fired since creation (next_exec frozen at 2026-07-05T01:00). No documented intended-enabled decision. Held per Failure mode K — needs a scoped-trigger re-verify before enabled:true.
5 audit-integrity-cron-nightly audit-integrity-cron.yaml yes Green 2026-08-23
6 audit-integrity-cron-monthly audit-integrity-cron.yaml yes Same file as 5, monthly schedule
7 audit-reconciler-cron audit-reconciler-cron.yaml yes Green 2026-08-23
8 account-merge-tombstone-cron account-merge-tombstone-cron.yaml yes #4077-era hold confirmed lifted — enabled continuously since 2026-07-08, green 2026-08-23
9 trace-integrity-cron trace-integrity-cron.yaml yes Green 2026-08-23
10 billing-retention-cron billing-retention-cron.yaml yes #4077-era hold confirmed lifted — enabled continuously since 2026-07-08, green 2026-08-23
11 heroku-config-health-nightly heroku-config-health-nightly.yaml yes Green 2026-08-23
12 mbt-assignment-at-expiry-cron mbt-assignment-at-expiry-cron.yaml yes
13 mbt-dividend-split-cron mbt-dividend-split-cron.yaml yes Green 2026-08-23
14 mbt-drift-daily-cron mbt-drift-daily-cron.yaml yes Green 2026-08-23
15 mbt-drift-per-symbol-weekly-cron mbt-drift-per-symbol-weekly-cron.yaml yes
16 mbt-resting-orders-cron mbt-resting-orders-cron.yaml yes
17 bcp-smoke-monthly bcp-smoke-monthly.yaml yes Check 4 (HEROKU_API_KEY missing) silent-SKIP gap fixed 2026-09-08 (#4591, §VAULT_ALLOW_MISSING audit below) — now reports FAIL, not a silent skip that could pass the run with only 4/5 checks executed. Re-verify via a real scoped trigger next monthly cycle.
18 e2e-smoke-nightly e2e-smoke.yaml yes Re-verified 2026-08-12 (#4488 lockfile fix); green 2026-08-23
19 freescout-backup-daily freescout-backup-daily.yaml yes Green 2026-08-23
20 historical-bars-warm-nightly historical-bars-warm-nightly.yaml no Held — root-cause fix (#4493, init_engine() bootstrap) is merged to main/release, but prod (api.raxx.app) is still on v1.10.0 per GET /api/system/version (checked 2026-08-23). Re-check before every future enable attempt; do not enable below v1.13.0.
21 historical-bars-reconcile historical-bars-warm-nightly.yaml no Same file/command as 20 (--reconcile flag), same init_engine() dependency — held under the identical prod-version gate as 20.
22 tickets-e2e-smoke-daily tickets-e2e-smoke.yaml yes
23 tickets-e2e-cleanup tickets-e2e-smoke.yaml yes
24 synthetic-gate synthetic-gate-cron.yaml no Never fired since creation (next_exec frozen at 2026-07-08T13:00). No documented intended-enabled decision — docs/architecture/ci-notification-posture.md flip-back checklist (which explicitly restores this cron's dense schedule) is still un-executed pre-launch. Held per Failure mode K.
25 waf-synthetic-probe waf-synthetic-probe-cron.yaml no — never enable Operator-trigger only, WAF probe per #4479. Do not flip via automation.
26 deploy-failure-monitor deploy-failure-monitor.yaml yes Fires every 30 min, green 2026-08-23. WOODPECKER_API_TOKEN fail-loud gap fixed 2026-09-08 (#4591, §VAULT_ALLOW_MISSING audit below) — a missing token previously degraded to a silent exit 0 (unauthenticated WP API call → fetch error → pre-existing non-fatal path); now logs ERROR - deploy-failure-monitor DARK and fails (alert run) before any network call; since 2026-11-04 it mails at most once per UTC day (see Failure mode T). Verified locally (subprocess run of the real heredoc source, not live-triggered) — see scripts/ci/tests/test_deploy_failure_monitor_pipeline.py. Re-verify via a real scoped trigger before the next WOODPECKER_API_TOKEN rotation to confirm the happy path is unaffected.
27 security-zap-weekly security-zap.yaml yes
28 ci-digest-cron ci-digest-cron.yaml yes Registered + verified 2026-08-23 (#4527). Real digest built and logged with live data (build_ci_digest.py). SLACK_WEBHOOK_URL not provisioned at /MooseQuest/slack/ — delivery is log-only (documented graceful-skip) until provisioned.
29 cron-heartbeat-monitor cron-heartbeat-monitor.yaml no Data-source bug fixed 2026-08-23 (#4531, PR #4538) — check_cron_heartbeats.py now queries the WP API instead of the retired GitHub Actions API. Re-verified via a real scoped trigger post-merge (WP pipeline #7083): correctly fails loud (check step exit 2) with a clear, specific error — WOODPECKER_API_TOKEN not provisioned ... Operator action: provision WOODPECKER_API_TOKEN in Infisical at /MooseQuest/woodpecker/ — instead of the old silent false-STALE (exit 0 masked by \|\| true). This is the correct, expected outcome per the fix's design; not green, so stays disabled. POSTMARK_SERVER_TOKEN was reported missing (#4534); a key of that name was seen at /MooseQuest/postmark/ in a 2026-10-30 names-only probe (#4881) but is unverified (see row 34) — delivery would be log-only even once the WP token lands.
30 daily-bot-token-smoke daily-bot-token-smoke.yaml no Registered + verify-triggered 2026-08-23 — RED. RAXX_OPS_BOT_PRIVATE_KEY at /MooseQuest/raxx-ops-bot/PRIVATE_KEY_PEM (prod) fails base64 decode ("Incorrect padding") — real broken credential, security-adjacent. See #4530.
31 daily-card-groomer daily-card-groomer.yaml no Exit-code masking fixed 2026-08-23 (#4532, PR #4538) — the step now captures run_card_groomer.py's real exit code (redirect to file, not \| tee) and only sends the "completed" Slack DM on real success. Re-verified via a real scoped trigger post-merge (WP pipeline #7084): the step now correctly surfaces red (groom step exit=1) on the same pre-existing RAXX_OPS_BOT_PRIVATE_KEY PEM-decode failure / real GitHub 403 (#4530, operator-gated) — the pass criterion for this fix, confirmed. Stays disabled pending #4530.
32 drift-orchestrator-cron drift-orchestrator-cron.yaml yes cd bug + exit-code masking fixed 2026-08-23 (#4532, #4533, PR #4538); confirmed working via a scoped re-trigger (WP pipeline #7080) but surfaced a third, previously-masked bug — wrong module invocation path (python3 -m app.services.drift_orchestrator run from inside console/ → ModuleNotFoundError: No module named 'console'; the script's own internal imports require console.app.services.drift_orchestrator run from the repo root, matching its own documented prog= invocation). Fixed in PR #4540. Re-verified via a second scoped trigger (WP pipeline #7088): imports succeed, all 3 registered reconcilers (flag_drift, asset_drift, secret_drift) genuinely execute and report specific real errors (not a masked no-op) — run complete — total=3 mismatches=0 failed=0, exit 0. Enabled 2026-08-23 (next_exec confirmed advanced to 2026-08-24T06:00 UTC). Prerequisite gaps that blocked the reconcilers from completing a real comparison were tracked separately (#4539, #4541) — orchestration mechanism itself works correctly, per the same enablement bar already applied to crons 40/41. Status of those gaps: (a) asset_drift "manifest not found (404)" was never a missing file — docs/asset-manifest.json exists on main; the 404 was GitHub's response to an unauthenticated Contents API call on a private repo. The run step now mints a raxx-ops-bot installation token (CI_FORGE_TOKEN fallback), exports GITHUB_TOKEN, and fails loud if both are empty (#4541). Note the manifest's url fields are still empty (surfaces not yet enrolled in the live-prod reconciler), so asset_drift fetches the manifest but compares 0 assets until URLs are enrolled. (b) secret_drift no longer reads a static INFISICAL_TOKEN; it logs in with Universal Auth (INFISICAL_CLIENT_ID/INFISICAL_CLIENT_SECRET, already loaded) and reads each enrolled secret via GET /api/v3/secrets/raw/{name} with workspaceId taken from INFISICAL_PROJECT_ID. The request carries viewSecretValue=false (ADR-0051 metadata-only reads), so no secret value is returned; the vault environment comes from INFISICAL_ENV (default prod, the fleet-wide name; INFISICAL_ENVIRONMENT is only a fallback). The four enrolled secrets live in different vault folders, so the step takes one INFISICAL_SECRET_PATH_<SECRET_NAME> env line per secret (see the comment in drift-orchestrator-cron.yaml), and reads the vault key named by INFISICAL_SECRET_KEY_<SECRET_NAME> (default: the secret's own name, except POSTMARK_SERVER_TOKEN, which reads its rotation source POSTMARK_SERVER_API_KEY). POSTMARK_SERVER_TOKEN (/MooseQuest/postmark/) and FREESCOUT_API_KEY (/MooseQuest/freescout/) folders are set; a names-only probe on 2026-10-30 saw a key named POSTMARK_SERVER_TOKEN and one named FREESCOUT_API_KEY present, but the Postmark rotation key POSTMARK_SERVER_API_KEY was not re-probed after the mapping, so a not found for it is the expected loud signal if it is absent. SLACK_BOT_TOKEN and CLOUDFLARE_API_TOKEN have no vault key of that exact name (the Slack token is stored as BOT_USER_SLACK_TRADING_MASTER_API; CLOUDFLARE_API_TOKEN is a CI alias for several CF_* keys), so they report vault metadata for <NAME> not found on every run. That error is visible in the cron 32 step log only (logger.error, read the run step output); it does NOT reach the daily drift digest or the drift_run_results table from cron 32 today, for two reasons: (i) the orchestrator's Flask bootstrap (from console.app import create_app in drift_orchestrator.main) fails when run from the repo root with ModuleNotFoundError: No module named 'app' (console/app/__init__.py imports app.models.db), so secret_drift._write_run_result logs failed to persist run result and writes no row, and _already_alerted fails open so the first real mismatch would re-alert daily; (ii) the run step loads only CONSOLE_DATABASE_URL, HEROKU_API_TOKEN, SLACK_BOT_TOKEN and CONSOLE_TOTP_ENCRYPTION_KEY from /MooseQuest/console/, so POSTMARK_SERVER_TOKEN is never in the step env and send_daily_drift_digest returns skipped=True, reason='POSTMARK_SERVER_TOKEN not set' before it reads the DB. The drift_digest.py change (listing secret_drift fetch errors and blocking the all-clear) is correct and unit-tested but inert from cron 32 until both are fixed. The #4541 ruling names only the Flask bootstrap warning (i) as a non-blocking follow-up; loading POSTMARK_SERVER_TOKEN into the step (ii) was never ruled on and is an open operator or code item. An operator decision on which vault key backs each destination config var is still open; once made, add an INFISICAL_SECRET_PATH_<NAME> / INFISICAL_SECRET_KEY_<NAME> line to the workflow, no code change. (c) flag_drift (and secret_drift's Heroku half) still need HEROKU_API_TOKEN provisioned at /MooseQuest/console/ (#4539, operator/vault action). The orchestrator logs a reconciler's report.error as a warning and still exits 0 with failed=0 (only a raised exception counts as failed), so read the step log for ERROR/fetch failed lines — a green cron 32 does not by itself prove real comparisons ran.
33 flag-drift-check flag-drift-check.yaml no Same cd bug + exit-code masking fixed 2026-08-23 (#4532, #4533, PR #4538); confirmed working via a real scoped re-trigger (WP pipeline #7081) — dependencies installed correctly, real reconciler code executed (run_flag_drift_check), real exit code captured (Flag-drift check exit code: 0, mismatch_count=0). That real run surfaced a fourth, previously-masked issue: the reconciler's actual Heroku-vs-console comparison silently fails (HEROKU_API_TOKEN not provisioned at /MooseQuest/console/, same root cause as cron 32's gap) and the reconciler's error-handling doesn't gate the exit code on report.error, only on mismatch_count — so a total fetch failure looks identical to "checked, zero drift." Filed as #4539 (touches both a vault-path question and application error-handling policy — not a mechanical CI fix). Error-handling half fixed (drawdown pass): the inline CLI in flag-drift-check.yaml now exits 3 (distinct from 1=real mismatch, 2=FlagDriftReconcilerError) whenever report.error is set, independent of mismatch_count — the same Failure-Mode-K principle already applied to check_cron_heartbeats.py (#4531). Credential half still open: /MooseQuest/console/HEROKU_API_TOKEN (read-only, scoped to raxx-console-prod + raxx-api-prod) has not been provisioned — that is a vault-mint action outside a code-only change and needs an sre-agent one-shot. Stays disabled until the credential is provisioned AND a re-trigger shows a real, successful Heroku-vs-console comparison (not just a distinguishable failure exit code).
34 postmark-delivery-monitor postmark-delivery-monitor.yaml no Registered + verify-triggered 2026-08-23 — BLOCKED. POSTMARK_SERVER_TOKEN not provisioned at /MooseQuest/postmark/ (prod); check never runs (ERROR: POSTMARK_SERVER_TOKEN is not set), masked green by \|\| true. See #4534. Vault key note (2026-10-31, #4881): the canonical vault entry behind the Heroku var POSTMARK_SERVER_TOKEN is /MooseQuest/postmark/POSTMARK_SERVER_API_KEY (docs/ops/runbooks/postmark.md, "Vault and credentials"); this workflow loads the CI-consumer name POSTMARK_SERVER_TOKEN from the same folder. A names-only probe on 2026-10-30 saw a key of that name there, but it has not been verified as a valid token or as a copy of the canonical entry, so this row stays no until a vault re-probe and a real delivery-monitor run confirm it. cron 32's secret_drift reads the canonical POSTMARK_SERVER_API_KEY (not the same-named key) for its updatedAt.
35 queue-zero-dyno-monitor queue-zero-dyno-monitor.yaml yes Registered + verified 2026-08-23 — HEROKU_API_KEY loads fine from /MooseQuest/heroku/; correctly flag-gates on FLAG_QUEUE_ZERO_DYNO_MONITOR (documented, clean skip logged — flag not yet set to 1 in vault, so the monitor is currently an intentional no-op pending operator flag flip).
36 bcp-vault-snapshot-daily bcp-vault-snapshot-daily.yaml no Registered + verify-triggered 2026-08-23 — BLOCKED. GPG_BACKUP_PUBLIC_KEY not provisioned at /MooseQuest/bcp/ (prod); vault-export step exits cleanly ("No encrypted backup created today") but performs zero real backup work. File header calls this job pageable — real, currently-silent DR gap. See #4535.
37 reminders-second-factor-nightly reminders-second-factor-nightly.yaml no Registered 2026-08-23, deliberately not scoped-triggered. Command has no dry-run mode and sends real customer 2FA-reminder emails via heroku run on raxx-api-prod if FLAG_REMINDER_SECOND_FACTOR is on; could not confirm the flag's current prod state this pass (vault read denied). Operator-verify required before any scoped trigger. See #4536.
38 deploy-queue-failure-monitor ~~deploy-queue-failure-monitor.yaml~~ (file removed 2026-10-30) yes (registration is stale, delete it) The pipeline file was a flag-gated duplicate of the Queue profile in deploy-failure-monitor.yaml: #4135 deleted it as a duplicate, #4342 re-added it unrelated to that work, and it never alerted (FLAG_DEPLOY_QUEUE_FAILURE_MONITOR was never set to 1, so every run logged a clean skip). Removed once the Queue profile in deploy-failure-monitor.yaml was made to agree with deploy-queue.yaml (workflow-level; see that file's header). The WP cron registration named deploy-queue-failure-monitor (every 2 hours, branch develop) still exists and now matches no workflow, so it creates nothing; delete it from the Woodpecker UI (repo settings, Crons) or with DELETE /api/repos/1/cron/38 using an admin token.
39 nightly-tf-email-drift terraform-email-delivery-stack.yaml no — intentional Registered 2026-08-23 per the known-exception list in #4527; pending #4115 operator action (plan/apply IAM keys not yet in vault at /MooseQuest/aws/terraform/email-delivery-stack-{plan,apply}/). Verify-trigger also found and fixed a Failure Mode E-4 bug in the same file (${TF_VER} blanked at WP parse time → 404 on the terraform binary download); fixed to $TF_VER / $${TF_VER} in the same pass. Stays disabled regardless of the E-4 fix, per #4115.
40 mbt-eod-snapshot-cron mbt-eod-snapshot-cron.yaml yes Registered + verified 2026-08-23 — DATABASE_URL loads fine from /Raxx/Raptor/ (confirms the #4069 operator action is complete). Correctly flag-gates on FLAG_PAPER_TRADING_V1 (off, clean skip logged: "FLAG_PAPER_TRADING_V1 is off — skipping paper_eod_snapshot").
41 mbt-retention-purge-cron mbt-retention-purge-cron.yaml yes Registered + verified 2026-08-23 — same clean flag-gate as 40; DATABASE_URL/INTERNAL_SERVICE_TOKEN load fine from /Raxx/Raptor/. GDPR-critical purge itself has not yet run for real (flag off) — re-verify with real deletions once FLAG_PAPER_TRADING_V1 flips on.
42 idempotency-cleanup-cron idempotency-cleanup-cron.yaml yes Registered + verified 2026-08-23 — full pass. Live POST /api/internal/jobs/idempotency-cleanup against prod (api.raxx.app) returned HTTP 200 with real JSON ({"deleted_count":0,"dry_run":false,"run_at":"2026-08-23T10:57:49.960Z"}).
— heroku-vault-drift-nightly heroku-vault-drift-nightly.yaml not yet registered Added 2026-09-08 (#4590) — sibling to cron 11 (heroku-config-health-nightly), which only checks ABSENT/EMPTY on the Heroku side. This job instead diffs the live Heroku config-var value against vault's canonical copy for every docs/ops/required-config-vars.yaml entry whose vault_path is a real Infisical path (not the managed-by-heroku sentinel — DATABASE_URL itself is intentionally never written to vault, see docs/ops/runbooks/raptor-db-credentials.md; RAPTOR_APP_DATABASE_URL and the other manifest secrets are the real drift-risk set). Exit behavior (#1725): every finding prints a DRIFT line, but the step exits 1 only on a critical-severity finding, a MISMATCH (vault and Heroku both set, values differ) at any severity, a failed Heroku config fetch (HEROKU_FETCH_FAILED), a failed vault read (VAULT_FETCH_FAILED: any vault error other than HTTP 404, at any severity) or a failed vault login. It also exits 1 at startup when HEROKU_API_KEY or the INFISICAL_* creds are unset, or when a manifest var has a severity other than critical/warning. A warning-severity var that is only absent (vault 404, or not set on Heroku) is reported and leaves the run green, matching the manifest's severity contract and cron 11. A run that exits 0 posts nothing; warning-severity DRIFT lines stay in the vault-drift-check step log, so read that log to see vars waiting on provisioning. The script's unit tests run in the lint-heroku-vault-drift step of lint-ci-config.yaml on PRs that touch the script, its tests, the manifest or either pipeline file. Cron registration (POST /api/repos/1/cron, proposed schedule 20 5 * * *) is a CI-mutating action outside a local-branch-only pass — register + scoped-verify-trigger before flipping enabled:true, same bar as every other row in this table.
— billing-trial-reminders-cron billing-trial-reminders-cron.yaml not yet registered Added 2026-10-31 (#4831 / #4892). Hourly POST /api/internal/billing/trial-reminders on Queue with the cron token. Operator registers it: schedule 0 * * * *, branch develop (see billing-test-tooling.md section 10). Fails on non-2xx and when configuration_error is above 0. Watched by cron-heartbeat-monitor (SLA 3h): until it is registered the monitor reports it stale.
— heroku-hygiene-cron heroku-hygiene-cron.yaml not yet registered Never registered since it was added (#2052), which is why 12 review apps outlived their closed PRs undetected (verified 2026-11-06; console-review-apps.md Failure mode K). Now runs heroku_review_app_hygiene.py --reap: it DESTROYS Heroku review apps whose PR closed 24h+ ago, so its scoped verify trigger and any bare manual trigger are real destructive runs. Preview with --reap --dry-run first. Operator registers it: POST /api/repos/1/cron {"name":"heroku-hygiene-cron","schedule":"0 7 * * *","branch":"develop"}, scoped-verify-trigger before flipping enabled:true.
— billing-dunning-downgrade-cron billing-dunning-downgrade-cron.yaml not yet registered Added 2026-10-31 (#4892, found while scheduling the reminder: the dunning job had no scheduler). Daily POST /api/internal/billing/dunning-downgrade on Queue with the cron token. Operator registers it: schedule 0 5 * * *, branch develop. A 200 no-op while FLAG_BILLING_DUNNING is off. Watched by cron-heartbeat-monitor (SLA 26h): until it is registered the monitor reports it stale.

Resolved: workflow files with event: cron and zero registered cron entry (#4527)

As of 2026-08-23, the 15 .woodpecker/*.yaml files that declared an event: cron trigger with no corresponding GET /api/repos/1/cron entry (silent scheduling gap since GHA retirement, ~6 weeks for 12 of the 15 — see

4527 for the full incident writeup) have all been registered (WP cron IDs

28–42, table above) and individually verified via a scoped trigger + real step-log read per Failure Mode K. 6 of the 15 (28, 35, 38, 40, 41, 42) passed verification and are enabled. 9 stay disabled: 6 are genuinely blocked (real bugs or missing vault secrets, tracked in #4530–#4535), 1 is an intentional exception pending #4115 operator action (39), and 1 is held for operator-verify due to a real customer-email risk with no dry-run path (37).

billing-collector-cron.yaml remains the one deliberate exception that is event: manual-only (deferred per #4077 pending #2159 Postgres migration) — not part of this cron-registration batch at all.

VAULT_ALLOW_MISSING audit (#4591, 2026-09-08)

VAULT_ALLOW_MISSING="true" tells scripts/ci/wpc_load_vault_secrets.py to tolerate a 404 for a given key (log a warning, leave the env var unset) rather than dying with Vault key '...' not found. That's the right default for a genuinely optional secret (an alert-delivery channel like POSTMARK_SERVER_TOKEN/SLACK_WEBHOOK_URL, or a feature-flag value that defaults off when absent). It's the wrong default, silently, for a secret the step's core logic actually needs — the loader not dying is not the same as the step correctly reporting failure when that secret turns out to matter.

As of 2026-09-08, 47 .woodpecker/*.yaml files carry VAULT_ALLOW_MISSING="true" across 83 occurrences (grep: grep -c 'VAULT_ALLOW_MISSING="true"' .woodpecker/*.yaml).

Meta-monitors first (priority per #4591)

The two crons that themselves watch other crons/deploys for failure are the highest-priority audit targets — if one of them degrades silently, every downstream alert it's supposed to fire also goes silent, and nothing else in the system would ever surface that.

Remaining 46 files — classification

A static pass (extract each VAULT_SECRETS= list paired with the VAULT_ALLOW_MISSING="true" line, then check for a nearby -z "$VAR" shell guard, a Python if not token: guard, or documentation language) plus manual spot-checks of every file the static pass couldn't classify automatically:

Audit method (repeatable)

grep -c 'VAULT_ALLOW_MISSING="true"' .woodpecker/*.yaml | grep -v ':0$'

then for each hit: read the VAULT_SECRETS= list on the preceding lines, classify by name (alert-channel vs core credential), and for core credentials confirm a guard exists (shell -z "$VAR" / Python if not var:) that turns "secret absent" into a loud failure (non-zero exit + an attempted alert) rather than a silent skip that could be mistaken for "checked, all clear."

Incident log

Date Duration Cause Fix applied
2026-07-04 ~2h WAF SizeRestrictions_BODY + WP v3 YAML ${{ parse error WAF override + pipeline YAML fixes (see RCA)
2026-07-11 ~5h 47m (08:00–13:47 UTC) woodpecker-restart.service Requires=woodpecker.service crash loop triggered by 08:00 timer — SEV-2 Removed Requires= line via SSM sed; service stable; IaC template patched
2026-08-12 N/A (verification pass, not an outage) e2e-smoke-nightly/historical-bars-warm-nightly crons re-verified per #4477/#4478; both found real pre-existing bugs (lockfile gap, missing init_engine()) rather than being safe to enable — SEV-3 Filed #4488/#4489 for feature-developer; left both crons disabled; filed #4490 for the cron-trigger co-scoping anomaly observed during the same pass
2026-08-23 N/A (routine sweep, not an outage) Launch-day cron enablement reconciliation — full GET /api/repos/1/cron inventory (27 entries) cross-referenced against .woodpecker/ and this runbook. All disabled crons held (4, 20, 21, 24 — none documented as intended-enabled or never fired; 25 — explicit never-enable). #4077-era hold on crons 8/10 confirmed already lifted. Found 15 .woodpecker/*.yaml files with an event: cron trigger and zero registered WP cron entry (12 with GHA already retired — silent scheduling gap since ~2026-07-10) — SEV-3 No enables applied (nothing met the bar). Filed a type:reliability ticket for the unregistered-cron gap; see §Cron inventory above
2026-08-23 N/A (registration/verification pass, not an outage) #4527 fix-forward: registered all 15 previously-unregistered crons (enabled:false), individually scoped-triggered + read real step logs per Failure Mode K. Found 6 distinct real defects: a malformed RAXX_OPS_BOT_PRIVATE_KEY vault secret (base64 decode fails); a cron-heartbeat-monitor data-source bug (queries retired-GHA workflow history instead of WP); a cd /woodpecker/src wrong-return-path bug shared by drift-orchestrator-cron/flag-drift-check (breaks console module import); a cmd \| tee file exit-code-masking pattern (no pipefail in /bin/sh) that hid both of the above as green; and two missing vault secrets (POSTMARK_SERVER_TOKEN, GPG_BACKUP_PUBLIC_KEY) — SEV-3 Also found + fixed a genuine Failure Mode E-4 bug in terraform-email-delivery-stack.yaml (${TF_VER} blanked at parse time). 6 of 15 crons enabled (28, 35, 38, 40, 41, 42); 9 held disabled — 6 blocked with named evidence (#4530–#4535), 1 intentional exception pending #4115 (39), 1 operator-verify held for customer-email risk (37, #4536)
2026-08-23 ~10m real-side-effect exposure window (11:29–11:39 UTC); ~1h total including cleanup + re-verification #4531/#4532/#4533 fix-forward on the 3 code-fixable defects found above, PR #4538: tee exit-code masking fixed (redirect to file, capture $? directly — dash has no pipefail) in drift-orchestrator-cron.yaml, flag-drift-check.yaml, daily-card-groomer.yaml; (cd console && ...) subshell fix for the cd-to-wrong-directory bug; check_cron_heartbeats.py rewritten to query the WP API directly. During pre-merge verification, a curl -sf trigger POST with empty local output was wrongly assumed to have failed — it had not, and created a real repo-wide manual-trigger fan-out (pipeline #7070) that ran to substantial completion (real prod billing-retention API call, no-op result; real cosmetic Heroku release on staging; reminders-second-factor-nightly failed before reaching its real send command) before being caught and cancelled — SEV-3, full RCA at docs/incidents/2026-08-23-cron-fix-verification-manual-trigger-fanout.md Post-merge scoped re-verification (Failure Mode K) found a further real bug in drift-orchestrator-cron (wrong module invocation path, fixed in follow-up PR #4540) and confirmed cron 31's masked-failure fix now surfaces red correctly. Final state: cron 32 enabled; crons 29/31/33 stay disabled with fresh, real evidence (see §Cron inventory rows above). New gaps filed: #4539, #4541. Root cause of the reminders-second-factor-nightly "heroku: not found" failure identified and fixed in the ci-pipelines sweep (2026-09-07): the Heroku CLI install (curl \| sh) ran on python:3.11-slim, which has no curl binary; under dash (no pipefail) the pipe's exit status was sh's (0, reading empty stdin), so the install line reported success while installing nothing, and every later heroku invocation died with "not found". Same root cause independently present in heroku-admin.yaml, launch-readiness-check.yaml, and ops-mint-preview-invite.yaml. Fixed in all four: apt-get install curl before the install line, plus a two-stage download-then-run (curl -o file && sh file) so a curl failure fails the line directly instead of degrading into an empty successful pipe; heroku --version added as a hard post-install assertion.
2026-11-03 22:00Z event; pipelines orphaned until cancelled The ci-agents-scale-in-evening schedule (min 1, desired 1) terminated 4 busy Woodpecker agents mid-job; the ASG had no termination lifecycle hook and the arbiter never scaled out on backlog (a busy fleet reads workers == 0, which it treated as cold-start lag) — SEV-3 #4576: ci-agent-drain terminating hook (45 min, CONTINUE) released by the arbiter after it cordons the agent (no_schedule) and sees no running task; burst rule desired = max(floor, ceil(pending / 3)) capped at 6; see §Scale-in drain hook and Failure mode T