RCA — WAF SizeRestrictions_BODY blocked all GitHub webhooks + WP v3 YAML parser broke all pipelines

Incident ID: 2026-07-04-waf-webhook-block-batch1-parse-error Date: 2026-07-04 Severity: SEV-2 Duration: ~40m total (detection → resolution; pipelines fully green by 00:41 UTC) Blast radius: All GitHub webhook-triggered Woodpecker pipelines for TradeMasterAPI; Wave A batch-1 PR-gate verification blocked. No production user traffic affected (CI-only). Author: sre-agent

Summary

Two compounding failures prevented any Woodpecker pipeline from running during the Wave A batch-1 CI migration verification window. First, the ALB WAF SizeRestrictions_BODY rule (8192-byte body cap in AWSManagedRulesCommonRuleSet) blocked all GitHub webhook deliveries to /api/hook with HTTP 403 — real PR/push payloads are 12–36 KB and exceeded the cap. Second, after the WAF fix was applied and PR #4043 merged, a GHA expression syntax fragment (${{) in a YAML comment inside .woodpecker/console-degraded-auto-file.yaml triggered the Woodpecker v3 substitution pre-processor and caused ALL pipelines to fail at compile time with "unable to parse variable name". Both issues were resolved by 00:41 UTC via Terraform apply (WAF) and three incremental PRs (#4046, #4048, #4049) to remove the offending patterns.

Timeline (all times UTC, 2026-07-04)

• 00:00 — Operator issued MIGRATION-WIDE UNBLOCK. Task: apply WAF fix (PR #4044), run terraform plan/apply, verify webhook delivery, merge PR #4043, verify batch-1 pipelines.
• 00:09 — PR #4044 (AllowGitHubWebhook WAF rule, priority 1, STARTS_WITH /api/hook) merged to develop via admin squash.
• 00:10 — Terraform plan completed: 0 to add, 1 to change (aws_wafv2_web_acl.ci), 0 to destroy. Safety gate confirmed (no server/RDS/ALB/ASG replacement).
• 00:11 — terraform apply completed. AllowGitHubWebhook rule active in AWS WAF.
• 00:11 — New push commit to test branch generated fresh webhook delivery. GitHub webhook history showed HTTP 204 — WAF block resolved.
• 00:12 — PR #4043 (console-degraded-auto-file vault-step split + INSTALL_ID dynamic lookup) merged to develop via admin squash.
• 00:14 — First Woodpecker pipeline triggered (Pipeline 28 on develop push). All steps returned "unable to parse variable name" at compile phase. No pipelines ran.
• 00:16 — Hypothesis 1: ${GH_TOKEN:-} / ${CF_CONSOLE_SVC_ID:-} shell default-value expansions triggering WP v3 substitution engine (:- not a valid identifier char). PR #4046 filed and merged (changed to ${GH_TOKEN:-} WP escape + ${CI_FORGE_TOKEN}).
• 00:20 — Pipeline 30 still failed with same "unable to parse variable name" error. Hypothesis 1 was incomplete — a second pattern was still present.
• 00:22 — Root cause identified: line 157 of console-degraded-auto-file.yaml contained GHA expression syntax in a YAML comment: # Mirrors: GH_TOKEN: ${{ steps.mint_bot_token.outputs.token || github.token }} WP v3 pre-processor scans ALL YAML text including comments. ${{ triggers ${...} parsing; the second { is not a valid identifier char → "unable to parse variable name". Because this is a global parse failure, ALL pipelines for every event failed.
• 00:25 — PR #4048 filed: changed comment to # (GHA syntax elided — ${{ }} is invalid in WP YAML). Error: the replacement text itself contained ${{. Merged (12f8ce39).
• 00:28 — Pipeline 31 still failed. ${{ pattern re-identified in the "fix" comment.
• 00:30 — PR #4049 filed: final clean replacement — # Mirrors: GH_TOKEN: (mint_bot_token.outputs.token || github.token) from GHA workflow. Confirmed zero ${{ in entire .woodpecker/ directory via grep -rc '\${{' .woodpecker/.
• 00:35 — PR #4049 required rebase (merge conflict with #4048). Rebased onto origin/develop, force-pushed with --force-with-lease. Merged (20bfbacf).
• 00:38 — Pipeline 29 (PR #4049) triggered. All 5 batch-1 PR-gate pipelines ran and returned SUCCESS: lint-cf-access-headers, lint-cf-pages-deploy-uniqueness, lint-cf-tokens, lint-workflow-secret-names, pii-scan.
• 00:40 — Pipeline 32 (manual trigger, console-degraded-auto-file): check step exit_code=0. Cron registered (id=4, */15 * * * *, branch develop).
• 00:41 — ASG ci-woodpecker-agents-asg scaled to desired=0. Test PR #4045 closed. Test branch deleted. Incident resolved.

Impact

• Users affected: 0 (CI-only; no production user traffic)
• User-visible symptoms: none
• Data integrity: ok
• Revenue / billing: ok
• CI impact: Wave A batch-1 webhook-triggered pipeline verification was blocked for ~40m. No pipelines triggered on PR/push events during the WAF-block window. All downstream Wave A batches (2–5) were blocked on batch-1 verification completing.

What went well

• Terraform safety gate (0-destroy check) ran before apply; no infrastructure was replaced.
• WAF fix was already written and PR-ready (#4044) before this session. Apply was fast.
• The ${{ root cause was isolated within 8 minutes of noticing hypothesis 1 was incomplete.
• Final fix PR (#4049) was confirmed clean via grep -rc before merge — no recurrence.
• Batch-1 pipeline verification was comprehensive: all 5 unconditional PR-gate pipelines passed; path-filtered pipelines correctly did not trigger (no matching file changes).

What didn't go well

• PR #4043 introduced ${{ in a YAML comment — the WP v3 pre-processor constraint was not documented in the runbook at the time of authoring.
• Hypothesis 1 (${VAR:-} escape) was plausible but incomplete. A grep -rc '\${{' .woodpecker/ scan before merging #4043 would have caught the second pattern.
• PR #4048 (the "fix" PR) re-introduced the exact pattern it was trying to remove — the fixer used the broken syntax in the description of why the syntax was broken.
• The admin:repo_hook scope was missing from the operator token, preventing webhook redeliver via API. This required generating a fresh webhook event as a workaround.

Root cause analysis

• Contributing factor 1: WAF SizeRestrictions_BODY cap (8192 bytes) vs real payload size (12–36 KB). The AllowGitHubWebhook priority-1 WAF rule (introduced in infra/ci/waf.tf PR #4044) was not present in the initial Terraform state. The system allowed deployment of a WAF configuration that blocked its own primary trigger mechanism (GitHub webhooks). The cap is a default-on managed rule with no escape hatch until an explicit allow rule is added at a lower priority number.

• Contributing factor 2: WP v3 pre-processor applies to ALL YAML text, including comments. Woodpecker v3 changed the substitution pre-processor to scan the full YAML buffer before parsing. The constraint ${{ is invalid was not captured in the runbook or in a pre-merge lint gate. The system allowed .woodpecker/*.yaml files to be merged without scanning for this pattern, causing a global pipeline compile failure.

• Contributing factor 3: No .woodpecker/ YAML lint gate in CI. There is no pre-merge check that validates .woodpecker/*.yaml files against WP v3 syntax constraints. A lint step catching ${{ and ${VAR:-} patterns would have blocked PR #4043 before merge and prevented the entire secondary incident.

Detection

• WAF block: detected immediately by checking GitHub webhook delivery status (403 responses visible in repository webhook history). Time from cause to detection: ~1m.
• WP v3 parse error: detected by observing the first pipeline (Pipeline 28) fail at compile phase with "unable to parse variable name". Time from cause to detection: ~2m.
• How to detect faster: a grep -rc '\${{' .woodpecker/ and grep -rn '\${[A-Z_]*:-' .woodpecker/ scan run as a CI lint step would detect before merge.

Resolution

• WAF fix: AllowGitHubWebhook rule applied via terraform apply in infra/ci/. Rule: priority 1, URI byte-match STARTS_WITH base64(/api/hook), action: allow {}. This is now permanent infrastructure — it survives future terraform apply runs.

• WP v3 YAML parse fix: PR #4049 (20bfbacf) removed all ${{ and ${VAR:-} patterns from .woodpecker/console-degraded-auto-file.yaml. The file now passes WP v3 pre-processor with zero substitution errors.

• Validation: Pipeline 29 (all 5 batch-1 PR-gate steps SUCCESS), Pipeline 32 (manual console-degraded check, exit_code=0), new webhook delivery (HTTP 204).

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md — Failure mode D-2, Failure mode E-4
• PR #4044: fix/ci-waf-github-webhook-sizerestrictions — WAF AllowGitHubWebhook rule
• PR #4043: console-degraded-auto-file vault-step split (introduced ${{ in comment)
• PR #4046: ${VAR:-} escape fix (hypothesis 1 — partial fix)
• PR #4048: First ${{ comment fix attempt (re-introduced the pattern)
• PR #4049 (20bfbacf): Final ${{ removal — clean
• Related incidents: docs/incidents/2026-07-04-woodpecker-v3-upgrade.md
• AWS WAF SizeRestrictions_BODY docs: https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-baseline.html
• Woodpecker v3 variable substitution: https://woodpecker-ci.org/docs/usage/environment