RCA — Unintended repo-wide WP manual trigger fan-out during cron-fix verification

Incident ID: 2026-08-23-cron-fix-verification-manual-trigger-fanout Date: 2026-08-23 Severity: SEV-3 (real production-adjacent side effects occurred; net impact was effectively nil, but that was not knowable in advance and one side effect — a real Heroku release on staging — did land) Blast radius: WP CI (ci.moosequest.net), raxx-api-staging (Heroku), api.raxx.app internal billing-retention endpoint (prod, no-op result). No customer-visible impact; no data changed; no emails sent. Author: sre-agent

Summary

While pre-merge-verifying the #4531/#4532/#4533 cron fixes on branch sre/4531-4532-4533-cron-fixes, two separate POST /api/repos/1/pipelines calls (a bare manual trigger, which fans out to every event: manual workflow in the repo — ~55 workflows) were made against that branch. The first call appeared to fail silently (empty local output); it had not — it created pipeline #7070, which ran to substantial completion before being noticed and cancelled. A second, deliberate attempt (pipeline #7071) was correctly observed and cancelled before any workflow started. Real side effects from #7070: a real (idempotent, no-op) prod billing-retention API call, and a real (cosmetic, config-var-only) Heroku release on raxx-api-staging. No customer emails sent, no code deployed.

Timeline (all times UTC)

• 11:29:36 — curl -sf -X POST .../api/repos/1/pipelines -d '{"branch":"sre/4531-4532-4533-cron-fixes"}' > /tmp/wp_trigger_result.json run to pre-merge-verify the cron fixes. Local output: none (script produced no stdout).
• 11:29:36 (approx, same request) — Unbeknownst to the operator agent: this POST succeeded server-side, creating WP pipeline #7070 (id 8549), a bare manual trigger that queued all ~55 event: manual workflows.
• 11:29:4x–11:39:26 — Pipeline #7070 runs. Notable real executions (see "Impact" below): billing-retention-cron (real prod API call, no-op result), reminders-second-factor-nightly (failed before reaching send logic), deploy-staging's deploy-raptor step (real heroku config:set RELEASE_VERSION, then killed before the actual git push --force heroku completed).
• ~11:39 — Local check of /tmp/wp_trigger_result.json finds it empty; concluded (incorrectly) that the first trigger attempt had failed with no effect.
• 11:31:16 (approx) — A second, deliberate curl -X POST (this time with -w to show the HTTP status) is run against the same branch. Response shows a new pipeline, #7071 (id 8550), also a bare manual trigger. This is correctly recognized as risky — it matches the runbook's pre-existing documented warning about manual triggers fanning out.
• Immediately after — POST /api/repos/1/pipelines/7071/cancel issued. Confirmed via GET: all 55 workflows in #7071 show skipped, pipeline status: canceled. Zero execution.
• Shortly after — While investigating whether #7071's fan-out could recur, GET /api/repos/1/pipelines/7071 and the WP queue (GET /api/queue/info) surface a separate, earlier, still-running pipeline #7070 on the same branch — the real first trigger from 11:29:36, previously believed to have failed.
• Immediately — POST /api/repos/1/pipelines/7070/cancel issued.
• Following minutes — Evidence gathered on #7070's actual per-workflow outcomes (see "Impact"). Confirmed: no customer emails sent, no code deployed to staging, one real-but-no-op prod API call, one real-but-cosmetic Heroku release.
• Resolved — runbook (docs/ops/runbooks/ci-woodpecker.md) strengthened with an explicit warning against this exact failure mode, in the same PR as the underlying cron fixes (#4538).

Impact

• Users affected: none.
• User-visible symptoms: none. deploy-staging's config-var change caused a brief Heroku dyno restart on raxx-api-staging (internal/staging only, not customer-facing).
• Data integrity: ok. billing-retention-cron executed a real, non-dry-run call to POST /api/internal/jobs/billing-retention (prod) but eligible_count: 0 / anonymized_count: 0 — no rows were touched.
• Revenue / billing: ok. billing-collector-cron (an internal AWS/Heroku/Cloudflare cost-attribution job, unrelated to customer billing) ran and failed cleanly (collector not available for all 5 collectors — a pre-existing, unrelated environment gap) before performing any write.
• Customer communications: none sent. reminders-second-factor-nightly (real customer 2FA reminder emails, explicitly flagged operator-gated in #4536 for exactly this reason) failed at heroku: not found (curl, needed to install the Heroku CLI, was missing from that step's base image) before it could invoke the actual heroku run ... send_second_factor_reminders command.
• Deploys: no code deployed. deploy-staging's deploy-raptor step ran heroku config:set RELEASE_VERSION=<wip-branch-version> (real, created Heroku release 1068 on raxx-api-staging) but was killed (cancel signal reached the running Docker container) before the subsequent git push --force heroku HEAD:refs/heads/main loop could execute or log a single attempt. Confirmed via heroku releases -a raxx-api-staging: no Deploy <sha> release exists after release 1067 (394b4529, from before this incident) — only the cosmetic Set RELEASE_VERSION config vars release 1068. That cosmetic mismatch was self-corrected within minutes by an unrelated, legitimate, concurrently-running tag-triggered deploy (release-2026.08.23-b3edbad, pipeline #7073, raxx-ops-bot[bot], triggered automatically by the unrelated #4537 merge to develop).
• Other workflows in #7070 that reached "failure" before being cancelled (heroku-admin, freescout-apply, deploy-console-shim, deploy-waf-log-shipper, deploy-customer-docs, deploy-internal-docs, ops-mint-preview-invite, bcp-smoke-monthly) were individually checked: each failed on its own manual-trigger input-validation guard (missing required CI_PIPELINE_EVENT_MANUAL_* variables) or a pre-existing, unrelated environment/toolchain gap (missing curl, Node.js too old for wrangler) — none reached a state where they attempted a real mutating action.

What went well

• The second (deliberate) trigger attempt was correctly recognized as risky against the runbook's pre-existing documented warning and cancelled within the same minute, before any of its 55 workflows started (skipped, not failure/success).
• Once the first trigger's real effect was discovered, cancellation and evidence-gathering were immediate and methodical — every "failure"/"success" workflow in the affected pipeline was individually checked for real mutating side effects rather than assuming the aggregate pipeline status told the whole story (the same Failure-Mode-K discipline this task was already applying to the underlying cron fixes).
• reminders-second-factor-nightly's lack of a dry-run path (flagged as a real risk in #4536, filed during the earlier #4527 sweep) meant this specific job was already on our radar as high-risk — its failure mode (missing heroku CLI) happened to prevent real customer impact, but that was luck, not design; the risk assessment that flagged it in #4536 was accurate.
• Zero secrets were committed to the repo or the PR at any point during this incident.

What didn't go well

• A curl -sf ... > file.json invocation with empty local output was taken as evidence that the underlying HTTP request never fired, rather than being independently verified with a follow-up GET. The request had, in fact, succeeded server-side.
• The bare manual-trigger endpoint was used at all for pre-merge verification, despite an existing, correctly-worded runbook warning (docs/ops/runbooks/ci-woodpecker.md, "Warning — manual trigger fires all 59 workflows") that predates this incident and describes exactly this failure mode. The warning existed; it wasn't consulted before acting.
• heroku releases:info <version> -a raxx-api-staging was run during incident triage to inspect a single release's metadata and inadvertently printed the app's entire live config-var snapshot (including DATABASE_URL, HEROKU_API_KEY, POSTMARK_SERVER_TOKEN, STRIPE_API_KEY, SECRET_KEY, SESSION_KEY, and others) into the agent's own tool output/transcript. heroku releases -a raxx-api-staging --json (already used earlier in triage) would have answered the actual question (was a new Deploy release created?) without ever needing the full config dump. No external system was exposed — the transcript is only readable by the operator, who already has full legitimate access to every value shown — but this was an avoidable, unnecessary broadening of where those values are echoed.

Root cause analysis

• Contributing factor 1 — no verification step after a "no output" trigger POST. The tooling pattern used (curl -sf ... > file.json, checked in a later, separate step) has no built-in confirmation that a mutating POST actually reached the server and succeeded. -f suppresses the response body on non-2xx but does not distinguish "request never sent" from "request sent, response written to a file that was later found empty for an unrelated reason." The system allowed a mutating action to be treated as a no-op based on absence of local evidence rather than presence of positive confirmation.
• Contributing factor 2 — an existing, accurate runbook warning was not consulted before acting. docs/ops/runbooks/ci-woodpecker.md already documented the exact fan-out risk (deploy-staging, cut-release-candidate, deploy-velvet among the risky workflows) from a prior incident. The warning was correct and sufficient; the gap was in the pre-action checklist, not the documentation. This is the same class of gap Failure Mode K exists to close for cron enablement — it did not yet exist for cron fix verification.
• Contributing factor 3 — reminders-second-factor-nightly has no dry-run path (pre-existing, tracked in #4536). This is what made the manual-trigger fan-out a real customer-communication risk rather than a purely internal one. It happened to fail for an unrelated reason (missing heroku CLI) before reaching the risky code path — that is not a structural safeguard and should not be relied on as one.

Detection

• What alerted us: the second, deliberate trigger's response was manually inspected (per standard practice, always check the response of a mutating API call) and immediately recognized as anomalous (fan-out to 55 workflows instead of 1).
• How long between cause (first trigger) and detection (discovering #7070 was real and running): approximately 10 minutes — the first trigger fired at 11:29:36, #7070 was discovered and cancelled around 11:39-11:40.
• How to detect faster next time: always follow a trigger POST with an immediate GET on the returned pipeline ID (or a GET /api/repos/1/pipelines?page=1 if the POST response itself is empty/ambiguous) before concluding a trigger attempt had no effect.

Resolution

• What was changed: POST /api/repos/1/pipelines/7070/cancel and POST /api/repos/1/pipelines/7071/cancel — both pipelines are now canceled/killed. No code or config changes were needed to "fix" anything (nothing was left in a broken state — the one real Heroku release, 1068, was cosmetic and has already been superseded by the correct value from the concurrent legitimate deploy). docs/ops/runbooks/ci-woodpecker.md was updated in the same PR (#4538) as the underlying cron fixes with an explicit warning against using the bare manual-trigger endpoint for pre-merge cron-fix verification, and the "verify via GET, don't trust local silence" lesson from this incident.
• Validation: GET /api/repos/1/pipelines/7070 and /7071 both show terminal, non-running states with every workflow accounted for (success/failure/killed/skipped, none running/pending). heroku releases -a raxx-api-staging confirms no unauthorized code deploy landed. billing-retention-cron's real API call response (eligible_count:0, anonymized_count:0) confirms zero data was touched.

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md (manual-trigger warning, updated in this incident's response)
• Related: #4536 (reminders-second-factor-nightly has no dry-run path — the risk this incident actually exercised)
• Related PR: #4538 (the #4531/#4532/#4533 cron-fix PR being verified when this incident occurred)