RCA — Stale WP org secret froze prod deploys for 5 weeks; console billing dark

Incident ID: 2026-08-29-deploy-prod-stale-wp-secret Date: 2026-08-29 Severity: SEV-2 (production deploy pipeline completely broken for 5 weeks; no data loss, no direct customer-facing exposure, but the entire billing sprint and 143 commits were undeployable) Duration: Root cause first manifested 2026-07-21 (rotation gap created); first observable failure 2026-08-22 (pipeline 7000); two operator-authorized recovery attempts (reseed HEROKU_PROD_API_KEY + restart, 2026-08-22 and 2026-08-27 — see Timeline) did not resolve it before pipeline 7800 also failed on 2026-08-27; confirmed root-caused and fixed 2026-08-29 — approximately 5 weeks from gap creation to fix, 7 days from first observed failure to fix. There is no comment evidence in https://github.com/raxx-app/TradeMasterAPI/issues/4502 confirming either authorized reseed was actually executed before its corresponding pipeline ran; see Timeline and Contributing factors. Blast radius: Production Heroku deploys (raxx-api-prod, raxx-console-prod, and raxx-queue-prod) via .woodpecker/deploy-prod.yaml and .woodpecker/deploy-queue.yaml's deploy-prod step — both consumed the same stale HEROKU_PROD_API_KEY static copy on the same event: tag gate before this PR. No customer-facing outage — prod stayed on its last-known-good release throughout. Operator-visible: console billing (console.raxx.app/billing) unavailable since 2026-07-25 because the fix for that (https://github.com/raxx-app/TradeMasterAPI/issues/4387) never reached prod. Author: raxx-dev-bot

Summary

Production had received no code deploy since 2026-07-21 (raxx-api-prod release #184, SHA 558c9eda). Two tag-triggered deploy-prod pipelines failed at the same step — deploy-raptor-prod, on the RELEASE_VERSION config-var PATCH — without ever reaching git push: pipeline 7000 (tag v1.13.0, cut 2026-08-22) and pipeline 7800 (tag v1.13.1, cut 2026-08-27T13:09:34Z, 99 seconds after that day's operator reseed+tag authorization — see Timeline; failure observed same day). Ground truth: https://api.raxx.app/api/system/version returned 1.10.0, a hardcoded literal deleted from the codebase on 2026-07-31 (ebc2e5f4a) — post-07-31 code cannot emit that string, confirming prod was frozen well before either failed pipeline. Staging (0.0.132-7073, correct shape) was unaffected — only the prod lane was dead. deploy-queue.yaml's tag-gated deploy-prod step reads the same secret family and was exposed to the same gap (see Blast radius); no confirmed pipeline record of a Queue-prod tag deploy in this window was found while writing this RCA.

Root cause: the Woodpecker org secret HEROKU_PROD_API_KEY (orgs/2, event-restricted to tag + manual) was a static copy of vault /MooseQuest/heroku/HEROKU_API_KEY, created 2026-07-06 by the Wave C port. The 2026-07-21 full credential rotation (docs/incidents/2026-07-21-full-credential-rotation.md:39) updated the vault value and the four Heroku app config vars, and its own log entry recorded "no static WP secret to update for this credential" — true for the vault-sourced HEROKU_API_KEY, false for the separately-named prod alias. Rotation coverage was name-matched, not consumer-matched, and no drift check covers WP org secrets, so the gap went undetected until the next tag-triggered prod deploy tried to use the now-revoked key. A secondary defect compounded diagnosis time: deploy-prod.yaml's RELEASE_VERSION PATCH ran as curl -fsS ... >/dev/null 2>&1, so the HTTP 401 response body was swallowed and both failed pipelines showed only curl exit 22 with no diagnostic.

Fixed by https://github.com/raxx-app/TradeMasterAPI/pull/4550 (closing https://github.com/raxx-app/TradeMasterAPI/issues/4544): HEROKU_API_KEY is now loaded from vault /MooseQuest/heroku/ at run time in both deploy-prod.yaml and deploy-queue.yaml's deploy-prod step — the same idiom already used by deploy-staging.yaml, deploy-velvet.yaml, and deploy-queue.yaml's staging step. The tag/manual when: gate and the in-step CI_COMMIT_TAG / DEPLOY_PROD_CONFIRM guards are unchanged and remain the prod authorization boundary. The RELEASE_VERSION PATCH now captures the HTTP status code and a truncated response body on failure and exits non-zero instead of swallowing the error.

Impact

• Users affected: 1 (operator; console.raxx.app is not customer-facing). No Raxx customer was directly affected — the frozen deploy pipeline blocked shipping changes, but the previously-deployed prod code kept serving traffic normally throughout.
• User-visible symptoms: console.raxx.app/billing and its sub-panels showed "temporarily unavailable" continuously since 2026-07-25, because the fix for that (https://github.com/raxx-app/TradeMasterAPI/issues/4387) landed on develop/main but never reached raxx-console-prod — this deploy pipeline was already dead by the time that fix was cut for release.
• raxx-queue-prod: the same stale HEROKU_PROD_API_KEY WP org secret was also the credential for deploy-queue.yaml's tag-gated deploy-prod step (docker login registry.heroku.com, heroku container:release, heroku run for Sqitch migrations). Any vX.Y.Z tag also fires that pipeline in parallel with deploy-prod.yaml. No queue-prod code deploy is confirmed to have landed in this window either, though this RCA does not have a specific failed-pipeline number for it (unlike 7000/7800 for Raptor/Console) — operator should confirm via the Woodpecker pipeline history for deploy-queue.yaml on v1.13.0/v1.13.1.
• Data integrity: ok — no writes affected; this is a deploy-pipeline defect, not a data-path defect.
• Revenue / billing: no direct revenue impact (billing data collection and the billing-retention cron were unaffected), but the entire billing sprint (https://github.com/raxx-app/TradeMasterAPI/issues/4512, https://github.com/raxx-app/TradeMasterAPI/issues/4520, https://github.com/raxx-app/TradeMasterAPI/issues/4521, https://github.com/raxx-app/TradeMasterAPI/issues/4522, https://github.com/raxx-app/TradeMasterAPI/issues/4523, https://github.com/raxx-app/TradeMasterAPI/issues/4526) and every other commit in v1.12.1..v1.13.1 (143 commits total) sat undeployable to prod for 5 weeks.
• CI-visible impact: deploy-prod pipelines 7000 and 7800 both show red, training the team to treat a failed prod tag deploy as "try the tag again" rather than "the credential itself is broken" — neither pipeline's failure mode pointed at the actual root cause without manual investigation.

Timeline (all times UTC)

• 2026-07-06 — Wave C creates the WP org secret HEROKU_PROD_API_KEY (orgs/2, event-restricted [tag, manual]) as a one-time copy of vault /MooseQuest/heroku/HEROKU_API_KEY, for use by deploy-prod.yaml.
• 2026-07-21 — Full credential rotation (docs/incidents/2026-07-21-full-credential-rotation.md) mints a new Heroku authorization, writes it to vault /MooseQuest/heroku/HEROKU_API_KEY, and propagates it to the four Heroku app config vars (raxx-api-staging, raxx-api-prod, raxx-console-staging, raxx-console-prod). The rotation's own record (RCA line 39) states "Confirmed Woodpecker reads HEROKU_API_KEY live from vault per-pipeline-run ... no static WP secret to update for this credential" — accurate for every consumer named HEROKU_API_KEY, but HEROKU_PROD_API_KEY is a distinct secret name and was never in scope. The old authorization backing HEROKU_PROD_API_KEY is revoked as part of the same rotation, silently invalidating the static copy.
• 2026-07-25 — Unrelated incident (docs/incidents/2026-07-25-console-billing-summary-unavailable.md) fixes a session-auth exemption gap for the Console billing routes on main. That fix is real and correct, but by this point deploy-prod cannot reach prod — the fix ships to develop/release but has no path to raxx-console-prod until this incident's fix lands.
• 2026-08-22T21:29:03Z — Tag v1.13.0 cut (tagger date, refs/tags/v1.13.0).
• 2026-08-22 (shortly after) — deploy-prod pipeline 7000 fails at deploy-raptor-prod on the RELEASE_VERSION config-var PATCH with curl exit 22 and no diagnostic body (the pre-fix curl -fsS ... >/dev/null 2>&1 swallowed the HTTP 401 — see https://github.com/raxx-app/TradeMasterAPI/pull/4550). Prod remains on release #184 / 558c9eda.
• 2026-08-22T22:28:36Z — Operator authorization posted on https://github.com/raxx-app/TradeMasterAPI/issues/4502 (comment 5382931400): reseed the WP org secret HEROKU_PROD_API_KEY from vault /MooseQuest/heroku/HEROKU_API_KEY and restart pipeline 7000, with the correct root cause already named in the authorization text ("this secret still held the Heroku authorization revoked in the 2026-07-21 fleet rotation"). No follow-up comment in issue https://github.com/raxx-app/TradeMasterAPI/issues/4502 confirms the reseed was executed or that pipeline 7000 was restarted and passed — the next comment on that issue is the 2026-08-25 readiness audit, three days later, which does not mention v1.13.0 as deployed.
• 2026-08-25T21:54:53Z — A billing production-readiness audit posted on issue https://github.com/raxx-app/TradeMasterAPI/issues/4502 concludes (incorrectly) that "prod already runs v1.13.0 content via out-of-band deploy."
• 2026-08-25T21:57:32Z (commit only, not a tag) — commit 612a8d3 (later tagged v1.13.1) is authored on main. This is the commit-author date, not the tag date — see the correction below.
• 2026-08-27T09:09:52Z — Correction posted on issue https://github.com/raxx-app/TradeMasterAPI/issues/4502: the "prod already runs v1.13.0" conclusion from 2026-08-25 was wrong. The Heroku release ledger (checked 2026-08-27) shows raxx-api-prod's last code deploy is still release #184 / 558c9eda from 2026-07-21; all releases since are config-var-only. The audit's inference failed on a known trap — on code that predates https://github.com/raxx-app/TradeMasterAPI/pull/4526, the generic 401 every /api/* route returns proves nothing about whether a route exists (see feedback_generic_401_proves_nothing_about_routes in project memory for the general pattern).
• 2026-08-27T13:07:55Z — Second operator authorization on issue https://github.com/raxx-app/TradeMasterAPI/issues/4502 (comment explicitly "supersedes the v1.13.0-scoped authorization"): (1) reseed HEROKU_PROD_API_KEY again, (2) cut and push tag v1.13.1 on main @ 612a8d333, (3) verify prod post-deploy. The "supersedes" wording indicates the first (2026-08-22) reseed authorization was not treated as completed/effective by the time this second one was issued.
• 2026-08-27T13:09:34Z — Tag v1.13.1 pushed (tagger date), 99 seconds after the authorization above — correcting this RCA's earlier draft, which had stated 2026-08-25 for both the commit and the tag.
• 2026-08-27 (shortly after) — deploy-prod pipeline 7800 for v1.13.1 fails at the same RELEASE_VERSION PATCH step as pipeline 7000, despite the reseed authorized 2 minutes before the tag was pushed. This RCA cannot confirm from available issue history whether the 2026-08-27 reseed was actually carried out before pipeline 7800 ran — if it was and 7800 still failed at the same PATCH, the stale-static-copy explanation alone is incomplete (a fresh reseed of the same secret name should have fixed a purely-staleness failure). No follow-up comment on issue https://github.com/raxx-app/TradeMasterAPI/issues/4502 reports the reseed's execution result either way. No comment on issue https://github.com/raxx-app/TradeMasterAPI/issues/4502 records escalation, paging, or further attempted recovery between this failure and 2026-08-29 — the next activity in that window is issue https://github.com/raxx-app/TradeMasterAPI/issues/4544 being opened.
• 2026-08-29 — Investigation dispatched per issue https://github.com/raxx-app/TradeMasterAPI/issues/4544. Root-caused via ground-truth check (/api/system/version returning a deleted-since-07-31 literal, 1.10.0) confirming prod was frozen well before the two visible pipeline failures, then tracing HEROKU_PROD_API_KEY's origin (2026-07-06 Wave C) against the 2026-07-21 rotation scope. This is the first point in the record where the root cause is identified as a distinct, separately named static-copy secret (HEROKU_PROD_API_KEY) rather than a simple staleness-of-the-same-secret problem — which is consistent with the 08-22 and 08-27 reseed authorizations (which named HEROKU_PROD_API_KEY correctly but may not have been carried out, or were carried out but the underlying static-copy architecture meant the fix needed to be structural, not a one-time reseed). Fix (this PR, https://github.com/raxx-app/TradeMasterAPI/pull/4550, closing issue https://github.com/raxx-app/TradeMasterAPI/issues/4544) opens the same day: deploy-prod.yaml and deploy-queue.yaml's deploy-prod step switch to a run-time vault load of HEROKU_API_KEY, permanently removing the static-copy failure mode rather than reseeding it again; the RELEASE_VERSION PATCH is made to fail loud; create-github-release gains a CI_FORGE_TOKEN fallback for the unrelated but co-located ops-bot PEM corruption (https://github.com/raxx-app/TradeMasterAPI/issues/4530); docs and this RCA are swept in the same PR.

Root cause

Rotation coverage was name-matched, not consumer-matched. The 2026-07-21 rotation correctly identified and updated every store literally named HEROKU_API_KEY (vault + 4 Heroku app config vars) and correctly noted that Woodpecker reads that name live from vault with no static copy. It had no mechanism to discover that a differently-named secret (HEROKU_PROD_API_KEY) existed as an out-of-band static copy of the same underlying credential, created by a separate migration effort (Wave C, 2026-07-06) six weeks earlier. There is no drift check or inventory that covers WP org-level secrets the way heroku-api-key-drift-recovery.md covers the vault ↔ Heroku-config-var relationship, so this class of gap has no automated detection.

Contributing factors

• No drift check on WP org secrets. The heroku-api-key-drift-recovery.md runbook's three-copies model (vault / GH secret / Heroku config vars, at the time) had no fourth entry for WP org secrets, because none should have existed as static copies in the first place — but one did, silently.
• Silent failure mode on the diagnostic. curl -fsS ... >/dev/null 2>&1 on the RELEASE_VERSION PATCH discarded the HTTP status and response body, turning a clearly-diagnosable 401 into an opaque curl exit 22. Both failed pipelines (7000, 7800) required manual reproduction to identify the actual cause instead of surfacing it directly in the build log.
• No card tracked the gap. Nothing flagged the existence of HEROKU_PROD_API_KEY as a rotation-scope risk between 2026-07-06 (creation) and 2026-08-29 (discovery) — it was invisible until the next tag-triggered prod deploy needed it.
• Pipeline failures did not carry the diagnostic to where the recovery decision was made. Corrected from an earlier draft of this RCA, which stated "no escalation" and "caught only when this issue was worked" — both were wrong. The operator was engaged twice on https://github.com/raxx-app/TradeMasterAPI/issues/4502, authorizing a targeted reseed + restart after pipeline 7000 failed (2026-08-22T22:28:36Z) and again after the 2026-08-25 readiness audit was corrected (2026-08-27T13:07:55Z, explicitly "supersedes" the first authorization). The actual gap was structural, not a lack of escalation: neither authorized reseed is confirmed executed in the issue record, no comment reports pipeline 7000 or 7800 passing, and — because HEROKU_PROD_API_KEY was an out-of-band static copy rather than a live vault read — even a correctly-executed reseed would only have fixed the symptom until the next rotation missed the same separately-named secret again. The gap that actually needed root-causing (a static-copy consumer invisible to the rotation runbook) was not identified until the dedicated 2026-08-29 investigation (https://github.com/raxx-app/TradeMasterAPI/issues/4544), roughly 36 hours after pipeline 7800's failure and the second authorization.

What went well

• Prod never received a bad deploy. Every failure happened before the git push to Heroku — the guard structure in deploy-prod.yaml (tag/confirm checks preceding any Heroku operation) worked exactly as designed. The credential was stale, not the authorization logic; prod stayed on its last good release throughout the entire 5-week window rather than receiving a broken or partial deploy.
• The ground-truth check was fast and unambiguous. Comparing /api/system/version's literal 1.10.0 against the codebase's known 2026-07-31 deletion of that hardcoded string immediately proved prod was frozen well before either visible pipeline failure — no guessing about whether this was a one-off transient error.
• The 2026-07-21 rotation's own documentation made root-causing fast. Its RCA explicitly recorded what it checked and why ("no static WP secret to update for this credential") — reading that single line against the existence of HEROKU_PROD_API_KEY immediately explained the mismatch once the two were placed side by side.

Remediation

This PR (https://github.com/raxx-app/TradeMasterAPI/pull/4550, closing https://github.com/raxx-app/TradeMasterAPI/issues/4544):

1. Retires the static HEROKU_PROD_API_KEY WP org secret. deploy-prod.yaml (deploy-raptor-prod, deploy-console-prod) and deploy-queue.yaml (deploy-prod) now load HEROKU_API_KEY from vault /MooseQuest/heroku/ at run time, identical to deploy-staging.yaml, deploy-velvet.yaml, and deploy-queue.yaml's own staging step. The tag/manual when: gate and in-step CI_COMMIT_TAG / DEPLOY_PROD_CONFIRM guards are preserved verbatim as the prod authorization boundary — only the credential-loading mechanism changed.
2. Makes the RELEASE_VERSION PATCH fail loud: captures the HTTP status code and a truncated response body on any non-2xx response and exits non-zero, instead of silently swallowing the error. Never prints a 2xx response body (which would echo all of the app's config vars, including secrets).
3. create-github-release falls back to CI_FORGE_TOKEN when the ops-bot App token cannot be minted (corrupt PEM, https://github.com/raxx-app/TradeMasterAPI/issues/4530), so a release-notes step failure can never mask a deploy that already succeeded. Note this fallback only guarantees the step stays green — CI_FORGE_TOKEN is documented elsewhere as read-only, so the fallback very likely still cannot create the actual GitHub Release (expect a logged non-fatal 403/404) until the PEM itself is repaired.
4. Docs swept in the same PR: docs/ops/runbooks/heroku-api-key-rotation.md (consumers table, static-copy sweep checklist item, dated rotation-log row), docs/ops/runbooks/heroku-api-key-drift-recovery.md (three-copies list, new symptom row), docs/architecture/adr/0135-woodpecker-ci-full-migration-phases3-5.md (dated amendment), docs/ops/runbooks/ci-woodpecker.md (dated amendment), and this RCA.

Action items

References

• Issue (this fix): https://github.com/raxx-app/TradeMasterAPI/issues/4544
• PR (this fix): https://github.com/raxx-app/TradeMasterAPI/pull/4550
• Follow-up context: https://github.com/raxx-app/TradeMasterAPI/issues/4502
• Related: ops-bot PEM corruption, https://github.com/raxx-app/TradeMasterAPI/issues/4530
• Prior rotation RCA: docs/incidents/2026-07-21-full-credential-rotation.md
• Prior billing-visibility RCA: docs/incidents/2026-07-25-console-billing-summary-unavailable.md
• Runbook: docs/ops/runbooks/heroku-api-key-rotation.md
• Runbook: docs/ops/runbooks/heroku-api-key-drift-recovery.md
• ADR: docs/architecture/adr/0135-woodpecker-ci-full-migration-phases3-5.md
• Woodpecker pipelines: 7000 (v1.13.0, failed), 7800 (v1.13.1, failed)