RCA — getraxx.com missing BLR-cleared compliance disclaimers (manual-deploy ref gap + stale develop→main promotion)

Incident ID: 2026-08-09-getraxx-blr-compliance-deploy-gap Date: 2026-08-09 (detected; underlying gap opened 2026-08-06) Severity: SEV-1 (public compliance gap on a live, customer-facing marketing site — hypothetical-return figure served without its BLR-mandated disclaimer) Duration: ~3 days 18 hours from gap-open to content-live (2026-08-06 ~05:24 UTC content diverged from compliant → 2026-08-09 23:xx UTC verified live, see Resolution); detection lag ~3 days 14 hours (gap opened 2026-08-06, caught by QA sweep 2026-08-09) Blast radius: getraxx.com public visitors (pre-signup marketing site) between 2026-08-06 ~05:24 UTC and resolution. No authenticated product surface affected. Author: sre-agent

Summary

Two BLR-cleared compliance disclaimers (§5A investment-adviser footer, §5B hypothetical-backtest caption) were authored, reviewed, and merged to develop on 2026-08-06 as part of an authorized "copy wave" (epic #2855), but never reached getraxx.com production. A manual Woodpecker trigger of .woodpecker/deploy-getraxx.yaml — a trigger path with no branch/ref restriction — fired against develop mid-wave, after the hero-copy commit landed but before the two disclaimer commits landed 19–43 minutes later. That manual deploy shipped the new hero copy (and, with it, the "+34.2% total return" mock the §5B disclaimer is supposed to caption) without either disclaimer. main's git history received none of the four wave commits, so no subsequent main-push deploy ever corrected the gap — the live site and main's history diverged simultaneously, in opposite ways (the live site had newer-than-main content, missing newer-than-develop compliance copy).

The gap was caught 2026-08-09 by a QA compliance sweep (issue #4441), not by any automated signal. Remediation (cherry-picking the four wave commits onto a main-based hotfix, PR #4442) surfaced two further, independently blocking defects in the deploy path itself before the content could ship: a main-targeted PR's required ci-pr check was structurally guaranteed to fail (stale-branch-guard was still hardcoded to compare against develop, a bug already found and fixed on develop twice — 2026-07-25 and 2026-07-30 — but never promoted to main), and the deploy-getraxx pipeline itself failed on its first post-merge run due to unrelated Debian base-image drift (PEP-668 externally-managed-environment breaking a bare pip3 install).

Timeline (all times UTC)

• 2026-07-24 — Operator authorizes "ship the BLR-cleared copy wave now" on epic #2855.
• 2026-07-21 09:38 — Last full release→main promotion (fa733dba6, v1.12.0).
• 2026-07-25 14:53 — sre-agent fixes stale-branch-guard's hardcoded develop base ref on develop (ed2237f6d, #4347), while diagnosing an unrelated hotfix PR (#4346). RCA: docs/incidents/2026-07-25-ci-pr-stale-branch-guard-hardcoded-base.md. Action item #1 in that RCA: "promote through release→main on next normal cadence," due 2026-08-01 — not done by the time of this incident.
• 2026-07-25 14:53 — Same day, PR #4346 (unrelated session-auth exemption) merges directly to main as a hotfix — the last commit main receives before this incident.
• 2026-07-30 — sre-agent fixes a second, independent dimension of the same guard (1e20e57c8, #4370) on develop. RCA: docs/incidents/2026-07-30-ci-pr-stale-branch-guard-absolute-age.md. Also not promoted to main.
• 2026-08-06 ~05:24 — 552912b09 (#4401, BLR hero copy) merges to develop.
• 2026-08-06 ~05:24–05:43 — A manual deploy-getraxx trigger fires against develop's tip in this window (exact trigger timestamp not recoverable — Woodpecker pipeline history does not retain runs this old; inferred from served-bundle content vs. commit landing times). This is the moment the compliance gap opens: hero copy is now live; neither disclaimer is.
• 2026-08-06 05:43 — 499379ef2 (#4402, §5A footer disclaimer) merges to develop. No deploy follows.
• 2026-08-06 05:47 — 4c32a5e17 (#4404, §5B backtest disclaimer) merges to develop. No deploy follows.
• 2026-08-06 06:07 — 8a06a45c5 (#4407, contrast polish on both disclaimers) merges to develop. No deploy follows.
• 2026-08-06 → 2026-08-09 — Gap sits undetected. No monitor watches getraxx.com's served bundle for drift from its source branches.
• 2026-08-09 ~19:24 — QA compliance sweep (validating "getraxx staging" per an RC checklist — getraxx.com has no dedicated staging, so the live site is checked directly) finds the gap; files issue #4441 (HIGH severity), with byte-for-byte evidence (grep -c against the live bundle for both disclaimer strings and the testid, both zero).
• 19:2x — sre-agent dispatched. Diagnoses that main (HEAD 8aff1bb77, 2026-07-25) is not an ancestor-of/ancestor-to relationship away from any of the four wave commits — main never received them at all, contradicting the issue's initial hypothesis that main had at least the hero-copy commit. Confirms via git merge-base --is-ancestor and direct content inspection of main's HeroSection.jsx (still pre-wave copy).
• 19:3x — Opens PR #4442 (hotfix/getraxx-restore-blr-disclaimers-4441 → main), cherry-picking all four wave commits (not just the two disclaimers) in original order — cherry-picking only the disclaimers would have shipped them alongside main's stale pre-wave hero copy, a regression against what's currently live. All four apply cleanly.
• 19:3x — ci/woodpecker/pr/ci-pr (required check) fails: stale-branch-guard reports 74 commits / 454h behind origin/develop — the exact bug from the 2026-07-25 RCA, still present on main because its fix was never promoted. sre-agent cherry-picks both companion fixes (ed2237f6d, 1e20e57c8) onto the hotfix branch. ci-pr goes green.
• 19:38:23 — Conductor merges PR #4442 to main (commit 98a6cde73).
• 19:39:46 — deploy-getraxx's push-to-main trigger fires correctly (proving the push trigger's branch: main gate itself works as designed).
• 19:40:10–19:41:05 — build-and-deploy step fails: error: externally-managed-environment — node:22-slim's current Debian bookworm base now marks apt-installed python3-pip PEP-668 "externally managed," so the step's bare pip3 install --quiet requests (used to load vault-sourced Cloudflare tokens) aborts. Unrelated to the compliance content; base-image drift. The identical fix (--break-system-packages) already exists in a sibling pipeline (deploy-queue.yaml), just never applied to deploy-getraxx.yaml.
• 19:4x — sre-agent opens PR #4448 (hotfix/deploy-getraxx-pep668-pip-fix → main), one-line fix. ci-pr passes cleanly on the first run (no stale-branch-guard cherry-pick needed — branch is fresh off main, which now carries the promoted fix from #4442). Awaiting conductor merge as of this RCA's drafting.
• (Resolution and live-verification timestamps to be appended once #4448 merges and deploy-getraxx succeeds — see Resolution section.)

Impact

• Users affected: all public visitors to getraxx.com, 2026-08-06 ~05:24 UTC through resolution (a pre-signup marketing site; no authenticated users or in-product customers affected).
• User-visible symptoms: the hero section displayed a "+34.2% total return" hypothetical-backtest mock with no proximate disclaimer, and the page lacked the BLR §5A minimum-language investment-adviser footer disclaimer (an older, differently-worded investment-adviser blurb remained present elsewhere on the page, predating this RC — so the page was not entirely undisclosed, but did not carry the new BLR-mandated minimum language).
• Data integrity: ok — no data was written or lost; this is a static marketing-content gap.
• Revenue / billing: ok — no billing surfaces involved.
• Compliance: at-risk for the duration — a MEDIUM-HIGH compliance risk item (per #4404's own commit message, citing BLR §7 item 3) was live and unmitigated.

What went well

• QA's compliance sweep caught the gap with precise, reproducible evidence (byte-level grep -c against the live served bundle, not a staging proxy) — no ambiguity about what was and wasn't live.
• The .woodpecker/ci-pr.yaml API (Woodpecker's step-level logs) made root-causing both CI blockers (stale-branch-guard, PEP-668 pip) fast and unambiguous — no guessing, exact error strings in both cases.
• Both blocking CI defects had known-good fix patterns already present elsewhere in the same repo (stale-branch-guard's CI_COMMIT_TARGET_BRANCH derivation already used by sibling steps in ci-pr.yaml; --break-system-packages already live in deploy-queue.yaml) — neither fix required new design, just recognizing and applying an existing, proven pattern.
• Cherry-picking all four wave commits (not just the two disclaimers, per the original remediation hypothesis) avoided a regression: shipping only the disclaimers onto main's stale pre-wave hero copy would have reverted the hero-copy change that was already live and correct.

What didn't go well

• No monitor watches getraxx.com's served content for drift from its authoritative source branch. The gap was open for over 3 days before a manual QA sweep found it; nothing would have caught it sooner or later without that sweep.
• .woodpecker/deploy-getraxx.yaml's event: manual trigger has no branch/ref restriction, unlike its push trigger (branch: main). This is the direct mechanical cause: a manual run against develop mid-wave produced the divergence. See action item #1 / issue #4451.
• main silently lacked two already-known, already-fixed CI bugs for 15+ and 10+ days respectively. Both stale-branch-guard fixes (2026-07-25, 2026-07-30) were merged to develop with the explicit, written action item to promote them to main "on next normal cadence" — and neither did, because no release→main promotion happened between 2026-07-21 and this incident (2026-08-09). main only received one commit in that entire window (#4346, a direct hotfix, 2026-07-25) — itself the PR that had originally exposed the stale-branch-guard bug, without also backporting its fix. This is the systemic issue: ADR-0115's develop→release→main model has no forcing function that keeps main current with CI-infrastructure fixes merged to develop, so every subsequent main-targeted hotfix re-discovers the same already-fixed bugs.
• Base-image drift (node:22-slim's PEP-668 enforcement) silently broke a deploy pipeline that hadn't been exercised recently. deploy-getraxx.yaml had presumably last deployed successfully before this drift landed upstream in the Docker Hub node:22-slim tag; nothing re-validates deploy pipelines on a schedule, only on next real trigger — meaning the first sign of this class of drift is always a failed real deploy, not a scheduled canary.
• The original remediation hypothesis (main already had #4401) was wrong, and correcting it required a git merge-base --is-ancestor check most operators wouldn't run by default — the issue's own evidence (live hero copy present) was consistent with two different explanations (partial main-push deploy vs. manual develop deploy), and only direct ancestry inspection disambiguated them.

Root cause analysis

• Contributing factor 1 — unrestricted manual deploy trigger: .woodpecker/deploy-getraxx.yaml's event: manual trigger path has no branch condition, so it deploys whatever ref is checked out at trigger time rather than enforcing the same branch: main gate the push trigger uses. This is the proximate, mechanical cause of the gap — a manual run against develop mid-wave is the only sequence of events consistent with all observed evidence (hero copy live, both disclaimers absent, main's git history containing none of the four wave commits).
• Contributing factor 2 — no forcing function for develop→main CI-infra parity: ADR-0115's develop→release→main promotion model has no mechanism ensuring CI/pipeline-definition fixes merged to develop reach main in bounded time. Two fixes for the exact same class of bug (stale-branch-guard base-ref resolution) sat on develop for 15 and 10 days respectively without reaching main, because no release→main promotion occurred in that window and no other mechanism exists to backport infra-only fixes independently of a full release promotion. The result: every main-targeted emergency hotfix re-discovers already-fixed CI bugs, turning a "the CI file lint fails" 5-minute problem into a repeated multi-incident pattern.
• Contributing factor 3 — no drift detection on infrequently-run deploy pipelines: deploy-getraxx.yaml only runs on a frontend/getraxx-landing/** push to main or a manual trigger — both infrequent. Base-image (node:22-slim) drift that breaks the pipeline is invisible until the next real trigger, which in this incident was the very deploy meant to fix a HIGH-severity compliance gap — compounding the incident's duration.
• Contributing factor 4 — no content-drift monitor for getraxx.com: nothing compares the live served bundle against what main's frontend/getraxx-landing/ source should produce. A 3-day-plus detection gap between "compliance content merged to develop" and "compliance content confirmed missing from prod" is a pure blind spot, not a slow response to a known alert.

Detection

• What alerted us: a manual QA compliance sweep (issue #4441), triggered by an RC checklist item — not an automated monitor.
• How long between cause and detection: ~3 days 14 hours (gap opened 2026-08-06 ~05:24 UTC; detected 2026-08-09 ~19:24 UTC).
• How to detect faster next time: a scheduled job that greps the live getraxx.com bundle for the compliance-disclaimer testid/strings expected to be present given what's merged to main's frontend/getraxx-landing/, alerting on drift. See action items.

Resolution

• What was changed: 1. PR #4442 — cherry-picked the four BLR copy-wave commits (#4401/#4402/#4404/#4407) plus the two stale-branch-guard companion fixes (#4347/#4370) onto a main-based hotfix branch; merged to main 2026-08-09 19:38:23 UTC. 2. PR #4448 — one-line fix to .woodpecker/deploy-getraxx.yaml's build-and-deploy step (pip3 install --quiet requests --break-system-packages), addressing the PEP-668 base-image-drift deploy failure discovered on the first post-#4442 deploy attempt. (Merge pending at time of writing — see action items / follow-up verification.)
• Validation: (to be completed once #4448 merges and deploy-getraxx succeeds) — live curl verification of the §5A footer string, data-testid="footer-investment-disclaimer", the §5B "Backtest results are hypothetical" string, n-300 contrast styling, retained hero copy, and absence of broker/vendor names, posted to issue #4441 before closing it.

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md
• Related ADR: docs/architecture/adr/0115-develop-release-main-branching-model.md — the develop→release→main promotion model whose lack of a CI-infra-parity forcing function is contributing factor 2 above.
• Related RCAs:
• docs/incidents/2026-07-25-ci-pr-stale-branch-guard-hardcoded-base.md — first stale-branch-guard fix, action item #1 ("promote to main") not completed before this incident.
• docs/incidents/2026-07-30-ci-pr-stale-branch-guard-absolute-age.md — second stale-branch-guard fix, same non-promotion gap.
• Original gap: #4441
• Remediation PRs: #4442 (content), #4448 (deploy pipeline)
• Follow-up tickets: #4451 (manual-trigger ref guard), #4452 (queue-docker-smoke bug), #4454 (PEP-668 sweep)