RCA — Cron re-verify-and-enable batch (#4477/#4478/#4479): all three found real drift, none enabled

Incident ID: 2026-08-12-cron-reverify-batch-e2e-warm-waf Date: 2026-08-12 Severity: SEV-3 (non-blocking degradation — three WP crons remain silently disabled; no customer-facing impact, no data-integrity impact) Duration: N/A — this is a verification pass, not a live-incident recovery. Underlying drift durations: e2e-smoke-nightly lockfile has been broken since 2026-05-27 (PR #2917); warm_historical_bars_1min.py's missing init_engine() bootstrap has been latent since 2026-05-29; waf-synthetic-probe verification was not completed (blocked, not drift). Blast radius: CI/ops tooling only. e2e-smoke-nightly, historical-bars-warm-nightly, and waf-synthetic-probe WP crons (ids 18, 20, 25) remain disabled after this pass, same as before it. No production traffic affected. Author: sre-agent

Summary

Three sibling cards (#4477, #4478, #4479 — split from #4469) asked sre-agent to re-verify and enable three WP crons that PR #4091 (2026-07-08) claimed were "functioning correctly" but never actually enabled (enabled:false default gotcha, documented in docs/ops/runbooks/ci-woodpecker.md §v3.x API changes). Following the

4450/#4429 precedent (never trust a month-old "functioning correctly"

claim blind — re-verify against current app behavior via a real scoped trigger), all three crons were triggered via POST /api/repos/1/cron/{id} (never a bare manual dispatch) and their results inspected in full.

Result: none were enabled. Two found genuine, pre-existing product bugs (not new drift, but never actually validated by PR #4091 either): e2e-smoke-nightly's Playwright job fails on npm ci due to an incomplete frontend/raxx-next/package-lock.json; and historical-bars-warm-nightly's warm job false-succeeds every time because warm_historical_bars_1min.py never bootstraps the SQLAlchemy engine before using it. Both are out of sre-agent's edit remit (product application code) and were filed precisely for feature-developer (#4488, #4489) rather than fixed directly. The third (waf-synthetic-probe) was independently confirmed side-effect-free but the live trigger itself was blocked by this session's own safety classifier as a security-adjacent production action; escalated to the operator (comment on #4479) rather than worked around.

A fourth, tangential finding (ci-boundary unexpectedly co-triggered by both cron endpoints despite its own when: filter not matching) was filed as #4490 and partially cross-referenced against an already-tracked same-day RCA (docs/incidents/2026-08-12-ci-boundary-gitleaks-scan-scope.md).

Timeline (all times UTC)

• 07:30 — Baseline GET /api/repos/1/cron/18 and /20: both enabled:false, next_exec frozen at 2026-07-08 creation-time values (zero fires confirmed for both).
• 07:37 — Cron 18 (e2e-smoke-nightly) triggered via POST /api/repos/1/cron/18 against develop HEAD (c56bf7d4d). WP queue was idle (0 pending/running) beforehand.
• 07:41 — Pipeline #5517 finishes: api-smoke PASS, e2e-smoke FAIL (npm ci EUSAGE — see Root cause 1). ci-boundary (unexpectedly included) also fails independently on gitleaks-report-upload (already-tracked vault-secret gap, see below).
• 08:07 — Cron 20 (historical-bars-warm-nightly) triggered via POST /api/repos/1/cron/20 against develop HEAD (139b5043d). WP queue idle beforehand.
• 08:11 — Pipeline #5524 finishes: historical-bars-warm-nightly reports success, but log inspection shows a real heroku run against raxx-api-prod (dyno run.9076) hit get_engine() called before init_engine(), caught and silently downgraded to "no symbols to warm" — a false success (see Root cause 2). ci-boundary again unexpectedly included and independently fails the same way as above.
• 08:15–08:25 — waf-synthetic-probe (cron 25) baseline confirmed (enabled:false, frozen next_exec). scripts/waf/probe.py independently reviewed and confirmed side-effect-free (12 GET flows + 1 intentionally-malformed POST against a vault auth endpoint, no token minted) — matches the card's own grooming-pass audit.
• 08:26 — Attempted the scoped trigger POST /api/repos/1/cron/25: blocked by the session's own auto-mode safety classifier (production/ security-adjacent action). Not worked around — escalated to operator instead, per SRE anti-pattern guidance ("never bypass a gate").
• 08:30–09:00 — Root-caused both real findings against the current develop tip (lockfile inspection + git grep cross-reference against sibling one-off commands), filed #4488 and #4489 with exact evidence and suggested fixes, filed #4490 for the ci-boundary co-trigger anomaly, cross-referenced #4490 against the pre-existing 2026-08-12 ci-boundary RCA to avoid a duplicate AWS-credential provisioning ask, posted evidence comments on #4477/#4478/#4479.

Impact

• Users affected: none.
• User-visible symptoms: none.
• Data integrity: ok. (historical_bars_1min cache staleness is unchanged by this pass — it was never being warmed by this path regardless of the cron's enabled state, per Root cause 2.)
• Revenue / billing: ok.
• Operational cost: three WP crons remain dark pending feature-developer fixes (#4488, #4489) and operator follow-through on the blocked waf-synthetic-probe trigger (#4479).

What went well

• The #4450/#4429 "never trust a month-old claim blind, re-verify via a real scoped trigger" precedent caught two genuine, previously-unvalidated bugs before either cron was armed — exactly the failure mode the precedent was designed to prevent (a cron that "runs green" on schedule while doing nothing or blocking on every fire, generating either false confidence or daily alert noise).
• The scoped single-workflow cron-trigger endpoint (POST /api/repos/1/cron/{id}) again avoided the ~65-workflow fan-out a bare manual dispatch would have caused, keeping WP agent load minimal during a live CI day (confirmed queue was idle before both triggers).
• Both root causes were fully diagnosable from the WP step log alone (decoded from the v3 log API's base64 line format) without needing local reproduction — though the frontend lockfile finding was cross-checked locally in the exact CI image (docker run ... playwright:v1.60.0-jammy) to confirm the platform-entry gap precisely and rule out a false lead.
• Sibling one-off command comparison (git grep -n "init_engine(" backend_v2/api/commands/) turned a vague "it silently no-ops" log line into an exact, three-line, copy-pasteable fix recommendation with three precedent files as reference.
• The blocked waf-synthetic-probe trigger was escalated rather than worked around, consistent with SRE policy on security-adjacent actions — even though the probe script itself was independently confirmed safe.

What didn't go well

• PR #4091 (2026-07-08) declared all of e2e-smoke-nightly, historical-bars-warm-nightly, and (per #4479) waf-synthetic-probe "functioning correctly" without ever confirming the underlying pipelines produced correct output — only that they didn't crash outright. historical-bars-warm-nightly's false-success (exit 0, zero rows written) would have passed any check that only inspected the step's exit code, which is exactly what likely happened.
• backend_v2/tests/test_warm_historical_bars_1min_command.py monkeypatches cmd.get_engine directly in every test, which means the real init_engine()/get_engine() singleton path — the exact thing that's broken — has zero test coverage. A test suite that patches around the one thing that's actually broken gives no useful signal.
• The cron-trigger endpoint's "scoped to a single workflow" behavior (established as a hard requirement by the #4429 precedent, specifically to avoid wasting CI capacity and confusing failure signals) did not hold for either cron 18 or cron 20 in this pass — ci-boundary rode along both times despite its when: block not matching. This contradicts the documented precedent and needs its own investigation (#4490) before it can be relied on as fully single-workflow-scoped again.

Root cause analysis

• Contributing factor 1 — frontend/raxx-next/package-lock.json was never regenerated after a partial manual pin. PR #2917 (2026-05-27) added @rollup/rollup-linux-x64-gnu to optionalDependencies in package.json to fix a macOS-generated-lockfile gap, but the lockfile itself was never regenerated via npm install to add matching entries for the ~40 other platform variants that rollup@4.60.4 and unrs-resolver declare as their own optionalDependencies. This has been silently broken for npm ci under npm ≥11 (the version in the mcr.microsoft.com/playwright:v1.60.0-jammy image) since that PR — it simply never surfaced because nothing exercised npm ci in that exact image/npm-version combination until this cron was triggered for real.
• Contributing factor 2 — warm_historical_bars_1min.py was written without the one-off-command DB bootstrap pattern used by every sibling command in the same directory. init_engine() must be called once per process before get_engine() will work; three other commands in backend_v2/api/commands/ do this correctly via an explicit init_engine(db_url) call guarded by except RuntimeError. This one does not, and its own error handling (except Exception around the active-strategies query) converts the resulting RuntimeError into an indistinguishable "no symbols" case rather than a fatal error — a design that specifically prevents this exact bug from ever being visible in the pipeline's exit code.
• Contributing factor 3 — no dependency/coupling check between "this command's only real invocation path is heroku run" and "does this command actually bootstrap what heroku run skips." Nothing in CI, code review, or the command's own tests would have caught this before a real heroku run was attempted — which, per PR #4091's stale claim, may never have actually happened before this pass.

Detection

• What alerted us: none of these were alert-driven — this was a scheduled/assigned re-verification task (#4477/#4478/#4479), not a reactive incident response. Both real bugs were caught by doing the verification the card asked for (a real scoped trigger) rather than trusting the prior "functioning correctly" claim.
• How long between cause and detection: e2e-smoke-nightly lockfile — ~2.5 months (2026-05-27 → 2026-08-12). historical-bars-warm-nightly init bug — ~2.5 months (2026-05-29 → 2026-08-12), though the command had effectively zero real invocations in that window (cron never enabled; no other caller exists).
• How to detect faster next time: the #4488/#4489 suggested-fix sections each include a verification step that re-triggers the specific cron and inspects real output (not just exit code) — once fixed, the now-enabled crons' own daily/weekly cadence becomes the ongoing detector. For the lockfile class of bug specifically: npm ci (not npm install) in every frontend CI/build pipeline, in the same image the "real" pipelines use, would have caught this the same day it was introduced.

Resolution

• What was changed: nothing in application code or CI config — by design, both real findings are outside sre-agent's edit remit (product application code: frontend/raxx-next/package-lock.json, backend_v2/api/commands/warm_historical_bars_1min.py). Filed precisely instead:

• 4488 — frontend/raxx-next lockfile regeneration, with exact

  reproduction, root cause, and a 4-step suggested fix.

• 4489 — warm_historical_bars_1min.py missing init_engine()

  bootstrap, with exact reproduction, root cause, the 3-line fix pattern copied from three sibling commands, and a test-coverage hardening suggestion.

• 4490 — ci-boundary cron-trigger co-scoping anomaly (novel;

  investigation only, not yet root-caused) + a now-corrected duplicate cross-reference to the already-tracked gitleaks-report-upload vault-secret provisioning gap (docs/incidents/2026-08-12-ci-boundary-gitleaks-scan-scope.md action item #1, already owned by operator, due 2026-08-19).

• Evidence comments posted on #4477, #4478, #4479 with pipeline links, decoded log excerpts, and explicit "left enabled:false" verdicts.
• Validation: N/A for this pass (no code changed). All three crons confirmed still enabled:false via a final GET after the triggered runs, matching their pre-trigger baseline — no unintended state change.
• Cards #4477, #4478, #4479 were deliberately not closed — each card's own AC requires either a clean verification-then-enable, or a fix-forward-then-re-verify-then-enable before closing, and sre-agent could do neither the fix (out of remit) nor the closing verification (would require enabling a known-broken cron). This is a deliberate deviation from the batch dispatch's default "close on completion" expectation — flagged explicitly to the operator.

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md (§v3.x API changes "Cron enabled field"; updated in this pass with a new Known failure mode)
• Precedent RCA: docs/incidents/2026-08-09-tickets-e2e-smoke-mailto-drift.md (#4429/#4450 — established the "never trust a month-old claim blind" pattern and the scoped single-workflow trigger method this pass reused)
• Related, same-day RCA: docs/incidents/2026-08-12-ci-boundary-gitleaks-scan-scope.md (already-tracked root cause for the gitleaks-report-upload vault-secret gap surfaced tangentially in this pass)
• Issues: #4477, #4478, #4479 (this pass's cards), #4488, #4489, #4490 (filed from this pass), #4070 / #4091 (original root cause), #3126 (parent epic)