RCA — Console Billing summary shows "temporarily unavailable" in production

Incident ID: 2026-07-25-console-billing-summary-unavailable Date: 2026-07-25 Severity: SEV-3 (non-blocking degradation — single feature impact, no data loss, no customer-facing exposure; console.raxx.app is operator-only) Duration: Unknown start (present since deploy of #765/#770, first observed and fixed same session) — fix window ~90m from investigation start to PR open Blast radius: Operator (Kristerpher) — console.raxx.app /billing, /billing/dashboard, /billing/customers pages; no customer-facing impact Author: sre-agent

Summary

console.raxx.app/billing rendered "Billing summary is temporarily unavailable" on every load. The Console→Raptor internal call (GET /api/_internal/billing/summary) was being rejected with HTTP 401 by Raptor's FLAG_SESSION_AUTH_MIDDLEWARE before the route's own X-Admin-Service-Token check ever ran. Fix: add /api/_internal/billing/ to the middleware's exemption prefix list. Same for the sibling /api/_internal/billing/alert-config* routes and the /billing/dashboard sub-panels (net-new subscriptions, live sessions, highlights, MRR sparkline) which share the same upstream 401.

Timeline (all times UTC)

• ~14:00 — Investigation started per operator FIX-mode dispatch.
• 14:0x — Confirmed FLAG_BILLING_SUMMARY_API=1 on raxx-console-prod (page loads, JS-level "unavailable" state, not a flag-off 404).
• 14:0x — Confirmed ADMIN_SERVICE_TOKEN present and identical on both raxx-console-prod and raxx-api-prod — ruled out token mismatch.
• 14:0x — Confirmed RAPTOR_INTERNAL_BASE_URL=https://api.raxx.app on raxx-console-prod — flagged as a possible origin-guard issue per standing cross-service rule, but not yet the confirmed root cause.
• 14:14:24 UTC — Pulled heroku logs -a raxx-console-prod; found the actual exception: ERROR in billing_summary: billing_summary_service: Raptor returned HTTP 401 — {"error":"Authentication required","reason":"missing"} This is the exact response shape emitted by session_auth.py's app-wide before_request hook, not a connection failure and not the route's own 403 {"error":"forbidden"}.
• 14:15:56–57 UTC — Confirmed the same 401 pattern on 5 more Console→Raptor billing-dashboard calls (/subscriptions/net-new, /sessions/live, /highlights/week, /mrr/sparkline, /subscriptions/summary) — same root cause, wider blast radius than just /summary.
• ~14:2x — Read backend_v2/api/middleware/session_auth.py; confirmed _EXEMPT_PREFIXES covers /api/internal/ (no underscore) routes only. billing.py's Blueprint registers under /_internal/billing (WITH underscore) — never matched by any existing exemption.
• ~14:3x — Confirmed the deployed prod SHA (heroku releases -a raxx-api-prod, last successful Deploy 558c9eda, 2026-07-21) and current main HEAD both lack the exemption — this is not a regression from a specific recent deploy, it has been broken since the billing summary route shipped (#765/#770) and the exemption list was never updated in that PR.
• ~14:4x — Wrote regression test (test_internal_billing_session_auth_exemption_4351.py); confirmed all 4 assertions fail against the current code (401 session-auth rejection) and pass after adding the exemption.
• ~14:5x — Ran full session_auth test suite (93 tests) post-fix — no regressions.
• Fix opened as PR against main per ADR-0115 emergency hotfix path (branched from main HEAD, label hotfix).

Impact

• Users affected: 1 (operator; console.raxx.app is not customer-facing)
• User-visible symptoms: Billing summary panel and Billing dashboard sub-panels (net-new subscriptions, live sessions, weekly highlights, MRR sparkline) showed "temporarily unavailable" / empty-error states instead of data
• Data integrity: ok — no writes were affected, this is a read-path auth misconfiguration
• Revenue / billing: ok — underlying vendor_billing_snapshots/vendor_billing_fixed data and the billing-retention cron (separately exempted under /api/internal/jobs/) were unaffected. This was purely an operator-visibility gap, not a billing-correctness issue.

What went well

• Heroku logs immediately surfaced the exact exception text and JSON body from billing_summary.py — no ambiguity between misconfig, upstream 5xx, or connection failure once logs were pulled.
• The distinctive {"error":"Authentication required","reason":"missing"} body is a documented, named signature in session_auth.py's own code comments, referencing 4 prior incidents of the identical failure class (#3305, #3253, #4166, #4283) — this made root-cause identification fast once the log line was in hand.
• Vault + CF Access + Heroku credential chain (per docs/ops/runbooks/vault-proxy.md) worked cleanly for a same-session sre-agent fetch — no operator interaction needed to pull the Heroku API key.
• A regression test existed to write against (test_session_auth_public_exemptions_3305.py provided the exact pattern), keeping the fix mechanical and low-risk.

What didn't go well

• The initial hypothesis in the dispatch (bare api.raxx.app host misconfiguration per the standing cross-service rule) was plausible and worth checking first, but was not the actual root cause here — RAPTOR_INTERNAL_BASE_URL=https://api.raxx.app did reach Raptor successfully (CF Access + origin guard did not block it); the 401 came from inside Raptor's own middleware stack. Diagnosis time would have been shorter if logs were pulled before forming a hypothesis about the config value. (Both are still worth fixing — see action items.)
• This is the 5th recurrence of the identical failure class (new internal/service-token route added, _EXEMPT_PREFIXES/_EXEMPT_EXACT not updated in the same PR) despite 4 prior documented incidents and prior action items. The billing.py/billing_alert_config.py routes predate the most recent prior fix (#4283, 2026-07-19) but the pattern recurring a 5th time indicates the review-time reminder ("add to exemption list when adding a new machine-to-machine route") is not being enforced mechanically.
• RAPTOR_INTERNAL_BASE_URL=https://api.raxx.app on raxx-console-prod is a separate, real risk (bare hostname per the standing cross-service rule — internal calls should target the hashed herokuapp host) that happened to not be the proximate cause this time, but remains latent and should be corrected regardless.

Root cause analysis

• Contributing factor 1 — missing exemption entry: backend_v2/api/routes/billing.py (issue #765) and backend_v2/api/routes/billing_alert_config.py (issue #770) register Blueprints under /_internal/billing (with a leading underscore). session_auth.py's _EXEMPT_PREFIXES list only contains /api/internal/... (no leading underscore) entries for other machine-to-machine routes. The two prefixes are visually similar but never match on the underscore, so every request to /api/_internal/billing/* was intercepted by the app-wide session-auth before_request hook and rejected with 401 before the route's own _check_service_token() (X-Admin-Service-Token check) ever executed. This is a system gap, not a one-off typo: the PR that introduced the billing summary route did not update the exemption list, and no CI check catches new Blueprints that are missing from the exemption list.
• Contributing factor 2 — no mechanical enforcement of the "add new internal routes to the exemption list" rule: the code comments in session_auth.py document this exact failure class occurring 4 times previously (#3305, #3253, #4166, #4283), each fixed by a one-line addition to _EXEMPT_PREFIXES. Each fix was manual and reactive (discovered after the route was already broken in prod), not caught by a pre-merge check. The system has no automated guard equivalent to route-migration-parity (which does exist for the Alembic-migration failure class documented in docs/ops/runbooks/raptor.md) for this failure class.
• Contributing factor 3 (latent, not proximate cause): RAPTOR_INTERNAL_BASE_URL on raxx-console-prod is set to the bare public hostname https://api.raxx.app rather than the hashed herokuapp origin. Per the standing cross-service rule, internal Console→Raptor calls should target the hashed host to avoid depending on the public CF-fronted path for internal traffic. This did not cause this incident (CF Access + origin guard let the request through, and the 401 originated inside Raptor), but it is still a config drift worth correcting as a follow-up — it adds unnecessary dependency on Cloudflare routing/Access policy for a call that should be able to bypass it entirely.

Detection

• What alerted us: Operator-reported symptom (manual navigation to console.raxx.app/billing) — no automated alert fired.
• How long between cause and detection: Unknown — likely broken since the billing summary route shipped (#765/#770); no monitoring covers Console→Raptor internal-call failure rates today.
• How to detect faster next time: Sentry (or a dedicated synthetic probe) on Console's billing_summary / billing_customer_dashboard logger names would have caught the ERROR in billing_summary / ERROR in billing_customer_dashboard log lines immediately after the first deploy that exercised this path. See action items.

Resolution

• What was changed: Added /api/_internal/billing/ to _EXEMPT_PREFIXES in backend_v2/api/middleware/session_auth.py, with a code comment documenting the root cause and cross-referencing this incident (mirrors the existing comment style for #3305/#3253/#4166/#4283). Added backend_v2/tests/test_internal_billing_session_auth_exemption_4351.py (4 tests) as a permanent regression guard.
• Validation:
• Confirmed all 4 new tests fail against pre-fix code (reproducing the exact 401 rejection seen in prod logs).
• Confirmed all 4 new tests pass post-fix.
• Ran the full session_auth-related test suite (test_session_auth.py, test_session_auth_public_exemptions_3305.py, test_internal_jobs_session_auth_exemption.py, plus the new file) — 93/93 passed, no regressions to other exemptions.
• Post-deploy verification (to be completed by operator/CI after merge + deploy): heroku run -a raxx-console-prod -- curl -s https://<raxx-api-prod hashed host>/api/_internal/billing/summary -H "X-Admin-Service-Token: $ADMIN_SERVICE_TOKEN" should return HTTP 200 with the aggregated summary JSON, and console.raxx.app/billing should render real data (or a genuine empty state) instead of "temporarily unavailable".

Action items

Action items 3 and 4 were not filed as tracked GitHub issues in this session per the standing "fix, don't file" policy — the fix itself is landed in the same PR. Operator should decide whether to formally track items 2–4.

References

• Runbook: docs/ops/runbooks/session-auth-exemptions.md (new — see below)
• Runbook: docs/ops/runbooks/raptor.md (Heroku app reference, deploy-smoke-checklist pattern this incident's action item #3 would extend)
• Runbook: docs/ops/runbooks/vault-proxy.md (credential path used for diagnosis)
• Related incidents (same failure class): #3305 (docs/ops/incidents/2026-05-25-signup-challenge-store-miss.md is unrelated; see PR history for #3305/#3253/#4166/#4283 — no dedicated RCA docs exist for those, only PR descriptions)
• Code: backend_v2/api/middleware/session_auth.py, backend_v2/api/routes/billing.py, backend_v2/api/routes/billing_alert_config.py