Session-auth exemption gap runbook

System: Raptor FLAG_SESSION_AUTH_MIDDLEWARE (backend_v2/api/middleware/session_auth.py) Owner: operator / sre-agent Last incident: 2026-07-25 (see docs/incidents/2026-07-25-console-billing-summary-unavailable.md) Last reviewed: 2026-07-25

Why this runbook exists

This is a recurring failure class, not a one-off bug. As of 2026-07-25 it has happened 5 times:

Every occurrence has the same shape: a new route is added that authenticates via something other than the raxx_session cookie (X-Admin-Service-Token, HMAC webhook signature, public pre-auth token), but _EXEMPT_PREFIXES / _EXEMPT_EXACT in session_auth.py is not updated in the same PR. The app-wide before_request hook intercepts the request and returns 401 before the route's own auth check ever runs. The route is not actually vulnerable — it has its own auth — but it is completely unreachable by its intended caller until the exemption is added.

How to tell it's broken

• Symptom 1: A machine-to-machine or pre-auth route that has its own token/HMAC check (not @require_session) returns HTTP 401 with body {"error":"Authentication required","reason":"missing"} for every request, even ones carrying a correct service token / HMAC signature / no cookie by design.
• Symptom 2: The caller (Console, a cron job, a webhook sender, a browser hitting a pre-auth page) logs an upstream 401 that does NOT match the route's own documented error shape (e.g. billing.py's own rejection is {"error":"forbidden"} at 403 — if you see {"error":"Authentication required","reason":"missing"} at 401 instead, session-auth intercepted before the route ran).
• Symptom 3: The route works fine locally / in tests with FLAG_SESSION_AUTH_MIDDLEWARE unset or 0, but fails in any environment where the flag is 1 (staging and prod have had it on since 2026-05-28).

How to diagnose (in order)

1. Confirm the exact response body and status code from the failing route (Heroku logs, or heroku run -a <app> -- curl -s -w '\n%{http_code}\n' <route>). {"error":"Authentication required","reason":"missing"} at 401 = session-auth rejection. Any other body/status = the route's own logic ran; look elsewhere.
2. Confirm FLAG_SESSION_AUTH_MIDDLEWARE is 1 on the target app: heroku config:get FLAG_SESSION_AUTH_MIDDLEWARE -a <app>.
3. Read backend_v2/api/middleware/session_auth.py _EXEMPT_PREFIXES and _EXEMPT_EXACT. Check whether the failing route's path is covered by any entry — character for character, including leading underscores. The 2026-07-25 incident was caused by /api/internal/ (no underscore) entries not matching /api/_internal/ (with underscore) — these look nearly identical when skimmed.
4. Confirm the route's own Blueprint url_prefix (grep the route file for Blueprint(...)) and cross-reference against the app factory's /api prefix rule (see docs/ops/runbooks/raptor.md §Blueprint authoring convention) to get the exact final path.

Known failure modes

Failure mode A: new internal/service-token route missing from exemption list

Symptom: New route added in a PR; authenticates via X-Admin-Service-Token, HMAC, or is intentionally pre-auth; returns session-auth's 401 for every caller.

Cause: _EXEMPT_PREFIXES/_EXEMPT_EXACT not updated in the same PR that added the route.

Fix: 1. Identify the exact final path (see diagnosis step 4). 2. Add the path (or its shared prefix) to _EXEMPT_PREFIXES (or _EXEMPT_EXACT for single exact paths) in backend_v2/api/middleware/session_auth.py. Scope the prefix as tightly as possible — do not add a broad prefix that would also exempt unrelated user-facing routes under the same path segment. 3. Add a regression test following the pattern in backend_v2/tests/test_internal_billing_session_auth_exemption_4351.py or backend_v2/tests/test_session_auth_public_exemptions_3305.py: assert the route does NOT return the session-auth 401 body when FLAG_SESSION_AUTH_MIDDLEWARE=1 and no cookie is sent. 4. Document the addition with a comment explaining why the route is exempt and what its own auth mechanism is (matches the existing style in the file — every entry has a 2–4 line comment citing the issue/incident).

Verification: New test passes; heroku run / curl against the target app returns the route's own expected status (200/403/404 per the route's own logic), never the session-auth 401 body.

Failure mode B: exemption added on develop but prod is still on main/release

Symptom: A fix for this failure class was merged (visible in git log --all) but the bug still reproduces against the deployed prod SHA.

Cause: Fixes normally land on develop first (per ADR-0115) and are promoted develop→release→main on a cadence, not immediately. If a route was added and exempted in the same develop PR, but a different internal route was added earlier and is still waiting on promotion, prod can lag by weeks. This is not a bug in the fix — it is normal cadence — but it means "the fix is merged" does NOT mean "the fix is in prod." Always check the deployed SHA (heroku releases -a <app>, find the last Deploy <sha> entry, git show <sha>:backend_v2/api/middleware/session_auth.py) before declaring an incident resolved.

Fix for a live production incident: use the ADR-0115 emergency hotfix path (branch from main HEAD, PR targeting main directly with label hotfix, then cherry-pick to release and develop after merge) rather than waiting for the next scheduled promotion.

Verification: git merge-base --is-ancestor <fix-sha> origin/main returns true; deployed SHA on the target Heroku app matches or descends from the fix.

Emergency stop

This middleware is a security control (session-cookie enforcement for user-facing routes) — do NOT disable FLAG_SESSION_AUTH_MIDDLEWARE to work around an exemption gap. That would remove auth enforcement from every user-facing route, not just the broken one. The correct emergency mitigation for a broken internal route is always to add the specific exemption, never to flip the flag off.

If a genuine emergency requires bypassing session auth app-wide (e.g. the middleware itself has a bug rejecting valid cookies), that is a SEV-1/SEV-2 and requires operator sign-off before flipping FLAG_SESSION_AUTH_MIDDLEWARE=0 — see docs/ops/runbooks/raptor.md §Session auth middleware for the CF Access interaction warning (CF Access must remain the gate if this flag is off).

Escalation

Wake the operator when: - The failing route's own auth mechanism is unclear or undocumented (don't guess at what prefix to exempt). - The exemption would need to cover a broad prefix that might also expose a user-facing route not intended to be public. - This is the 6th+ recurrence and a systemic fix (CI lint, see action item #3 in the 2026-07-25 RCA) has still not been scoped.

Cross-references

• Middleware: backend_v2/api/middleware/session_auth.py
• Regression tests: backend_v2/tests/test_session_auth.py, backend_v2/tests/test_session_auth_public_exemptions_3305.py, backend_v2/tests/test_internal_jobs_session_auth_exemption.py, backend_v2/tests/test_internal_billing_session_auth_exemption_4351.py
• Blueprint authoring convention (url_prefix footgun): docs/ops/runbooks/raptor.md
• Emergency hotfix path: docs/architecture/adr/0115-develop-release-main-branching-model.md
• RCA: docs/incidents/2026-07-25-console-billing-summary-unavailable.md