RCA — WebAuthn validator boot regression on raxx-api-prod

Incident ID: 2026-05-21-prod-webauthn-boot-fail Date: 2026-05-21 Severity: SEV-1 Duration: 6m 18s (19:00:01 UTC first crash → 19:06:09 UTC rollback complete) Blast radius: raxx-api-prod web dyno; operator-only (CF Access perimeter active; no customer traffic) Author: sre-agent

Summary

Flipping FLAG_WEBAUTHN_REGISTRATION=1 + FLAG_AUTH_WEBAUTHN_LOGIN=1 on raxx-api-prod caused the web dyno to crash-loop at boot. The validator in create_app() raised RuntimeError: WebAuthn config validation failed — refusing to start: WEBAUTHN_RP_ID is not set even though WEBAUTHN_RP_ID=raxx.app was present in the Heroku config and visible to one-off dynos. The crash was caused by app.config.from_pyfile('config.py', silent=True) overwriting the correctly-mapped WEBAUTHN_RP_ID value with '' after from_mapping() set it — a code path that the PR #2590 regression tests did not exercise because they used the test_config code path, not the production from_pyfile path. The dyno was restored in 6 minutes via heroku config:unset.

Timeline (all times UTC)

• 07:24 — Release v32: WEBAUTHN_RP_ID=raxx.app, WEBAUTHN_ORIGIN=https://raxx.app, WEBAUTHN_RP_NAME=Raxx set on raxx-api-prod (no flags flipped; dyno stable).
• 18:59:51 — heroku config:set BOOTSTRAP_TOKEN_SIGNING_KEY=<64-hex> FLAG_WEBAUTHN_REGISTRATION=1 FLAG_AUTH_WEBAUTHN_LOGIN=1 -a raxx-api-prod
• 18:59:53 — Old web.1 received SIGTERM, exited cleanly.
• 19:00:01 — New web.1 boot attempt. create_app() raised: RuntimeError: WebAuthn config validation failed — refusing to start: WEBAUTHN_RP_ID is not set. Set WEBAUTHN_RP_ID=raxx.app in your environment.
• 19:00:11 — Second restart attempt, identical crash.
• 19:02:00 — SRE investigation started. heroku run python -c "import os; print(repr(os.environ.get('WEBAUTHN_RP_ID')))" returned 'raxx.app' confirming env var is present on the dyno.
• 19:04:00 — Root cause identified: from_pyfile('config.py', silent=True) overwrites app.config['WEBAUTHN_RP_ID'] with '' after from_mapping() sets it correctly. Confirmed via local reproduction.
• 19:06:09 — heroku config:unset FLAG_WEBAUTHN_REGISTRATION FLAG_AUTH_WEBAUTHN_LOGIN -a raxx-api-prod. Web dyno recovered.

Impact

• Users affected: 0 (CF Access perimeter; launch posture is operator-testing-only)
• User-visible symptoms: none
• Data integrity: ok
• Revenue / billing: ok

What went well

• Rollback was clean and fast (config:unset is instant; no code deploy needed).
• CF Access perimeter contained blast radius to operator only.
• The error message from the validator was precise and pointed directly at the right config key.
• heroku run one-off dyno check confirmed os.environ state quickly.

What didn't go well

• The same failure class (WebAuthn boot crash) recurred 24 hours after PR #2590 was merged to fix the staging variant. The fix existed in main but its test coverage had a gap.
• PR #2590's regression tests exercised create_app(test_config={...}) (the test code path), which calls app.config.from_mapping(test_config) and skips from_pyfile('config.py', silent=True). The production path was never tested.
• No pre-flip checklist existed to verify: (a) the fix is deployed to prod, and (b) the production boot path is smoke-tested.
• BOOTSTRAP_TOKEN_SIGNING_KEY was set in the same config:set call as the flag flip. That key had not yet been written to Infisical vault (operator note: still outstanding at rollback time).

Root cause analysis

• Contributing factor 1: from_pyfile silently overwrites critical config keys — create_app() calls app.config.from_mapping(...) at line 66 (correctly setting WEBAUTHN_RP_ID=os.environ.get(...)) and then calls app.config.from_pyfile('config.py', silent=True) at line 121. from_pyfile loads {instance_path}/config.py if it exists. If that file contains WEBAUTHN_RP_ID = '', Flask's config system silently overwrites the correctly-set value with ''. The silent=True parameter suppresses the FileNotFoundError when the file is absent, but does NOT suppress loading when the file IS present. Any file present in backend_v2/instance/config.py (e.g., from a prior local dev session that was accidentally included in the slug, or from a migration tool creating a partial config) would have triggered this overwrite.

• Contributing factor 2: PR #2590's test suite only covered the test code path — The tests in test_app_factory.py::TestWebAuthnConfigWithFlagEnabled::test_boot_succeeds_when_valid_config_and_flag_on called create_app(test_config={...}). With a non-None test_config, the factory skips from_pyfile and calls from_mapping(test_config) instead. This meant the from_pyfile → overwrite → crash scenario was never exercised in CI. The test passed because there was no from_pyfile call on that code path.

• Contributing factor 3: No prod deployment gate before flag flip — PR #2590 fixed the staging incident (2026-05-20) and was merged to main, which auto-deploys to staging. Prod requires workflow_dispatch + human approval per ADR-0020. It is possible that at the time of the flag flip (18:59:51 UTC 2026-05-21), the prod slug was still running a build that pre-dated the #2590 fix (the from_mapping fix was absent entirely). If so, WEBAUTHN_RP_ID was never mapped into app.config at all — the original failure mode from 2026-05-20. Either the prod slug lacked the fix, or instance/config.py triggered the overwrite. The evidence is consistent with both; the fix addresses both.

Detection

• What alerted us: Heroku dyno crash log surfaced immediately after config:set.
• Time between cause and detection: ~10 seconds (dyno boot attempt at 19:00:01, crash visible instantly in log stream).
• How to detect faster next time: a boot smoke test (e.g., heroku run python -c "from api import create_app; create_app()") before the flag flip would have surfaced this before dyno traffic was affected. Add this as a pre-flip runbook step.

Resolution

• What was changed: heroku config:unset FLAG_WEBAUTHN_REGISTRATION FLAG_AUTH_WEBAUTHN_LOGIN -a raxx-api-prod (rollback). No code change at incident time.
• Validation: web dyno came up cleanly. heroku ps -a raxx-api-prod showed web.1: up.
• Follow-on fix (this PR): added if test_config is None: re-pin block in create_app() that re-reads WEBAUTHN_RP_ID, WEBAUTHN_RP_NAME, WEBAUTHN_ORIGIN from os.environ after the from_pyfile call, so no instance/config.py can overwrite them. Added regression tests that exercise the production code path and the override scenario.

Action items

References

• Runbook: docs/ops/runbooks/raptor.md (to be updated with pre-flip smoke step)
• Prior incident: docs/incidents/2026-05-20-staging-webauthn-boot-fail.md
• Fix PR: fix(api): WebAuthn validator regression on flag-gated boot — RCA + smoke gate
• ADR-0005: WebAuthn RP ID policy
• ADR-0012: RP ID isolation (raxx.app vs console.raxx.app)
• ADR-0020: Prod deploy gate (workflow_dispatch + human approval)