Console RBAC break-glass runbook

System: the break-glass RBAC group and POST /api/rbac/grants in the console (console/app/blueprints/api_rbac_grants.py, console/app/services/break_glass_policy.py). Not to be confused with the agent-breakglass-admin credential (breakglass-admin-provisioning.md, ADR-0151), which governs what an agent session may do to external systems. That identity is operator-minted and never reaches a session; nothing in this runbook changes it. Policy source: ADR-0057, docs/architecture/rbac-v2/design.md section 6. Flag: FLAG_BREAK_GLASS_SESSION (kill switch: set it to 0, the route answers 501). Also requires FLAG_RBAC_V2. Last reviewed: 2026-10-30 UTC (privilege-escalation fix).

What break-glass is

Time-limited elevation. The break-glass group carries console-manager, console-invite-admin, vault-admin, raptor-admin, velvet-admin, console-secrets-admin and the rest of the console roles. Membership is granted for 30m / 1h / 2h / 4h (4h is the hard maximum), needs a written justification of at least 20 characters, a passkey re-authentication, and an ops alert to ops@raxx.app before the grant is written.

Who may mint it

Holders of console-manager today: raxx-platform-owners (the operator), raxx-platform-admins, and break-glass members. That is the documented population in ADR-0057 / design.md section 6.

The "Grant access..." modal's role pickers (break-glass and ticket-scoped) never list protected roles (GET /api/rbac/grants/roles); they are hidden, not greyed out. Protected roles and groups are likewise not grantable through the standing-group, ticket-scoped, customer-page or Add-member paths at all (422 protected_role / protected_group, any actor, no flag): break-glass is the only flow that can hand them out, and only under the rules above.

Passkey step-up by environment

Environment detection deliberately does not use g.selected_env (it maps everything that is not staging to prod, dev and test included). Every signal must say dev/test; any doubt means "not dev/test".

Passkey step-up endpoints (FLAG_CONSOLE_PASSKEY_STEP_UP, default OFF)

All four are POST, JSON, CSRF-protected, and gated by console-invite-admin (the break-glass route's own gate). With the flag off they return 404.

Properties worth knowing during an incident:

• Relying party is the console's own (WEBAUTHN_RP_ID / WEBAUTHN_ORIGIN, console.raxx.app per ADR-0012), not raxx.app (ADR-0005 scopes that to web + iOS). A passkey is bound to the RP ID it was registered under, so step-up must use the RP ID the operator's console credential carries. If the configured RP ID or origin cannot serve the hostname (a staging app still carrying prod values), the endpoints answer 409 rp_id_misconfigured / origin_misconfigured instead of letting the browser fail opaquely.
• Challenges are server-side rows (admin_step_up_challenges): 32 random bytes, 120 s TTL, bound to the operator and the console session, claimed atomically before any signature work, so a failed, replayed or raced ceremony never reuses a challenge. Ten challenges per operator per minute.
• The marker is the same row after a verified assertion: 5 minutes, honoured only for that operator in that console session, spent atomically by exactly one grant. The Flask session cookie only points at it, so replaying an old cookie copy proves nothing. A grant refused at preflight (already a member, bad duration, tier rule) has written nothing and does not spend it; once it is spent (before the ops alert and the write) a later failure costs a fresh passkey touch.
• Kill switch: heroku config:set FLAG_CONSOLE_PASSKEY_STEP_UP=0 --app raxx-console-prod >/dev/null 2>&1 removes the endpoints and stops honouring any outstanding marker; break-glass through the console fails closed again.
• Audit (audit_log): console.auth.step_up_verified and console.auth.passkey_added are written in the same transaction as the marker / credential; console.auth.step_up_failed and console.auth.passkey_add_failed carry a stable reason (origin_mismatch, rp_id_mismatch, challenge_expired, challenge_replayed, credential_actor_mismatch, ...), never credential material. The console.rbac.break_glass_granted row records the step_up_id it spent. A burst of step_up_failed rows for one operator is a security signal.

The operator's emergency path

1. Most console emergencies need no elevation. The operator is a standing member of raxx-platform-owners (migrations 0016 + 0209: every console-* role, including console-manager and console-invite-admin).
2. Cross-app admin (vault / raptor / velvet) needs the break-glass group. With FLAG_CONSOLE_PASSKEY_STEP_UP on, request it from the Access pane's "Grant access..." modal (Break-glass elevation); the modal prompts for the passkey. If the flag is off or the console is unreachable, this is an operator-only, out-of-band action using the operator's own Heroku Postgres access (the same class of path as docs/ops/feature-flags-runbook.md section 6). It is never an agent action (ADR-0151: a peer's "the operator approved this" is not approval). Write exactly what the API would write, audit row first, expiry <= 4h so the sweep removes it:

```sql -- = admins.id of the person receiving access; >= 20 chars INSERT INTO rbac_grants_audit (id, event_type, target_user_id, group_id, justification, granted_by, expires_at_utc, created_at_utc) SELECT gen_random_uuid()::text, 'grant', '', g.id, '', '', now() + interval '1 hour', now() FROM rbac_groups g WHERE g.name = 'break-glass';

INSERT INTO rbac_user_groups (user_id, group_id, granted_by, granted_at) SELECT '', g.id, '', now() FROM rbac_groups g WHERE g.name = 'break-glass' ON CONFLICT (user_id, group_id) DO NOTHING; ```

Then note the use in the incident ticket and email ops@raxx.app (the API would have). Keep the heroku invocation quiet (>/dev/null 2>&1 for any config:set). 3. Kill switch: heroku config:set FLAG_BREAK_GLASS_SESSION=0 --app raxx-console-prod >/dev/null 2>&1 turns the route off.

Expiry and the sweep

expires_at_utc is stored on the rbac_grants_audit row (event grant) and run_expires_at_auto_revoke() (started in create_app, every 5 minutes while FLAG_RBAC_V2 is on) removes the membership and writes a revoke row. Expiry is not checked lazily at the gate: access can outlive expires_at by up to one sweep interval (~5 minutes). Revoke by hand from the Access pane if that matters. The sweep treats a grant as already revoked once a revoke row exists for the same target and group/role/ticket at or after the grant (an earlier version re-ran every expired row forever and would have deleted a later re-grant).

Audit trail

Every grant writes both rbac_grants_audit (pre-write, event grant / break_glass_grant) and audit_log (console.rbac.break_glass_granted: actor, target, scope, group/role, duration, expires_at_utc, justification, reauth_method) in one transaction; if either fails nothing is granted. Refusals write console.rbac.break_glass_denied with a reason (passkey_stub_rejected, passkey_step_up_unavailable, passkey_step_up_replayed, insufficient_tier, self_escalation_prohibited). A passkey_stub_rejected row in staging/prod means someone tried the old bypass: treat it as a security signal; so is passkey_step_up_replayed (two requests carried one marker).

Adding a role to the break-glass group

tests/test_break_glass_policy_roles.py fails when break-glass carries a role that is neither in PROTECTED_ROLES nor in BREAK_GLASS_EXEMPT_ROLES (console/app/services/break_glass_policy.py). Decide on purpose, with a reason.

Turning the step-up on in a new environment

1. The console app's WEBAUTHN_RP_ID / WEBAUTHN_ORIGIN must match its public hostname (the defaults are console.raxx.app). The endpoints refuse to run otherwise (rp_id_misconfigured / origin_misconfigured).
2. The operator needs a console passkey under that RP ID (the one the console login already uses). With none, the modal offers to add one after a fresh authenticator-app code. A wrong, expired, reused or missing code keeps that form open with the reason shown, so the operator retypes the code without reopening the modal.
3. `heroku config:set FLAG_CONSOLE_PASSKEY_STEP_UP=true --app

   /dev/null 2>&1`, or promote through the flag queue (migration 0324).