Raxx · internal docs

internal · gated ↑ index

Console Cockpit Pattern — Operator Runbook

Owner: Operator (Kristerpher) + agents Last updated: 2026-05-03 Anchor epic: #871 Anchor card (this doc): #877

What this runbook covers

The Raxx Console operates in a "two cockpits, one cabin" model — console-staging is the daily-driver, console-prod is the wall-display backup, and there is no session-level env switcher. The host you visit determines the env. This document captures that model + the promotion flow that replaced the old dropdown.

If you reach for the old "switch env" dropdown — it's gone. See §3 for the replacement flow.

§1 — Single console deployment per environment

Two separate Heroku apps:

URL Heroku app Postgres DB Purpose
console.raxx.app raxx-console-prod d860pa1ulpacce Production cockpit (wall display, attestation surface)
console-staging.raxx.app raxx-console-staging d76no3asiepgbk Daily-ops cockpit (where operators live)

Both apps deploy the same code from main. Staging auto-deploys on every push to main. Prod requires explicit dispatch (gh workflow run deploy-console.yml -f environment=production) OR the console version manager (§3).

Both use separate Postgres add-ons on the same RDS cluster — different logical DBs, no cross-env data leak. The "Recent rotations" / "Recent deploys" panels query their own app's DB only.

Verified the separate-DB model on 2026-05-03 03:50 UTC: heroku config:get DATABASE_URL on each app returns different DB names.

§2 — Banner + env pill (post-env-switcher)

The banner above the dashboard reads PRODUCTION (red) or STAGING (purple). It is derived from the hostname at request time — request.host matches against the known mapping. The pill is not clickable. There is no dropdown.

Implementation: - console/app/middleware/env_guard.py writes g.env from request.host (the Flask request hook ships in PR #919; was previously read from a session-stored value). - The audit envelope still carries selected_env for backward compatibility; the value is the hostname-derived env, not a session toggle.

If you want to operate against the other env, open a different browser tab to the other host.

§3 — Promoting console code (staging → prod)

The previous behaviour: env-switcher dropdown changed your session env, then "Deploy" routed to whatever surface in that session. Removed in PR #919 because surface_id alone determines target env (verified by PR #889 / #873 — every entry in SURFACE_WORKFLOW_MAP carries an explicit target_env).

The new behaviour:

  1. Open console-staging.raxx.app/admin/console-versions (in flight as #874 / #875 — see §6 for current break-glass path)
  2. The page shows two columns: "console-staging current SHA" and "console-prod current SHA"
  3. Click "Promote staging → prod" → confirmation modal → type PROMOTE to confirm + 6-digit TOTP code
  4. The console-prod deployment is triggered automatically targeting the same main SHA staging is running
  5. Audit row written: console.version.promoted with {from_sha, to_sha, requested_by_admin_id, promoted_at_utc} (sub-card #876)

Until #874 / #875 ship, use the break-glass path in §6.

§4 — Where deploys actually go

Each surface_id has a fixed workflow + target env baked into console/app/services/deploys.py:SURFACE_WORKFLOW_MAP. Six current surfaces:

surface_id workflow target_env Heroku app
api-prod deploy-heroku.yml production raxx-api-prod
api-staging deploy-heroku.yml staging raxx-api-staging
console-prod deploy-console.yml production raxx-console-prod
console-staging deploy-console.yml staging raxx-console-staging
getraxx deploy-customer-docs.yml production CF Pages (raxx-getraxx)
raxx-mockups deploy-customer-docs.yml production CF Pages (raxx-mockups)

The map is the canonical env-binding authority (PR #889 #890). The probe layer queries /actions/workflows/{file}/runs filtered by workflow file + dispatch input environment to pin builds to the right tile.

§5 — Audit trail

Every console action writes to console_audit_log (the per-app DB; see §1 about per-env separation). Three event families relevant to deploy + promotion:

Break-glass deploys (§6) write console.deploy.intent.breakglass with the actor's email captured at dispatch time.

§6 — Break-glass deploy path

When the console version manager is down, locked, or you need to deploy a specific non-main ref:

# Deploy console code (this app) to prod
gh workflow run deploy-console.yml -f environment=production -f ref=main

# Deploy console code to staging (rare — staging auto-deploys on push)
gh workflow run deploy-console.yml -f environment=staging -f ref=main

# Deploy backend (api) — same workflow_dispatch pattern
gh workflow run deploy-heroku.yml -f environment=production -f ref=main

If HEROKU_API_KEY is rejected as invalid in the workflow output, you have token drift — see heroku-api-key-drift-recovery.md.

§7 — Code promotion is not flag promotion

Console code promotion (this doc) — staging-SHA → prod-SHA via the version manager.

Console feature flag promotion (#798 / /flags/promotions) — separate flow. Each flag has its own staging-on / prod-on state. Flag flips happen WITHIN an env, not across envs. Do not conflate.

The two flows: - Code promotion uses the deploy workflow (Heroku push), updates the running binary - Flag promotion uses Heroku config var writes, takes effect on next request

If you want to flip a flag in prod, you don't need a code promotion — just toggle the flag in the flag manager.

§8 — Kill switches

Switch Where Effect
FLAG_CONSOLE_DEPLOY_UI Heroku config Disables the entire deploy modal flow; break-glass only
FLAG_CONSOLE_DEPLOY_ASYNC Heroku config When true, dispatch returns 202 immediately + lazy run_id resolution. When false, falls back to synchronous 30s run_id poll (the H12 risk path, default until staging soak passes)
FLAG_CONSOLE_DEPLOY_CHIP Heroku config Cross-page in-flight deploy indicator (#892)
FLAG_CONSOLE_ROTATE_UI Heroku config Enables the rotate-from-console button on /secrets rows
FLAG_CONSOLE_ROTATION_MODE_A Heroku config Per-secret Mode A handler dispatch (auto-rotate vs manual SOP)

Toggle via heroku config:set <flag>=true -a <app> >/dev/null 2>&1 (always silence stdout — heroku config:set echoes the secret in unredacted form by default; SECRET_KEY leaked from a default echo on 2026-05-01).

§9 — Refs