Runbook: Remove dormant FLAG_CONSOLE_* Heroku config vars (post-GA)

Issue: #1061 Prereq cleared: PR #1712 (§9 GA gate — all 46 os.environ.get("FLAG_*") callsites migrated to flags.is_on() DB-backed resolution) Deprecation table: docs/ops/feature-flags-heroku-deprecation.md Full ops runbook: docs/ops/feature-flags-runbook.md Apps in scope: raxx-console-prod, raxx-console-staging Owner: operator

Why this exists

FLAG_CONSOLE_* Heroku config vars were the original flip mechanism for feature flags before the DB-backed flag layer shipped. With FLAG_CONSOLE_FEATURE_FLAGS_DB=1 set on both apps and all code callsites migrated to flags.is_on() (PR #1712), the Heroku env vars are dormant for any flag that has a DB row. They still exist on Heroku but have no effect while the DB row is present.

This runbook gives the operator a pre-flight checklist, the exact heroku config:unset commands, and a post-unset verification checklist.

Part 1: Pre-flight — confirm DB rows exist before removing anything

Run this for each flag you intend to unset. Connect to the console DB (via heroku pg:psql):

heroku pg:psql --app raxx-console-prod
heroku pg:psql --app raxx-console-staging

Query to confirm DB rows exist:

-- Confirm all FLAG_CONSOLE_* flags have DB rows on each env
SELECT flag_key, env, value, updated_at
FROM console_feature_flags
WHERE flag_key LIKE 'console_%'
ORDER BY flag_key, env;

For each flag key listed in Part 2 below, you need a row for both env='prod' and env='staging' before removing that flag's Heroku config var from the corresponding app.

Query to confirm audit trail (at least one flip per env per flag):

SELECT target_id AS flag_key,
       payload->>'env' AS env,
       COUNT(*) AS flip_count,
       MAX(created_at) AS last_flip
FROM console_audit_log
WHERE action = 'console.flag.flip'
GROUP BY target_id, payload->>'env'
ORDER BY target_id, env;

Do not proceed with unset for any flag that has no DB row or no audit entry.

Part 2: Flags to unset

2a. Confirmed can-remove now (no code dependency remaining)

These flags have no remaining code reference in console/app/:

2b. Dormant after PR #1712 — remove after DB row confirmed

All of the following are resolved exclusively via flags.is_on() after PR #1712. Their Heroku env vars are dormant once the DB row exists. Confirm DB row + audit entry for each env before running the unset.

2c. Do NOT remove

Part 3: The unset commands

All heroku config:unset calls must silence stdout to avoid echoing the full config including adjacent secrets (>/dev/null 2>&1).

Use the script at scripts/console/cleanup-dormant-flags.sh for the full automated listing pass.

For manual targeted removal:

# Unset a single flag on both apps (replace FLAG_CONSOLE_EXAMPLE):
heroku config:unset FLAG_CONSOLE_EXAMPLE --app raxx-console-prod >/dev/null 2>&1
heroku config:unset FLAG_CONSOLE_EXAMPLE --app raxx-console-staging >/dev/null 2>&1

To remove all flags from Part 2a + 2b in one pass (run only after Part 1 confirms DB rows):

# --- staging first ---
heroku config:unset \
  FLAG_CONSOLE_ENV_SWITCHER_BANNER \
  FLAG_CONSOLE_ADMINS_ONLINE \
  FLAG_CONSOLE_ALERTS_AUTO_TICKET \
  FLAG_CONSOLE_ALERTS_BELL \
  FLAG_CONSOLE_ALERTS_DRAWER_MULTISOURCE \
  FLAG_CONSOLE_BILLING \
  FLAG_CONSOLE_CF_PAGES_BUILD_STATUS \
  FLAG_CONSOLE_CLAUDE_MENU \
  FLAG_CONSOLE_CUSTOMER_ADMIN \
  FLAG_CONSOLE_CUSTOMER_AUDIT_VIEWER \
  FLAG_CONSOLE_DASHBOARD_HOME \
  FLAG_CONSOLE_DEPLOY_ASYNC \
  FLAG_CONSOLE_DEPLOY_AUDIT_INGEST \
  FLAG_CONSOLE_DEPLOY_CHIP \
  FLAG_CONSOLE_DEPLOY_KV_WRITE \
  FLAG_CONSOLE_DEPLOY_OBSERVABILITY \
  FLAG_CONSOLE_DEPLOY_RESUME \
  FLAG_CONSOLE_DEPLOY_UI \
  FLAG_CONSOLE_DEPLOY_VIEW_LOGS \
  FLAG_CONSOLE_DEPLOY_XENV_READ \
  FLAG_CONSOLE_DRAWER_ACTIONS \
  FLAG_CONSOLE_DRIFT_WIDGET \
  FLAG_CONSOLE_ENV_GATE \
  FLAG_CONSOLE_FLAG_DRIFT_WIDGET \
  FLAG_CONSOLE_FLAG_PROMOTION_WATCHDOG \
  FLAG_CONSOLE_FLAG_PROMOTIONS \
  FLAG_CONSOLE_FLAGS_BY_SURFACE \
  FLAG_CONSOLE_GITHUB_BUILD_STATUS \
  FLAG_CONSOLE_GOOGLE_OAUTH_FALLBACK \
  FLAG_CONSOLE_HEROKU_DYNO_MONITOR \
  FLAG_CONSOLE_HEROKU_LOG_DRAIN_ALERTING \
  FLAG_CONSOLE_INVESTIGATE_FROM_STATUS \
  FLAG_CONSOLE_IP_GEO_WIDGET \
  FLAG_CONSOLE_LOCATION_GEOCODE \
  FLAG_CONSOLE_MINIMIZABLE_MODAL \
  FLAG_CONSOLE_NAV_V2 \
  FLAG_CONSOLE_POST_PROMOTION_HEALTH \
  FLAG_CONSOLE_PROMOTION_EXPIRY_SCHEDULER \
  FLAG_CONSOLE_PROMOTIONS_READY_BADGE \
  FLAG_CONSOLE_RECENTS_AUTOCOLLAPSE \
  FLAG_CONSOLE_REPLAY_RBAC_GATE \
  FLAG_CONSOLE_REPLAY_SPLIT_VIEW \
  FLAG_CONSOLE_REPLAY_TIMELINE \
  FLAG_CONSOLE_ROLLBACK_SIGNAL_LISTENER \
  FLAG_CONSOLE_ROTATE_UI \
  FLAG_CONSOLE_ROTATION_MODE_A \
  FLAG_CONSOLE_SECRET_ONBOARDING_AUTO \
  FLAG_CONSOLE_SENTRY_24H \
  FLAG_CONSOLE_SITE_LATENCY_CHART \
  FLAG_CONSOLE_STATUS_API \
  FLAG_CONSOLE_STATUS_CONFIG \
  FLAG_CONSOLE_TELEMETRY_HARDENING \
  FLAG_CONSOLE_TILE_DRAWER \
  FLAG_CONSOLE_TILE_HOVER_SPARKLINE \
  FLAG_CONSOLE_VAULT_ENV_AWARE \
  FLAG_CONSOLE_VERSION_MANAGER \
  --app raxx-console-staging >/dev/null 2>&1

# Wait 5 minutes; confirm staging healthy before proceeding to prod
# (see Part 4 verification checklist)

# --- prod after staging soak ---
heroku config:unset \
  FLAG_CONSOLE_ENV_SWITCHER_BANNER \
  FLAG_CONSOLE_ADMINS_ONLINE \
  FLAG_CONSOLE_ALERTS_AUTO_TICKET \
  FLAG_CONSOLE_ALERTS_BELL \
  FLAG_CONSOLE_ALERTS_DRAWER_MULTISOURCE \
  FLAG_CONSOLE_BILLING \
  FLAG_CONSOLE_CF_PAGES_BUILD_STATUS \
  FLAG_CONSOLE_CLAUDE_MENU \
  FLAG_CONSOLE_CUSTOMER_ADMIN \
  FLAG_CONSOLE_CUSTOMER_AUDIT_VIEWER \
  FLAG_CONSOLE_DASHBOARD_HOME \
  FLAG_CONSOLE_DEPLOY_ASYNC \
  FLAG_CONSOLE_DEPLOY_AUDIT_INGEST \
  FLAG_CONSOLE_DEPLOY_CHIP \
  FLAG_CONSOLE_DEPLOY_KV_WRITE \
  FLAG_CONSOLE_DEPLOY_OBSERVABILITY \
  FLAG_CONSOLE_DEPLOY_RESUME \
  FLAG_CONSOLE_DEPLOY_UI \
  FLAG_CONSOLE_DEPLOY_VIEW_LOGS \
  FLAG_CONSOLE_DEPLOY_XENV_READ \
  FLAG_CONSOLE_DRAWER_ACTIONS \
  FLAG_CONSOLE_DRIFT_WIDGET \
  FLAG_CONSOLE_ENV_GATE \
  FLAG_CONSOLE_FLAG_DRIFT_WIDGET \
  FLAG_CONSOLE_FLAG_PROMOTION_WATCHDOG \
  FLAG_CONSOLE_FLAG_PROMOTIONS \
  FLAG_CONSOLE_FLAGS_BY_SURFACE \
  FLAG_CONSOLE_GITHUB_BUILD_STATUS \
  FLAG_CONSOLE_GOOGLE_OAUTH_FALLBACK \
  FLAG_CONSOLE_HEROKU_DYNO_MONITOR \
  FLAG_CONSOLE_HEROKU_LOG_DRAIN_ALERTING \
  FLAG_CONSOLE_INVESTIGATE_FROM_STATUS \
  FLAG_CONSOLE_IP_GEO_WIDGET \
  FLAG_CONSOLE_LOCATION_GEOCODE \
  FLAG_CONSOLE_MINIMIZABLE_MODAL \
  FLAG_CONSOLE_NAV_V2 \
  FLAG_CONSOLE_POST_PROMOTION_HEALTH \
  FLAG_CONSOLE_PROMOTION_EXPIRY_SCHEDULER \
  FLAG_CONSOLE_PROMOTIONS_READY_BADGE \
  FLAG_CONSOLE_RECENTS_AUTOCOLLAPSE \
  FLAG_CONSOLE_REPLAY_RBAC_GATE \
  FLAG_CONSOLE_REPLAY_SPLIT_VIEW \
  FLAG_CONSOLE_REPLAY_TIMELINE \
  FLAG_CONSOLE_ROLLBACK_SIGNAL_LISTENER \
  FLAG_CONSOLE_ROTATE_UI \
  FLAG_CONSOLE_ROTATION_MODE_A \
  FLAG_CONSOLE_SECRET_ONBOARDING_AUTO \
  FLAG_CONSOLE_SENTRY_24H \
  FLAG_CONSOLE_SITE_LATENCY_CHART \
  FLAG_CONSOLE_STATUS_API \
  FLAG_CONSOLE_STATUS_CONFIG \
  FLAG_CONSOLE_TELEMETRY_HARDENING \
  FLAG_CONSOLE_TILE_DRAWER \
  FLAG_CONSOLE_TILE_HOVER_SPARKLINE \
  FLAG_CONSOLE_VAULT_ENV_AWARE \
  FLAG_CONSOLE_VERSION_MANAGER \
  --app raxx-console-prod >/dev/null 2>&1

Part 4: Verify after unset

Run these checks within 5 minutes of the unset on each app.

4a. Confirm the vars are gone

# Should print nothing (or "No such config var") for each var you unset:
heroku config --app raxx-console-staging 2>/dev/null | grep "^FLAG_CONSOLE_" || echo "No FLAG_CONSOLE_ vars found"
heroku config --app raxx-console-prod    2>/dev/null | grep "^FLAG_CONSOLE_" || echo "No FLAG_CONSOLE_ vars found"

Expected: only FLAG_CONSOLE_FEATURE_FLAGS_DB and FLAG_CONSOLE_VELVET_INTEGRATION remain. Everything else should be absent.

4b. Confirm flag resolution still works

# Flags page should load without errors (requires session cookie):
curl -sI https://console.raxx.app/console/flags | grep "HTTP/"
# Expected: HTTP/2 200  (or 302 redirect to login if session not set)

curl -sI https://console-staging.raxx.app/console/flags | grep "HTTP/"
# Expected: same

4c. Confirm critical flags resolve from DB (not fallen back to env/yaml defaults)

Connect to the DB and verify the rows are still present and unchanged after the unset:

-- Run on both prod and staging after config:unset
SELECT flag_key, env, value, updated_at
FROM console_feature_flags
WHERE flag_key IN (
  'console_billing',
  'console_dashboard_home',
  'console_flag_promotions',
  'console_env_gate',
  'console_deploy_ui',
  'console_feature_flags_db'
)
ORDER BY flag_key, env;

Expected: rows present and value unchanged. If a row is absent for a flag that was expected to be on, the flag resolves to its YAML default. Check feature_flags.yaml to see what that default is.

4d. Confirm dyno restart completed (Heroku restarts dynos on config:unset)

heroku ps --app raxx-console-staging
heroku ps --app raxx-console-prod
# Expected: web.1: up  (may take 30-60s post-unset to show up)

If dynos are cycling or in error state, check:

heroku logs --tail --app raxx-console-prod

4e. Smoke the dashboard

Load the console dashboard on both staging and prod. Confirm: - Dashboard tiles render (no blank page, no 500) - Nav items present (Security / Status / Issues) - Flag management page (/console/flags) accessible and shows current flag states

Part 5: Post-completion

After unset and verification:

1. Update docs/ops/feature-flags-heroku-deprecation.md — change each removed var's status to removed and add the removal date.
2. Close issue #1061.
3. Note: FLAG_CONSOLE_VELVET_INTEGRATION cleanup is tracked separately (needs flags.is_on() migration in velvet_client.py first).

Rollback

If anything breaks after the unset, the env vars can be restored immediately:

heroku config:set FLAG_CONSOLE_<KEY>=1 --app raxx-console-prod >/dev/null 2>&1
heroku config:set FLAG_CONSOLE_<KEY>=1 --app raxx-console-staging >/dev/null 2>&1

The DB row still wins over the env var while FLAG_CONSOLE_FEATURE_FLAGS_DB=1, so restoring the env var does not change flag behavior for any flag that has a DB row. To force a specific flag value when a DB row exists, update the DB row directly (see docs/ops/feature-flags-runbook.md §6).