Raxx · internal docs

internal · gated

Flag reconciler backfill runbook

Script: console/scripts/flag_reconciler_backfill.py Issue: #2012 Design: [PR #2005](https://github.com/raxx-app/TradeMasterAPI/pull/2005) — docs/architecture/flag-reconciler-bidirectional-sync-2026-05-13.md section 9.2 Epic: #798 Last updated: 2026-05-21 UTC

Purpose

Before the flag reconciler (Card 2) enforces drift detection, every existing FLAG_* Heroku config var must have a corresponding row in console_flag_promotions with synced=true.

Without this backfill, the reconciler's kill-switch (Card 4) would see all 64 pre-existing rows as synced=false and interpret them as drift — disabling the entire promotion UI.

This script reads all 4 Heroku apps, diffs against console_flag_promotions, and creates or updates rows for any untracked FLAG_* vars.

CRITICAL ordering constraint: Card 4 (kill-switch enforcement) MUST NOT be deployed until this script has been run and confirmed zero unexpected pending-state rows.

Prerequisites

  1. Migration 0078 has been applied on the target DB (console_flag_promotions has the synced, last_heroku_value, last_synced_at, drift_detected_at, drift_reason, sync_source columns).
  2. DATABASE_URL is set to the target console DB (staging or prod).
  3. HEROKU_API_TOKEN is set to a read-scoped Heroku token. Read from vault:

HEROKU_API_TOKEN=<read from vault at HEROKU_API_TOKEN path>

  1. Python dependencies are installed:

pip install requests sqlalchemy

Procedure

Step 1 — dry run (always first)

cd console/
DATABASE_URL="<target>" HEROKU_API_TOKEN="<token>" \
  python scripts/flag_reconciler_backfill.py --dry-run

Review the JSON output. For each app, verify: - flags_found matches your expectation (based on the comment in issue #2012: ~5 on raxx-api-prod, ~53 on raxx-console-prod, ~38 on raxx-api-staging, ~76 on raxx-console-staging). - planned_inserts covers flags that you know are live on Heroku but absent from console_flag_promotions. - planned_updates covers rows where synced=false (all pre-migration-0078 rows default to false). - skipped_existing is the count of flags already confirmed synced. - No error fields in any app block.

Step 2 — apply

Once the dry-run output looks correct:

cd console/
DATABASE_URL="<target>" HEROKU_API_TOKEN="<token>" \
  python scripts/flag_reconciler_backfill.py --apply --confirm

The --confirm flag is required as a defense-in-depth guard against accidental writes.

Step 3 — verify

After --apply, run --dry-run again to confirm idempotency:

DATABASE_URL="<target>" HEROKU_API_TOKEN="<token>" \
  python scripts/flag_reconciler_backfill.py --dry-run

Expected output: planned_inserts=0, planned_updates=0 for all apps. All previously-untracked flags are now skipped_existing.

Optionally query the DB directly:

SELECT count(*) FROM console_flag_promotions
WHERE synced = false AND state = 'deployed';

Expected: 0 rows.

Expected output format

The script emits JSON to stdout and log lines to stderr.

{
  "mode": "DRY-RUN",
  "run_at_utc": "2026-05-21T12:00:00+00:00",
  "apps": [
    {
      "app": "raxx-console-staging",
      "flags_found": 76,
      "planned_inserts": 24,
      "planned_updates": 52,
      "skipped_existing": 0,
      "applied_inserts": null,
      "applied_updates": null,
      "inserts": [...],
      "updates": [...],
      "skips": [],
      "error": null
    }
  ],
  "totals": {
    "flags_found": 172,
    "planned_inserts": 50,
    "planned_updates": 122,
    "skipped_existing": 0
  }
}

After --apply, the mode field is "APPLY" and applied_inserts / applied_updates replace the null values.

What to do if the script exits non-zero

Exit code 1 means one or more Heroku API fetch calls failed. The script logs the failing app name and error before exiting.

Steps: 1. Check the log line for fetch failed for <app>: .... 2. Verify HEROKU_API_TOKEN is valid and has read scope on that app. 3. Re-run the script; it is idempotent — any apps that succeeded in the prior run will have skipped_existing > 0 and will produce no new writes. 4. If the error persists, check Heroku status at https://status.heroku.com and retry.

Rollback procedure

The script only creates rows in console_flag_promotions with state='deployed' and created_by='system_backfill'. To undo:

DELETE FROM console_flag_promotions
WHERE created_by = 'system_backfill'
  AND sync_source = 'heroku_cli_backfill'
  AND state = 'deployed';

This is destructive — only do this if the backfill produced incorrect rows and you have reviewed the impact on the reconciler.

Non-goals