Flag Reconciler — Bidirectional Sync (Drift-as-Kill-Switch)

Status: Proposed
Owner: software-architect
Date: 2026-05-13 UTC
Related docs:
- console-flag-promotion-flow.md — prior staging-to-prod promotion design
- [ADR-0085](https://internal-docs.raxx.app/architecture/adr/0085-flag-reconciler-bidirectional-sync.html) — decision record for this design

1. Goal

Establish a bidirectional, audited sync between console_flag_promotions (the DB source of intent) and Heroku FLAG_* environment variables (the runtime source of truth) across all four Raxx apps (raxx-console-staging, raxx-console-prod, raxx-api-staging, raxx-api-prod). A 5-minute reconciler job compares both sides and marks rows synced=true or synced=false. When drift is detected, the console UI disables promote/rollback actions for the affected flag and surfaces a banner with a side-by-side diff. The operator resolves drift by choosing which side wins ("DB is correct" or "Heroku is correct") via a "Mark as synced" action. Only after drift is resolved does the normal promotion workflow resume. Drift is therefore a kill-switch on forward progress — not a warning.

2. Current State and Problem

Heroku has been the canonical source of truth for feature flags for approximately 9 months. Flag values are set via heroku config:set directly (per feedback_bootstrap_via_heroku.md). The console_flag_promotions table exists and records intent (pending, staged, deploying, deployed, failed) but the synced column does not exist. The /console/flags page is gated behind FLAG_CONSOLE_FLAG_PROMOTIONS which is currently OFF on both staging and prod (per project_console_flag_toggle.md), meaning all flag flips still go via CLI.

This creates three types of drift today:

1. Orphan-on-Heroku: A FLAG_* var exists on Heroku with no corresponding DB promotion row. This is the dominant case (~50+ flags set in prod; ~7 with DB rows). The flag_drift reconciler (console/app/services/drift_reconcilers/flag_drift.py) already detects this as orphan_on_heroku and writes to drift_run_results, but takes no action beyond alerting.
2. Missing-from-Heroku: A promoted DB row exists but Heroku never got the value. The reconciler detects this as missing_from_heroku at severity critical.
3. Value-mismatch: DB row says true; Heroku says false (or vice versa). Reconciler detects as value_mismatch.

The existing reconciler is read-only. This design makes it write-capable (mark synced) and wires it into a UI-level kill-switch.

3. Invariants

All platform invariants apply. Reconciler-specific:

1. No stored credentials. The Heroku API token lives in the env/secret store. The reconciler reads it at call time. It must not be written to any DB column, log line, or response body. Log only a SHA-256[:12] hash for token health checks.
2. Audit trail for every state change. Every write to console_flag_promotions (promote, rollback, mark-synced) emits a row to console_audit_log. The reconciler's automated updates (setting synced, updating last_heroku_value) also emit audit rows.
3. Drift is a kill-switch, not a warning. When synced=false, the promote and rollback actions for that flag are disabled at the API layer (not just the UI layer). A request to promote a drifted row returns 409 {"error": "flag_drifted"}.
4. Mark-synced requires operator intent. Automated resolution is not permitted. The operator must explicitly choose which side wins. Auto-reconcile (reconciler silently pushing DB value to Heroku) is out of scope for v1.
5. Paper-first gating is not a feature flag. The reconciler must never manage the paper-first gate. Any FLAG_* var that corresponds to a paper-first path must be treated as untouchable and logged with a paper_first_protected guard.
6. GDPR baseline. This system stores no PII. last_heroku_value stores a boolean as text. drift_reason stores an enum-like string. No user data.

4. Data Model

4.1 New columns on console_flag_promotions

The following columns are added to the existing table. All are nullable to allow zero-downtime additive migration.

4.2 Index additions

CREATE INDEX idx_cfp_synced ON console_flag_promotions (synced)
    WHERE synced = false;

A partial index on the drifted rows enables the drift-banner query to be fast even as the table grows.

4.3 Unchanged columns

The existing state machine columns (state, created_at, created_by, target_app, new_value, soak_seconds, etc.) are unchanged. The synced column is orthogonal to state — a row can be in state deployed and synced=false if Heroku was subsequently changed via CLI.

5. APIs and Contracts

5.1 Existing endpoint changes

POST /console/flags/promotions/<id>/promote — existing route.
New pre-condition: if the console_flag_promotions row has synced=false, return 409 {"error": "flag_drifted", "drift_reason": "<reason>", "last_heroku_value": "<val>"} before any other processing. This is enforced in PromotionService.promote_async(), not just the blueprint.

POST /console/flags/promotions/<id>/reject — existing route.
Same pre-condition: synced=false returns 409 {"error": "flag_drifted"} for rollback operations as well.

5.2 New endpoints

POST /console/flags/<flag_key>/promote
Shortcut that creates a pending promotion row and immediately calls Heroku API (for flags that need a direct UI-driven flip without the soak queue). Writes DB row with sync_source='console_ui_promote', calls Heroku PATCH /apps/{app}/config-vars, then sets synced=true and last_synced_at=now on success or synced=false + drift_reason='heroku_unset' on Heroku failure.

• Auth: @require_role("superadmin") + @require_totp_elevation
• Body: {"value": true|false, "target_app": "raxx-console-prod"}
• Response 202: {"state": "deploying", "status_url": "..."} (follows existing async pattern)
• Response 409: {"error": "flag_drifted"} if row is already drifted

As built (#2016, 2026-10-30) — surfaced in the single-pane console's Release flag drawer ("Write Heroku config var…" confirm modal), behind FLAG_CONSOLE_FLAG_HEROKU_WRITE (default off) AND FLAG_CONSOLE_FLAG_PROMOTIONS. Differences from the sketch above: auth is console-manager (RBAC V2 — narrower than the Release pane's own visibility) + require_totp_elevation; the body also carries a required typed reason (8–280 chars) and target_app is optional (the server derives it and rejects a mismatch); the 409 body returns a normalised observed ON/OFF word, never the raw last_heroku_value; a second write while one is deploying is 409 promotion_in_flight; GET /console/flags/<key>/promote/<id>/status is the poll target. A blocking pre-write audit row (console.flag.promote_via_ui.requested) precedes the PATCH and console.flag.promoted_via_ui records the outcome. The Heroku call (app/services/heroku_config_client.py) writes only allow-listed FLAG_* names on the two prod apps with HEROKU_API_TOKEN_WRITE, and never logs, returns or audits the PATCH payload or Heroku's response (which is the app's full config-var map).

Which flags, and which app (#2016 QA F1, revised in #4853 QA rounds 2 and 3) — the target app is the flag's declared surface: in feature_flags.yaml, never the key's spelling (32 console-surface flags have no console_ prefix, and the old prefix rule sent every one of them to the API app). The decision, in promotions.SURFACE_TARGET_APPS: console writes raxx-console-prod; raptor, multi, freescout, antlers, queue, getraxx and status-page write raxx-api-prod; one flag-level exception, freescout_console_customer_link (surface freescout, but read only by the console), writes raxx-console-prod, and (round 4, below) three flags declared console that Raptor gates write raxx-api-prod (promotions.FLAG_TARGET_APP_OVERRIDES, applied only to a flag the YAML declares). The first version of this note refused multi and freescout along with velvet, queue and ci on the grounds that no console or API dyno reads them; that was wrong. Raptor gates paper_trading_v1 (routes/trading.py), 13 of the 14 multi flags are checked in backend_v2, the console also reads options_data_feed and beta_rules_lab, and migration 0243 records that support_portal_api and support_operator_reply_email gate Raptor-side code. The queue's mark-promote had always put those on raxx-api-prod, so refusing them removed a working path (POST /flags/<key>/mark-promote answered 422; the drawer showed "Target app: none (multi surface)"). The model is one app per promotion row, so a multi flag that the console also reads (options_data_feed, beta_rules_lab, ...) has its console copy set separately; this change does not add a both-apps write. Round 3 found the same wrong premise under antlers, queue, getraxx and status-page: the surface names a product area, not the process that reads the flag. Raptor gates 21 antlers flags (e.g. quebec_geoblock, signup_geoblock_eu, trade_window_v1, graduate_gate, onboarding_wizard_v1), 2 queue flags (queue_billing in middleware/tier_gate.py, founders_checkout in routes/billing_checkout.py), 1 getraxx flag (getraxx_waitlist in routes/marketing.py) and 1 status-page flag (status_3p_poller in api/__init__.py) directly through feature_flags.is_enabled, which reads FLAG_<KEY> from the API dyno; GET /api/flags serves every YAML flag with that dyno's overrides applied; the B1 migrations 0025, 0032, 0056 and 0070 record raxx-api-prod for them; and on develop the queue derived raxx-api-prod for all of them. Refusing them removed a working mark-promote path for at least those 25 flags (the first version of this change refused 111 YAML flags in total). They now write raxx-api-prod. A flag that Antlers also reads from a CF Pages env var has that copy set separately, like the console copy of a multi flag. console/tests/test_promotions_service_stub.py scans backend_v2 for is_enabled() reads and fails if a flag Raptor gates has no app. Refused with 422 surface_not_supported (and the drawer hides both actions): velvet and ci, internal-docs, legacy and any key the YAML does not declare. "Refused" means the surface maps to an app the console does not write, not that the flag has no Heroku app: the Velvet service reads FLAG_VELVET_API_V1 from its own env (velvet/routes/tokens.py) and the repo's B1 migration 0300_promote_velvet_broker_credentials.py records raxx-velvet-prod, an app outside heroku_config_client.ALLOWED_TARGET_APPS (the two apps above). The 422 body says so and names raxx-velvet-prod for a velvet flag, and the drawer's "Target app" stat reads not written by the console (velvet surface); no direct read of a ci flag was found. Develop sent these flags to raxx-api-prod, which never reads them, so refusing them stays correct. The queue-based mark-promote uses the same derivation (promotions.derive_target_app), so both paths agree. The authenticator code is checked last, after the body and state checks, so a 400/409/422 does not spend it; only one deploying row can exist per flag and app (partial unique index uq_cfp_one_deploying_per_flag_app, migration 0325).

Three console-surface flags that Raptor gates (#4853 QA round 4) — the surface names where a flag is managed, not the process that reads it (the same premise round 3 corrected for antlers, queue, getraxx and status-page), and three flags declared console are gated by Raptor. support_history_endpoints: only Raptor reads it (routes/admin_customer_support_history.py, 503 when off); nothing in console/app references it, so a raxx-console-prod write set a var no console code reads, and mark-promote, the drawer's "Write Heroku config var…" action and its "Target app" stat all pointed at that app. It now derives raxx-api-prod, as on develop. account_merge (Raptor routes/account_merge.py, routes/internal_merges.py, routes/auth.py, middleware/session_auth.py; console blueprints/merges.py and the nav) and beta_ghost_reset (Raptor routes/internal_ghost_reset.py; console blueprints/beta.py and the people pane) are read by both services. This is the single-app limitation documented above for multi: a promotion row names one app, so one write cannot set both copies. Decision: raxx-api-prod for both. Raptor is the side that enforces the gate and mutates data (account merges and tombstones, ghost-user deletes), so FLAG_<KEY>=false there stops the effect whatever the console shows; it is what develop derived and what the queue has always promoted, so no working path moves; and the console copy is the operator-UI gate, set separately on raxx-console-prod (FLAG_ACCOUNT_MERGE, FLAG_BETA_GHOST_RESET) exactly like the console copy of a multi flag. The B1 migrations do not settle it (0213 records account_merge on raxx-api-prod, 0222 records beta_ghost_reset and 0237 records support_history_endpoints on raxx-console-prod); they record the app a queued row was created for, not which service must be written. console/tests/test_promotions_service_stub.py scans console/app for references and fails if a console-surface flag that Raptor gates and the console never references derives anything but raxx-api-prod, and pins the two both-read flags by name.

Staging console (#4853 QA round 2) — both target apps are prod apps, and a console with CONSOLE_ENV=staging answers every prod-targeted write with 403 promote-from-prod-only (_require_prod_for_prod_target). The drawer now uses the same predicate (promotions.staging_console_refuses) and hides the "Write Heroku config var…" action, the "Config var" stat and the "Console config writes" history there, instead of offering an action that can only be refused after the operator has typed a reason and a code.

Failed-write copy (#4853 QA round 2) — the modal says "Heroku did not apply the write … Nothing was changed on Heroku" only for failure codes that prove it: the console refused to send (token_missing, target_app_not_allowed, var_name_not_allowed) or Heroku answered with a non-2xx status (heroku_auth_rejected, heroku_app_not_found, heroku_rejected_value, heroku_rate_limited, heroku_unavailable, heroku_http_<n>). For heroku_timeout (the 15 s limit can fire after the PATCH was sent), heroku_unreachable (a connection error mid-request), unexpected_error and any code not listed, the title is "Result not confirmed" and the text matches the unconfirmed state: "The write may still apply. Reopen the flag and check “Console config writes” before trying again."

Enabling, and recovering from a stuck deploying row (#2016)

Enable. Prerequisites: HEROKU_API_TOKEN_WRITE is set on raxx-console-prod (the write-scoped token; there is no fallback to a broader key), FLAG_CONSOLE_FLAG_PROMOTIONS is on, and the operator holds console-manager with an authenticator enrolled. Then heroku config:set FLAG_CONSOLE_FLAG_HEROKU_WRITE=true -a raxx-console-prod >/dev/null 2>&1 (never let config:set echo). The flag is the kill switch: set it back to false and the route 404s and the drawer action disappears.

Why a row can stick. A write to raxx-console-prod makes Heroku restart the console's own dynos. The settle step runs in a daemon thread; if the process is killed after Heroku accepted the PATCH but before the worker commits deployed/failed, the row stays deploying (the window is milliseconds). Because the outcome audit row (console.flag.promoted_via_ui) is best-effort, that case may also have no outcome row; the trail is then the console.flag.promote_via_ui.requested row (which carries the typed reason), the promotion row itself, and whatever you do next.

What you see. The drawer's "Console config writes" history shows the write as DEPLOYING long after it should have settled, "Write Heroku config var…" is hidden, a further write returns 409 promotion_in_flight, and the queue's Promote for the same flag and app returns 409 too (the unique index allows one deploying row per flag and app).

Recover.

1. Check what Heroku actually holds. heroku config:get FLAG_<NAME> -a <app> prints one boolean (it is the only var to read; never config without a name). Compare it with the row's "Set to" value in the history.
2. Once the row has been deploying for 5 minutes, the drawer offers Force-complete promotion and Force-fail promotion to holders of console-prod-services-write (authenticator code required). If Heroku already holds the value the operator asked for, use Force-complete (it re-verifies against the Heroku Release API); if it does not, use Force-fail with a one-line failure reason and an optional RCA link. Force-fail replaces the row's reason column with the failure text; the operator's typed reason stays in the ...requested audit row. These routes are behind FLAG_CONSOLE_FLAG_PROMOTION_WATCHDOG; with it off they return 503 feature_disabled and the buttons do not appear.
3. With the watchdog flag on, nothing needs doing after 24 hours: the sweep warns in Slack at 12 hours, then at 24 hours (PROMOTION_DEPLOYING_TIMEOUT_S overrides) re-checks the Heroku Release API and marks the row deployed if the release is confirmed, or fails it with a critical Slack alert if not. The watchdog flag defaults off, so until it is enabled a stuck row stays stuck.
4. If neither the buttons nor the watchdog are available, the SRE fallback is the one used in docs/ops/incidents/2026-05-11-stuck-flag-promotion-console-heroku-log-drain-alerting.md: verify the value with heroku config:get, settle the row to deployed (value landed) or failed (it did not) with a manual UPDATE, and write an audit row for it.

When the row leaves deploying the index no longer blocks anything and the drawer action returns on the next open. If you Force-fail a write that had in fact landed, the console's row says failed while Heroku holds the new value; resolve that with "Accept observed value" in the drawer.

POST /console/flags/<flag_key>/mark-synced
Operator resolves a drift conflict by declaring which side wins.

• Auth: @require_role("superadmin") + @require_totp_elevation
• Body: {"winner": "db" | "heroku", "totp_code": "..."}
• winner="db": Sets sync_source='mark_synced_db_wins'. Schedules an immediate Heroku PATCH to push the DB value. On success: synced=true, last_synced_at=now.
• winner="heroku": Reads current last_heroku_value, updates the DB row's new_value to match, sets synced=true, sync_source='mark_synced_heroku_wins', last_synced_at=now. No Heroku write needed.
• Response 200: {"synced": true, "resolved_value": true|false, "winner": "db"|"heroku"}
• Response 409: if Heroku PATCH fails during winner="db" path, returns {"error": "heroku_push_failed"} and leaves synced=false.
• Emits audit row: action="console.flag.mark_synced", payload {flag_key, winner, resolved_value, previous_db_value, previous_heroku_value}.

GET /console/flags/drift
Returns all currently-drifted rows for the UI drift banner.

• Auth: @require_role("ops", "superadmin")
• Response 200: {"drifted": [{"flag_key": "...", "target_app": "...", "db_value": true|false, "heroku_value": "true"|"false"|null, "drift_reason": "...", "drift_detected_at": "...UTC..."}]}
• Returns [] when no drift exists (banner hides itself).

6. Architecture

+---------------------------+     POST /flags/<key>/mark-synced
|   Console UI              |<---------------------------------+
|   /console/flags          |                                  |
|   - Drift banner          |--GET /console/flags/drift------->|
|   - Per-row drift badge   |                                  |
|   - Disabled promote btn  |                                  |
+---------------------------+                                  |
            |                                                  |
            | POST /flags/promotions/<id>/promote              |
            v                                                  |
+---------------------------+          +---------------------+ |
|   Console API             |          |   Heroku Platform   | |
|   blueprints/flags.py     |--------->|   API               | |
|   services/promotions.py  |  PATCH   |   PATCH config-vars | |
|                           |<---------|   GET  config-vars  | |
+---------------------------+          +---------------------+ |
            |                                    ^             |
            | DB write                           |             |
            v                                    |             |
+---------------------------+     reads          |             |
|   console DB              |<-------------------+             |
|   console_flag_promotions |   (reconciler reads             |
|   + synced columns        |    Heroku + DB, diffs,          |
+---------------------------+    writes synced col)            |
            ^                                                  |
            |                                                  |
+---------------------------+                                  |
|   Reconciler Job          |----------------------------------+
|   services/flag_reconciler|  mark-synced API when
|   .py                     |  winner resolution needed
|   (every 5 min via APSched)|
+---------------------------+
            |
            v
+---------------------------+
|   console_audit_log       |
|   (every state change)    |
+---------------------------+

The reconciler is a separate scheduler job, not a request-scoped service. It runs independently of HTTP traffic.

7. State Machine

7.1 Row states (flag_key x target_app pair)

Note: drifted is not a separate state column value — it is a derived view based on synced=false. The underlying state column retains its current values (deployed, pending, etc.). The UI derives "drifted" from synced=false.

7.2 Transition table

7.3 State machine diagram

stateDiagram-v2
    [*] --> pending : mark-promote
    pending --> deployed : promote (soak elapsed)
    pending --> rejected : operator rejects
    pending --> expired : 7-day TTL

    deployed --> drifted : reconciler detects mismatch
    drifted --> deployed : mark-synced resolves conflict

    deployed --> [*] : terminal

    note right of drifted
        promote/rollback buttons
        DISABLED while in this state.
        Operator must resolve via
        mark-synced before proceeding.
    end note

8. Reconciler Service Shape

File: console/app/services/flag_reconciler.py

This service is a standalone module called by the APScheduler job registered in the console app factory. It does not import from blueprint handlers.

# Pseudo-Python — shapes the public interface; implementation is feature-developer's

from dataclasses import dataclass
from typing import Optional

APPS = [
    "raxx-console-staging",
    "raxx-console-prod",
    "raxx-api-staging",
    "raxx-api-prod",
]

@dataclass
class ReconcileResult:
    app_name: str
    synced_count: int        # rows marked synced=true this run
    drifted_count: int       # rows marked synced=false this run
    skipped_count: int       # rows skipped (e.g. paper-first protected)
    error: Optional[str]     # non-None if Heroku fetch failed for this app


def reconcile_all_apps() -> list[ReconcileResult]:
    """
    Called every 5 minutes by APScheduler.
    For each app in APPS:
      1. Fetch FLAG_* config vars via Heroku Platform API
         GET /apps/{app}/config-vars
         Auth: HEROKU_API_TOKEN (or per-app token from env, see promotions.py)
      2. Read all console_flag_promotions rows for this target_app
         that are in state IN ('deployed', 'pending', 'staged')
      3. Diff: for each DB row, check if Heroku value matches new_value
      4. Write synced=true/false + last_heroku_value + timestamps
      5. Write audit row for each change to synced
    Never raises — errors are captured in ReconcileResult.error.
    Returns list of ReconcileResult, one per app.
    """
    ...


def _fetch_heroku_config_vars(app_name: str) -> dict[str, str]:
    """
    Calls GET /apps/{app_name}/config-vars.
    Returns raw dict of all env vars.
    Raises HerokuFetchError on 4xx/5xx or network timeout.
    Token sourced from env: HEROKU_API_TOKEN or
    per-app override (HEROKU_API_TOKEN_CONSOLE_PROD etc.).
    15-second timeout. No retries in the reconciler path
    (job will retry in 5 minutes automatically).
    """
    ...


def _extract_flag_vars(raw_config: dict) -> dict[str, bool]:
    """
    Filter raw config to FLAG_* keys.
    Returns {flag_key_lowercase: bool} e.g. {"console_flag_promotions": True}.
    Key is lower(key.removeprefix("FLAG_")).
    Value is case-insensitive "true" -> True, anything else -> False.
    """
    ...


def _diff_and_update(
    db_rows: list,          # ConsoleFlagPromotion rows for one target_app
    heroku_vars: dict,      # from _extract_flag_vars
    target_app: str,
    dry_run: bool = False,
) -> tuple[int, int]:
    """
    Compare each DB row against heroku_vars.
    Write synced, last_heroku_value, drift_reason, timestamps.
    Emit audit rows for each change.
    Returns (synced_count, drifted_count).
    """
    ...

One applied row per flag speaks for Heroku (#4853 QA, 2026-10-31) — every console write and every queue promote adds a NEW deployed row, but Heroku holds only the last value written. Comparing every old deployed row against Heroku made a flip-flop (write ON, then OFF) look like drift on the ON row; after three cycles that row was synced=false, the Release "Drifted" count and the drawer banner reported it, and Revert / "Accept observed value" (which prefer an unsynced row) acted on it, re-applying the stale ON. The reconciler now splits an app's rows (_split_rows) into compared, superseded and skipped: for each flag_key only the newest applied row (state in deployed, untracked_drift; newest by created_at, then id — the order the console write's drift kill-switch uses, flag_config_write.latest_applied_row) is compared; older applied rows are superseded and are not compared. A newer failed / deploying / rejected / expired row is skipped and supersedes nothing, so a rejected write leaves the last row that really was applied as the one checked, and a write that timed out but did land is still reported as drift on that row. pending / staged rows are queued intent, not applied state: they neither supersede nor are superseded, and are compared exactly as before. Databases written before this change can hold a superseded row that is already synced=false with drift_detected_at set; _settle_superseded puts such a row in the state mark-synced leaves a resolved row in (synced=true, drift_reason cleared, drift_cycle_count=0; drift_detected_at kept as history), writes one console.flag.sync_updated audit row with drift_reason=superseded_by_newer_row, and does nothing for a row that is already clean. ReconcileResult.superseded_count reports how many rows an app had superseded. The two routes that act on a drifted row (mark-synced, resolve-drift) now rank a row the reconciler reported as drifted (synced=false and drift_detected_at set — the Release pane's own predicate) before any other unsynced row, so a newer failed write (unsynced, never reported) can no longer be the row Revert changes while the real drift stays.

Queued intent never trips the console write's kill-switch (#4853 QA round 3) — a queue promotion is created synced=false and the reconciler leaves pending / staged rows alone, so after three cycles with Heroku holding the current value a queued row carries drift_cycle_count=3 with drift_detected_at NULL: queued intent, not drift. The console write's kill-switch (flag_config_write.drift_blocking_row) used to read the flag's latest row of any state, so that queued row made the drawer say "in sync" while the write action vanished and POST /console/flags/<key>/promote answered 409 flag_drifted; a queued row that happened to match Heroku could equally hide real drift on the applied row beneath it. The kill-switch and the drawer's blocked check now read only the newest applied row (deployed / untracked_drift, the row the reconciler compares): a pending / staged row neither blocks nor masks, and a newer failed attempt does not clear drift on the last applied row (resolve it with Revert / Accept observed, which act on the reported row). The reconciler's handling of pending / staged rows is unchanged.

A queue promote settles its row in sync (#4853 QA round 4) — the round-3 rule left a transient version of the same symptom. When the queue settled a long-queued row to deployed it set state and nothing else, so the row kept synced=false, drift_cycle_count=3 and drift_detected_at NULL, and as an applied row it was now what the kill-switch reads: drawer Drift "in sync", banner ok, no "Write Heroku config var" action, and POST /console/flags/<key>/promote answered 409 flag_drifted until one reconcile with Heroku holding the new value healed the row (at most one reconciler interval, about 5 minutes; it bit an operator flipping a flag straight back after a queue promote). Every path that settles a queue promotion to deployed — promote, the async worker (_do_heroku_work), the watchdog's PENDING_COMMIT and 24h resolutions, the startup scanner and force_complete — now calls promotions._settle_in_sync: synced=true, last_synced_at, drift_cycle_count=0, drift_reason cleared, exactly what the console's own write records. The value has just been applied (for force_complete an operator attested it), so the row is in sync now, and because it starts synced the reconciler can also report a later divergence (it flips a row to drifted only from synced=true; a row left synced=false could never be reported). The other option, making drift_blocking_row require drift_detected_at IS NOT NULL, was not taken: an applied row that is unsynced past the threshold but was never reported would stop blocking, and the kill-switch tests that seed exactly that state would have to change. Rows already deployed with that state before this change still heal on the next reconcile.

Scheduler registration (in console/app/__init__.py or equivalent scheduler setup):

scheduler.add_job(
    func=reconcile_all_apps,
    trigger="interval",
    minutes=5,
    id="flag_reconciler",
    replace_existing=True,
    misfire_grace_time=60,   # tolerate up to 60s delay before treating as missed
)

Heroku API dependency: The reconciler uses the same GET /apps/{app}/config-vars endpoint already used by flag_drift.py. It can share the fetch_heroku_flag_vars helper from drift_reconcilers/flag_drift.py (or extract it to a shared heroku_client.py).

Token: HEROKU_API_TOKEN (generic) or per-app tokens HEROKU_API_TOKEN_CONSOLE_PROD, HEROKU_API_TOKEN_API_PROD etc. already in Infisical. Rotatable without redeploy.

9. Migrations

9.1 Migration 0055 — add sync columns (additive)

File: console/migrations/versions/0055_cfp_sync_columns.py

-- up
ALTER TABLE console_flag_promotions
  ADD COLUMN synced           BOOLEAN NOT NULL DEFAULT FALSE;
ALTER TABLE console_flag_promotions
  ADD COLUMN last_heroku_value TEXT;
ALTER TABLE console_flag_promotions
  ADD COLUMN last_synced_at   TIMESTAMPTZ;
ALTER TABLE console_flag_promotions
  ADD COLUMN drift_detected_at TIMESTAMPTZ;
ALTER TABLE console_flag_promotions
  ADD COLUMN drift_reason     TEXT;
ALTER TABLE console_flag_promotions
  ADD COLUMN sync_source      TEXT;

CREATE INDEX idx_cfp_synced_false
  ON console_flag_promotions (synced)
  WHERE synced = false;

-- down
DROP INDEX IF EXISTS idx_cfp_synced_false;
-- SQLite: no DROP COLUMN; Postgres supports it
ALTER TABLE console_flag_promotions DROP COLUMN IF EXISTS sync_source;
ALTER TABLE console_flag_promotions DROP COLUMN IF EXISTS drift_reason;
ALTER TABLE console_flag_promotions DROP COLUMN IF EXISTS drift_detected_at;
ALTER TABLE console_flag_promotions DROP COLUMN IF EXISTS last_synced_at;
ALTER TABLE console_flag_promotions DROP COLUMN IF EXISTS last_heroku_value;
ALTER TABLE console_flag_promotions DROP COLUMN IF EXISTS synced;

Depends on: migration 0054 applied. Zero-downtime: additive columns with nullable defaults. Existing code that does not read these columns is unaffected.

SQLite note: SQLite supports ADD COLUMN but not DROP COLUMN before version 3.35.0. The down migration for SQLite environments will use batch_alter_table (Alembic) to recreate the table without the columns, consistent with migration 0054 pattern.

9.2 Migration 0056 — backfill (data migration)

File: console/migrations/versions/0056_cfp_heroku_backfill.py

This migration does NOT run inline. It is implemented as a one-shot script invoked by a sub-card (see Implementation Cards). The migration file defines the schema structure for the backfill operation and a heroku_cli_backfill marker.

The backfill script: 1. Reads all FLAG_* vars from each of the 4 Heroku apps via API. 2. For each FLAG_* var with no existing console_flag_promotions row for that (flag_key, target_app) pair, inserts a new row with: - state='deployed' (it is live on Heroku) - new_value = current Heroku boolean - synced=true - sync_source='heroku_cli_backfill' - created_by='system_backfill' - last_heroku_value = string representation - last_synced_at = now (UTC) 3. For existing rows where synced is false (all rows post-0055 migration), reconciles against current Heroku state.

The backfill is idempotent. Running it twice produces the same result.

10. Rollout Plan

Phase 0 and Phase 1 are safe to ship before 2026-05-23 launch — they add columns and read-only behavior. Phase 2 (kill-switch enforcement) should also land pre-launch if the team has bandwidth; it closes a gap where the UI could act on stale state. Phase 3 (UI-triggered Heroku writes) is lower urgency pre-launch and can be deferred post-launch if needed.

11. Failure Modes

12. Security Considerations

• PII collected: None. last_heroku_value is a boolean string. drift_reason is an enum. sync_source is an enum. No user identifiers beyond admin UUIDs (existing created_by).
• Retention: New columns on console_flag_promotions follow the same 90-day operational retention as other columns. console_audit_log entries for mark-synced follow the 2-year audit retention.
• DSR erasure: No new PII columns. Existing created_by / approved_by erasure behavior is unchanged.
• Credential posture: HEROKU_API_TOKEN (and per-app variants) live in Infisical. The reconciler reads them from env at call time. Never logged (hash only). Rotatable without redeploy.
• Audit trail: Every state change to synced column (reconciler write or mark-synced) emits a console_audit_log row with action="console.flag.sync_updated" or action="console.flag.mark_synced". The sync_source column on the promotion row gives a denormalized trace without needing to query the audit log for the common case.
• Kill-switch enforcement at service layer: The synced=false gate is enforced in PromotionService, not just the template. A direct API call to the promote endpoint will also hit the gate. This prevents bypass via curl/scripts.
• Mark-synced requires TOTP elevation: Because mark-synced can push a DB value to Heroku (modifying the runtime), it requires the same @require_totp_elevation as the existing promote/reject endpoints.
• No auto-resolution: The reconciler never silently resolves drift. It only reads and marks. Resolution always requires explicit operator action.
• Breach posture: The new columns contain flag names and boolean values. No user PII. Exfiltration of console_flag_promotions reveals feature flag state and operator activity — operational sensitivity, not GDPR-relevant. Existing breach-notification automation applies.

13. Open Questions

These require a decision from Kristerpher before sub-cards in Phase 2 and Phase 3 can be claimed:

1. Reconciler token scope. The existing flag_drift.py reconciler uses a single HEROKU_API_TOKEN or falls back to per-app tokens. The new reconciler will also need write access for the POST /apps/{app}/config-vars call (mark-synced winner=db path). Should the write token be the same as the read token, or should a separate write-scoped Heroku token be provisioned for the reconciler? Separate tokens provide blast-radius isolation but require vault setup.

2. Backfill timing. The backfill will create ~50+ new console_flag_promotions rows for prod (one per live FLAG_* var with no existing DB row). Should this run as a one-shot admin script, or as an Alembic data migration that runs automatically on deploy? The risk of automatic migration is that it hits the Heroku API during deploy, which could fail and block the migration chain.

3. Drift banner placement pre-FLAG_CONSOLE_FLAG_PROMOTIONS-on. The drift banner (Phase 1/2) requires FLAG_CONSOLE_FLAG_PROMOTIONS=1 to access /console/flags. Should the drift endpoint (GET /console/flags/drift) be accessible without the main promotion flag, so a drift banner can appear on the existing dashboard before the full promotion UI ships? Or is it acceptable to gate all of this behind the single flag?

4. Grace period before kill-switch fires. The moment a CLI operator sets a FLAG_* var directly on Heroku, the next 5-minute reconciler cycle will mark the row drifted and disable the UI promote button. Is 5 minutes the right grace period, or should the kill-switch only fire after N consecutive drifted cycles (e.g., 15 minutes = 3 cycles) to avoid transient failures from Heroku API flakiness?

5. Untracked flags: backfill-only or persistent drift. Flags set directly on Heroku with no DB row are currently orphan_on_heroku warnings. After this design ships, should they create synthetic console_flag_promotions rows automatically on first reconciler detection (enabling the drift banner to surface them), or should they only be onboarded via the backfill script? Automatic creation is more visible but adds complexity to the reconciler.

14. Implementation Cards

PM can lift these cards directly. Each card is sized for a single PR by feature-developer. Cards are ordered by dependency chain.

Card 1 — Schema migration: add sync columns to console_flag_promotions

Title: feat(console): migration 0055 — add sync columns to console_flag_promotions

Body: Add six new columns to console_flag_promotions that support the bidirectional sync design. See design doc: docs/architecture/flag-reconciler-bidirectional-sync-2026-05-13.md section 4 and section 9.1.

Columns: synced (BOOLEAN, default false), last_heroku_value (TEXT nullable), last_synced_at (TIMESTAMPTZ nullable), drift_detected_at (TIMESTAMPTZ nullable), drift_reason (TEXT nullable), sync_source (TEXT nullable).

Partial index: idx_cfp_synced_false on (synced) WHERE synced = false.

Use batch_alter_table pattern (consistent with migration 0054) for dialect-neutral SQLite/Postgres compatibility.

Acceptance criteria: - [ ] Migration 0055 applies cleanly on top of 0054 with no data loss. - [ ] Down migration removes columns and index without error. - [ ] ConsoleFlagPromotion model in app/models/flag_promotion.py updated to include new fields. - [ ] Existing tests pass. - [ ] New test: test_0055_migration.py verifies columns exist post-up and do not exist post-down.

Size: S
Risk: Low — additive columns only.
Dependencies: None (depends on 0054 already applied, which it is).

Card 2 — Reconciler service + scheduler registration

Title: feat(console): flag_reconciler.py — 5-minute bidirectional sync job

Body: Implement console/app/services/flag_reconciler.py as specified in design doc section 8. The reconciler: 1. Iterates all 4 Heroku apps. 2. Fetches FLAG_* config vars via GET /apps/{app}/config-vars. 3. Reads console_flag_promotions rows for that target_app where state NOT IN ('deploying', 'rejected', 'expired', 'failed'). 4. Diffs each row's new_value against the Heroku boolean. 5. Writes synced, last_heroku_value, drift_reason, and timestamp columns. 6. Emits console_audit_log rows with action="console.flag.sync_updated" for each change to synced. 7. Never raises — errors captured per-app in ReconcileResult.

Register in APScheduler: interval=5m, misfire_grace_time=60s, id="flag_reconciler". Skip rows in state='deploying' to avoid race condition during in-flight promotes.

Reuse _fetch_heroku_flag_vars / HerokuConfigFetchError from drift_reconcilers/flag_drift.py or extract to a shared heroku_client.py.

Acceptance criteria: - [ ] reconcile_all_apps() returns one ReconcileResult per app. - [ ] synced=true set when DB new_value matches Heroku value. - [ ] synced=false, drift_reason, drift_detected_at set when mismatch. - [ ] state='deploying' rows are skipped. - [ ] Heroku API failure captured in ReconcileResult.error; no exception propagated. - [ ] Audit row emitted for each synced column change. - [ ] APScheduler job registered in app factory; visible in scheduler job list. - [ ] Unit tests with mocked Heroku API covering: clean run, mismatch, orphan-on-Heroku, Heroku failure. - [ ] No token values in any log output (hash only).

Size: M
Risk: Medium — introduces write behavior; must not corrupt existing promotion state.
Dependencies: Card 1.

Card 3 — Backfill data migration script

Title: feat(console): one-shot backfill — seed console_flag_promotions rows for existing Heroku FLAG_* vars

Body: Implement scripts/flag_reconciler_backfill.py (not an Alembic migration; invoked manually by operator). See design doc section 9.2.

Script logic: 1. For each of the 4 Heroku apps, fetch all FLAG_* config vars. 2. For each var with no matching (flag_key, target_app) row in console_flag_promotions, insert a new row: state='deployed', new_value=<heroku_boolean>, synced=true, sync_source='heroku_cli_backfill', created_by='system_backfill', last_heroku_value=<str>, last_synced_at=<utcnow>. 3. For existing rows in console_flag_promotions for this (flag_key, target_app) where synced=false (NULL or explicit false), diff against Heroku and update. 4. Print a summary: N rows inserted, M rows updated, K errors. 5. Idempotent: running twice produces the same result.

Include a --dry-run flag that prints the diff without writing.

Acceptance criteria: - [ ] --dry-run prints all planned inserts/updates without writing. - [ ] Script is idempotent (run twice = same result). - [ ] All 4 apps processed. - [ ] Rows inserted have sync_source='heroku_cli_backfill'. - [ ] Script exits non-zero on Heroku API failure after logging the error. - [ ] Unit test with mocked Heroku API verifying idempotency and insert/update logic. - [ ] Runbook entry in docs/runbooks/ describing when and how to invoke.

Size: S
Risk: Low — script is separate from main app; uses --dry-run for validation.
Dependencies: Card 1.

Card 4 — Mark-synced API endpoints

Title: feat(console): POST /flags/<key>/mark-synced + GET /flags/drift endpoints

Body: Add two new endpoints to console/app/blueprints/flags.py. See design doc section 5.2.

GET /console/flags/drift: Returns all rows where synced=false. Auth: @require_role("ops", "superadmin"). Response is a JSON list of drifted rows (flag_key, target_app, db_value, heroku_value, drift_reason, drift_detected_at).

POST /console/flags/<flag_key>/mark-synced: Resolves a drift conflict. Body: {"winner": "db"|"heroku", "totp_code": "..."}. Auth: @require_role("superadmin") + @require_totp_elevation. Winner=db: calls Heroku PATCH then sets synced=true. Winner=heroku: updates DB new_value, sets synced=true. Both paths emit audit row action="console.flag.mark_synced". Returns 409 if Heroku PATCH fails (winner=db path).

Also add the drift kill-switch to PromotionService.promote_async() and PromotionService.reject(): return 409 {"error": "flag_drifted"} if the target row has synced=false at the start of the operation.

Acceptance criteria: - [ ] GET /flags/drift returns empty list when no drift. - [ ] GET /flags/drift returns correct rows when drift exists. - [ ] POST /mark-synced winner=heroku updates DB value and sets synced=true. - [ ] POST /mark-synced winner=db calls Heroku PATCH and sets synced=true on success. - [ ] POST /mark-synced winner=db returns 409 heroku_push_failed on Heroku failure. - [ ] POST /mark-synced emits audit row for both winner paths. - [ ] Promote endpoint returns 409 flag_drifted when synced=false. - [ ] Reject endpoint returns 409 flag_drifted when synced=false. - [ ] TOTP elevation enforced on mark-synced. - [ ] Unit tests for each path; mock Heroku API.

Size: M
Risk: Medium — introduces drift kill-switch enforcement; must not break existing promote flow for non-drifted rows.
Dependencies: Cards 1 and 2.

Card 5 — Drift banner UI

Title: feat(console): /console/flags drift banner and per-row drift badge

Body: Update /console/flags page and /console/flags/promotions page. See design doc section on UI changes.

On both pages: add a banner above the table that queries GET /console/flags/drift. If response is non-empty, show a yellow/amber banner: "N flag(s) are drifted — DB and Heroku disagree. Promote and rollback actions are disabled for affected flags until drift is resolved." Each drifted flag in the banner links to its row.

Per-row in the flags table: add a drift badge (e.g., "DRIFTED") in the status column when the row has synced=false. The badge should show drift_reason on hover (tooltip).

Promote/rollback buttons for drifted rows: render as disabled (greyed out, no click handler). Show tooltip: "Flag is drifted — resolve drift first."

Banner auto-hides via HTMX polling (reuse existing 5-second poll pattern from promotions page). If GET /flags/drift returns [], banner removes itself.

Acceptance criteria: - [ ] Banner appears when at least one drifted row exists. - [ ] Banner disappears when no drifted rows remain. - [ ] Per-row drift badge visible for drifted flags. - [ ] Promote/rollback buttons disabled on drifted rows (not just hidden). - [ ] Banner links to correct rows. - [ ] HTMX poll refreshes banner without full page reload. - [ ] Works with existing FLAG_CONSOLE_FLAG_PROMOTIONS gate (404 when off). - [ ] Snapshot test or Playwright test confirms banner renders and hides.

Size: S
Risk: Low — UI-only; no new write paths.
Dependencies: Card 4.

Card 6 — "Mark as synced" modal with side-by-side diff

Title: feat(console): mark-as-synced modal — side-by-side diff + winner selection

Body: Add a modal to the /console/flags page triggered by a "Resolve drift" button on drifted rows. See design doc section on UI changes.

Modal content: - Header: "Resolve drift for <flag_key>" - Two-column diff table: - Left column: "DB value" — shows new_value from the promotion row (true/false) - Right column: "Heroku value" — shows last_heroku_value from the row - drift_reason displayed as a human-readable explanation (e.g., "Heroku value was changed directly via CLI after the last promotion") - drift_detected_at timestamp - Two action buttons: "DB value is correct (push to Heroku)" and "Heroku value is correct (update DB)" - TOTP field (required before either action fires) - Confirmation: high-risk flags (per YAML risk: high) require typing the flag key before confirming

On confirmation: calls POST /console/flags/<flag_key>/mark-synced with winner: "db"|"heroku". On success: modal closes, row refreshes via HTMX to show synced=true state, promote/rollback buttons re-enable. On 409 error: modal shows error message inline.

Acceptance criteria: - [ ] Modal opens on "Resolve drift" button click. - [ ] Side-by-side diff shows correct DB and Heroku values. - [ ] DB wins path: calls mark-synced, shows success, re-enables promote button. - [ ] Heroku wins path: calls mark-synced, shows updated DB value, re-enables promote button. - [ ] TOTP required before action fires. - [ ] High-risk flag confirmation phrase required. - [ ] 409 error displayed inline in modal (not full page redirect). - [ ] Modal is accessible (keyboard navigable, ARIA labels).

Size: M
Risk: Low-Medium — UI complexity; depends on Card 4 for API.
Dependencies: Card 4.

Card 7 — Promote-via-UI calls Heroku API directly

Title: feat(console): POST /flags/<key>/promote — UI-triggered Heroku config-var write

Body: Add the new POST /console/flags/<flag_key>/promote endpoint (distinct from the existing POST /flags/promotions/<id>/promote). See design doc section 5.2.

This endpoint is for direct UI-driven flag flips that bypass the soak queue. It: 1. Validates the flag is not drifted (synced=true or no existing row). 2. Creates a console_flag_promotions row with state='deploying', sync_source='console_ui_promote'. 3. Calls PATCH /apps/{target_app}/config-vars on Heroku (async pattern, same as existing promote flow). 4. On success: sets state='deployed', synced=true, last_synced_at=now. 5. On failure: sets state='failed', synced=false, drift_reason='heroku_unset'. 6. Emits audit row.

Return 202 with status_url for frontend polling (same pattern as existing promote_async).

This is the "Phase 3" endpoint that finally enables operators to flip flags from the UI without touching the Heroku CLI.

Acceptance criteria: - [ ] Returns 202 with status_url on valid request. - [ ] Returns 409 flag_drifted if existing row is drifted. - [ ] On Heroku success: state='deployed', synced=true. - [ ] On Heroku failure: state='failed', synced=false, drift_reason='heroku_unset'. - [ ] Audit row emitted. - [ ] TOTP elevation required. - [ ] End-to-end test with mocked Heroku API. - [ ] FLAG_CONSOLE_FLAG_PROMOTIONS gate enforced.

Size: M
Risk: Medium-High — this is the first endpoint that writes to Heroku from the UI. Requires thorough testing. Must land after Cards 1-6 are stable on staging.
Dependencies: Cards 1, 2, 4.

Card 8 — Audit log integration for all new actions

Title: feat(console): audit trail — emit trace_events for sync_updated and mark_synced

Body: Ensure all new state changes emit to both console_audit_log and trace_events (dual-write). See design doc section 12 (security considerations).

New audit actions to register: - console.flag.sync_updated — emitted by reconciler each time synced changes (true→false or false→true). Payload: {flag_key, target_app, previous_synced, new_synced, drift_reason, last_heroku_value}. - console.flag.mark_synced — emitted by mark-synced endpoint. Payload: {flag_key, target_app, winner, resolved_value, previous_db_value, previous_heroku_value, admin_id}.

Verify that existing audit actions from the promote flow (console.flag.promoted, console.flag.mark_promote) already dual-write to trace_events. If not, add dual-write in this card.

Verify retention policy: console_audit_log rows for these actions must be covered by the existing 2-year retention job.

Acceptance criteria: - [ ] console.flag.sync_updated written on every reconciler synced column change. - [ ] console.flag.mark_synced written on every mark-synced endpoint call. - [ ] Both actions dual-write to trace_events table. - [ ] Payload fields match specification above. - [ ] admin_id is 'system_reconciler' for automated reconciler writes. - [ ] Retention policy confirmed to cover new action types. - [ ] Unit test: reconciler run triggers correct audit row. - [ ] Unit test: mark-synced triggers correct audit row.

Size: S
Risk: Low — audit additions only; does not change existing behavior.
Dependencies: Cards 2 and 4.