ADR 0133 — Console Ghost-Reset Tool: cross-service delete + audit-before-mutate pattern

Status: Accepted
Date: 2026-06-26 UTC
Deciders: software-architect
Scope: Console (beta.py + audit service), Raptor (internal API), Console DB, Raptor DB

Context

A beta tester reaches a "ghost" state when the Console walkthrough token is consumed but the Raptor users row and/or webauthn_credentials row was never persisted (passkey ceremony failed mid-flight; root cause fixed in #2771). The Console "Resend" action refuses because the token is marked used. The operator has resolved this 3x via raw SQL. This ADR records the key design choices for productizing that procedure.

Constraints: - No stored credentials. The ghost shell has 0 passkey rows; the reset must not read, store, or replay anything credential-shaped. - Audit trail before mutation. The platform invariant requires an audit event to be written before any state-change that affects permissions or account existence. - Cross-service operation. Console owns beta_testers and beta_walkthrough_tokens. Raptor owns users and webauthn_credentials. The delete must traverse the service boundary safely. - RBAC gate. The action is restricted to superadmin (Console) or console-users-write (RBAC v2, when provisioned).

Decision

Use a two-step cross-service protocol: Console calls Raptor internal endpoints (/api/internal/users/ghost-check then /api/internal/users/ghost-delete), with audit-before-mutate on both sides.

Console writes console.beta.ghost_reset.initiated to its own audit log before calling Raptor. Raptor writes account.ghost_reset to its audit log inside a database transaction before executing DELETE FROM users WHERE id = :uid. Console writes console.beta.ghost_reset.completed after success.

If the Console audit write fails before calling Raptor, the action does not proceed. If Raptor's internal audit write fails, Raptor rolls back and returns 500; Console surfaces the error without any mutation having occurred.

The ghost-check predicate is a single query that classifies the account as one of four states: ghost_no_users_row, ghost_empty_shell, mid_enrollment, or healthy. The predicate runs twice — once for the UI-visible state, and once inside the Raptor delete transaction (belt-and-suspenders). The 5-minute grace window on users.created_at disambiguates a ceremony-in-flight from a true abandoned shell.

The Console cascade on beta_walkthrough_tokens (revoke all, mint fresh) reuses the existing generate_walkthrough_token() service function. An idempotency check (query for a token minted within the last 60 seconds) prevents double-minting on retry.

Language choice rationale

This ADR does not introduce a new service. The ghost-reset tool is implemented as new routes and endpoint handlers within the existing Console (Python/Flask) and Raptor (Python/Flask) services. No language classification is required.

Consequences

Positive

• Operator can unblock a stuck beta tester in under 2 minutes without touching SQL.
• Audit-before-mutate invariant satisfied on both sides of the service boundary.
• Reuses existing patterns: HMAC machine auth, write_audit, generate_walkthrough_token, _send_welcome_email — no new abstractions invented.
• Ghost predicate is expressed as a single SQL query with a formal definition; easy to test as a unit.
• The 5-minute grace window prevents racing against an in-flight passkey ceremony without requiring a distributed lock.
• Feature flag acts as kill switch without redeploy.

Negative / risks

• Two round-trips (check then delete) across the Console-Raptor boundary introduce a TOCTOU window. Mitigated by the in-transaction predicate re-check inside Raptor.
• If Console succeeds at step 6 (Raptor delete) but fails at step 10 (email send), the account is deleted and a token is minted but the user receives no email. The retry path (run ghost-reset again) will detect ghost_no_users_row and re-mint + re-email. The idempotency check and the audit trail make the gap visible.
• The 5-minute grace window is a heuristic. A passkey ceremony that stalls for more than 5 minutes (e.g. extremely slow device) could be incorrectly classified as a ghost. In practice the WebAuthn ceremony times out in < 60 seconds on all known browsers.

Neutral

• No schema migrations required in either Raptor or Console for the core feature.
• The beta_nda_acknowledgements table (keyed by email, not users.id) survives the delete automatically — no special handling needed.
• audit_log.actor_user_id is TEXT NULL with no FK; audit rows for the deleted user persist correctly with the historical email/user_id reference intact.

Alternatives considered

Alternative A: Console directly connects to Raptor DB

Rejected because it violates service-boundary isolation. Console should not hold a connection to Raptor's Postgres; the two databases run under separate roles with separate credentials. Adding a cross-DB connection would also bypass Raptor's application-layer audit logic.

Alternative B: Single distributed transaction (two-phase commit)

Rejected because the codebase has no two-phase commit infrastructure, and the cost of introducing XA for one operator action is disproportionate. The two-step protocol (check → delete) is safe within the constraints: the worst-case failure leaves a partly-cleared ghost (token revoked but no re-email) that is recoverable by a retry.

Alternative C: Soft-delete the users row (set deleted_at)

Rejected because the ghost shell contains no real data; a hard delete is correct and simpler. Soft-delete would require the re-enrollment flow to detect and clobber the deleted row on next signup, adding complexity. Existing GDPR erasure path for healthy accounts uses soft-delete + scheduled purge, but that path is for accounts with real data that must be retained for the cooling period. A ghost shell has no such data.

Alternative D: Async job / background queue

Rejected for an operator-action tool. The operator needs immediate feedback (success or failure). Synchronous in-request is correct at this scale (one reset at a time, triggered by a human). Bulk/automated sweep (Phase 2) can use a background job when that card is picked up.