Paper Trading v1 — Decomposition and Engine-Gap Map

Status: Design complete; implementation complete; 3 follow-on gaps filed as sub-cards. Date: 2026-07-12 Owner: software-architect Parent epic: #3483 — Paper Trading v1 (close MBT engine gaps + ship unified Simulate surface) Refs: ADR-0108 (MBT fill engine), ADR-0110 (intraday bar feed), ADR-0138 (idempotency key), docs/architecture/mbt-paper-trading-engine.md, docs/architecture/2026-06-11-in-house-paper-trading.md

1. Context

Epic #3483 decomposed the paper-trading build into 15 architect sub-cards (SC-1 through SC-15) and 5 data-scientist gap cards (GAP-1 through GAP-5). All 20 cards are closed and merged. This document is the synthesis artifact: it maps what shipped, reconciles the design against the live code, records the 3 remaining gaps not covered by the original card set, and serves as the Wave 4 exit-criteria reference document (see docs/plan-to-production.md §Wave 4).

2. Invariants That Apply

All invariants from ADR-0108 apply without exception:

• No stored credentials. MBT users hold zero broker tokens. ALPACA_MARKET_DATA_KEY lives in env/SSM only. ADR-0002 CI grep enforces.
• Paper-first gating. Live trading remains unreachable until the paper-profitable gate clears or an operator records an audited override.
• GDPR by default. Every paper_* table row is financial PII. Tier-bound retention enforced by daily cron; users.id CASCADE DELETE covers all paper tables. DSR export runs through existing GDPR machinery.
• Audit trail for every state change. Every fill, rejection, expiration, dividend credit, assignment, and account reset writes an audit_log row.
• Credentials into infra. ALPACA_MARKET_DATA_KEY and ALPACA_MARKET_DATA_SECRET are env/SSM only; never in DB or shipped files.
• Data licensing. Every market-data fetch writes a row to market_data_access_log(user_id, symbol, data_type, fetched_at, source). Cron-level fetches log user_id=0 as a sentinel (individual user IDs unavailable at bars-provider construction time; log volume still satisfies redistribution audit AC).
• Idempotency on money paths. Per ADR-0138 (#4161/#4168), every order-submit endpoint must carry @idempotency_key(fail_closed=True) and accept a caller_conn for transactional atomicity. Inherited by SC-5/6/7/8 (see §6).
• Alpaca redistribution gate. FLAG_PAPER_TRADING_V1 stays off on prod until operator files a comment on #3483 confirming the redistribution posture decision (ADJ-14 in docs/plan-to-production.md).

3. Data Model

Schema shipped across migrations 0022 (initial MBT tables), 0027 (position group ID), and 0035–0039 (SC-1 through SC-6 columns):

Reused tables (tax engine, no changes): tax_lots, wash_sale_lots, holding_period_classifications, lcc_strategy_config.

Migration invariants: All PL/pgSQL blocks carry the -- POSTGRES-ONLY sentinel. All migrations define op.drop_table rollback. Real-Postgres smoke tests ran for each migration per project policy.

4. Engine-Gap Map (All Closed)

Note on GAP-2 and the synchronous market order path: The resting-order cron (process_resting_orders.py) injects Alpaca 1-min bars carrying live bid/ask snapshots. The synchronous market order path in trading.py injects EOD bars via fetch_or_load_bars (Phase 1, "Phase 2 will use 1-min bars"). When EOD bars lack real-time bid/ask, _fill_market() falls back to the midpoint of (high+low). This is a fill-quality gap addressed by PAPER-23.

5. APIs and Contracts

All endpoints live under /api/trading/ (Blueprint trading, backend_v2/api/routes/trading.py), gated on FLAG_PAPER_TRADING_V1:

Response envelope: {"data": ..., "meta": {...}}. Errors: {"error_code": "...", "message": "...", "request_id": "..."}.

6. Money-Path Idempotency Inheritance (ADR-0138)

Per ADJ-2 in docs/plan-to-production.md: Idempotency-Key is required on Tier-1 (paper order submit, order replace). The caller_conn parameter is wired on all three MbtFillEngine submission methods, making the paper order INSERT and the idempotency completion UPDATE atomic on a single connection.

Replay returns HTTP 201 with the original response body. Replay events logged to idempotency_log separately from audit_log.

7. Simulate Surface Component Tree (UI-bearing cards)

Gated at the page level: app/(authed)/simulate/page.tsx redirects to /dashboard when FLAG_PAPER_TRADING_V1 is off (hide, never gray — per project feedback).

app/(authed)/simulate/page.tsx          — Server Component; flag gate + auth guard (SC-11)
  └── components/simulate/SimulateView.tsx   — 'use client'; URL toggle + 5s polling (SC-11)
        ├── components/backtest/BacktestView.tsx — backward mode; no regression (SC-11)
        ├── components/simulate/TradeHistoryTab.tsx — orders + tax lot summary (SC-12)
        ├── components/simulate/ResetButton.tsx    — 30-day cooldown + two-step confirm (SC-13)
        └── components/simulate/SwitchToLiveCta.tsx — graduation gate CTA (SC-14)

Toggle state (?mode=forward|backward) and active tab (?tab=portfolio|history) are URL-persisted for deep-linking. Polling pauses via Page Visibility API when the tab is hidden. Missed-bar banner appears when mbt_catchup.missed_bars > 5 (logged by evaluate_resting_orders).

UI-bearing cards: SC-11, SC-12, SC-13, SC-14. Engine/API-only (no frontend work): SC-1 through SC-10, SC-15, GAP-1 through GAP-5.

8. Batch Job Schedule

All jobs ship as Python command modules invoked by Woodpecker cron pipelines:

9. Remaining Gaps — New Sub-Cards Filed

Three items were not covered by the original SC-1..SC-15 / GAP-1..GAP-5 card set:

PAPER-21: EOD snapshot command + Woodpecker pipeline

mbt-paper-trading-engine.md §8 specifies paper.eod_snapshot at 21:00 UTC Mon-Fri, writing one mbt_eod_snapshots row per user (equity, cash, positions_json JSONB). No command module or WP pipeline exists. Required for: historical equity tracking, drift comparison baseline, and GDPR data-export completeness. Blocks the Wave 4 7-day soak baseline calculation.

PAPER-22: Tier retention purge command + Woodpecker pipeline

Specified as paper.retention_purge at 02:00 UTC daily, soft-deleting paper_orders, paper_fills, and closed paper_positions older than tier cap (Free 90d / Pro 3yr / Pro+ skip). The market_data_access_log 13-month purge is already present in SC-3; this card covers the paper trading tables only.

PAPER-23: Market order real-time quote (Phase 2 bars provider)

The _bars_provider in trading.py:_get_mbt_engine() uses fetch_or_load_bars (EOD bars). When EOD bars have no bid/ask, _fill_market() uses the midpoint of (high+low) from the most recent bar rather than a live quote. AlpacaMarketDataService.get_latest_quote() exists and returns real-time bid/ask. Phase 2 wires the snapshot quote into the synchronous market order path so fills are at live bid/ask. Simulation quality gap, not a correctness bug. Must land before beta phase; acceptable at dogfood if operator is aware.

10. Dependency Chain

SC-15 (flag + B1 migration) — ships first
  SC-1, SC-2, SC-3 (schema) — parallel after SC-15
    SC-4 (Alpaca client)
      GAP-1 (1-min bars into resting-order cron)
        GAP-2 (bid/ask fill — active in _fill_market)
          SC-5 (market orders)
            SC-6 (limit orders + resting eval)
            SC-7 (IC builder)
            SC-8 (LCC roll)
            SC-9 + GAP-3 (expiry sweep)
            SC-10 (dividend + split)
            GAP-5 (equity MTM)
      GAP-4 (tax hook, Founders gate)
        SC-12 (trade history + tax section)
  SC-11 (Simulate UI forward mode)
    SC-12 (Trade History tab)
  SC-13 (reset endpoint + UI)
  SC-14 (CTA placeholder)

PAPER-21 (EOD snapshot) — no predecessor required
PAPER-22 (tier retention purge) — no predecessor required
PAPER-23 (market order real-time quote) — after SC-4; before beta flip

11. Rollout Plan

12. Security and GDPR Checklist

• PII collected: user_id, symbol, fill prices, lot cost basis. No names or contact data in paper trading tables.
• Retention: Free 90d / Pro 3yr / Pro+ unlimited. market_data_access_log 13-month retention. PAPER-22 implements the paper trading tier purge.
• DSR deletion: CASCADE DELETE on users.id for all paper_* tables. 30-day cooling per ADR-0003. audit_log rows persist with hashed actor ID for 2 years.
• Audit logging: Every fill, rejection, expiry, dividend credit, assignment, reset → audit_log row; 7-year retention for trade-affecting rows.
• Stored credentials: None. ADR-0002 CI grep enforces.
• Breach path: Paper tables contain simulated trade history only; no real money, no credentials. Standard ADR-0003 72h notification path.
• Secrets rotation: ALPACA_MARKET_DATA_KEY rotatable without redeploy.
• Kill switches: MBT_TRADING_DISABLED=1 (503 all order submits), MBT_NEW_ORDERS_DISABLED=1 (blocks new submits only), FLAG_PAPER_TRADING_V1=off (full revert).

13. Open Questions

1. Alpaca redistribution posture — Operator comment on #3483 required before beta or prod flip (ADJ-14). Not a code card.
2. PAPER-23 dogfood timing — Can dogfood proceed with EOD-bar fills and defer real-time quote wiring to before beta? Operator decision, not a design decision.
3. Securities attorney review (#197) — Must resolve before beta. Not a code card; gates SC-11 and SC-12 from reaching real users.

See ADR-0108 for the fill engine spec, ADR-0110 for the intraday bar feed design, ADR-0138 for the idempotency key mechanism, and 2026-06-11-in-house-paper-trading.md for the complete per-card acceptance criteria.