Raxx · internal docs

internal · gated

Email Verification Rollout Runbook

Issue: #3272
Flags: FLAG_AUTH_EMAIL_VERIFICATION (Raptor) + FLAG_EMAIL_VERIFICATION_UI (Antlers)
B1 migrations: 0146 (auth_email_verification), 0147 (email_verification_ui)

What ships

Layer 1 — syntactic validation at signup (always on; improves data quality).
Layer 2 — MX-record lookup on signup email domain (gated by FLAG_AUTH_EMAIL_VERIFICATION).
Layer 3 — signed-link email verification via Postmark; email_verified flips on click.
Trading gate — POST paper/live order routes return 403 when user is unverified.
Waitlist carryover — if waitlist double-opt-in was confirmed, new user starts verified.

Pre-flip checklist

Before flipping either flag, verify:

  1. Postmark DKIM authenticated on raxx.app — check Postmark dashboard > Sender signatures.
  2. POSTMARK_SERVER_TOKEN is set on both raxx-api-staging and raxx-api-prod. heroku config:get POSTMARK_SERVER_TOKEN --app raxx-api-staging
  3. APP_BASE_URL is set correctly on each app (used to build verify links): - Staging: https://raxx-app-staging.raxx.app (or your staging frontend URL) - Prod: https://raxx.app heroku config:get APP_BASE_URL --app raxx-api-staging heroku config:set APP_BASE_URL=https://raxx.app --app raxx-api-prod >/dev/null 2>&1
  4. Run local smoke before staging retest: bash scripts/smoke/email_verification_smoke.sh http://localhost:5001

Staging flip sequence

# Step 1 — flip backend flag on Raptor staging
heroku config:set FLAG_AUTH_EMAIL_VERIFICATION=1 --app raxx-api-staging >/dev/null 2>&1

# Step 2 — flip frontend flag on Antlers staging
heroku config:set NEXT_PUBLIC_FLAG_EMAIL_VERIFICATION_UI=1 --app raxx-app-staging >/dev/null 2>&1
# OR for the CRA-based Antlers:
heroku config:set REACT_APP_FLAGS=email_verification_ui --app raxx-app-staging >/dev/null 2>&1

Staging verification steps

  1. Sign up with a real email on staging.
  2. Confirm the verification email arrives at the inbox (sender: no-reply@raxx.app).
  3. Click the link — confirm the page shows "Email verified" and redirects to dashboard.
  4. Check that email_verified_at_utc is set: heroku pg:psql --app raxx-api-staging -c \ "SELECT email, email_verified_at_utc FROM users WHERE email = 'your@email.com';"
  5. With a separate unverified test account, navigate to /trading. - Trade-action buttons must be HIDDEN (not grayed). - The "Verify your email to start trading" card must be visible.
  6. After verifying the second account: trade-action buttons appear.
  7. Try submitting a trade order with an unverified session (e.g. via curl with a valid unverified session cookie). - Expected: 403 {"error": "email_verification_required"}.
  8. Confirm MX check: try signing up with test@this-domain-does-not-exist.invalid. - Expected: 400 {"error": "email_domain_unreachable"}.

Soak period

Minimum 48 hours on staging with no error spike before prod flip.

Monitor Sentry for: - auth_email.* route errors - email_gate.blocked log events - Postmark bounce/complaint rate

Prod flip sequence

After 48h soak:

heroku config:set FLAG_AUTH_EMAIL_VERIFICATION=1 --app raxx-api-prod >/dev/null 2>&1
heroku config:set NEXT_PUBLIC_FLAG_EMAIL_VERIFICATION_UI=1 --app raxx-app-prod >/dev/null 2>&1

Rollback

heroku config:set FLAG_AUTH_EMAIL_VERIFICATION=0 --app raxx-api-prod >/dev/null 2>&1
heroku config:set NEXT_PUBLIC_FLAG_EMAIL_VERIFICATION_UI=0 --app raxx-app-prod >/dev/null 2>&1

Rollback is safe — flag off restores the pre-#3272 behaviour: - Endpoints return 404. - Trading gate is a no-op. - MX check skipped. - Trade-action buttons show regardless of verification status.

Existing email_verified_at_utc rows are preserved. Re-enabling the flag continues from where it left off.

Edge cases

Waitlist users signing up

If the user was on the waitlist and clicked the double-opt-in confirmation, their confirmed_at is NOT NULL. The signup route detects this and seeds email_verified_at_utc on user creation. They skip verification.

Resend rate limiting

3 resends per hour per IP. If a user hits the limit, the frontend shows a cooldown message. They can verify via the original email link or wait for the cooldown.

Token expiry

Links expire after 24 hours. Expired links show the "Link expired" page with a "Sign in to resend" CTA. The user signs in and clicks "Resend verification email" from the pending page or from the trading gate card.

MX check transient failure

Transient DNS failures (Timeout, ServFail) are logged at WARNING and do not block signup (fail open). Only definitive NXDOMAIN / NoAnswer responses block. Cache TTL is 5 minutes.

Contacts

Postmark dashboard: https://account.postmarkapp.com
DKIM status: check under Sender Signatures for raxx.app
Support: support@raxx.app

History — 2026-06-04 MBT paper tables + waitlist double opt-in (migrations 0022/0023)

System: Raptor (raxx-api-prod, raxx-api-staging) — waitlist double opt-in + MBT schema
Owner: operator / sre-agent
Last incident: 2026-06-04 (escalation + authorized deploy; see timeline below)
Last reviewed: 2026-06-04 UTC

What this covers

Two schema changes landed together in the 2026-06-04 production deploy (v101 → alembic head 0027):

  1. Migration 0022 — MBT paper tables (destructive schema swap). Dropped the legacy broker-mirror paper_orders table created by migration 0002 and replaced it with the MBT-engine-owned paper_orders, paper_accounts, and paper_positions tables.

  2. Migration 0023 — waitlist_signups double opt-in columns. Added confirmed_at, confirmation_sent_at, last_resend_at, resend_count, unsubscribed_at and a partial index to support the double opt-in email confirmation flow.

Migration 0022 — paper_orders schema swap: what happened and why it is safe

Background

Migration 0002 (2026-01 era) created a broker-mirror paper_orders table keyed on broker_order_id, intended for Alpaca live-broker audit. That table had no user_id foreign key; it was a pass-through mirror of the broker's own order state.

Migration 0022 (2026-05-29) drops that table and creates a new paper_orders with a user_id FK, strategy_id FK, fill_price, slippage, legs, and asset_class columns — the schema required by the MBT (Market Back-Test) paper-trading engine (ADR-0108).

Why the DROP was authorized (2026-06-04 operator decision)

The operator confirmed on 2026-06-04:

"All the paper trading we originally did was with Alpaca. So nothing is needed to keep there."

All Alpaca paper-trading data of record lived in Alpaca's broker-side records. The broker-mirror paper_orders table on raxx-api-prod had zero rows of value at time of migration. The DROP destroyed throwaway schema, not user data.

The authorization is recorded here as a durable SRE artifact. Any future operator who sees the table is gone and wonders why should read this note.

Downgrade path

If a rollback past 0022 is ever needed, the 0022 downgrade() function recreates the 0002-era broker-mirror paper_orders schema exactly (see 0022_mbt_paper_tables.py docstring). The MBT-owned tables (paper_accounts, paper_positions, new paper_orders) are dropped on downgrade.

How to tell the rollout is broken

How to diagnose (in order)

  1. Check the Heroku release log for migration output: gh run list --repo raxx-app/TradeMasterAPI --workflow deploy-heroku.yml --limit 10 gh run view --log --job <job-id> | grep "Running upgrade" Expected: last line says Running upgrade 0026 -> 0027 ....

  2. Verify alembic head via Heroku one-off dyno: heroku run "cd /app && alembic -c backend_v2/alembic.ini current" --app raxx-api-prod Expected output: 0027 (head).

  3. Verify waitlist_signups schema: heroku run "cd /app && python -c \" from sqlalchemy import create_engine, text, inspect; import os e = create_engine(os.environ['DATABASE_URL']) cols = [c['name'] for c in inspect(e).get_columns('waitlist_signups')] print(cols) \"" --app raxx-api-prod Expected: confirmed_at, confirmation_sent_at, last_resend_at, resend_count, unsubscribed_at all present.

  4. Verify paper_orders is the MBT schema (has user_id, not broker_order_id): heroku run "cd /app && python -c \" from sqlalchemy import create_engine, inspect; import os e = create_engine(os.environ['DATABASE_URL']) cols = [c['name'] for c in inspect(e).get_columns('paper_orders')] print(cols) \"" --app raxx-api-prod Expected: user_id, strategy_id, fill_price, slippage, legs, asset_class present. broker_order_id absent.

Known failure modes

Failure mode A: migration ran but 500 on /api/waitlist/signup

Symptom: Alembic head is 0023+ but POST to /api/waitlist/signup returns 500.

Cause: The waitlist_service.py or a downstream dependency (Postmark, rate-limiter) is raising an unhandled exception. Schema is correct; this is an application bug.

Diagnosis: - Check Sentry project raxx-backend, filter last 1 hour, sort by count. - Check Heroku logs: heroku logs --tail --app raxx-api-prod | grep waitlist

Fix: Escalate to feature-developer with the Sentry issue and stack trace.

Failure mode B: migration 0022 left paper_orders with wrong schema

Symptom: inspect('paper_orders') shows broker_order_id column (legacy schema) or the table is absent.

Cause: Migration 0022 upgrade() may have failed partway through (e.g., the DROP succeeded but the CREATE failed, leaving no paper_orders table).

Fix: 1. Check alembic current — if head is 0021 the upgrade aborted cleanly; if it is 0022 but the table is wrong, there is data inconsistency. 2. Do NOT run alembic upgrade head manually without checking for FK dependencies first. 3. Escalate to operator with the exact inspect() output and the error from Heroku logs.

Failure mode C: post-deploy health check fails (H10/H12)

Symptom: GH Actions Post-deploy health check step returns non-200 after 5 attempts. Deploy job fails. Release version is not promoted.

Cause: Dyno crash on boot (H10) — most commonly a missing env var or a migration that left the DB in a state the app can't handle on startup.

Diagnosis: 1. heroku logs --tail --app raxx-api-prod | grep "app\[web\]" — look for the exception. 2. heroku ps --app raxx-api-prod — confirm dyno state. 3. Check Sentry for boot-time errors.

Fix: Per docs/ops/runbooks/raptor.md H10/H12 section.

Enabling double opt-in flow (flag gate)

The waitlist double opt-in flow is gated behind FLAG_WAITLIST_DOUBLE_OPTIN. Schema is deployed but the feature is inert until the flag is turned on.

To enable on prod:

heroku config:set FLAG_WAITLIST_DOUBLE_OPTIN=1 --app raxx-api-prod >/dev/null 2>&1

Verify: POST /api/waitlist/signup with a new email should return 202 and trigger a confirmation email via Postmark. Check waitlist_signups.confirmation_sent_at is set for the row.

To disable:

heroku config:set FLAG_WAITLIST_DOUBLE_OPTIN=0 --app raxx-api-prod >/dev/null 2>&1

Deploy history

Date (UTC) Version Alembic head Notes
2026-06-04 15:43 v101 0027 First prod deploy of 0022+0023. Operator authorized DROP of legacy paper_orders.

References

Escalation

Escalate to operator when: - Alembic head is below 0023 after a deploy that showed migration lines in the log. - paper_orders table is absent or has neither schema (legacy nor MBT). - Any 5xx from /api/waitlist/signup that persists after a dyno restart.

Contact: Kristerpher (GitHub: MooseQuest)