Raxx · internal docs

internal · gated

Launch-day playbook — v1 go-live

Owner: operator (Kristerpher) + sre-agent (EyeTok on-call) Created: 2026-05-30 UTC Go/no-go gate: #3151 Parent epics: #3126 (Pre-launch readiness), #94 (v1.0 Production Release) Last reviewed: 2026-05-30 UTC

This runbook is the operator's step-by-step guide for the actual launch day. It does not define what "ready" means — that is #3151. It assumes every checkbox in #3151 is confirmed before Section 3 (T-0) is executed.

Section 1 — T-7 day pre-launch verification

Run this section 7 days before the planned launch date. Every item must be "confirmed done" before the T-0 sequence starts. If any item is unresolved on T-7, escalate immediately — some items (attorney, FinCEN) have external lead times that cannot be compressed.

1A — Secrets + infrastructure

bash heroku config:get WAITLIST_OPT_IN_SIGNING_KEY --app raxx-api-prod | wc -c # Expect > 5

bash heroku config:get POSTMARK_TEMPLATE_REMINDER_2FA --app raxx-api-prod | wc -c # Expect > 5

bash heroku config:get POSTMARK_TEMPLATE_WAITLIST_CONFIRMATION --app raxx-api-prod | wc -c # Expect > 5

bash gh secret list --repo raxx-app/TradeMasterAPI | grep FREESCOUT_API_KEY # Expect: FREESCOUT_API_KEY listed with a recent Updated date

bash gh secret list --repo raxx-app/TradeMasterAPI | grep GPG_BACKUP_PUBLIC_KEY # Expect: present. If missing, follow docs/ops/runbooks/bcp-smoke-monthly-review.md §Operator action item 1. gh run list --workflow bcp-vault-snapshot-daily.yml --limit 1 --json conclusion -q '.[0].conclusion' # Expect: success

bash # From docs/ops/runbooks/github-app-credentials.md python3 scripts/agents/mint_github_token.py --bot raxx-dev-bot --export | grep -c TOKEN # Expect: 1 (token line present, no errors)

bash python3 scripts/agents/mint_github_token.py --bot raxx-pm-bot --export | grep -c TOKEN # Expect: 1

bash gh api repos/raxx-app/TradeMasterAPI/environments/staging/secrets \ | python3 -c "import sys,json; [print(s['name']) for s in json.load(sys.stdin)['secrets']]" \ | grep STAGING_DATABASE_URL # Expect: STAGING_DATABASE_URL

bash python3 scripts/ops/create_sentry_alert_rule.py --dry-run # Expect: existing rule ID printed, exit 0 — see docs/ops/runbooks/sentry-alert-rules.md

bash curl -sI https://raxx.app/ | head -3 # Expect: HTTP/2 200 — NOT a 302 to cloudflareaccess.com curl -sI https://api.raxx.app/health | head -3 # Expect: HTTP/2 200

If either still returns 302: follow docs/ops/runbooks/cf-access.md.

All items below are operator-confirmed manual steps. There is no automated check. Record confirmation in a comment on #3151.

1C — Quality + stability

bash heroku run --app raxx-freescout-prod -- php artisan raxx:freescout-version 2>/dev/null \ || curl -s https://tickets.raxx.app/admin | grep -i "FreeScout" # Record version. Cross-check against https://github.com/freescout-help-desk/freescout/releases

Full procedure: docs/ops/runbooks/freescout.md §Version + CVE audit.

Manual: log into https://tickets.raxx.app/admin and confirm each setting.

Confirm option was selected and implemented per operator comment on #3139. (#3139 closed.)

bash curl -s "https://tickets.raxx.app/admin" | grep -i "version\|1\.8\." | head -3

bash gh run list --workflow bcp-smoke-monthly.yml --limit 3 --json conclusion,createdAt \ | python3 -m json.tool # Expect: at least 2 entries with "conclusion": "success" # If fewer than 2 successful runs: confirm expedited drill outcome with operator sign-off in #3140.

bash gh run list --workflow bcp-smoke-monthly.yml --limit 1 --json conclusion -q '.[0].conclusion' # Expect: success # Also check operator sign-off comment on #3140.

Confirmed done — #3148 is CLOSED. Verify no follow-on bug cards are open:

bash gh issue list --label "area:mbt,bug" --state open # Expect: 0 results, or all known minor items are accepted by operator

Confirm operator sign-off comment on #3147.

Confirmed done — #3150 is CLOSED.

bash # Check via Sentry web UI — filter project=trademaster-raptor, environment=production, # last 72h, sort by first-seen desc. No P0/P1 (unresolved CRITICAL/ERROR tagged as launch-blocker). # URL: https://sentry.io/organizations/moosequest/issues/?project=trademaster-raptor

bash curl -sI https://console.raxx.app/api/internal/deploy-freeze | grep -i "x-deploy-freeze" # If header shows frozen: release via https://console.raxx.app/console/deploy-freeze gh run list --workflow deploy-heroku.yml --limit 1 --json conclusion,headBranch \ | python3 -m json.tool # Expect: conclusion=success, headBranch=main (or most recent prod deploy)

Section 2 — T-24h checks

Run this section the evening before launch day.

bash heroku releases --app raxx-api-prod --num 5 heroku releases --app raxx-console-prod --num 5 # Expect: no failed releases in the last 5 entries

bash gh run list --workflow synthetic-gate.yml --limit 5 --json conclusion,createdAt \ | python3 -m json.tool # Expect: all conclusions=success within the last 24h

Any failure: follow docs/ops/runbooks/synthetic-check-diagnosis.md.

bash gh issue list --label "priority:critical" --state open --limit 20 # Expect: 0 or 0 that are NOT already operator-accepted

bash heroku pg:backups --app raxx-api-prod # Most recent "Completed" backup should be < 24h old

If no recent backup: heroku pg:backups:capture --app raxx-api-prod

bash # Sentry web UI: https://sentry.io/organizations/moosequest/issues/?project=trademaster-raptor # Filter: last 24h, sort by first-seen. No unexpected new issue classes.

bash curl -sI https://console.raxx.app/api/internal/deploy-freeze | grep -i "x-deploy-freeze" # Expect: header absent OR value=false

bash RAXX_SMOKE_ALLOW_PROD=1 make smoke-signup-prod # Expect: SMOKE VERDICT: PASS # Full runbook: docs/ops/runbooks/signup-smoke.md

Section 3 — T-0 launch trigger sequence

Execute in order. Each step has a verification. Do not proceed to the next step until the current step is confirmed.

Before starting: announce in ops Slack channel that the launch sequence is live.

Step 1 — Remove noindex from docs.raxx.app and getraxx.com

Follow docs/ops/runbooks/launch-day-cutover.md §1–§3 exactly:

  1. In frontend/docs/_headers, change X-Robots-Tag: noindex, nofollow to X-Robots-Tag: index, follow.
  2. In scripts/build-customer-docs.py, change content="noindex, nofollow" to content="index, follow".
  3. In frontend/getraxx-landing/public/_headers, change X-Robots-Tag: noindex, nofollow, noarchive, nosnippet to X-Robots-Tag: index, follow.
  4. Open one PR for all three changes, merge to main.

Verification (after CI deploy — ~2–5 min):

curl -sI https://docs.raxx.app/ | grep -i x-robots
# Expect: x-robots-tag: index, follow

curl -sI https://getraxx.com/ | grep -i x-robots
# Expect: x-robots-tag: index, follow

Step 2 — Remove CF Access gate from getraxx.com (if not already removed)

Follow docs/ops/runbooks/getraxx-launch-day-cf-access-removal.md exactly:

# Source credentials
export CLOUDFLARE_API_TOKEN=$(infisical secrets get CF_ACCESS_MGMT \
  --path /MooseQuest/cloudflare/ --plain)
export TF_VAR_cf_access_account_id=$(infisical secrets get CF_ACCESS_ACCOUNT_ID_MOOSEQUEST \
  --path /MooseQuest/cloudflare/ --plain)

# Plan — confirm 3 resources to destroy
cd terraform/modules/cf-access-getraxx
terraform plan -destroy

# Destroy
terraform destroy

Wait 30 seconds for CF propagation.

Verification:

curl -sI https://getraxx.com/ | head -3
# Expect: HTTP/2 200 (NOT 302 to cloudflareaccess.com)

curl -sI https://www.getraxx.com/ | head -3
# Expect: HTTP/2 301 (www → apex redirect; this is correct)

If still 302 after 90 seconds: follow the emergency rollback procedure in docs/ops/runbooks/getraxx-launch-day-cf-access-removal.md §Emergency rollback.

Step 3 — Flip waitlist flag open

heroku config:set FLAG_WAITLIST_DATASTORE=1 --app raxx-api-prod >/dev/null 2>&1
heroku config:set FLAG_GETRAXX_WAITLIST=1 --app raxx-api-prod >/dev/null 2>&1

Verification:

heroku config:get FLAG_WAITLIST_DATASTORE --app raxx-api-prod
# Expect: 1

heroku config:get FLAG_GETRAXX_WAITLIST --app raxx-api-prod
# Expect: 1

Test from a fresh incognito browser: visit https://getraxx.com/ and confirm the waitlist form is visible and interactive.

Step 4 — Confirm console flag reconciler shows no drift

# Flag reconciler runs on a 5-min cycle. After the config:set in Step 3,
# wait 5 minutes then verify no drift alert fires.
# Check: https://console.raxx.app/console/flags
# Expect: all flags show "synced" (not "drift"). No operator-decision modal.

Full procedure: docs/ops/feature-flags-runbook.md.

Step 5 — Trigger waitlist email blast (operator action)

This is a manual operator step. Postmark dashboard or the waitlist blast script.

Prerequisite: POSTMARK_TEMPLATE_WAITLIST_CONFIRMATION confirmed set (Section 1A, item 3).

# Verify Postmark token is active before blast
heroku run --app raxx-api-prod \
  python -c "from api.services.postmark_client import test_postmark_token; test_postmark_token()"
# Expect: Postmark token valid

After blast: monitor ops@raxx.app for any bounce / delivery failure notifications.

Step 6 — Post-launch announcement (operator action)

Operator-owned. Raxx is live when Step 1–5 are complete. Communicate externally per the operator's plan. The sre-agent does not own external-facing communications.

Section 4 — T+1h smoke

Run from a fresh anonymous browser session (incognito, no stored credentials):

bash # Backend validation via signup smoke (parallel to browser test) RAXX_SMOKE_ALLOW_PROD=1 make smoke-signup-prod # Expect: SMOKE VERDICT: PASS

bash curl -sI https://tickets.raxx.app/ | head -3 # Expect: HTTP/2 200

bash # Drift modal fires when Heroku config vs YAML is out of sync. # After Steps 3–4 above, all flags should be synced.

bash # Sentry web UI: https://sentry.io/organizations/moosequest/issues/?project=trademaster-raptor # Filter: last 1h, level=critical. Expect: 0 results.

bash heroku logs --app raxx-api-prod --tail --num 50 | grep -E "H1[0-9]|Error R1[0-9]" # Expect: no H-series router errors

Section 5 — T+24h posture flip

Digest posture switches to per-event the moment real customers exist. Execute these changes within 24 hours of the first customer signup.

Sentry alert routing: digest → immediate

Sentry is already configured to email ops@raxx.app on skipped_no_postmark_token (#3135). For general CRITICAL/ERROR issues, verify the alert frequency is set to immediate (not digest):

# Sentry web UI: https://sentry.io/organizations/moosequest/alerts/rules/
# For each rule in trademaster-raptor: set frequency to "Immediately" (not "Once per day").
# See docs/ops/runbooks/sentry-alert-rules.md §Adding new alert rules

Synthetic gate failures: immediate Slack notification

The synthetic gate already posts to #raxx-ops-alert-sev2 on every failure. No change needed if the channel is monitored. Confirm the webhook is active:

gh secret list --repo raxx-app/TradeMasterAPI | grep SLACK_WEBHOOK_URL
# Expect: present with a recent Updated date

Heroku rate-limit alert threshold

Current ceiling: 9,000 requests/hr (raised 2026-05-14). Alert threshold: 6,000/hr (66% ceiling). If the request rate approaches this level, investigate before it becomes a 429 incident.

heroku logs --app raxx-api-prod --num 1000 | grep "router" | wc -l
# Rough requests-per-log-page estimate; use Heroku metrics dashboard for precise rate

Section 6 — Rollback procedure

If any step in Section 3 breaks and the launch must be aborted:

6A — Restore noindex headers

Revert the docs + getraxx headers PR (one-line revert per surface):

# In frontend/docs/_headers
X-Robots-Tag: noindex, nofollow

# In frontend/getraxx-landing/public/_headers
X-Robots-Tag: noindex, nofollow, noarchive, nosnippet

Open a revert PR to main, merge immediately. CI redeploys in ~2–5 min.

Verification:

curl -sI https://docs.raxx.app/ | grep -i x-robots
# Expect: x-robots-tag: noindex, nofollow

curl -sI https://getraxx.com/ | grep -i x-robots
# Expect: x-robots-tag: noindex, nofollow, noarchive, nosnippet

6B — Re-apply CF Access gate on getraxx.com

cd terraform/modules/cf-access-getraxx
terraform apply
# Recreates the access application and policy from tfvars.

Expected: ~30s for CF propagation. Verify:

curl -sI https://getraxx.com/ | head -3
# Expect: HTTP/2 302 to cloudflareaccess.com

If terraform state has drifted: follow docs/ops/runbooks/terraform-cf-access-state-imports.md.

6C — Flip launch-cohort flags back

heroku config:set FLAG_WAITLIST_DATASTORE=0 --app raxx-api-prod >/dev/null 2>&1
heroku config:set FLAG_GETRAXX_WAITLIST=0 --app raxx-api-prod >/dev/null 2>&1

Verification:

heroku config:get FLAG_WAITLIST_DATASTORE --app raxx-api-prod
# Expect: 0 (or absent)

6D — Incident posture

  1. Post a one-line update in the ops Slack channel: reason + estimated hold time.
  2. Open a SEV1 GH issue with the timeline: first signal → now, steps attempted, current state.
  3. Do not re-attempt T-0 until all Section 1 items are re-verified.

Rollback SLA: 6B (CF Access) should be complete within 10 minutes of the decision to abort. 6A (noindex) CI deploy takes an additional 2–5 minutes.

Section 7 — T+1 week soak

Check these items 7 days after T-0.

7A — MBT drift

# Open #3148 or the drift-orchestrator workflow logs
gh run list --workflow drift-orchestrator-cron.yml --limit 10 --json conclusion,createdAt \
  | python3 -m json.tool
# Expect: all conclusions=success; no drift > 2%/day in any run

7B — Sentry top issues by user impact

# Sentry web UI: https://sentry.io/organizations/moosequest/issues/?project=trademaster-raptor
# Sort by: Users (distinct users affected, not event count) — last 7 days
# Expected: no P0/P1 issues in top 5; any new issue classes triaged and assigned

7C — Postmark delivery health

7D — First customer support queue

curl -s "https://tickets.raxx.app/api/v1/conversations" \
  -H "X-FreeScout-API-Key: $FREESCOUT_API_KEY" \
  | python3 -c "import sys,json; data=json.load(sys.stdin); print('open:', data.get('meta',{}).get('pagination',{}).get('count','?'))"
# Review open ticket trend: is the volume expected? Any patterns indicating a UX gap?

7E — Heroku cost check

heroku apps:info --app raxx-api-prod | grep -i "dyno\|addons"
# Review the current dyno count and add-on plan against the cost model in
# docs/architecture/heroku-cost-model.md (if present)
# Expect: no unexpected tier upgrades from traffic spikes

7F — Signup funnel

heroku run --app raxx-api-prod -- python -c "
import sys; sys.path.insert(0, '/app'); import os; os.chdir('/app')
from backend_v2.db.engine import get_engine
from sqlalchemy import text
with get_engine().connect() as conn:
    total = conn.execute(text('SELECT COUNT(*) FROM waitlist_emails')).scalar()
    confirmed = conn.execute(text(\"SELECT COUNT(*) FROM waitlist_emails WHERE confirmed = true\")).scalar()
    print(f'total={total} confirmed={confirmed} conversion_rate={confirmed/max(total,1)*100:.1f}%')
"

7G — Stripe first invoices (when billing is wired)

Not applicable at v1 if Stripe is not yet live. Verify no accidental Stripe charges have fired in test mode by checking the Stripe dashboard.

7H — Churn signal

Any signup that enrolled a passkey but has not returned to /dashboard within 7 days is a potential early-churn signal. This is informational — pass to the operator for product review, not an SRE action item.

References