Staging testing guide — operator walk + automated pre-checks

System: staging-nextjs.raxx.app (Antlers) + api-staging.raxx.app (Raptor) + console-staging.raxx.app (Console) Owner: operator / qa-agent Verified against: build 0.0.128-7042, RC release-2026.08.23, develop 333be5eb1 — qa-agent staging walk, 2026-08-23 Last reviewed: 2026-08-23 Basis for: Phase 3 human-tester program (staging walks below are the template testers will run)

Staleness warning

This guide is pinned to a specific staging build. Before relying on it, confirm the build currently live on staging still matches (or is a superset of) the verified build above:

curl -s https://api-staging.raxx.app/api/system/version

If the returned build string is newer than 0.0.128-7042, the automated checks in §2 are still a reasonable starting battery (the routes they exercise are stable), but any FAIL should be treated as "needs re-triage against current code", not automatically as a regression. If the build is older than 0.0.128-7042, staging has not yet received the RC this guide was verified against — do not use this guide to sign off that RC.

1. What staging is

• Scope: api (Raptor, raxx-api-staging) + console (raxx-console-staging) only. Queue and Velvet are not part of staging — Queue-side behavior (e.g. dunning handlers, billing webhook processing) cannot be walked on staging. See docs/ops/runbooks/billing-test-tooling.md §1 for the Queue-staging billing topology if that's what you're actually trying to test.
• Deploy trigger: staging deploys only from release-YYYY.MM.DD tags on develop (.woodpecker/deploy-staging.yaml, event: tag on refs/tags/release-*) — tag-scoped, 8 pipeline steps, followed by an automatic no-fast-forward promote of develop → release. A merge to develop alone does not reach staging until the next release-candidate tag is cut. See docs/ops/runbooks/gatekeeper-develop-to-release.md for the cut/promote mechanics and docs/ops/runbooks/ci-woodpecker.md for Woodpecker pipeline behavior generally (manual-trigger side effects, failure modes, etc.) — this guide does not duplicate either.
• Version ground truth — always check this first, not last:

bash curl -s https://api-staging.raxx.app/api/system/version

The build suffix in the response must equal the Woodpecker pipeline number that ran deploy-staging. A green deploy-staging pipeline in Woodpecker is necessary but not sufficient — the Heroku release phase (migrations) runs after the slug push, and on failure Heroku keeps serving the old slug while every health probe keeps returning 200. Do not stop at "pipeline was green" or "/health returns 200" — those alone do not prove the new code landed. If in doubt, cross-check heroku releases -a raxx-api-staging for a release NOT marked "release command failed" (never heroku releases:info — it dumps the full plaintext config snapshot). The same three-check pattern (release status + DB migration head + a code-signature probe) used for prod releases is documented in docs/ops/runbooks/post-merge-prod-state-checklist.md §Step 1 / §Step 5 — apply the same logic to staging, substituting raxx-api-staging for raxx-api-prod.

2. Automated pre-walk checks (sandbox-executable)

Everything in this section is a plain curl — no browser, no credentials, no device. Run these before any human walk in §3; if the golden path is broken, there is no point walking the UI on top of it.

All commands assume no session cookie is sent (i.e. run from a clean shell/sandbox, not a browser dev-tools console with an active login).

2.1 Public front door

curl -s -o /dev/null -w "%{http_code}\n" https://staging-nextjs.raxx.app/
curl -s -o /dev/null -w "%{http_code}\n" https://staging-nextjs.raxx.app/login
curl -s -o /dev/null -w "%{http_code}\n" https://staging-nextjs.raxx.app/signup

PASS: all three return 200, no 5xx.

curl -s https://staging-nextjs.raxx.app/ | grep -o '<title>[^<]*</title>'

PASS: <title>Raxx</title>.

for p in / /login /signup; do
  curl -s "https://staging-nextjs.raxx.app${p}" | grep -Eio 'alpaca|tradier|schwab|interactive brokers|plaid'
done

PASS: zero output (no vendor-name leakage into customer-facing copy — broker is plumbing, never customer-visible branding).

curl -s -X POST https://staging-nextjs.raxx.app/api/waitlist/signup -d '{}' -H 'Content-Type: application/json'

PASS: 400 {"error":"email is required"} — pre-auth reachable, route's own validation runs (this endpoint is always-on, double-opt-in).

curl -s -o /dev/null -w "%{http_code}\n" -X POST https://staging-nextjs.raxx.app/api/waitlist/subscribe

PASS: 404 — FLAG_GETRAXX_WAITLIST is default OFF; this is expected, not a bug.

2.2 #4526 code-signature probe — regression sentinel (run this every time)

This is the 8th recurrence of the session-auth exemption failure class (see docs/ops/runbooks/session-auth-exemptions.md for the full history and root cause). Treat a FAIL here as high-priority — it means a new/moved internal route regressed the exemption list, not a one-off staging flake.

curl -A "raxx-staging-probe/1.0" -X POST \
  https://api-staging.raxx.app/api/internal/billing/mirror-sync \
  -H "Content-Type: application/json" \
  -d '{"customer_id":1,"event_type":"tier_changed","plan_tier":"pro"}'

• PASS: 403 {"error":"forbidden"} — the route's own service-token check ran and rejected the unauthenticated request.
• FAIL / regression: 401 {"error":"Authentication required","reason":"missing"} — this shape means the session-auth middleware intercepted the request before the route's own logic ran, i.e. the exemption is gone or was never added for the current path. Go straight to docs/ops/runbooks/session-auth-exemptions.md §"How to diagnose" rather than re-running the probe.

Companion checks (same failure class, cheap to run alongside):

curl -s -X POST https://api-staging.raxx.app/api/subscriptions/apple/notifications \
  -H "Content-Type: application/json" -d '{}'

PASS: 400 bad_request {"error":"Request body must be JSON"}-shaped response (exact wording may vary) — the route's own validation ran, not the session-auth 401.

curl -s -o /dev/null -w "%{http_code}\n" https://api-staging.raxx.app/api/settings

PASS (control): 401 generic session-auth shape — this route is supposed to be session-gated, so a plain 401 here is correct and confirms the middleware itself is alive (i.e. the mirror-sync PASS above isn't just "middleware is entirely down").

2.3 Golden path (all confirmed on 0.0.128-7042)

curl -s https://api-staging.raxx.app/health

PASS: 200 {"status":"ok"} — this is the real liveness probe (see known-issue footnote at the bottom of this guide re: a dead /api/system/health entry).

curl -s https://api-staging.raxx.app/api/system/version

PASS: 200, non-empty build string. This is the deploy ground-truth check from §1 — run it here too if you skipped straight to §2.

curl -s -o /dev/null -w "%{http_code}\n" https://api-staging.raxx.app/health/migrations

PASS: 404 — FLAG_RAPTOR_MIGRATION_HEAD is off on staging. Expected, not a bug.

curl -s -X POST https://api-staging.raxx.app/api/auth/login/options \
  -H "Content-Type: application/json" \
  -d '{"email":"nonexistent@example.com"}'

PASS: 200, a valid WebAuthn challenge object, rpId: "raxx.app" (ADR-0005), allowCredentials: []. This must be a constant 200 regardless of whether the email exists — a 404/400 here is a user-enumeration regression.

for r in /api/billing/checkout-session /api/billing/refund /api/billing/dsr/123/initiate; do
  curl -s -o /dev/null -w "%{http_code} $r\n" -X POST "https://api-staging.raxx.app${r}"
done

PASS: all three 401 (generic session-gated shape) — not 404. A 404 here would mean the route isn't registered / mounted, which is a different and worse failure than "not authenticated".

curl -s -o /dev/null -w "%{http_code}\n" https://api-staging.raxx.app/api/dashboard/summary
curl -s -o /dev/null -w "%{http_code}\n" -X PATCH https://api-staging.raxx.app/api/settings
curl -s -o /dev/null -w "%{http_code}\n" https://api-staging.raxx.app/api/settings

PASS: all 401 generic — authed-prefix control.

curl -s -X POST https://api-staging.raxx.app/webhooks/postmark/delivery \
  -H "Content-Type: application/json" -d '{}'

PASS: 401 {"error":"invalid_token"} — this route lives outside the /api prefix by design (session_auth.py L158) and has its own HMAC check; an unsigned request should fail that check, not fall through as if unauthenticated-but-reachable.

curl -s -X POST https://api-staging.raxx.app/api/webhooks/freescout/support \
  -H "Content-Type: application/json" -d '{}'

PASS: 401 {"error":"invalid signature"} — route's own signature check.

curl -s -o /dev/null -w "%{http_code}\n" https://console-staging.raxx.app/

PASS: 302 (to CF Access login). Console is fully CF-Access-gated; a 302 here means "up", not "broken" — do not chase this as a bug.

3. Human walks (not sandbox-executable)

Run these only after §2 is fully green. Each walk lists preconditions, steps, expected result, and environment-specific quirks to not mistake for bugs.

Walk 1 — CF Access console walk

Preconditions: CF Access identity (email-OTP) for an authorized operator.

Steps: 1. Navigate to https://console-staging.raxx.app/. 2. Log in via the CF Access email-OTP prompt. 3. Walk the Security / Status / Issues nav (docs/memory/project_console_nav_structure.md).

Expected: Console loads after CF Access auth, nav renders, no 5xx on any of the three sections.

Environment quirk: CF Access gates the whole host (§2.3 confirmed the bare 302 already). Bot Fight Mode is OFF on the raxx.app family of hosts, so there is no bot-challenge friction to work around during login.

Walk 2 — Real signup with passkey

Preconditions: Either a real device with Face ID / Touch ID / a security key, or the synthetic-credential smoke harness (no real device needed).

Steps — on-device: Tap through the signup flow at https://staging-nextjs.raxx.app/signup normally.

Steps — synthetic harness (sandbox-executable substitute for this walk):

RAXX_SMOKE_ALLOW_PROD=1 \
RAXX_SMOKE_API_URL=https://api-staging.raxx.app \
HEROKU_APP=raxx-api-staging \
RAXX_SMOKE_ORIGIN=https://staging-nextjs.raxx.app \
python -m pytest backend_v2/tests/smoke/test_signup_e2e_prod.py -s -v

Expected: Passkey registers; users + webauthn_credentials rows land; harness prints SMOKE VERDICT: PASS.

Environment quirks: - RAXX_SMOKE_ORIGIN must be https://staging-nextjs.raxx.app exactly — omitting it or leaving the prod default reproduces the #4444 origin-mismatch false-failure, not a real regression. Full detail: docs/ops/runbooks/signup-smoke.md §"Running against staging". - RP ID stays raxx.app regardless of environment (ADR-0005) — only the origin changes per environment. - The harness needs heroku CLI auth against raxx-api-staging for the DB-write confirmation step. - Passkeys fail inside in-app/webview browsers — a real browser tab is required for the on-device path.

Walk 3 — Billing page with flag on

Preconditions: FLAG_ANTLERS_BILLING_SELF_SERVICE=1 set on staging (default OFF, soak 72h, risk high — confirm it's actually flipped before walking, don't assume); an authenticated session; an existing customer with a subscription. B1 console promotion migration 0306 must be applied.

Steps: 1. Log in at https://staging-nextjs.raxx.app/. 2. Navigate to https://staging-nextjs.raxx.app/account/billing. 3. Verify plan, payment method, and invoice history all render.

Expected: Confidence Engine styling throughout; plan + payment method + invoice history populated; no 5xx; correct tier gating; no vendor names anywhere in the rendered copy.

Environment quirk: The flag is OFF by default — this is the single most likely "walk fails" cause for this scenario and should be checked first, not last. For the Stripe test-mode data setup behind this walk (test customer/subscription provisioning), see docs/ops/runbooks/billing-test-tooling.md §4 — do not reinvent that setup here.

Walk 4 — Paper-trade flow

Preconditions: FLAG_PAPER_TRADING_V1=1 AND, for options coverage, FLAG_OPTIONS_PAPER_TRADING=1 (AND-semantics — both must be on for the options path). Authenticated session. Both flags are gated behind the Alpaca Data API redistribution clause and #197 securities-attorney sign-off — confirm that's cleared before flipping either flag on staging.

Steps: 1. In Antlers, open the "Simulate" tab. 2. Place a paper order (POST /api/trading/simulate/*). 3. Verify GET /api/trading/paper/portfolio shows the $100k notional auto-init on first request. 4. Verify order history via GET /api/trading/paper/orders. 5. Cancel/reset via DELETE /api/trading/paper/orders and POST /api/trading/paper/reset.

Expected: Order fills via the MBT engine; portfolio and order-history reads work; reset clears state cleanly.

Environment quirk: Kill switches MBT_TRADING_DISABLED=1 / MBT_NEW_ORDERS_DISABLED=1 exist for maintenance windows — if this walk unexpectedly no-ops, check those before assuming a code regression.

4. Environment quirks appendix

• Bot Fight Mode: OFF on raxx.app + moosequest.app, ON on getraxx (docs/memory/project_bfm_off_pre_launch.md). Non-browser probes should still set a distinctive User-Agent — a generic/empty UA can trip CF error 1010 on hosts where BFM is on.
• CF Access bypasses exist for /login, /help/passkeys*, /beta/walk/*, /beta/preview/* on raxx.app — don't be surprised these don't prompt for CF Access even though the rest of the host does.
• Synthetic-credential convention: synthetic accounts only, never real PII — see docs/ops/runbooks/signup-smoke.md §"Prod-write safety" for the naming and teardown convention (applies the same way on staging).
• Staging scope: api + console only — Queue and Velvet are not deployed to a staging-equivalent for this program; anything that depends on Queue-side state (dunning, webhook processing) cannot be walked here.
• Deploy cadence: staging only moves on a release-YYYY.MM.DD tag cut, not on every develop merge — see §1.
• Deploy ground truth: /api/system/version build suffix must equal the Woodpecker pipeline number — "WP green" alone is not proof the code landed; see §1.

5. Troubleshooting pointers

This guide intentionally does not duplicate the content of these runbooks — read them directly for the full procedure:

Known issue (not a regression, do not file)

session_auth.py's exemption list documents /api/system/health as a pre-auth route ("frontend/synthetic probes"), but no such route is registered (backend_v2/api/routes/system.py, api/__init__.py) — it 404s via the SPA fallback. This is a dead exemption entry, not a live gap: /health (root) is the actual liveness probe and is what §2.3 checks. Confirmed present on 0.0.126 as well, so it predates this RC.