Billing test tooling reference

System: Queue billing (Stripe) + Apple IAP Owner: sre-agent Companion SOP: docs/ops/runbooks/billing-e2e-test-sop.md — the scenario matrix and pass/fail criteria live there; this doc is deliberately just the concrete test card numbers, Test Clock commands, vault paths, and verification shell commands the SOP references. Where this file and the SOP could both describe the same thing (e.g. Scenario 9b's idempotency-replay steps), this file links to the SOP rather than duplicating it — see feedback_stale_groomer_comments_signal / feedback_docs_sweep_on_decision on why doc drift between the two is treated as a real finding, not cosmetic. Last reviewed: 2026-08-21 (#4502 follow-up — refreshed two MEDIUM findings from the #4502 QA gate: this doc was stale, and there was no billing-scope synthetic session harness; see §9 below and test_billing_replay_staging.py) Refs: ADR-0007 (iOS IAP), ADR-0071 (Queue as billing authority), ADR-0075 (operator override), ADR-0138 (idempotency key mechanism), docs/architecture/queue-stripe-webhook-design-2026-05-14.md, docs/ops/2026-05-13-stripe-test-mode-verification.md, #4502 (this refresh),

4149 / PR #4499 (idempotency Wave 3 on billing mutations), PR #4485

(staging-aware signup smoke harness), #4512 (dunning flow), #4520 (Antlers customer-facing billing page)

1. Test environment topology and isolation

Service map

Recommended test environment

Use raxx-queue-staging + Stripe test mode. This is the only safe, isolated billing test environment.

Rationale: - Queue staging has its own Postgres instance — test rows never touch prod Queue-DB. - Stripe test-mode events are completely isolated from live events in Stripe's data plane. - The staging webhook endpoint URL is distinct from prod — Stripe never confuses the two when both endpoints are registered in the same account (test dashboard vs. live dashboard). - Raptor staging (raxx-api-staging) receives mirror fan-out from Queue staging, so the full stack can be exercised without touching prod.

Hard rules — DO NOT VIOLATE

NEVER use Stripe test cards (4242..., 4000...) against live Stripe keys.
NEVER register a test webhook endpoint in the Stripe Live dashboard.
NEVER register a live webhook endpoint in the Stripe Test dashboard.
NEVER point the prod Queue app (raxx-queue-prod) at test-mode Stripe keys.
NEVER use real credit card numbers in Stripe test mode dashboards.

A live key (prefix rk_live_ or sk_live_) accepts test card numbers without error but creates real charges. This is irreversible without a refund cycle.

Confirming test mode before any test run

# 1. Check the key prefix in Heroku config on the target app:
heroku config:get STRIPE_RESTRICTED_KEY --app raxx-queue-staging
# Must start with rk_test_  (NOT rk_live_)

# 2. Check the webhook secret prefix:
heroku config:get STRIPE_WEBHOOK_SECRET --app raxx-queue-staging
# Must start with whsec_  — but the associated dashboard MUST be the Test dashboard.
# The whsec_ prefix alone does not distinguish test vs. live; key prefix is the authoritative check.

# 3. In Stripe Dashboard: top-right corner shows "Test mode" toggle (orange "Test" badge).
# If it shows "Live mode" (no badge), you are looking at the wrong context.
# URL confirmation: test dashboard URL contains /test/ — e.g. dashboard.stripe.com/test/customers

Current gap (refreshed 2026-08-21, #4502 follow-up)

See the SOP's own "Current billing state" table for the full picture — not duplicated here. The 2026-06-17 snapshot below marked everything MISSING. That was stale — the SOP's 2026-08-19 refresh (#4502) independently re-probed staging directly and found several of these items live. This table reflects what has actually been confirmed since, and is explicit about what still has NOT been re-verified (most agent sandboxes cannot run heroku config:get — see feedback_main_loop_vault_limit — so "not re-verified" is a real gap, not false modesty):

The companion SOP (billing-e2e-test-sop.md) documents the setup sequence and carries the authoritative, more granular "Current billing state" table (refreshed 2026-08-19) — read it first if the environment is not yet wired, and treat it as the source of truth if it and this table ever disagree (the "Doc-drift convention" at the top of the SOP is what should catch that before it happens again).

2. Stripe test mode setup

Vault paths (key names only — values are never logged or printed)

Legacy path /MooseQuest/stripe/ also holds test keys (env: dev). Per queue-stripe-webhook-design-2026-05-14.md §I-10, the canonical path is /Raxx/Queue/Billing/Stripe/. Operator must promote keys from the legacy path to the canonical path before claiming E-1.

Test webhook endpoint URL

https://raxx-queue-staging-403c1aa5941f.herokuapp.com/api/v1/billing/webhook

This is the direct Heroku URL (no CF proxy on staging). Register it in the Stripe Test mode dashboard at:

Stripe Dashboard (Test mode) → Developers → Webhooks → Add endpoint

Events to subscribe (minimum for E2E billing tests): - customer.created, customer.updated, customer.deleted - customer.subscription.created, customer.subscription.updated, customer.subscription.deleted - invoice.created, invoice.payment_succeeded, invoice.payment_failed - checkout.session.completed

After saving, click Reveal signing secret (shown once) — this is your STRIPE_WEBHOOK_SECRET value.

Switching Stripe Dashboard to Test mode

Top-right corner of dashboard.stripe.com — click the Test mode toggle. URL changes to include /test/. All data (customers, subscriptions, events) shown is isolated to test mode. No test-mode operation creates charges, emails customers, or appears in live-mode dashboards.

3. Stripe test card catalog

All cards use expiry 12/34 (any future date), CVC 123, ZIP 10001 unless noted. Do NOT use real card numbers.

Source: https://docs.stripe.com/testing#cards

SCA / 3DS notes

Cards triggering 3DS (3155, 3220) require a browser interaction to complete the authentication flow. In automated test environments, use the Stripe test mode next_action.redirect_to_url and complete it via the test authentication endpoint Stripe provides. The Stripe CLI cannot auto-complete 3DS — use the Stripe Dashboard → Payments → Authenticate button.

For server-side-only tests (webhook handler), skip 3DS cards and use 4242... or 0341. 3DS testing is relevant only when testing the Antlers subscribe UI (FLAG_ANTLERS_SUBSCRIBE).

4. Test accounts and customers

Naming convention

test+billing-<YYYYMMDD>@raxx.app

Examples: - test+billing-20260617@raxx.app — daily test customer - test+billing-founders-20260617@raxx.app — Founders tier test customer - test+billing-dunning-20260617@raxx.app — dunning scenario

This convention makes test customers findable in the Stripe Test dashboard via email search and purgeable by date range.

Note: @raxx.app email addresses in Stripe test mode are safe — Stripe does not send email to test-mode customers. Real no-reply@raxx.app Postmark sends are only triggered if FLAG_QUEUE_BILLING=true AND Postmark is wired (check Queue staging Postmark env vars before running).

Creating a test customer via Stripe Dashboard

1. Stripe Dashboard (Test mode) → Customers → Create customer
2. Email: test+billing-<YYYYMMDD>@raxx.app
3. Name: Test Customer <YYYYMMDD>
4. Payment method: Add card → use 4242 4242 4242 4242, exp 12/34, CVC 123
5. Save. Note the cus_... customer ID.

Creating a test customer via Stripe API (copy-paste)

RKEY=$(infisical secrets get STRIPE_RESTRICTED_KEY \
  --path "/Raxx/Queue/Billing/Stripe" --env staging --plain)

curl -s -u "${RKEY}:" https://api.stripe.com/v1/customers \
  -d "email=test+billing-$(date +%Y%m%d)@raxx.app" \
  -d "name=Test Customer $(date +%Y%m%d)" \
  -d "description=sre-test created $(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  | python3 -m json.tool | grep '"id"'
# → "id": "cus_..."

Attaching a test payment method

# Create a PaymentMethod from a test card number
PM=$(curl -s -u "${RKEY}:" https://api.stripe.com/v1/payment_methods \
  -d "type=card" \
  -d "card[number]=4242424242424242" \
  -d "card[exp_month]=12" \
  -d "card[exp_year]=34" \
  -d "card[cvc]=123" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")

# Attach to the customer
curl -s -u "${RKEY}:" https://api.stripe.com/v1/payment_methods/${PM}/attach \
  -d "customer=cus_YOUR_CUSTOMER_ID" >/dev/null

# Set as default
curl -s -u "${RKEY}:" https://api.stripe.com/v1/customers/cus_YOUR_CUSTOMER_ID \
  -d "invoice_settings[default_payment_method]=${PM}" >/dev/null

Subscribing to a Raxx tier

Raxx pricing tiers and their test-mode price ID env vars (set on raxx-queue-staging):

Fetch the price ID and create a subscription:

PRICE_ID=$(heroku config:get STRIPE_PRICE_ID_PRO --app raxx-queue-staging)

curl -s -u "${RKEY}:" https://api.stripe.com/v1/subscriptions \
  -d "customer=cus_YOUR_CUSTOMER_ID" \
  -d "items[0][price]=${PRICE_ID}" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('sub:', d['id'], 'status:', d['status'])"
# → sub: sub_...  status: active

Cleanup — purging test customers

# List test customers created by this convention
curl -s -u "${RKEY}:" \
  "https://api.stripe.com/v1/customers?email=test%2Bbilling-$(date +%Y%m%d)%40raxx.app&limit=10" \
  | python3 -c "import sys,json; [print(c['id']) for c in json.load(sys.stdin)['data']]"

# Delete a specific test customer (also cancels active subscriptions)
curl -s -u "${RKEY}:" -X DELETE \
  "https://api.stripe.com/v1/customers/cus_YOUR_CUSTOMER_ID" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('deleted:', d.get('deleted'), d.get('id'))"

Test-mode customer deletions are permanent. In test mode only, Stripe allows customer deletion regardless of subscription state. In live mode, active subscriptions must be cancelled first.

After deleting test customers from Stripe, purge the corresponding rows from Queue staging DB:

heroku pg:psql --app raxx-queue-staging -c \
  "DELETE FROM billing_customer WHERE billing_email LIKE 'test+billing-%@raxx.app';"
# The CASCADE on billing_subscription and billing_invoice removes child rows.
# billing_action_log rows are append-only and are not deleted (audit trail).

5. Stripe Test Clocks

Test Clocks allow time-acceleration for subscription lifecycle events (renewals, trials, dunning, cancellations) without waiting real calendar time.

Reference: https://docs.stripe.com/billing/testing/test-clocks

5.1 Create a test clock (Dashboard)

1. Stripe Dashboard (Test mode) → Billing → Test clocks → + Create test clock
2. Name: raxx-billing-test-<YYYYMMDD>
3. Frozen time: set to today's date (e.g. 2026-06-17)
4. Save. Note the clock_... ID.

5.2 Create a test clock (API)

CLOCK=$(curl -s -u "${RKEY}:" https://api.stripe.com/v1/test_helpers/test_clocks \
  -d "frozen_time=$(date +%s)" \
  -d "name=raxx-billing-test-$(date +%Y%m%d)" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
echo "Clock: $CLOCK"

5.3 Create a test customer attached to the clock

CUST=$(curl -s -u "${RKEY}:" https://api.stripe.com/v1/customers \
  -d "email=test+billing-clock-$(date +%Y%m%d)@raxx.app" \
  -d "name=Clock Test $(date +%Y%m%d)" \
  -d "test_clock=${CLOCK}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
echo "Customer: $CUST"

Customers on a test clock advance in time WITH the clock. All their subscriptions, invoices, and events are generated at the simulated time.

5.4 Advance time — scenario playbook

All time-advance calls use the same pattern:

# Helper: advance clock by N days
advance_clock() {
  local clock=$1
  local days=$2
  local new_ts=$(( $(date +%s) + days * 86400 ))
  curl -s -u "${RKEY}:" \
    https://api.stripe.com/v1/test_helpers/test_clocks/${clock}/advance \
    -d "frozen_time=${new_ts}" \
    | python3 -c "import sys,json; d=json.load(sys.stdin); print('clock status:', d['status'], 'time:', d['frozen_time'])"
}

Wait ~5s after each advance for Stripe to process generated events before checking webhook delivery.

Scenario A: Successful renewal (invoice.payment_succeeded)

# Advance 31 days past subscription start → triggers renewal invoice
advance_clock $CLOCK 31
# Expected webhook events at staging endpoint:
#   invoice.created
#   invoice.finalized
#   invoice.payment_succeeded
#   customer.subscription.updated (current_period_end advanced)

Scenario B: Payment failure + dunning

# First: create subscription with card 4000 0000 0000 0341 (attaches, charges fail)
# Then advance past renewal:
advance_clock $CLOCK 31
# Expected: invoice.payment_failed → Queue sets subscription status past_due
# Advance 3 more days (within Stripe Smart Retries window):
advance_clock $CLOCK 3
# Expected: retry attempt, another invoice.payment_failed
# Advance to Smart Retries exhaustion (default ~4 attempts over ~30 days):
advance_clock $CLOCK 30
# Expected: invoice.payment_failed (final), customer.subscription.updated (status=unpaid)

Dunning flow now has a real Queue-side implementation (#4512, FLAG_BILLING_DUNNING, default OFF) — added 2026-08-21, #4502 follow-up. Scenario B above describes Stripe's own Smart Retries mechanism, which is independent of this. #4512 adds a second, Queue-side state machine on top:

• invoice.payment_failed starts an idempotent 7-day grace period by writing dunning_started_at / dunning_grace_ends_at on billing_subscription (this write happens regardless of the flag, so the clock stays accurate for whenever the flag is later flipped on).
• A nightly job, POST /api/internal/billing/dunning-downgrade, re-verifies live Stripe status immediately before downgrading a still-past_due subscription to free at grace expiry (protects against downgrading a subscription that actually recovered between the webhook and the nightly run).
• Two new Postmark emails fire at day 0 and day 6 of the grace period (subject to FLAG_BILLING_EMAIL_DISPATCH, same gate as the existing payment-failed email — see the SOP's "Email dispatch" row).
• Founders-tier downgrades leave founders_started_at untouched (operator decision, 2026-08-19) — the 6-month intro-pricing clock keeps running independent of tier state; a resubscribe within the original 6-month window resumes at the $29 rate for the remaining balance.

Flag state: FLAG_BILLING_DUNNING is OFF by default and, as of this refresh, has not been confirmed ON anywhere — the grace-period columns write unconditionally, but the nightly downgrade job no-ops (200, zero counters) and neither dunning email dispatches while OFF. Per #4512's own scope, this is pending a dedicated Stripe Test Clock E2E run (a genuine Scenario B extension: create a past_due subscription, advance the clock 7+ days, and confirm the downgrade job actually flips plan_tier to free and dispatches the day-6 email) — that run has not happened yet as of this doc refresh; do not assume dunning is validated end-to-end. Confirm current state with:

heroku config:get FLAG_BILLING_DUNNING --app raxx-queue-staging

Scenario C: Trial expiry

# Create subscription with trial_period_days=14
PRICE_ID=$(heroku config:get STRIPE_PRICE_ID_PRO --app raxx-queue-staging)
SUB=$(curl -s -u "${RKEY}:" https://api.stripe.com/v1/subscriptions \
  -d "customer=${CUST}" \
  -d "items[0][price]=${PRICE_ID}" \
  -d "trial_period_days=14" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")

# Advance 15 days → trial ends, first real invoice fires
advance_clock $CLOCK 15
# Expected: customer.subscription.trial_will_end (at day 13), invoice.payment_succeeded (day 15)

Scenario D: Cancellation at period end

# Mark subscription to cancel at end of current period
curl -s -u "${RKEY}:" https://api.stripe.com/v1/subscriptions/${SUB} \
  -d "cancel_at_period_end=true" | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print('cancel_at_period_end:', d['cancel_at_period_end'])"

# Advance past period end
advance_clock $CLOCK 32
# Expected: customer.subscription.deleted (status=canceled)
#           Queue sets billing_subscription.status='canceled', runs downgrade detector
#           Mirror fan-out → Raptor billing_subscription_mirror updated → 402 on next paywall check

Scenario E: Immediate cancellation

curl -s -u "${RKEY}:" -X DELETE \
  https://api.stripe.com/v1/subscriptions/${SUB} \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('status:', d['status'])"
# Expected: customer.subscription.deleted fires immediately (no clock advance needed)

5.5 Delete a test clock

Test clocks cannot be reused once advanced — they are immutable after being advanced past a point. Delete and recreate for each test run:

curl -s -u "${RKEY}:" -X DELETE \
  https://api.stripe.com/v1/test_helpers/test_clocks/${CLOCK}

6. Verification commands

6.1 Send a test webhook with correct HMAC signature

The Stripe CLI stripe trigger or stripe events resend are the canonical tools. They use the correct STRIPE_WEBHOOK_SECRET and compute the Stripe-Signature header automatically.

# Install Stripe CLI if not present: https://docs.stripe.com/stripe-cli

# Authenticate against test mode (uses the rk_test_ key)
stripe login --api-key $(infisical secrets get STRIPE_RESTRICTED_KEY \
  --path "/Raxx/Queue/Billing/Stripe" --env staging --plain)

# Send a test customer.created event to the staging webhook endpoint
stripe trigger customer.created \
  --webhook-endpoint https://raxx-queue-staging-403c1aa5941f.herokuapp.com/api/v1/billing/webhook
# Expected: HTTP 200 response. Check Queue staging logs for "webhook processed" or event ID.

Alternative — send a manually constructed event using the stored webhook secret:

WHSEC=$(infisical secrets get STRIPE_WEBHOOK_SECRET \
  --path "/Raxx/Queue/Billing/Stripe" --env staging --plain)

PAYLOAD='{"id":"evt_test_sre_manual_001","type":"customer.created","created":'"$(date +%s)"',"data":{"object":{"id":"cus_test_sre_001","object":"customer","email":"test+billing-verify@raxx.app"}}}'
TIMESTAMP=$(date +%s)
SIG_PAYLOAD="${TIMESTAMP}.${PAYLOAD}"
SIG=$(echo -n "${SIG_PAYLOAD}" | openssl dgst -sha256 -hmac "${WHSEC}" -hex | awk '{print $2}')

curl -s -X POST \
  "https://raxx-queue-staging-403c1aa5941f.herokuapp.com/api/v1/billing/webhook" \
  -H "Content-Type: application/json" \
  -H "Stripe-Signature: t=${TIMESTAMP},v1=${SIG}" \
  -d "${PAYLOAD}" \
  -w "\nHTTP %{http_code}"
# Expected: HTTP 200

6.2 Confirm billing tables exist in Queue staging DB

heroku pg:psql --app raxx-queue-staging -c "\dt billing_*"
# Expected output:
#              List of relations
#  Schema |           Name              | Type  | Owner
# --------+-----------------------------+-------+-------
#  public | billing_action_log          | table | ...
#  public | billing_customer            | table | ...
#  public | billing_invoice             | table | ...
#  public | billing_subscription        | table | ...
#  public | processed_stripe_events     | table | ...

If no tables appear: sqitch migrations have not run. Check Queue staging release logs:

heroku releases --app raxx-queue-staging --num 5
heroku logs --app raxx-queue-staging --num 50 | grep -i "sqitch\|migrate\|schema"

6.3 Confirm test customer rows are written

After sending a customer.created webhook event:

heroku pg:psql --app raxx-queue-staging -c \
  "SELECT id, stripe_customer_id, billing_email, created_at
   FROM billing_customer
   ORDER BY created_at DESC LIMIT 5;"

After a subscription event (customer.subscription.created):

heroku pg:psql --app raxx-queue-staging -c \
  "SELECT id, stripe_subscription_id, plan_tier, status, current_period_end
   FROM billing_subscription
   ORDER BY created_at DESC LIMIT 5;"

Confirm dedup table captured the event ID:

heroku pg:psql --app raxx-queue-staging -c \
  "SELECT event_id, created_at FROM processed_stripe_events ORDER BY created_at DESC LIMIT 5;"

After an invoice.payment_succeeded event:

heroku pg:psql --app raxx-queue-staging -c \
  "SELECT id, stripe_invoice_id, status, paid_at FROM billing_invoice
   WHERE status = 'paid' ORDER BY paid_at DESC LIMIT 5;"

6.4 Confirm Raptor mirror received fan-out

After Queue processes a subscription event and fires POST /api/internal/billing/mirror-sync to Raptor staging:

heroku pg:psql --app raxx-api-staging -c \
  "SELECT queue_customer_id, plan_tier, status, current_period_end, updated_at
   FROM billing_subscription_mirror ORDER BY updated_at DESC LIMIT 5;"

If the table does not exist: the Raptor E-4 migration (billing_subscription_mirror) has not been applied. Check:

heroku run --app raxx-api-staging -- python -c "
from backend_v2.db import engine
from sqlalchemy import inspect
print(inspect(engine).has_table('billing_subscription_mirror'))
"

6.5 Confirm Console billing view reflects test customer

After Queue has a test customer and subscription:

# Console staging reads Queue via internal API
# Check that Queue internal endpoint is reachable from console
QUEUE_TOKEN=$(infisical secrets get QUEUE_INTERNAL_SERVICE_TOKEN \
  --path "/Raxx/Queue/Internal" --env staging --plain 2>/dev/null || echo "MISSING")

curl -s \
  -H "Authorization: Bearer ${QUEUE_TOKEN}" \
  "https://raxx-queue-staging-403c1aa5941f.herokuapp.com/api/v1/internal/billing/customers?limit=5" \
  | python3 -m json.tool | head -20
# Expected: JSON array of billing_customer rows including your test customer

If QUEUE_INTERNAL_SERVICE_TOKEN is MISSING, the internal API is not yet wired. Check docs/architecture/queue-stripe-webhook-design-2026-05-14.md §4.5 for the mirror-sync token provisioning step.

6.6 Check Queue staging logs for webhook processing

heroku logs --app raxx-queue-staging --tail --num 50 | grep -E "webhook|billing|stripe|event"
# Happy path lines to look for:
#   "webhook event received: evt_..."
#   "processed event: customer.created cus_..."
#   "mirror sync dispatched"
# Error lines to triage:
#   "HMAC verification failed"        → STRIPE_WEBHOOK_SECRET mismatch
#   "stale timestamp"                  → system clock drift or replayed event >5min
#   "FLAG_QUEUE_BILLING is false"      → flag not enabled; see §1 gap table
#   "DB write failed"                  → Postgres connection issue; check DATABASE_URL

6.7 Confirm the customer-facing Antlers billing page reflects test customer state (#4504, added 2026-08-21)

New since 2026-06-17: /account/billing (frontend/raxx-next) is a read-only, self-service view of the caller's own plan, payment method (brand + last4 only), and last 12 invoices — see PR #4520 (Closes #4504). Gated behind FLAG_ANTLERS_BILLING_SELF_SERVICE (default OFF). This is distinct from both the Console customer-detail view (#409, operator-facing, §6.5 above) and the SOP's Scenario 1 steps 7-8 — this is the customer's own self-service surface, composed server-side by a new Raptor route from Queue's existing internal billing read endpoints, scoped to the caller's own queue_customer_id.

# Confirm the flag before attempting the walk:
heroku config:get FLAG_ANTLERS_BILLING_SELF_SERVICE --app raxx-api-staging
# Expected: 1 (OFF/unset → the route 404s, hide-don't-gray per
# feedback_hide_dont_gray_unavailable_features)

Backend routes to spot-check directly (as the test customer's own session — NOT an operator/internal-token call):

# GET the composed billing view (plan + payment method + invoices):
GET /api/account/billing
# Expected 200: { plan: {...}, payment_method: {brand, last4}, invoices: [...] }
# (only reachable as an authenticated session for the customer being checked —
# see §9.3 below for how to mint one without a real browser)

# POST to start a Stripe Customer Portal session ("Update payment method" /
# "Manage or cancel your plan" both land here):
POST /api/account/billing/portal-session
# Expected 200: { url: "https://billing.stripe.com/..." } in TEST MODE

Frontend walk: sign in as the test customer, navigate to /account/billing, and confirm the page renders one of its 7 documented states correctly for the test customer's current tier — active, founders-intro, grace period/past_due (pairs well with the dunning scenario above once FLAG_BILLING_DUNNING is on), canceled, free tier, loading, error. Frozen mockups: docs/design/billing-page/.

No new billing schema — this is a pure read composer over the existing billing_customer / billing_subscription / billing_invoice tables (live since #405/#1682); nothing here writes new rows, so it does not affect the "Confirm no test artifacts leaked to prod" check in §8.

7. Apple IAP / StoreKit 2 sandbox

The iOS billing path uses Apple's IAP system, not Stripe. Sandbox purchases never charge real money. Reference: ADR-0007, docs/architecture/adr/0007-ios-subscription-billing-iap.md.

Full sandbox runbook: docs/ops/runbooks/ios-iap-sandbox-testing.md — covers the StoreKit Configuration file, scheme wiring, sandbox tester setup, device purchase flow, renewal cadence, and the server-side verification gap. Read that first; this section is an overview for cross-reference.

7.1 Create Sandbox Tester accounts

Location: App Store Connect → Users and Access → Sandbox → Testers → (+)

Fields to fill: - First name / Last name: Test User (or any name) - Email: use a real address you can receive verification at, OR use an @raxx.app alias forwarded to ops@raxx.app - Password: meets Apple complexity requirements; store in Infisical at /Raxx/iOS/Sandbox/SANDBOX_TESTER_PASSWORD - Secret question / answer: required by Apple; store answer in Infisical - Date of birth: any past date - Store region: United States (matches Raxx's primary market; change if testing international tax behavior) - Subscription renewal rate: Accelerated (see §7.2)

After creation, verify the email address using the link Apple sends.

One sandbox tester account can be used for multiple test runs. Create separate testers for separate test scenarios if you need isolated state (e.g. a tester that has never purchased, vs one mid-subscription).

7.2 Accelerated sandbox renewal cadence

Apple's sandbox accelerates subscription renewal periods so you can test a full subscription year in minutes:

Accelerated renewals repeat automatically. A sandbox subscription renews approximately every 5 minutes until the tester cancels or the max renewal count is reached (Apple allows up to 12 renewals per sandbox subscription).

7.3 Test a sandbox purchase

On a physical iOS device (Simulator does not support IAP):

1. Sign out of the real Apple ID in Settings → [your name] → Sign Out.
2. Do NOT sign in with a sandbox tester account at the system level. Instead:
3. Open the Raxx iOS app → navigate to the subscription screen.
4. Tap Subscribe on any tier.
5. When Apple prompts for Apple ID, enter the sandbox tester credentials.
6. Apple's sandbox payment sheet appears with [Environment: Sandbox] indicator.
7. Confirm purchase.

The purchase completes with no real charge. The tester's sandbox account shows the purchase in the App Store sandbox environment.

7.4 Server-side transaction verification

After an IAP purchase, Apple sends a server-to-server notification to the backend. The backend endpoint is:

POST /api/subscriptions/apple/notifications

(See ADR-0007 compliance checklist — this endpoint must implement JWS validation of Apple's signed transaction data.)

Verify the notification arrived and was processed:

# Check Raptor staging logs for Apple notification receipt
heroku logs --app raxx-api-staging --num 50 | grep -i "apple\|iap\|storekit\|notification"

# Check the subscriptions table for the sandbox transaction
heroku pg:psql --app raxx-api-staging -c \
  "SELECT id, subscription_source, plan_tier, status, original_transaction_id
   FROM billing_subscription_mirror
   WHERE subscription_source = 'ios'
   ORDER BY created_at DESC LIMIT 5;"

The original_transaction_id from StoreKit 2 is the durable identifier across all renewals (not the rolling transaction_id). Verify it is stored, not the rolling transaction ID.

7.5 Manage sandbox subscriptions

On the iOS device: - View/cancel: Settings → [sandbox tester name] → Subscriptions (only appears after a sandbox purchase) - Cancel: tap subscription → Cancel Subscription

In App Store Connect: - Sandbox → Testers → [tester] → Manage: shows purchase history, allows clearing purchase history for a clean-slate test

7.6 Key differences from Stripe path

iOS subscribers do NOT appear in the Stripe dashboard. The Raxx Console billing view must aggregate from both sources. A Stripe-only view will undercount total subscribers.

8. Cleanup and safety

Pre-flight checklist — confirm test mode before any test

[ ] heroku config:get STRIPE_RESTRICTED_KEY --app raxx-queue-staging
    → value starts with rk_test_  (STOP if rk_live_)

[ ] Stripe Dashboard URL contains /test/ (top right: orange "Test" badge visible)

[ ] heroku config:get FLAG_QUEUE_BILLING --app raxx-queue-staging
    → value is true

[ ] heroku config:get STRIPE_WEBHOOK_SECRET --app raxx-queue-staging
    → value is set (non-empty); registered endpoint URL in Stripe Test dashboard matches
      https://raxx-queue-staging-403c1aa5941f.herokuapp.com/api/v1/billing/webhook

[ ] You are NOT looking at raxx-queue-prod in any terminal tab or Heroku dashboard

[ ] Test clock is at the expected frozen time (if using Test Clocks)

Purge test customers from Stripe (test mode)

# List all test+billing-* customers created this week
curl -s -u "${RKEY}:" \
  "https://api.stripe.com/v1/customers?limit=100" \
  | python3 -c "
import sys, json
data = json.load(sys.stdin)['data']
test_customers = [c for c in data if c.get('email','').startswith('test+billing')]
for c in test_customers:
    print(c['id'], c['email'])
"

# Delete each (also cancels subscriptions — test mode only)
# Replace cus_XXX with each customer ID from the above list:
curl -s -u "${RKEY}:" -X DELETE "https://api.stripe.com/v1/customers/cus_XXX"

Confirm no test artifacts leaked to prod Queue-DB

# Should return 0 rows — any test+billing- email in prod is a contamination event
heroku pg:psql --app raxx-queue-prod -c \
  "SELECT COUNT(*) FROM billing_customer WHERE billing_email LIKE 'test+billing-%';"
# Expected: 0

# Also check processed_stripe_events for evt_test_ prefixes (Stripe test events have this prefix)
heroku pg:psql --app raxx-queue-prod -c \
  "SELECT COUNT(*) FROM processed_stripe_events WHERE event_id LIKE 'evt_test_%';"
# Expected: 0

If either query returns non-zero: STOP. This is a SEV-2 data contamination incident. Do not delete the rows yet — preserve state and escalate to the operator. The incident requires an RCA and audit of how test data reached the prod DB.

Delete test clocks after each run

# List active test clocks
curl -s -u "${RKEY}:" "https://api.stripe.com/v1/test_helpers/test_clocks" \
  | python3 -c "import sys,json; [print(c['id'], c['name']) for c in json.load(sys.stdin)['data']]"

# Delete each by ID
curl -s -u "${RKEY}:" -X DELETE \
  "https://api.stripe.com/v1/test_helpers/test_clocks/clock_XXX"

Test clocks are automatically garbage-collected by Stripe after 72 hours of inactivity, but explicit deletion keeps the test dashboard clean.

Apple sandbox tester cleanup

Apple does not allow permanent deletion of sandbox testers. To reset a tester's purchase history: - App Store Connect → Users and Access → Sandbox → Testers → [tester] → Clear Purchase History

This allows the same tester account to be reused for a fresh-start test.

9. Idempotency-Key replay testing (billing mutations, #4149 Wave 3 — added 2026-08-21, #4502 follow-up)

Five billing/subscription mutation routes carry an optional Idempotency-Key: <uuid4> request header (Raptor-layer, @idempotency_key decorator — api/middleware/idempotency.py), independent of Stripe's own event-level dedup (the SOP's Scenario 9): POST /api/billing/checkout-session, POST /api/billing/refund, POST /api/subscription/upgrade, POST /api/subscription/downgrade, POST /api/subscription/cancel. A duplicate call with the same key + request body replays the stored response ("idempotency_replayed": true injected) instead of re-executing the mutation. Full mechanism + pass criteria: SOP Scenario 9b — this section is just the concrete commands, per this doc's stated role (see the top-of-file note); do not duplicate the scenario's step-by-step or pass criteria here.

9.1 Two prerequisites Scenario 9b depends on — check both before any run

FLAG_IDEMPOTENCY_MIDDLEWARE — the shared master flag for all three idempotency waves (order endpoints, wave 2, and this billing wave 3); default OFF. When OFF the decorator is a transparent passthrough — every call re-executes, idempotency_replayed is never injected, and nothing is written to the idempotency_keys table. Confirm:

heroku config:get FLAG_IDEMPOTENCY_MIDDLEWARE --app raxx-api-staging

IDEMPOTENCY_RESPONSE_KMS_KEY_ARN — a plain Heroku config var (AWS workload config, not an Infisical secret — feedback_aws_workloads_use_ssm_not_vault), read at call time by api/crypto/envelope.py. The five billing routes above are the only idempotency-covered routes that set encrypt_response=True (resolves ADR-0138 open question 2 — the stored response_body for a billing mutation must never be plaintext at rest). If this ARN is unset, the failure is silent from the client's point of view: per PR #4499's own operational note, the completion write fails encryption, the row stays in_flight, and it stale-heals to failed after 30s — the caller's actual HTTP response is unaffected (they still get their real 200), but the second call will never see idempotency_replayed: true, and will instead either re-execute (if the stale-heal already fired) or 409 idempotency_previous_failed (if it landed on the now-failed row). This looks exactly like "the middleware is broken" if you don't check this first. Confirm:

heroku config:get IDEMPOTENCY_RESPONSE_KMS_KEY_ARN --app raxx-api-staging

Neither of these has been directly confirmed on raxx-api-staging from an agent sandbox as of this doc's last refresh (no heroku CLI credentials available — see feedback_main_loop_vault_limit); both are on the operator one-liner list. FLAG_IDEMPOTENCY_MIDDLEWARE=1 was reported set on staging 2026-08-19 in the #4502 issue thread by sre-agent (with a Heroku session), but that report has not been independently re-verified since, and the KMS ARN has never been confirmed either way.

9.2 Manual replay recipe (curl, once you have a session cookie)

IDEM_KEY=$(python3 -c "import uuid; print(uuid.uuid4())")

# First call — real execution:
curl -s -X POST https://api-staging.raxx.app/api/subscription/cancel \
  -H "Cookie: raxx_session=${SESSION_COOKIE}" \
  -H "Idempotency-Key: ${IDEM_KEY}" \
  -H "Content-Type: application/json" -d '{}' | python3 -m json.tool
# Expected: 200, no "idempotency_replayed" key (or explicitly false)

# Second call — SAME key, SAME body:
curl -s -X POST https://api-staging.raxx.app/api/subscription/cancel \
  -H "Cookie: raxx_session=${SESSION_COOKIE}" \
  -H "Idempotency-Key: ${IDEM_KEY}" \
  -H "Content-Type: application/json" -d '{}' | python3 -m json.tool
# Expected: 200, "idempotency_replayed": true, body otherwise identical

The remaining piece — getting SESSION_COOKIE for a real customer-scoped session without a browser or Face ID — is exactly what §9.3 automates.

9.3 Automated harness — test_billing_replay_staging.py (closes #4502 residual B)

backend_v2/tests/smoke/test_billing_replay_staging.py makes the manual recipe above a single command. It mints a synthetic authenticated session the same way docs/ops/runbooks/signup-smoke.md does (steps 1-4 of the signup flow — register/verify-with-token mints and sets the raxx_session cookie on success, so no separate login/assertion phase is needed), then runs the 9.2 replay recipe against POST /api/subscription/cancel — chosen because it is the only one of the five routes that never calls Queue or Stripe at all (see the test file's module docstring for the full route-selection rationale), making it usable without any Stripe TEST MODE credentials.

RAXX_BILLING_REPLAY=1 python3 -m pytest \
  backend_v2/tests/smoke/test_billing_replay_staging.py -s -v

Defaults to staging (api-staging.raxx.app / raxx-api-staging / staging-nextjs.raxx.app origin) — unlike the signup/login smoke harnesses, this one does not default to prod, and hard-fails (an assertion, not a silent skip) the moment the live test actually runs unless RAXX_SMOKE_API_URL and HEROKU_APP exactly match the known staging host/app (case-normalized allowlist — api-staging.raxx.app / staging-nextjs.raxx.app / raxx-api-staging — not a "contains 'staging'" heuristic; tightened 2026-08-21 per a pre-merge QA finding that the original substring check let adversarial-but-plausible identifiers like STAGING.API.RAXX.APP or raxx-api-prod-staging through). Skips by default (no RAXX_BILLING_REPLAY=1 → skipped, not run) — safe to leave in the normal test suite; only the live scenario is skip-gated, the always-on dry tests below always collect and run. See the module docstring for the full override table (RAXX_SMOKE_EMAIL, RAXX_SMOKE_API_URL, HEROKU_APP, RAXX_SMOKE_ORIGIN — same variables as signup-smoke.md, same conventions).

Requires Heroku CLI credentials (steps 1 and the teardown step both shell out to heroku run) — this is the same access boundary #4502's QA thread already documented for the signup harness; an agent sandbox without Heroku-scoped credentials will fail cleanly at the token-mint step with a clear error, not silently no-op. Run it as the operator or as sre-agent with a provisioned Heroku session.

Also ships a set of always-on dry/unit tests (Test*Dry classes in the same file — staging-host guard logic, synthetic credential shape, cookie extraction, and a cross-check against the real api.middleware.idempotency._is_valid_uuid validator) that run in normal CI regardless of RAXX_BILLING_REPLAY — these do not require network or Heroku access and are what actually gates this PR's CI, since the live scenario itself could not be executed from this environment (see the test file's "Execution status" docstring section for exactly what was and wasn't run).

References

• Companion SOP: docs/ops/runbooks/billing-e2e-test-sop.md (see its Scenario 9b for the full idempotency-replay spec; §9 above is just the concrete commands)
• Signup smoke runbook (session-mint primitive reused by §9.3): docs/ops/runbooks/signup-smoke.md
• iOS IAP sandbox testing runbook: docs/ops/runbooks/ios-iap-sandbox-testing.md
• Queue service runbook: docs/ops/runbooks/queue.md
• Stripe founders setup SOP: docs/ops/runbooks/stripe-founders-setup.md
• Stripe restricted key rotation: docs/ops/runbooks/rotation/stripe-restricted-key.md
• Gap analysis (2026-05-13): docs/ops/2026-05-13-stripe-test-mode-verification.md
• Webhook design: docs/architecture/queue-stripe-webhook-design-2026-05-14.md
• ADR-0007 (iOS IAP): docs/architecture/adr/0007-ios-subscription-billing-iap.md
• ADR-0071 (Queue as authority): docs/architecture/adr/0071-stripe-billing-queue-as-authority.md
• ADR-0138 (idempotency key mechanism): docs/architecture/adr/0138-idempotency-key-mechanism.md
• Pricing tiers locked: docs/memory/project_pricing_tiers_locked.md
• Stripe test cards: https://docs.stripe.com/testing#cards
• Stripe Test Clocks: https://docs.stripe.com/billing/testing/test-clocks
• Apple sandbox: https://developer.apple.com/documentation/storekit/testing_in_xcode_and_the_sandbox_environment
• Apple server notifications v2: https://developer.apple.com/documentation/appstoreservernotifications

• 4502 (this refresh + test_billing_replay_staging.py harness — closes the

  two MEDIUM findings from the #4502 QA gate: this doc's staleness, and the missing billing-scope synthetic session harness)
• PR #4499 (Closes #4149 — idempotency Wave 3 on the five billing/subscription routes)
• PR #4485 (staging-aware signup smoke harness — RAXX_SMOKE_ORIGIN / RAXX_SMOKE_API_URL)
• PR #4512 (Closes #4503 — dunning grace-period + auto-downgrade)
• PR #4520 (Closes #4504 — Antlers customer-facing billing page)