Billing E2E Test SOP — Subscription Lifecycle Validation

Owner: sre-agent / Kristerpher Last updated: 2026-06-17 UTC Refs: #403 (billing console epic), #405 (data model), #407 (webhook handler), #409 (customer detail view — pending), #410 (manual ops — pending), #1633 (MRR dashboard — pending) Tooling companion: docs/ops/runbooks/billing-test-tooling.md — all concrete test card numbers, Stripe Test Clock commands, test account credentials, IAP sandbox setup, vault test paths, and verification shell commands live there. This SOP references that doc for specifics; do not duplicate values here.

Purpose and scope

This is the repeatable, operator-executable procedure for exercising the complete Raxx subscription lifecycle against test accounts and test credit cards — before any real charge is processed. It is the authoritative pre-charge acceptance gate. No real billing is enabled until every scenario in this document passes in Stripe TEST MODE, and the prod smoke at the end of this SOP runs clean.

The SOP is grounded in the live codebase state as of 2026-06-17. Scenarios that depend on unbuilt surfaces are marked explicitly. Run only what is buildable today; hold the acceptance gate open until blocked scenarios are unblocked.

What is in scope: - Stripe web/desktop subscription path (Queue webhook handler, billing tables, Raptor mirror) - Apple IAP/StoreKit 2 iOS path (separate flow; reconciliation with Stripe path) - Console billing views as they become available

What is out of scope: - Vendor-spend billing (Heroku, AWS, Cloudflare cost collection — separate billing-readonly-tokens.md) - Alpaca/broker wiring - Stripe Tax filing / PA-3 return (operator action, tracked in #2743)

Architecture summary (know before you test)

Queue is the authoritative billing store. All Stripe webhook events arrive at POST https://queue.raxx.app/api/v1/billing/webhook, are HMAC-verified, deduped via processed_stripe_events, and written to billing_customer / billing_subscription / billing_invoice in Queue's Postgres. Queue then fans out a PII-free mirror row to Raptor's billing_subscription_mirror table. Console reads Queue's internal HTTP API; it has no local billing tables.

The feature flag FLAG_QUEUE_BILLING is the kill-switch. When false, the webhook endpoint returns 503 immediately. All testing requires FLAG_QUEUE_BILLING=true on the target environment.

Tier ranking in the codebase: free(0) < founders(1) < pro(2) < pro_plus(3). Downgrade detection compares these ranks; feature_locked_at is set on the first detected downgrade. Plan tier comes from subscription.metadata.plan_tier or subscription.items[0].price.metadata.plan_tier in the Stripe event payload.

Current billing state (verified 2026-06-17)

Pre-test prerequisites

Complete every item before running any scenario. Failing to do so will produce misleading results.

[ ] 1. Stripe dashboard is in TEST MODE (toggle in top-right; verify "Test" label is visible)
[ ] 2. STRIPE_WEBHOOK_SECRET on raxx-queue-staging matches the staging webhook endpoint's whsec_* value
      (see billing-test-tooling.md § "Webhook secrets")
[ ] 3. FLAG_QUEUE_BILLING=true on the target environment (staging for test-mode run; prod for smoke)
      heroku config:get FLAG_QUEUE_BILLING --app raxx-queue-staging
      Expected: true or 1
[ ] 4. Queue sqitch migrations 01–06 are applied to the target Queue DB
      (see billing-test-tooling.md § "Verifying migration state")
[ ] 5. raxx-queue-staging dyno is up and healthy
      curl -s https://raxx-queue-staging-403c1aa5941f.herokuapp.com/health | python3 -m json.tool
      Expected: {"service":"raxx-queue","status":"ok","timestamp":"...","version":"..."}
[ ] 6. RAPTOR_BASE_URL and QUEUE_TO_RAPTOR_INTERNAL_TOKEN set on raxx-queue-staging
      (required for mirror fan-out; confirm with billing-test-tooling.md § "Internal service tokens")
[ ] 7. Test customer account created in Stripe TEST MODE
      (see billing-test-tooling.md § "Test accounts and customers")
[ ] 8. Stripe test price IDs configured for all four tiers (free, founders, pro, pro_plus)
      (see billing-test-tooling.md § "Test price IDs and products")

Scenario matrix

Each scenario specifies: preconditions, the test card/account to use (referencing the tooling doc), step-by-step actions, and expected results at every layer.

The layers checked in each scenario are: - Stripe state — what the Stripe TEST MODE dashboard shows - Queue billing tables — billing_customer, billing_subscription, billing_invoice, processed_stripe_events - Raptor mirror — billing_subscription_mirror - Console — where applicable; marked "pending #NNN" when the feature is not built

Scenario 1 — New subscription: happy path

Purpose: Verify the end-to-end checkout → subscription active → webhook ingestion → table population flow.

Preconditions: - Test customer exists in Stripe TEST MODE with no active subscription - FLAG_QUEUE_BILLING=true on target env - Staging webhook endpoint registered in Stripe and receiving events

Test card: success card from billing-test-tooling.md § "Success cards" (the standard 4242... card with any future expiry)

Steps:

1. In Stripe TEST MODE dashboard, create a checkout session for the test customer with the Pro price ID. Alternatively use stripe trigger checkout.session.completed if Stripe CLI is available (see billing-test-tooling.md § "Stripe CLI triggers").
2. Complete checkout using the success test card.
3. Wait 5–10 seconds for Stripe to deliver webhooks.
4. In the Stripe dashboard, navigate to Developers → Webhooks → your staging endpoint → Recent deliveries. Confirm delivery of: - checkout.session.completed — HTTP 200 - customer.subscription.created — HTTP 200 - invoice.payment_succeeded — HTTP 200
5. Check Queue billing tables (see billing-test-tooling.md § "DB verification commands"):

-- On Queue staging DB:
SELECT stripe_customer_id, billing_email, customer_segment
FROM billing_customer
WHERE stripe_customer_id = '<test_stripe_cus_id>';
-- Expected: 1 row with billing_email and customer_segment = 'organic'

SELECT stripe_subscription_id, plan_tier, status,
       current_period_start, current_period_end, cancel_at_period_end
FROM billing_subscription
WHERE stripe_customer_id = '<test_stripe_cus_id>';
-- Expected: 1 row, status = 'active', plan_tier = 'pro', cancel_at_period_end = false

SELECT stripe_invoice_id, amount_due, amount_paid, status, invoice_event_type, paid_at
FROM billing_invoice
WHERE stripe_customer_id = '<test_stripe_cus_id>'
ORDER BY created_at DESC LIMIT 5;
-- Expected: at least 1 row, status = 'paid', invoice_event_type = 'invoice.payment_succeeded', paid_at NOT NULL

SELECT event_id, created_at
FROM processed_stripe_events
ORDER BY created_at DESC LIMIT 5;
-- Expected: event IDs for the three webhook events above; all deduped (no duplicates)

6. Check Raptor mirror (see billing-test-tooling.md § "Raptor mirror check"):

-- On Raptor DB:
SELECT queue_customer_id, plan_tier, status, current_period_end, updated_at
FROM billing_subscription_mirror
WHERE queue_customer_id = '<queue_customer_id_for_test_user>';
-- Expected: 1 row, status = 'active', plan_tier = 'pro'
-- If row is absent, fan-out failed — check Queue logs for 'billing.mirror.fan_out_failure'

7. Console customer detail — pending #409: Once #409 ships, navigate to /console/customers/<customer_id> and verify the Subscription section shows plan tier = Pro, status = Active, correct period dates.

8. Console MRR dashboard — pending #1633: Once #1633 ships, navigate to /console/billing/dashboard and verify MRR tile has increased by the Pro monthly amount.

Pass criteria: - [ ] All three Stripe webhook events delivered HTTP 200 - [ ] billing_customer row exists with correct email - [ ] billing_subscription row: status = 'active', plan_tier = 'pro' - [ ] billing_invoice row: status = 'paid', paid_at populated - [ ] processed_stripe_events contains all three event IDs - [ ] billing_subscription_mirror row: status = 'active', plan_tier = 'pro' - [ ] (Pending #409) Console customer detail shows correct tier and state - [ ] (Pending #1633) Console MRR tile updated

Scenario 2 — Tier coverage: Free / Pro / Pro+ / Founders

Purpose: Verify each pricing tier maps correctly through the webhook pipeline to the right plan_tier value.

Preconditions: Four separate test customers, one per tier. Free-tier customer has no Stripe subscription (or a free plan subscription with no price).

Test accounts/prices: See billing-test-tooling.md § "Tier test matrix" for the price ID for each tier.

Steps (repeat for each tier):

1. For each of Pro, Pro+, and Founders: create a subscription for the test customer using the respective price ID. For Free: confirm a test customer with no active subscription is treated as free.

2. For Pro, Pro+, and Founders: deliver customer.subscription.created to the staging webhook (trigger from Stripe dashboard or Stripe CLI).

3. After each event, verify the billing_subscription table:

-- For each tier, confirm:
SELECT plan_tier, status, stripe_price_id
FROM billing_subscription
WHERE stripe_customer_id = '<tier_test_cus_id>';
-- Expected plan_tier values:
--   Pro:      'pro'
--   Pro+:     'pro_plus'
--   Founders: 'founders'
-- Free: no billing_subscription row (or row with plan_tier = 'free' if provisioned)

4. For Founders, also verify customer_segment in billing_customer:

SELECT customer_segment FROM billing_customer
WHERE stripe_customer_id = '<founders_cus_id>';
-- Expected: 'founders'
-- NOTE: the webhook handler defaults customer_segment to 'organic' on insert.
-- If Founders should be 'founders', this must be set at customer creation time or
-- updated via a separate event/workflow. Verify expected behavior against [ADR-0076](https://internal-docs.raxx.app/architecture/adr/0076-queue-phase1-billing-v1-aggressive-12day.html).

Pass criteria: - [ ] plan_tier = 'pro' for Pro price event - [ ] plan_tier = 'pro_plus' for Pro+ price event - [ ] plan_tier = 'founders' for Founders price event - [ ] Free-tier customer: no active billing_subscription row OR row with plan_tier = 'free' - [ ] Raptor mirror updated for each paid tier

Known gap: The plan_tier is read from subscription.metadata.plan_tier or subscription.items[0].price.metadata.plan_tier in the webhook payload. Confirm these metadata fields are set on each Stripe Price object at creation time (see billing-test-tooling.md § "Price metadata setup"). If the metadata field is absent, plan_tier defaults to 'free' and the tier will be wrong silently.

Scenario 3 — Renewal: Stripe Test Clock advance

Purpose: Verify that subscription renewal fires invoice.payment_succeeded and that current_period_start / current_period_end update correctly in billing_subscription.

Preconditions: - Active Pro subscription from Scenario 1 - Stripe Test Clock created and attached to the test customer - billing-test-tooling.md § "Test Clocks" has the setup commands

Note on Test Clocks: Stripe Test Clocks are independent of real time. A Test Clock advances a simulated timeline for associated customers. The test customer must have been created with the Test Clock attached — you cannot retroactively attach a Test Clock to a live customer. Create a new test customer via the Test Clock UI or Stripe CLI before running Scenario 3.

Steps:

1. Create a Test Clock (see billing-test-tooling.md § "Test Clock creation").
2. Create a test customer under the Test Clock; subscribe to the Pro price.
3. Verify Scenario 1 pass criteria for this new customer.
4. Advance the Test Clock to the renewal date (one billing period past the current current_period_end):

# Stripe CLI:
stripe test_helpers test_clocks advance \
  --test-clock-id <clock_id> \
  --frozen-time <ISO-8601 timestamp one month after subscription start>

5. Wait for Stripe to deliver: - invoice.created — HTTP 200 - invoice.payment_succeeded — HTTP 200 (assuming success card) - customer.subscription.updated — HTTP 200 (period dates updated)

6. Check billing_subscription:

SELECT current_period_start, current_period_end, updated_at
FROM billing_subscription
WHERE stripe_customer_id = '<test_clock_cus_id>';
-- Expected: current_period_start and current_period_end advanced by one billing period
-- updated_at should be newer than the value from Scenario 1

7. Check billing_invoice for the renewal invoice:

SELECT stripe_invoice_id, amount_due, amount_paid, status, paid_at
FROM billing_invoice
WHERE stripe_customer_id = '<test_clock_cus_id>'
ORDER BY created_at DESC LIMIT 3;
-- Expected: 2 paid invoices (initial + renewal), latest paid_at is within the last few minutes

8. Check billing_subscription_mirror:

SELECT current_period_end, updated_at
FROM billing_subscription_mirror
WHERE queue_customer_id = '<queue_customer_id>';
-- Expected: current_period_end matches the new period end from billing_subscription

9. Console MRR tile — pending #1633: Verify MRR does not double-count the renewal.

Pass criteria: - [ ] Renewal invoice.payment_succeeded delivered and returns HTTP 200 - [ ] billing_subscription.current_period_start and current_period_end advanced correctly - [ ] New billing_invoice row: status = 'paid', paid_at set - [ ] billing_subscription_mirror.current_period_end updated to match - [ ] No duplicate billing_invoice row for the same renewal event (idempotency)

Scenario 4 — Payment failure and dunning

Purpose: Verify that a declined card transitions the subscription to past_due, fires the expected events, and that the system recovers correctly when the card is fixed.

Preconditions: - Active Pro subscription - Test Clock (recommended, so you can control retry timing) or use Stripe Smart Retry with the real clock

Test cards: See billing-test-tooling.md § "Decline cards" — use the insufficient-funds card and the generic-decline card.

Steps — inducing failure:

1. Update the test customer's default payment method to the decline test card (see billing-test-tooling.md § "Attaching test cards").
2. Advance the Test Clock past the renewal date, or use the Stripe CLI to trigger a payment attempt on an existing open invoice.
3. Confirm Stripe delivers invoice.payment_failed.
4. Verify billing_subscription:

SELECT status, plan_tier
FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: status = 'past_due' (Stripe sets this after first failed attempt)

5. Verify billing_invoice:

SELECT stripe_invoice_id, status, amount_due, amount_remaining, invoice_event_type
FROM billing_invoice
WHERE stripe_customer_id = '<test_cus_id>'
ORDER BY created_at DESC LIMIT 3;
-- Expected: at least one row with invoice_event_type = 'invoice.payment_failed',
--           status = 'open', amount_remaining = amount_due

6. Check Raptor mirror:

SELECT status FROM billing_subscription_mirror
WHERE queue_customer_id = '<queue_customer_id>';
-- Expected: status = 'past_due'
-- A 'past_due' mirror means Raptor's paywall will enforce a fail-closed 402
-- on gated endpoints — verify this is the intended behavior for past_due users

7. Email trigger — not wired (#1686): Confirm the Queue log shows the payment-failed email trigger log line (billing.webhook: payment-failed email trigger for event_id=...) but no actual email is sent.

Steps — recovery:

8. Update the test customer's payment method back to the success test card.
9. Retry the failed invoice via Stripe dashboard (Customers → [test customer] → Invoices → [failed invoice] → Retry).
10. Confirm Stripe delivers invoice.payment_succeeded.
11. Verify billing_subscription:

SELECT status FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: status = 'active'

12. Verify billing_subscription_mirror.status = 'active'.

Pass criteria: - [ ] invoice.payment_failed delivered HTTP 200 - [ ] billing_subscription.status = 'past_due' after failure - [ ] billing_invoice row updated with failure event type - [ ] billing_subscription_mirror.status = 'past_due' during failure window - [ ] After card fix and retry: invoice.payment_succeeded delivered HTTP 200 - [ ] billing_subscription.status reverts to 'active' - [ ] billing_subscription_mirror.status reverts to 'active' - [ ] Log line confirms payment-failed email trigger (actual email send is pending #1686)

Scenario 5 — 3DS / SCA authentication

Purpose: Verify that a 3DS-required card does not get rejected silently and that the authentication challenge path completes successfully.

Preconditions: - Test customer with no active subscription

Test card: See billing-test-tooling.md § "3DS / SCA cards" — use the "authentication required" test card.

Steps:

1. Create a checkout session for the test customer with the Pro price using the 3DS test card.
2. In the Stripe test mode checkout page, you will see a simulated 3DS authentication dialog. Click "Complete authentication" (or equivalent test-mode button).
3. Confirm Stripe delivers customer.subscription.created and invoice.payment_succeeded after authentication completes.
4. Verify billing_subscription.status = 'active' and billing_invoice.status = 'paid' as in Scenario 1.

Failure path (optional):

5. Repeat step 1–2 but click "Fail authentication" in the 3DS dialog.
6. Confirm Stripe delivers invoice.payment_failed (or checkout.session.expired) and the subscription is NOT created (or is incomplete_expired).
7. Verify no active billing_subscription row exists for this customer.

Pass criteria: - [ ] 3DS success path: subscription created, invoice paid, tables populated as in Scenario 1 - [ ] 3DS failure path: no active subscription row; billing_subscription absent or status = 'incomplete_expired'

Note on incomplete status: When a subscription requires 3DS and the customer hasn't completed authentication yet, Stripe briefly sets the subscription to incomplete. The webhook handler's status CHECK constraint accepts incomplete and incomplete_expired. Confirm the handler does not reject these status values silently.

Scenario 6 — Cancellation: cancel-at-period-end and immediate

Purpose: Verify both cancellation paths write the correct state and the customer's access is downgraded at the right time.

Preconditions: Active Pro subscription (from Scenario 1 or a fresh one).

6a — Cancel at period end

Steps:

1. In Stripe dashboard, on the test customer's subscription, click "Cancel subscription" → "Cancel at end of billing period."
2. Confirm Stripe delivers customer.subscription.updated.
3. Verify billing_subscription:

SELECT status, cancel_at_period_end, canceled_at
FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: status = 'active', cancel_at_period_end = true, canceled_at = NULL
-- (subscription is still active until period ends)

4. Advance Test Clock past the period end. Confirm Stripe delivers customer.subscription.deleted.
5. Verify:

SELECT status, cancel_at_period_end, canceled_at
FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: status = 'canceled', canceled_at IS NOT NULL

6. Verify billing_subscription_mirror:

SELECT status FROM billing_subscription_mirror
WHERE queue_customer_id = '<queue_customer_id>';
-- Expected: status = 'canceled'
-- Raptor's paywall will return 402 for gated routes for this customer

6b — Immediate cancellation

Steps:

1. Start with a fresh active Pro subscription.
2. In Stripe dashboard, cancel the subscription immediately (not at period end).
3. Confirm Stripe delivers customer.subscription.deleted.
4. Verify billing_subscription.status = 'canceled' and canceled_at IS NOT NULL immediately.
5. Verify billing_subscription_mirror.status = 'canceled'.
6. Verify Raptor paywall: make an authenticated API request as the test user to a Pro-gated endpoint. Confirm 402 Payment Required.

Pass criteria: - [ ] Cancel-at-period-end: cancel_at_period_end = true while still active - [ ] After period end: status = 'canceled', canceled_at populated - [ ] Immediate cancel: status = 'canceled' within seconds of Stripe event - [ ] Mirror updated to 'canceled' in both cases - [ ] Raptor returns 402 on gated endpoints after cancellation (fail-closed) - [ ] (Pending #409) Console customer detail reflects canceled state

Founders note: The Founders tier has a 6-month minimum commitment enforced application-side (not Stripe-side). Customer-initiated cancellation before month 6 is supposed to route through a dunning flow rather than firing subscription.delete directly. This enforcement logic is not documented as shipped. Verify whether the cancellation path in Scenario 6 correctly blocks early Founders cancellation or whether that is a gap to file as a finding.

Scenario 7 — Refund: full and partial

Purpose: Verify that refund events are recorded and reconcile correctly in billing tables.

Preconditions: Paid invoice from Scenario 1 or any completed payment.

Steps — full refund:

1. In Stripe TEST MODE dashboard, navigate to the test customer → Payments → [successful charge] → Refund.
2. Enter the full charge amount and click Refund.
3. Confirm Stripe delivers charge.refunded.

Note: charge.refunded is not currently a handled event type in webhook_handler.cpp (the handler routes customer.*, customer.subscription.*, invoice.*, and checkout.session.completed). The charge.refunded event will be deduped and return HTTP 200 with {"received":true}, but no billing table mutation will occur.

4. Verify processed_stripe_events contains the charge.refunded event ID (confirming dedup works).
5. Verify that billing_invoice is NOT automatically updated (no refund amount field exists in the current schema). This is a gap — document it.
6. Console manual-ops — pending #410: Once #410 ships, the refund workflow runs through Console. Until then, all refunds are done via Stripe dashboard directly.

Steps — partial refund:

7. Repeat steps 1–5 with a partial amount (e.g., 50% of the charge).
8. Same result: processed_stripe_events records the event; billing_invoice is not updated.

Pass criteria: - [ ] charge.refunded event delivers HTTP 200 (dedup + return, no mutation) - [ ] processed_stripe_events contains the refund event ID - [ ] No duplicate entry if the same charge.refunded event is re-delivered - [ ] (Pending #410) Console manual-ops refund flow

Gap finding: The current billing_invoice schema has no amount_refunded column and no charge.refunded handler that writes back to the invoice. Refunds are visible in Stripe's dashboard but are not reflected in Queue's billing tables. This must be resolved before the acceptance gate can close for Scenario 7. File a follow-up card for the charge.refunded handler + amount_refunded column.

Scenario 8 — Plan change and proration

Purpose: Verify upgrade and downgrade between tiers write the correct plan_tier and prior_tier values, and that proration invoices appear correctly.

Preconditions: Active Pro subscription.

8a — Upgrade (Pro → Pro+)

Steps:

1. In Stripe dashboard, update the test customer's subscription to the Pro+ price.
2. Confirm Stripe delivers customer.subscription.updated and a proration invoice.payment_succeeded.
3. Verify:

SELECT plan_tier, prior_tier, status, feature_locked_at
FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: plan_tier = 'pro_plus', prior_tier = NULL (upgrade, not downgrade),
--           feature_locked_at = NULL

4. Verify proration invoice in billing_invoice:

SELECT amount_due, amount_paid, status
FROM billing_invoice
WHERE stripe_customer_id = '<test_cus_id>'
ORDER BY created_at DESC LIMIT 3;
-- Expected: a new invoice row with amount_due reflecting the prorated upgrade charge,
--           status = 'paid'

8b — Downgrade (Pro+ → Pro)

Steps:

5. From an active Pro+ subscription, update to the Pro price.
6. Confirm Stripe delivers customer.subscription.updated.
7. Verify the downgrade detection in billing_subscription:

SELECT plan_tier, prior_tier, feature_locked_at, status
FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: plan_tier = 'pro', prior_tier = 'pro_plus',
--           feature_locked_at IS NOT NULL (set on first downgrade),
--           status = 'active'

8. Re-deliver the same customer.subscription.updated event (duplicate delivery test — see Scenario 9). Verify feature_locked_at is NOT overwritten with a newer timestamp (the LWW guard and already_locked check should prevent this).

9. Verify billing_subscription_mirror.plan_tier = 'pro' after downgrade.

Pass criteria: - [ ] Upgrade: plan_tier updated to higher tier, prior_tier = NULL, feature_locked_at = NULL - [ ] Upgrade proration invoice: new billing_invoice row with correct amounts, status = 'paid' - [ ] Downgrade: plan_tier updated to lower tier, prior_tier set to previous tier, feature_locked_at IS NOT NULL - [ ] Downgrade: re-delivery does NOT update feature_locked_at again - [ ] Mirror updated after both upgrade and downgrade

Scenario 9 — Idempotency: duplicate webhook delivery

Purpose: Verify that re-delivering the same Stripe event does not produce duplicate rows or double mutations.

Preconditions: Any processed Stripe event from a prior scenario (note its event_id).

Steps:

1. Re-deliver the same event using the Stripe dashboard (Developers → Webhooks → [endpoint] → [delivery] → Resend).
2. Confirm Queue returns HTTP 200 with {"received":true}.
3. Verify the Queue log shows the idempotent-dedup log line:

billing.webhook: duplicate event_id=<id> → 200 (idempotent)

4. Verify processed_stripe_events contains exactly ONE row for the event ID (no duplicates):

SELECT COUNT(*) FROM processed_stripe_events WHERE event_id = '<event_id>';
-- Expected: 1

5. Verify billing_subscription and billing_invoice were NOT mutated by the duplicate delivery (row updated_at unchanged from after the first delivery).

Concurrent duplicate test (optional, advanced):

6. Use two simultaneous curl requests to post the same signed Stripe event payload to the webhook endpoint at exactly the same time. Confirm that the pqxx::unique_violation catch path fires for the second request and it also returns HTTP 200 (not 500 or 409). Check Queue logs for the concurrent duplicate event_id=<id> — 200 (idempotent) log line.

Pass criteria: - [ ] Re-delivered event returns HTTP 200 ({"received":true}) - [ ] Log shows idempotent dedup path, not the processing path - [ ] processed_stripe_events has exactly 1 row for the event ID - [ ] No mutations to billing tables from the duplicate delivery - [ ] Concurrent duplicate: second request returns 200 (not 500)

Scenario 10 — Apple IAP / StoreKit 2 (iOS path)

Status: BLOCKED — see below

Purpose: Verify the iOS billing path — sandbox tester purchases a subscription, the server validates the transaction, entitlement is granted, sandbox renewal and cancellation are reflected.

Current state: This scenario is fully blocked. As of 2026-06-17, the following components required for this scenario do not exist in the codebase:

Design references: ADR-0007 (docs/architecture/adr/0007-ios-subscription-billing-iap.md) accepted this approach. Downstream implementation cards were filed under Epic #167. None have shipped.

Reconciliation posture (when built): Apple is the source of truth for iOS subscriptions. A user who has an active iOS subscription via Apple IAP and also attempts to subscribe via web Stripe should be refused the second subscription (same-user dual-subscription path). This guard also does not exist yet.

Actions before this scenario can run: 1. Implement the Apple S2S notification handler (ADR-0007 downstream card). 2. Add subscription_source column to billing_subscription (or a separate IAP table). 3. Set up App Store Connect sandbox environment (requires Apple Developer org account — confirm status with Kristerpher before starting). 4. Create a sandbox tester account in App Store Connect. 5. Register at least one auto-renewable subscription product in App Store Connect for the Pro tier.

When unblocked, the steps will be:

1. On a development or TestFlight iOS build, sign in as the sandbox tester.
2. Purchase a Pro subscription via the in-app subscription UI (StoreKit 2 flow).
3. Confirm Apple calls /api/subscriptions/apple/notifications with a DID_CHANGE_RENEWAL_STATUS or SUBSCRIBED notification.
4. Verify entitlement is granted: user's Raptor-side session claims or mirror table shows Pro access.
5. Advance sandbox time (Apple sandbox subscriptions have accelerated renewal periods).
6. Confirm renewal notification arrives and entitlement is extended.
7. Cancel the sandbox subscription; confirm DID_CHANGE_RENEWAL_STATUS fires and access is revoked.
8. Cross-billing check: same user with active iOS sub should receive a 409 Conflict when attempting to subscribe via web Stripe checkout.

Pass criteria (for when built): - [ ] S2S notification validated (JWS signature verified) - [ ] original_transaction_id stored (not transaction_id) - [ ] subscription_source = 'ios' on the subscription row - [ ] Entitlement granted after SUBSCRIBED notification - [ ] Renewal extends entitlement period - [ ] Cancellation revokes access - [ ] Dual-subscription attempt returns 409 (not a silent double-billing)

Production smoke (minimal, disposable test customer)

Run this section only after every testable scenario above has PASSED in Stripe TEST MODE.

Purpose: Confirm the live webhook endpoint receives and processes a real Stripe event in production. This uses a real card charge that is refunded immediately — a controlled, minimal, real-money transaction.

Prerequisites: - All TEST MODE scenarios above: PASS - Stripe live-mode keys in vault prod path (see billing-test-tooling.md § "Live mode vault paths") - FLAG_QUEUE_BILLING=true on raxx-queue-prod - Stripe dashboard switched to LIVE MODE - Disposable test customer account available (see billing-test-tooling.md § "Prod smoke customer")

Steps:

1. Switch Stripe dashboard to LIVE MODE.
2. Create a new customer in live mode with a dedicated test email (e.g., billing-smoke-YYYY-MM-DD@raxx.app). Do not use a real customer email.
3. Subscribe the test customer to the Pro plan using the operator's own card (or a pre-authorized test payment method).
4. Confirm live-mode webhook delivery at https://queue.raxx.app/api/v1/billing/webhook: - customer.subscription.created — HTTP 200 - invoice.payment_succeeded — HTTP 200
5. Verify Queue prod tables:

-- On Queue prod DB (read-only credentials from billing-test-tooling.md):
SELECT billing_email, stripe_customer_id FROM billing_customer
WHERE billing_email = 'billing-smoke-YYYY-MM-DD@raxx.app';
-- Expected: 1 row

SELECT status, plan_tier FROM billing_subscription
WHERE stripe_customer_id = '<prod_smoke_cus_id>';
-- Expected: status = 'active', plan_tier = 'pro'

6. Verify billing_subscription_mirror on Raptor prod DB:

SELECT status, plan_tier FROM billing_subscription_mirror
WHERE queue_customer_id = '<prod_smoke_queue_cus_id>';
-- Expected: status = 'active', plan_tier = 'pro'

7. Immediately cancel the smoke subscription in live Stripe dashboard and issue a full refund of the charge. Do not let it bill into a second period.
8. Confirm cancellation and refund land in prod billing tables.
9. Optionally: delete the smoke customer record from Stripe (not required, but keeps live-mode data clean).

Pass criteria: - [ ] Live webhook delivers HTTP 200 for subscription creation and invoice payment - [ ] billing_customer and billing_subscription rows exist in prod DB - [ ] billing_subscription_mirror row exists in Raptor prod DB - [ ] Subscription canceled and refund issued immediately after validation - [ ] No Sentry errors during the smoke window

If the prod smoke fails: Do NOT proceed to enabling real customer billing. Roll back FLAG_QUEUE_BILLING=false on raxx-queue-prod, investigate, and repeat the test-mode scenarios to isolate the regression before re-running the smoke.

Pre-charge acceptance gate

Every line below must be checked before real customer billing is enabled or a real charge is processed. This checklist is the gate.

Test-mode scenarios (Stripe TEST MODE)

[ ] Scenario 1 (happy path): PASS — subscription active, all webhooks 200, tables populated, mirror populated
[ ] Scenario 2 (tier coverage): PASS — all four tiers map to correct plan_tier values
[ ] Scenario 3 (renewal): PASS — Test Clock advance triggers renewal, period dates updated
[ ] Scenario 4 (payment failure + dunning): PASS — past_due on fail, active on recovery
[ ] Scenario 5 (3DS / SCA): PASS — authentication challenge path completes; failure path rejected cleanly
[ ] Scenario 6a (cancel at period end): PASS — cancel_at_period_end=true → status=canceled after period
[ ] Scenario 6b (immediate cancel): PASS — status=canceled immediately, Raptor 402 confirmed
[ ] Scenario 7 (refund): PARTIAL PASS ACCEPTABLE — charge.refunded deduped correctly; billing_invoice gap
       acknowledged and tracked on a filed follow-up card before gate closes
[ ] Scenario 8 (plan change / proration): PASS — upgrade no lock; downgrade sets prior_tier + feature_locked_at
[ ] Scenario 9 (idempotency): PASS — duplicate delivery returns 200, no double mutation
[ ] Scenario 10 (Apple IAP): BLOCKED — not built; gate HOLDS OPEN until built and passing
       (Apple IAP is NOT required before web/Stripe billing can go live; they are independent paths;
       the gate for web billing can close without Scenario 10 if confirmed by operator)

Console billing surfaces

[ ] #409 (Console customer detail): PASS or explicitly deferred by operator — mark deferred date
[ ] #1633 (Console MRR dashboard): PASS or explicitly deferred by operator — mark deferred date
[ ] #410 (Console manual ops / refund): PASS or explicitly deferred by operator — mark deferred date
[ ] (Deferred surfaces require a follow-up date and a separate mini-gate when they ship)

Infrastructure and security

[ ] FLAG_QUEUE_BILLING=true on raxx-queue-prod confirmed with no rollback in preceding 48h
[ ] STRIPE_WEBHOOK_SECRET on raxx-queue-prod is the current live-mode whsec_* value (not test mode, not stale)
[ ] STRIPE_RESTRICTED_KEY (or equivalent) on raxx-queue-prod uses rk_live_* restricted key with correct scopes
[ ] Stripe test keys are NOT present on raxx-queue-prod (only live keys on prod)
[ ] billing_subscription_mirror fan-out is confirmed working (RAPTOR_BASE_URL + QUEUE_TO_RAPTOR_INTERNAL_TOKEN set on prod)
[ ] No Sentry billing.webhook.hmac_failure events in the last 24h on prod
[ ] No billing.mirror.fan_out_failure log lines on prod in the last 24h
[ ] PA SaaS tax posture confirmed with CPA (#2743) OR explicitly deferred with documented operator decision

Production smoke

[ ] Prod smoke: PASS (see "Production smoke" section above)
[ ] Prod smoke subscription canceled and refunded immediately after validation
[ ] No Sentry errors during smoke window

Operator sign-off

[ ] Operator (Kristerpher) has reviewed this checklist and confirmed all checked items
[ ] Operator-deferred items documented with explicit deferral date and follow-up issue number
[ ] This checklist signed off by: _______________ Date (UTC): _______________

Known gaps to file as follow-up cards

These are gaps found during authoring of this SOP. Each should become a filed issue before or alongside the acceptance gate closing.

Rollback procedure

If any scenario fails during the acceptance gate run and the operator has already enabled FLAG_QUEUE_BILLING=true on prod:

heroku config:set FLAG_QUEUE_BILLING=false --app raxx-queue-prod >/dev/null 2>&1
# Confirm:
heroku config:get FLAG_QUEUE_BILLING --app raxx-queue-prod
# Expected: false

This disables the webhook endpoint (returns 503 to Stripe; Stripe will retry). It does not affect existing subscription rows already written to Queue DB. Re-enable only after the regression is identified and fixed in staging.

References

• docs/ops/runbooks/billing-test-tooling.md — test cards, test clock commands, vault paths, DB verification commands
• docs/ops/runbooks/stripe-founders-setup.md — Stripe product/price/webhook provisioning SOP
• docs/architecture/stripe-customer-billing.md — billing data model (v5)
• docs/architecture/queue-stripe-webhook-design-2026-05-14.md — webhook handler design
• docs/architecture/adr/0007-ios-subscription-billing-iap.md — Apple IAP decision
• docs/architecture/adr/0071-stripe-billing-queue-as-authority.md — Queue as billing authority
• docs/architecture/adr/0127-webhook-idempotency-5xx-not-local-queue.md — idempotency design
• docs/ops/runbooks/queue.md — Queue service runbook (health checks, failure modes)
• docs/ops/2026-05-13-stripe-test-mode-verification.md — historical gap analysis (most gaps now closed)
• docs/architecture/issue-409-customer-detail-view.md — Console customer detail design
• project_pricing_tiers_locked.md — Free / Pro / Pro+ / Founders $29/6mo pricing
• project_ios_billing_iap.md — iOS billing is Apple IAP, not Stripe
• Epic #403 — Console billing epic
• Epic #167 — iOS billing epic