Billing E2E Test SOP — Subscription Lifecycle Validation

Owner: sre-agent / Kristerpher Last updated: 2026-08-19 UTC (refresh + partial gate execution — #4502; prior full authoring 2026-06-17 UTC — PR #3624) Refs: #403 (billing console epic), #405 (data model), #407 (webhook handler), #409 (customer detail view — shipped, PR #3860), #410 (manual ops — shipped, PR #3860 + #4341), #1633 (MRR dashboard — shipped, PR #3277 + #4386 + #4388), #4149 / PR #4499 (idempotency Wave 3), #4502 (this refresh) Tooling companion: docs/ops/runbooks/billing-test-tooling.md — all concrete test card numbers, Stripe Test Clock commands, test account credentials, IAP sandbox setup, vault test paths, and verification shell commands live there. This SOP references that doc for specifics; do not duplicate values here.

Doc-drift convention (added 2026-08-19, #4502 risk mitigation): Any PR that closes a #NNN referenced in the "Current billing state" table or the "Known gaps" table below must update this doc in the same PR, or file a follow-up doc-refresh card within 48h, per feedback_docs_sweep_on_decision. A stale "NOT BUILT" row for a shipped feature is itself a QA finding on the next sweep.

Purpose and scope

This is the repeatable, operator-executable procedure for exercising the complete Raxx subscription lifecycle against test accounts and test credit cards — before any real charge is processed. It is the authoritative pre-charge acceptance gate. No real billing is enabled until every scenario in this document passes in Stripe TEST MODE, and the prod smoke at the end of this SOP runs clean.

The SOP is grounded in the live codebase state as of 2026-08-19 (origin/develop @ e87574011). Scenarios that depend on unbuilt surfaces are marked explicitly. Run only what is buildable today; hold the acceptance gate open until blocked scenarios are unblocked.

What is in scope: - Stripe web/desktop subscription path (Queue webhook handler, billing tables, Raptor mirror) - Apple IAP/StoreKit 2 iOS path (separate flow; reconciliation with Stripe path) - Console billing views (customer detail, MRR dashboard, manual ops — all shipped; see "Current billing state")

What is out of scope: - Vendor-spend billing (Heroku, AWS, Cloudflare cost collection — separate billing-readonly-tokens.md) - Alpaca/broker wiring - Stripe Tax filing / PA-3 return (operator action, tracked in #2743)

Architecture summary (know before you test)

Queue is the authoritative billing store. All Stripe webhook events arrive at POST https://queue.raxx.app/api/v1/billing/webhook, are HMAC-verified, deduped via processed_stripe_events, and written to billing_customer / billing_subscription / billing_invoice in Queue's Postgres. Queue then fans out a PII-free mirror row to Raptor's billing_subscription_mirror table. Console reads Queue's internal HTTP API; it has no local billing tables.

The feature flag FLAG_QUEUE_BILLING is the kill-switch. When false, the webhook endpoint returns 503 immediately. All testing requires FLAG_QUEUE_BILLING=true on the target environment.

Tier ranking (corrected 2026-08-19 — see below): free(0) < pro(1) < founders(2) == pro_plus(2). Downgrade detection compares these ranks; feature_locked_at is set on the first detected downgrade. Plan tier comes from subscription.metadata.plan_tier or subscription.items[0].price.metadata.plan_tier in the Stripe event payload.

Correction (2026-08-19, #4502 groomer addendum): this section previously stated free(0) < founders(1) < pro(2) < pro_plus(3). That predates #3890 (2026-06-29, operator-locked: "Founders is just Pro+ at a discount"). Verified directly against backend_v2/api/middleware/tier_gate.py _TIER_RANK (current origin/develop): python _TIER_RANK: dict[str, int] = { "free": 0, "pro": 1, "founders": 2, # billing cohort; feature access == pro_plus (#3890) "pro_plus": 2, } Founders is a billing/cohort label only for analytics and rate-lock purposes (validate_founders_rate() in the same module) — it is stored as plan_tier = 'founders' in billing_subscription (unchanged), but for @require_tier(...) feature-gate purposes a Founders subscriber passes every gate a Pro+ subscriber passes. Any scenario below asserting Founders feature access must be checked against founders == pro_plus rank, not the old founders < pro ordering.

Current billing state (verified 2026-08-19 against origin/develop @ e87574011)

Pre-test prerequisites

Complete every item before running any scenario. Failing to do so will produce misleading results.

[ ] 1. Stripe dashboard is in TEST MODE (toggle in top-right; verify "Test" label is visible)
[ ] 2. STRIPE_WEBHOOK_SECRET on raxx-queue-staging matches the staging webhook endpoint's whsec_* value
      (see billing-test-tooling.md § "Webhook secrets")
[ ] 3. FLAG_QUEUE_BILLING=true on the target environment (staging for test-mode run; prod for smoke)
      heroku config:get FLAG_QUEUE_BILLING --app raxx-queue-staging
      Expected: true or 1
      (2026-08-19: confirmed true by behavior — see Execution log; heroku config:get itself not run, no CLI access this session)
[ ] 4. Queue sqitch migrations 01–08 are applied to the target Queue DB
      (see billing-test-tooling.md § "Verifying migration state"; 07/08 are new since 2026-06-17)
[ ] 5. raxx-queue-staging dyno is up and healthy
      curl -s https://raxx-queue-staging-403c1aa5941f.herokuapp.com/health | python3 -m json.tool
      Expected: {"service":"raxx-queue","status":"ok","timestamp":"...","version":"..."}
      (2026-08-19: confirmed PASS)
[ ] 6. RAPTOR_BASE_URL and QUEUE_TO_RAPTOR_INTERNAL_TOKEN set on raxx-queue-staging
      (required for mirror fan-out; confirm with billing-test-tooling.md § "Internal service tokens")
[ ] 7. Test customer account created in Stripe TEST MODE
      (see billing-test-tooling.md § "Test accounts and customers")
[ ] 8. Stripe test price IDs configured for all four tiers (free, founders, pro, pro_plus)
      (see billing-test-tooling.md § "Test price IDs and products")
[ ] 9. FLAG_BILLING_EMAIL_DISPATCH on raxx-queue-staging confirmed OFF (or, if ON, all test customer
      emails are addresses you control and are prepared to receive real Postmark sends — NEW as of
      2026-08-19; this flag did not exist as a live-send path on 2026-06-17)

Scenario matrix

Each scenario specifies: preconditions, the test card/account to use (referencing the tooling doc), step-by-step actions, and expected results at every layer.

The layers checked in each scenario are: - Stripe state — what the Stripe TEST MODE dashboard shows - Queue billing tables — billing_customer, billing_subscription, billing_invoice, processed_stripe_events - Raptor mirror — billing_subscription_mirror - Console — customer detail (#409), MRR dashboard (#1633), and manual ops (#410) are all shipped as of this refresh; scenario steps below say "shipped — verify" rather than "pending #NNN"

Scenario 1 — New subscription: happy path

Purpose: Verify the end-to-end checkout → subscription active → webhook ingestion → table population flow.

Preconditions: - Test customer exists in Stripe TEST MODE with no active subscription - FLAG_QUEUE_BILLING=true on target env - Staging webhook endpoint registered in Stripe and receiving events

Test card: success card from billing-test-tooling.md § "Success cards" (the standard 4242... card with any future expiry)

Steps:

1. In Stripe TEST MODE dashboard, create a checkout session for the test customer with the Pro price ID. Alternatively use stripe trigger checkout.session.completed if Stripe CLI is available (see billing-test-tooling.md § "Stripe CLI triggers").
2. Complete checkout using the success test card.
3. Wait 5–10 seconds for Stripe to deliver webhooks.
4. In the Stripe dashboard, navigate to Developers → Webhooks → your staging endpoint → Recent deliveries. Confirm delivery of: - checkout.session.completed — HTTP 200 - customer.subscription.created — HTTP 200 - invoice.payment_succeeded — HTTP 200
5. Check Queue billing tables (see billing-test-tooling.md § "DB verification commands"):

-- On Queue staging DB:
SELECT stripe_customer_id, billing_email, customer_segment
FROM billing_customer
WHERE stripe_customer_id = '<test_stripe_cus_id>';
-- Expected: 1 row with billing_email and customer_segment = 'organic'

SELECT stripe_subscription_id, plan_tier, status,
       current_period_start, current_period_end, cancel_at_period_end
FROM billing_subscription
WHERE stripe_customer_id = '<test_stripe_cus_id>';
-- Expected: 1 row, status = 'active', plan_tier = 'pro', cancel_at_period_end = false

SELECT stripe_invoice_id, amount_due, amount_paid, status, invoice_event_type, paid_at
FROM billing_invoice
WHERE stripe_customer_id = '<test_stripe_cus_id>'
ORDER BY created_at DESC LIMIT 5;
-- Expected: at least 1 row, status = 'paid', invoice_event_type = 'invoice.payment_succeeded', paid_at NOT NULL

SELECT event_id, created_at
FROM processed_stripe_events
ORDER BY created_at DESC LIMIT 5;
-- Expected: event IDs for the three webhook events above; all deduped (no duplicates)

6. Check Raptor mirror (see billing-test-tooling.md § "Raptor mirror check"):

-- On Raptor DB:
SELECT queue_customer_id, plan_tier, status, current_period_end, updated_at
FROM billing_subscription_mirror
WHERE queue_customer_id = '<queue_customer_id_for_test_user>';
-- Expected: 1 row, status = 'active', plan_tier = 'pro'
-- If row is absent, fan-out failed — check Queue logs for 'billing.mirror.fan_out_failure'

7. Console customer detail — shipped (#409) — verify: navigate to /console/customers/<customer_id> and verify the Subscription section shows plan tier = Pro, status = Active, correct period dates.

8. Console MRR dashboard — shipped (#1633) — verify: navigate to /console/billing/dashboard and verify MRR tile has increased by the Pro monthly amount.

Pass criteria: - [ ] All three Stripe webhook events delivered HTTP 200 - [ ] billing_customer row exists with correct email - [ ] billing_subscription row: status = 'active', plan_tier = 'pro' - [ ] billing_invoice row: status = 'paid', paid_at populated - [ ] processed_stripe_events contains all three event IDs - [ ] billing_subscription_mirror row: status = 'active', plan_tier = 'pro' - [ ] Console customer detail shows correct tier and state - [ ] Console MRR tile updated

Scenario 2 — Tier coverage: Free / Pro / Pro+ / Founders

Purpose: Verify each pricing tier maps correctly through the webhook pipeline to the right plan_tier value.

Preconditions: Four separate test customers, one per tier. Free-tier customer has no Stripe subscription (or a free plan subscription with no price).

Test accounts/prices: See billing-test-tooling.md § "Tier test matrix" for the price ID for each tier.

Steps (repeat for each tier):

1. For each of Pro, Pro+, and Founders: create a subscription for the test customer using the respective price ID. For Free: confirm a test customer with no active subscription is treated as free.

2. For Pro, Pro+, and Founders: deliver customer.subscription.created to the staging webhook (trigger from Stripe dashboard or Stripe CLI).

3. After each event, verify the billing_subscription table:

-- For each tier, confirm:
SELECT plan_tier, status, stripe_price_id
FROM billing_subscription
WHERE stripe_customer_id = '<tier_test_cus_id>';
-- Expected plan_tier values:
--   Pro:      'pro'
--   Pro+:     'pro_plus'
--   Founders: 'founders'
-- Free: no billing_subscription row (or row with plan_tier = 'free' if provisioned)

4. For Founders, also verify customer_segment in billing_customer:

SELECT customer_segment FROM billing_customer
WHERE stripe_customer_id = '<founders_cus_id>';
-- Expected: 'founders'

Pass criteria: - [ ] plan_tier = 'pro' for Pro price event - [ ] plan_tier = 'pro_plus' for Pro+ price event - [ ] plan_tier = 'founders' for Founders price event - [ ] Free-tier customer: no active billing_subscription row OR row with plan_tier = 'free' - [ ] Raptor mirror updated for each paid tier - [ ] customer_segment auto-updates to 'founders' on the Founders subscription event

Resolved (2026-08-19, was "Known gap" as of 2026-06-17): customer_segment = 'founders' auto-set shipped via #3630. webhook_handler.cpp now runs, on plan_tier='founders' events:

UPDATE billing_customer SET customer_segment = 'founders', updated_at = CURRENT_TIMESTAMP
WHERE stripe_customer_id = ... AND customer_segment = 'organic'

It only overwrites the default 'organic' value (does not clobber a non-organic segment), and does not revert customer_segment back to 'organic' on downgrade away from Founders — confirm this one-way behavior is the intended semantics when running this scenario.

Known gap still open: The plan_tier is read from subscription.metadata.plan_tier or subscription.items[0].price.metadata.plan_tier in the webhook payload. Confirm these metadata fields are set on each Stripe Price object at creation time (see billing-test-tooling.md § "Price metadata setup"). If the metadata field is absent, plan_tier defaults to 'free' and the tier will be wrong silently.

Scenario 3 — Renewal: Stripe Test Clock advance

Purpose: Verify that subscription renewal fires invoice.payment_succeeded and that current_period_start / current_period_end update correctly in billing_subscription.

Preconditions: - Active Pro subscription from Scenario 1 - Stripe Test Clock created and attached to the test customer - billing-test-tooling.md § "Test Clocks" has the setup commands

Note on Test Clocks: Stripe Test Clocks are independent of real time. A Test Clock advances a simulated timeline for associated customers. The test customer must have been created with the Test Clock attached — you cannot retroactively attach a Test Clock to a live customer. Create a new test customer via the Test Clock UI or Stripe CLI before running Scenario 3.

Steps:

1. Create a Test Clock (see billing-test-tooling.md § "Test Clock creation").
2. Create a test customer under the Test Clock; subscribe to the Pro price.
3. Verify Scenario 1 pass criteria for this new customer.
4. Advance the Test Clock to the renewal date (one billing period past the current current_period_end):

# Stripe CLI:
stripe test_helpers test_clocks advance \
  --test-clock-id <clock_id> \
  --frozen-time <ISO-8601 timestamp one month after subscription start>

5. Wait for Stripe to deliver: - invoice.created — HTTP 200 - invoice.payment_succeeded — HTTP 200 (assuming success card) - customer.subscription.updated — HTTP 200 (period dates updated)

6. Check billing_subscription:

SELECT current_period_start, current_period_end, updated_at
FROM billing_subscription
WHERE stripe_customer_id = '<test_clock_cus_id>';
-- Expected: current_period_start and current_period_end advanced by one billing period
-- updated_at should be newer than the value from Scenario 1

7. Check billing_invoice for the renewal invoice:

SELECT stripe_invoice_id, amount_due, amount_paid, status, paid_at
FROM billing_invoice
WHERE stripe_customer_id = '<test_clock_cus_id>'
ORDER BY created_at DESC LIMIT 3;
-- Expected: 2 paid invoices (initial + renewal), latest paid_at is within the last few minutes

8. Check billing_subscription_mirror:

SELECT current_period_end, updated_at
FROM billing_subscription_mirror
WHERE queue_customer_id = '<queue_customer_id>';
-- Expected: current_period_end matches the new period end from billing_subscription

9. Console MRR tile — shipped (#1633) — verify: Verify MRR does not double-count the renewal.

Pass criteria: - [ ] Renewal invoice.payment_succeeded delivered and returns HTTP 200 - [ ] billing_subscription.current_period_start and current_period_end advanced correctly - [ ] New billing_invoice row: status = 'paid', paid_at set - [ ] billing_subscription_mirror.current_period_end updated to match - [ ] No duplicate billing_invoice row for the same renewal event (idempotency)

Scenario 4 — Payment failure and dunning

Purpose: Verify that a declined card transitions the subscription to past_due, fires the expected events, and that the system recovers correctly when the card is fixed.

Preconditions: - Active Pro subscription - Test Clock (recommended, so you can control retry timing) or use Stripe Smart Retry with the real clock

Test cards: See billing-test-tooling.md § "Decline cards" — use the insufficient-funds card and the generic-decline card.

Steps — inducing failure:

1. Update the test customer's default payment method to the decline test card (see billing-test-tooling.md § "Attaching test cards").
2. Advance the Test Clock past the renewal date, or use the Stripe CLI to trigger a payment attempt on an existing open invoice.
3. Confirm Stripe delivers invoice.payment_failed.
4. Verify billing_subscription:

SELECT status, plan_tier
FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: status = 'past_due' (Stripe sets this after first failed attempt)

5. Verify billing_invoice:

SELECT stripe_invoice_id, status, amount_due, amount_remaining, invoice_event_type
FROM billing_invoice
WHERE stripe_customer_id = '<test_cus_id>'
ORDER BY created_at DESC LIMIT 3;
-- Expected: at least one row with invoice_event_type = 'invoice.payment_failed',
--           status = 'open', amount_remaining = amount_due

6. Check Raptor mirror:

SELECT status FROM billing_subscription_mirror
WHERE queue_customer_id = '<queue_customer_id>';
-- Expected: status = 'past_due'
-- A 'past_due' mirror means Raptor's paywall will enforce a fail-closed 402
-- on gated endpoints — verify this is the intended behavior for past_due users

7. Email trigger — shipped (code) behind FLAG_BILLING_EMAIL_DISPATCH, default OFF (#3631/#3816/#3819): If the flag is OFF, confirm the Queue log shows the payment-failed email trigger log line (billing.webhook: payment-failed email trigger for event_id=... (FLAG_BILLING_EMAIL_DISPATCH=false — #3631)) but no actual email is sent. If the flag is ON, a real Postmark send fires via POSTMARK_TEMPLATE_BILLING_PAYMENT_FAILED — confirm the target test customer email is one you control before running this step with the flag ON (see Pre-test prerequisite #9).

Steps — recovery:

8. Update the test customer's payment method back to the success test card.
9. Retry the failed invoice via Stripe dashboard (Customers → [test customer] → Invoices → [failed invoice] → Retry).
10. Confirm Stripe delivers invoice.payment_succeeded.
11. Verify billing_subscription:

SELECT status FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: status = 'active'

12. Verify billing_subscription_mirror.status = 'active'.

Pass criteria: - [ ] invoice.payment_failed delivered HTTP 200 - [ ] billing_subscription.status = 'past_due' after failure - [ ] billing_invoice row updated with failure event type - [ ] billing_subscription_mirror.status = 'past_due' during failure window - [ ] After card fix and retry: invoice.payment_succeeded delivered HTTP 200 - [ ] billing_subscription.status reverts to 'active' - [ ] billing_subscription_mirror.status reverts to 'active' - [ ] Log line (flag OFF) or real Postmark send (flag ON) confirms payment-failed email trigger; verify against current FLAG_BILLING_EMAIL_DISPATCH staging value before running

Scenario 5 — 3DS / SCA authentication

Purpose: Verify that a 3DS-required card does not get rejected silently and that the authentication challenge path completes successfully.

Preconditions: - Test customer with no active subscription

Test card: See billing-test-tooling.md § "3DS / SCA cards" — use the "authentication required" test card.

Steps:

1. Create a checkout session for the test customer with the Pro price using the 3DS test card.
2. In the Stripe test mode checkout page, you will see a simulated 3DS authentication dialog. Click "Complete authentication" (or equivalent test-mode button).
3. Confirm Stripe delivers customer.subscription.created and invoice.payment_succeeded after authentication completes.
4. Verify billing_subscription.status = 'active' and billing_invoice.status = 'paid' as in Scenario 1.

Failure path (optional):

5. Repeat step 1–2 but click "Fail authentication" in the 3DS dialog.
6. Confirm Stripe delivers invoice.payment_failed (or checkout.session.expired) and the subscription is NOT created (or is incomplete_expired).
7. Verify no active billing_subscription row exists for this customer.

Pass criteria: - [ ] 3DS success path: subscription created, invoice paid, tables populated as in Scenario 1 - [ ] 3DS failure path: no active subscription row; billing_subscription absent or status = 'incomplete_expired'

Note on incomplete status: When a subscription requires 3DS and the customer hasn't completed authentication yet, Stripe briefly sets the subscription to incomplete. The webhook handler's status CHECK constraint accepts incomplete and incomplete_expired. Confirm the handler does not reject these status values silently.

Scenario 6 — Cancellation: cancel-at-period-end and immediate

Purpose: Verify both cancellation paths write the correct state and the customer's access is downgraded at the right time.

Preconditions: Active Pro subscription (from Scenario 1 or a fresh one).

6a — Cancel at period end

Steps:

1. In Stripe dashboard, on the test customer's subscription, click "Cancel subscription" → "Cancel at end of billing period."
2. Confirm Stripe delivers customer.subscription.updated.
3. Verify billing_subscription:

SELECT status, cancel_at_period_end, canceled_at
FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: status = 'active', cancel_at_period_end = true, canceled_at = NULL
-- (subscription is still active until period ends)

4. Advance Test Clock past the period end. Confirm Stripe delivers customer.subscription.deleted.
5. Verify:

SELECT status, cancel_at_period_end, canceled_at
FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: status = 'canceled', canceled_at IS NOT NULL

6. Verify billing_subscription_mirror:

SELECT status FROM billing_subscription_mirror
WHERE queue_customer_id = '<queue_customer_id>';
-- Expected: status = 'canceled'
-- Raptor's paywall will return 402 for gated routes for this customer

6b — Immediate cancellation

Steps:

1. Start with a fresh active Pro subscription.
2. In Stripe dashboard, cancel the subscription immediately (not at period end).
3. Confirm Stripe delivers customer.subscription.deleted.
4. Verify billing_subscription.status = 'canceled' and canceled_at IS NOT NULL immediately.
5. Verify billing_subscription_mirror.status = 'canceled'.
6. Verify Raptor paywall: make an authenticated API request as the test user to a Pro-gated endpoint. Confirm 402 Payment Required.

Pass criteria: - [ ] Cancel-at-period-end: cancel_at_period_end = true while still active - [ ] After period end: status = 'canceled', canceled_at populated - [ ] Immediate cancel: status = 'canceled' within seconds of Stripe event - [ ] Mirror updated to 'canceled' in both cases - [ ] Raptor returns 402 on gated endpoints after cancellation (fail-closed) - [ ] Console customer detail (#409, shipped) reflects canceled state — verify - [ ] Founders early-cancel guard fires correctly for Founders-tier test subscriptions (see note below)

Founders note — RESOLVED (2026-08-19, was an open question as of 2026-06-17): The Founders 6-month minimum commitment is enforced application-side and shipped via #3628. Confirmed directly in webhook_handler.cpp (current origin/develop): - On customer.subscription.updated with cancel_at_period_end=true, plan_tier='founders', and elapsed months since subscription start < 6: the handler sets founders_early_cancel_blocked = true, forces cancel_at_period_end back to false in the stored row (does not let the early cancel-at-period-end request take effect), writes a billing_action_log entry with action = 'founders.early_cancel_blocked', and logs a Sentry warning ("Founders early cancel blocked in Queue DB — Stripe state diverged"). - On customer.subscription.deleted with plan_tier='founders' and elapsed < 6 months (i.e., an immediate cancel that bypassed the block above, e.g. actioned directly in Stripe): the handler records founders_early_cancel_detected = true and writes action = 'founders.early_cancel_detected' — this is a detection-only path, it cannot un-cancel a subscription Stripe has already terminated; it exists to surface the drift for operator follow-up, not to prevent it. - This scenario now needs a Founders-specific sub-case added to the test run: create a Founders test subscription, attempt cancel-at-period-end before month 6, and confirm (a) cancel_at_period_end stays false in billing_subscription, (b) the founders.early_cancel_blocked audit row is written when FLAG_BILLING_AUDIT_WRITES=true, (c) Stripe-side state vs Queue-side state divergence is visible for operator reconciliation. Not executed this run — requires a live Stripe TEST MODE session (see Execution log).

Scenario 7 — Refund: full and partial

Purpose: Verify that refund events are recorded and reconcile correctly in billing tables, and that the Console manual refund flow initiates and reconciles Stripe-side refunds correctly.

Preconditions: Paid invoice from Scenario 1 or any completed payment.

Resolved (2026-08-19, was a documented gap as of 2026-06-17): charge.refunded handling and the billing_invoice.amount_refunded column both shipped via #3627 (sqitch migration 07-billing-invoice-refund-column.sql). This scenario's steps and pass criteria are rewritten below to reflect the real handler behavior; the prior "no mutation occurs" framing is stale.

Steps — full refund (Stripe-initiated):

1. In Stripe TEST MODE dashboard, navigate to the test customer → Payments → [successful charge] → Refund.
2. Enter the full charge amount and click Refund.
3. Confirm Stripe delivers charge.refunded.
4. Verify billing_invoice is updated:

SELECT stripe_invoice_id, amount_paid, amount_refunded, status
FROM billing_invoice
WHERE stripe_customer_id = '<test_cus_id>'
ORDER BY created_at DESC LIMIT 3;
-- Expected: amount_refunded incremented to match the refunded amount
-- (handler prefers data.refunds.data[0].amount for the incremental delta;
--  falls back to data.amount_refunded — the cumulative total — only when
--  the delta field is absent)

5. Verify processed_stripe_events contains the charge.refunded event ID (confirming dedup works, and re-delivery does not double-increment amount_refunded).
6. Edge case to test: a charge.refunded event arriving before the underlying invoice.payment_succeeded event (out-of-order delivery) — confirm the handler logs a warning and does not silently drop the refund (check for billing.webhook: charge.refunded — invoice not found; confirm the reconciliation path, e.g. a written audit-log placeholder row, per current handler code around the "invoice not found" branch).

Steps — Console-initiated refund (#410, shipped):

7. In Console, navigate to /console/billing/customers/<customer_id> and use the "Refund" manual-ops action (POST /billing/customers/<id>/ops/refund).
8. Confirm the refund is created against the correct Stripe charge and that billing_invoice.amount_refunded reflects it (same verification as step 4, sourced from the resulting charge.refunded webhook).
9. Confirm a Console-side audit trail entry is written for the operator-initiated refund action (check billing_action_log and/or Console's own audit surface).

Steps — partial refund:

10. Repeat steps 1–4 with a partial amount (e.g., 50% of the charge). Confirm amount_refunded reflects only the partial amount, status stays 'paid' (not refunded-in-full), and a second partial refund on the same charge correctly accumulates rather than overwrites.

Pass criteria: - [ ] charge.refunded event delivers HTTP 200 - [ ] billing_invoice.amount_refunded populated and correct (full and partial cases) - [ ] processed_stripe_events contains the refund event ID; re-delivery does not double-increment - [ ] Console manual-ops refund flow (/ops/refund) initiates a real Stripe refund and the resulting webhook reconciles correctly - [ ] Out-of-order charge.refunded (before invoice exists) does not silently drop the refund

Not executed this run — requires a live Stripe TEST MODE session and a Console-authenticated browser session; see Execution log.

Scenario 8 — Plan change and proration

Purpose: Verify upgrade and downgrade between tiers write the correct plan_tier and prior_tier values, and that proration invoices appear correctly.

Preconditions: Active Pro subscription.

8a — Upgrade (Pro → Pro+)

Steps:

1. In Stripe dashboard, update the test customer's subscription to the Pro+ price.
2. Confirm Stripe delivers customer.subscription.updated and a proration invoice.payment_succeeded.
3. Verify:

SELECT plan_tier, prior_tier, status, feature_locked_at
FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: plan_tier = 'pro_plus', prior_tier = NULL (upgrade, not downgrade),
--           feature_locked_at = NULL

4. Verify proration invoice in billing_invoice:

SELECT amount_due, amount_paid, status
FROM billing_invoice
WHERE stripe_customer_id = '<test_cus_id>'
ORDER BY created_at DESC LIMIT 3;
-- Expected: a new invoice row with amount_due reflecting the prorated upgrade charge,
--           status = 'paid'

8b — Downgrade (Pro+ → Pro)

Steps:

5. From an active Pro+ subscription, update to the Pro price.
6. Confirm Stripe delivers customer.subscription.updated.
7. Verify the downgrade detection in billing_subscription:

SELECT plan_tier, prior_tier, feature_locked_at, status
FROM billing_subscription
WHERE stripe_customer_id = '<test_cus_id>';
-- Expected: plan_tier = 'pro', prior_tier = 'pro_plus',
--           feature_locked_at IS NOT NULL (set on first downgrade),
--           status = 'active'

8. Re-deliver the same customer.subscription.updated event (duplicate delivery test — see Scenario 9). Verify feature_locked_at is NOT overwritten with a newer timestamp (the LWW guard and already_locked check should prevent this).

9. Verify billing_subscription_mirror.plan_tier = 'pro' after downgrade.

Note (tier-rank correction applies here too): A downgrade FROM Founders to Pro is a real feature-rank downgrade (founders rank 2 → pro rank 1) and should set feature_locked_at exactly like a Pro+ → Pro downgrade. A "downgrade" from Founders to Pro+ (or vice versa) is NOT a feature-rank change (founders == pro_plus rank 2) — confirm feature_locked_at is NOT set in that case, since _tier_rank(tier) >= _tier_rank(min_tier) for require_tier gates does not change. This is a genuinely new edge case introduced by the #3890 tier-rank decision; it was not covered in the 2026-06-17 version of this scenario.

Pass criteria: - [ ] Upgrade: plan_tier updated to higher tier, prior_tier = NULL, feature_locked_at = NULL - [ ] Upgrade proration invoice: new billing_invoice row with correct amounts, status = 'paid' - [ ] Downgrade: plan_tier updated to lower tier, prior_tier set to previous tier, feature_locked_at IS NOT NULL - [ ] Downgrade: re-delivery does NOT update feature_locked_at again - [ ] Mirror updated after both upgrade and downgrade - [ ] Founders ⇄ Pro+ lateral moves do NOT set feature_locked_at (rank-equal, not a downgrade)

Scenario 9 — Idempotency: duplicate Stripe webhook delivery

Purpose: Verify that re-delivering the same Stripe event does not produce duplicate rows or double mutations. This scenario is about Stripe-event-level dedup (processed_stripe_events, Queue-side). For Raptor-layer, application-level Idempotency-Key header replay on the billing mutation endpoints, see Scenario 9b below — these are two different mechanisms and both need coverage.

Preconditions: Any processed Stripe event from a prior scenario (note its event_id).

Steps:

1. Re-deliver the same event using the Stripe dashboard (Developers → Webhooks → [endpoint] → [delivery] → Resend).
2. Confirm Queue returns HTTP 200 with {"received":true}.
3. Verify the Queue log shows the idempotent-dedup log line:

billing.webhook: duplicate event_id=<id> → 200 (idempotent)

4. Verify processed_stripe_events contains exactly ONE row for the event ID (no duplicates):

SELECT COUNT(*) FROM processed_stripe_events WHERE event_id = '<event_id>';
-- Expected: 1

5. Verify billing_subscription and billing_invoice were NOT mutated by the duplicate delivery (row updated_at unchanged from after the first delivery).

Concurrent duplicate test (optional, advanced):

6. Use two simultaneous curl requests to post the same signed Stripe event payload to the webhook endpoint at exactly the same time. Confirm that the pqxx::unique_violation catch path fires for the second request and it also returns HTTP 200 (not 500 or 409). Check Queue logs for the concurrent duplicate event_id=<id> — 200 (idempotent) log line.

Pass criteria: - [ ] Re-delivered event returns HTTP 200 ({"received":true}) - [ ] Log shows idempotent dedup path, not the processing path - [ ] processed_stripe_events has exactly 1 row for the event ID - [ ] No mutations to billing tables from the duplicate delivery - [ ] Concurrent duplicate: second request returns 200 (not 500)

Partial verification 2026-08-19: the signature-verification front door of this endpoint was probed directly against raxx-queue-staging (no valid signed payload available — see Execution log) and correctly rejected a missing-signature request (400 missing_signature) and a garbage-signature request (400 stale_timestamp, timestamp-tolerance check fired before signature comparison). This confirms the endpoint is live and enforcing HMAC verification, but does not exercise the dedup path itself, which requires a validly-signed event.

Scenario 9b — Idempotency-Key header replay on billing mutations (Raptor layer, #4149 Wave 3, PR #4499)

NEW scenario, added 2026-08-19 per #4502 scope.

Status: NOT YET VALIDATABLE ON STAGING — see finding below. Do not mark this PASS until re-run after the code is actually deployed to staging.

Purpose: Verify the Raptor-layer, application-level Idempotency-Key request header (distinct from Stripe's own event-level idempotency covered in Scenario 9) correctly dedupes on the five Wave 3 billing/subscription mutation endpoints: POST /api/billing/checkout-session, POST /api/billing/refund, POST /api/subscription/upgrade, POST /api/subscription/downgrade, POST /api/subscription/cancel.

Finding (2026-08-19): Wave 3 is merged to develop (e87574011, PR #4499, closes #4149) but has NOT been promoted to release, and staging (raxx-api-staging) deploys only from release-tagged builds (.woodpecker/deploy-staging.yaml, event: tag on refs/tags/release-*, or a manual re-deploy of current workspace HEAD). Verified directly:

git merge-base --is-ancestor e87574011 origin/release   → NOT an ancestor (not on release)

The most recent release promotion at the time of this check was release-2026.08.09 (91c55b8ac chore(release): promote release-2026.08.09 develop->release), which predates the Wave 3 merge. Unless an operator triggered a manual Woodpecker re-deploy of develop HEAD directly to staging (bypassing the normal tag-gated flow) between the Wave 3 merge and this check, staging is running pre-Wave-3 code for these five routes. This sandbox has no access to ci.moosequest.net (401/302, no credentials) or heroku releases --app raxx-api-staging to confirm or rule out a manual deploy.

Consequence for this card's stated premise: #4502's Background states "Idempotency Wave 3 on billing mutations (#4149) just merged today (PR #4499)... a fresh run is the right validation point" — this is only true once Wave 3 reaches staging. Recommend: confirm via heroku releases --app raxx-api-staging | head -5 (or equivalent) which commit is actually live before attempting this scenario; if pre-Wave-3, either wait for the next release-* promotion + tag-gated deploy, or have an operator trigger the Woodpecker manual-deploy path intentionally for this validation.

Steps (to run once deployment is confirmed):

1. Generate a UUID for Idempotency-Key.
2. POST /api/billing/checkout-session with a valid session cookie/token and Idempotency-Key: <uuid> header. Record the response body ({"url": "..."}).
3. Repeat the exact same request (same body, same Idempotency-Key). Confirm the second response includes "idempotency_replayed": true and the exact same url as the first call — and confirm in the Stripe TEST MODE dashboard that only ONE Checkout Session was created (no second session/customer-facing URL).
4. Repeat steps 1–3 for POST /api/billing/refund against a real refundable charge; confirm "idempotency_replayed": true on replay and that only ONE refund appears in Stripe.
5. Confirm the stored response_body for these two routes' idempotency_keys rows (Raptor DB) is NOT plaintext — per PR #4499, it should be an envelope: {"_idem_encrypted": true, "kms_key_id", "iv", "wrapped_dek", "ciphertext"}. Operational prerequisite: PR #4499 explicitly flags that IDEMPOTENCY_RESPONSE_KMS_KEY_ARN was not provisioned in prod at merge time; confirm it is set on raxx-api-staging before running this step — if unset, the completion write fails encryption, the row stays in_flight, and it stale-heals after 30s without ever becoming replayable (the client's actual HTTP response is unaffected either way, but the replay behavior in step 3/4 above will not be observable).
6. Repeat for subscription/upgrade, subscription/downgrade, subscription/cancel with equivalent idempotency-replay checks.
7. Fail-open check: with the idempotency store artificially unavailable (or FLAG_IDEMPOTENCY_MIDDLEWARE=false), confirm all five routes still process the request normally (fail-open per ADR-0138 Tier 2 — these are NOT fail-closed like the order endpoints).

Pass criteria: - [ ] FLAG_IDEMPOTENCY_MIDDLEWARE confirmed ON on staging before running (shared master flag; default OFF per feature_flags.yaml) - [ ] IDEMPOTENCY_RESPONSE_KMS_KEY_ARN confirmed set on raxx-api-staging - [ ] Duplicate checkout-session call with same key replays without creating a second Stripe Checkout Session - [ ] Duplicate refund call with same key replays without issuing a second Stripe refund - [ ] Stored response_body for these routes is the encrypted envelope, not plaintext - [ ] subscription/upgrade, /downgrade, /cancel all replay correctly on duplicate key - [ ] Fail-open confirmed: store unavailable does not block the mutation

Not executed this run. Blocked on (a) confirming/achieving staging deployment of Wave 3 and (b) an authenticated staging session or Stripe TEST MODE credentials — see Execution log below for the precise access gaps.

Scenario 10 — Apple IAP / StoreKit 2 (iOS path)

Status: BLOCKED — see below

Purpose: Verify the iOS billing path — sandbox tester purchases a subscription, the server validates the transaction, entitlement is granted, sandbox renewal and cancellation are reflected.

Current state: This scenario is fully blocked. As of 2026-08-19, the following components required for this scenario still do not exist in the codebase (re-confirmed against current origin/develop; unchanged from 2026-06-17):

Design references: ADR-0007 (docs/architecture/adr/0007-ios-subscription-billing-iap.md) accepted this approach. Downstream implementation cards were filed under Epic #167. None have shipped.

Reconciliation posture (when built): Apple is the source of truth for iOS subscriptions. A user who has an active iOS subscription via Apple IAP and also attempts to subscribe via web Stripe should be refused the second subscription (same-user dual-subscription path). This guard also does not exist yet.

Actions before this scenario can run: 1. Implement the Apple S2S notification handler (ADR-0007 downstream card). 2. Add subscription_source column to billing_subscription (or a separate IAP table). 3. Set up App Store Connect sandbox environment (requires Apple Developer org account — confirm current status with Kristerpher before starting). 4. Create a sandbox tester account in App Store Connect. 5. Register at least one auto-renewable subscription product in App Store Connect for the Pro tier.

When unblocked, the steps will be:

1. On a development or TestFlight iOS build, sign in as the sandbox tester.
2. Purchase a Pro subscription via the in-app subscription UI (StoreKit 2 flow).
3. Confirm Apple calls /api/subscriptions/apple/notifications with a DID_CHANGE_RENEWAL_STATUS or SUBSCRIBED notification.
4. Verify entitlement is granted: user's Raptor-side session claims or mirror table shows Pro access.
5. Advance sandbox time (Apple sandbox subscriptions have accelerated renewal periods).
6. Confirm renewal notification arrives and entitlement is extended.
7. Cancel the sandbox subscription; confirm DID_CHANGE_RENEWAL_STATUS fires and access is revoked.
8. Cross-billing check: same user with active iOS sub should receive a 409 Conflict when attempting to subscribe via web Stripe checkout.

Pass criteria (for when built): - [ ] S2S notification validated (JWS signature verified) - [ ] original_transaction_id stored (not transaction_id) - [ ] subscription_source = 'ios' on the subscription row - [ ] Entitlement granted after SUBSCRIBED notification - [ ] Renewal extends entitlement period - [ ] Cancellation revokes access - [ ] Dual-subscription attempt returns 409 (not a silent double-billing)

Production smoke (minimal, disposable test customer)

Run this section only after every testable scenario above has PASSED in Stripe TEST MODE.

Purpose: Confirm the live webhook endpoint receives and processes a real Stripe event in production. This uses a real card charge that is refunded immediately — a controlled, minimal, real-money transaction.

Prerequisites: - All TEST MODE scenarios above: PASS - Stripe live-mode keys in vault prod path (see billing-test-tooling.md § "Live mode vault paths") - FLAG_QUEUE_BILLING=true on raxx-queue-prod - Stripe dashboard switched to LIVE MODE - Disposable test customer account available (see billing-test-tooling.md § "Prod smoke customer")

Steps:

1. Switch Stripe dashboard to LIVE MODE.
2. Create a new customer in live mode with a dedicated test email (e.g., billing-smoke-YYYY-MM-DD@raxx.app). Do not use a real customer email.
3. Subscribe the test customer to the Pro plan using the operator's own card (or a pre-authorized test payment method).
4. Confirm live-mode webhook delivery at https://queue.raxx.app/api/v1/billing/webhook: - customer.subscription.created — HTTP 200 - invoice.payment_succeeded — HTTP 200
5. Verify Queue prod tables:

-- On Queue prod DB (read-only credentials from billing-test-tooling.md):
SELECT billing_email, stripe_customer_id FROM billing_customer
WHERE billing_email = 'billing-smoke-YYYY-MM-DD@raxx.app';
-- Expected: 1 row

SELECT status, plan_tier FROM billing_subscription
WHERE stripe_customer_id = '<prod_smoke_cus_id>';
-- Expected: status = 'active', plan_tier = 'pro'

6. Verify billing_subscription_mirror on Raptor prod DB:

SELECT status, plan_tier FROM billing_subscription_mirror
WHERE queue_customer_id = '<prod_smoke_queue_cus_id>';
-- Expected: status = 'active', plan_tier = 'pro'

7. Immediately cancel the smoke subscription in live Stripe dashboard and issue a full refund of the charge. Do not let it bill into a second period.
8. Confirm cancellation and refund land in prod billing tables.
9. Optionally: delete the smoke customer record from Stripe (not required, but keeps live-mode data clean).

Pass criteria: - [ ] Live webhook delivers HTTP 200 for subscription creation and invoice payment - [ ] billing_customer and billing_subscription rows exist in prod DB - [ ] billing_subscription_mirror row exists in Raptor prod DB - [ ] Subscription canceled and refund issued immediately after validation - [ ] No Sentry errors during the smoke window

If the prod smoke fails: Do NOT proceed to enabling real customer billing. Roll back FLAG_QUEUE_BILLING=false on raxx-queue-prod, investigate, and repeat the test-mode scenarios to isolate the regression before re-running the smoke.

Pre-charge acceptance gate

Every line below must be checked before real customer billing is enabled or a real charge is processed. This checklist is the gate.

Test-mode scenarios (Stripe TEST MODE)

[ ] Scenario 1 (happy path): NOT EXECUTED this run — infra reachable, Stripe TEST MODE session blocked (see Execution log)
[ ] Scenario 2 (tier coverage): NOT EXECUTED this run — same blocker
[ ] Scenario 3 (renewal): NOT EXECUTED this run — same blocker
[ ] Scenario 4 (payment failure + dunning): NOT EXECUTED this run — same blocker
[ ] Scenario 5 (3DS / SCA): NOT EXECUTED this run — same blocker
[ ] Scenario 6a/6b (cancellation, incl. new Founders early-cancel sub-case): NOT EXECUTED this run — same blocker
[ ] Scenario 7 (refund, incl. Console manual-ops path): NOT EXECUTED this run — same blocker
[ ] Scenario 8 (plan change / proration, incl. new Founders⇄Pro+ lateral-move check): NOT EXECUTED this run — same blocker
[ ] Scenario 9 (Stripe-event idempotency): PARTIAL — signature-verification front door confirmed live and enforcing;
       dedup path itself not exercised (needs a validly-signed event)
[ ] Scenario 9b (Raptor Idempotency-Key replay, NEW #4149 Wave 3): BLOCKED — code not confirmed deployed to staging
       (merged to develop, not on release; see finding in Scenario 9b and Execution log)
[ ] Scenario 10 (Apple IAP): BLOCKED — not built; gate HOLDS OPEN until built and passing
       (Apple IAP is NOT required before web/Stripe billing can go live; they are independent paths;
       the gate for web billing can close without Scenario 10 if confirmed by operator)

This gate is NOT closed by this run. See "Execution log — 2026-08-19" below for exactly what was and was not reachable, and what is needed to complete the run.

Console billing surfaces

[x] #409 (Console customer detail): SHIPPED in code ([PR #3860](https://github.com/raxx-app/TradeMasterAPI/pull/3860)) — UI walk not executed this run (no CF-Access-authenticated
       browser session available in this sandbox)
[x] #1633 (Console MRR dashboard): SHIPPED in code ([PR #3277](https://github.com/raxx-app/TradeMasterAPI/pull/3277), #4386, #4388) — UI walk not executed this run
[x] #410 (Console manual ops / refund): SHIPPED in code ([PR #3860](https://github.com/raxx-app/TradeMasterAPI/pull/3860) Console + #4341 Queue) — UI walk not executed this run
[ ] (A follow-up Console UI walk via Playwright MCP, per feedback_playwright_for_live_verification, is recommended
       once a CF-Access-authenticated staging session is available to this agent or a human operator runs it)

Infrastructure and security

[x] FLAG_QUEUE_BILLING=true on raxx-queue-staging — confirmed by live behavior 2026-08-19 (webhook endpoint enforces
       HMAC verification rather than returning 503); raxx-queue-prod value NOT checked this run (out of scope —
       staging-only per this card)
[ ] STRIPE_WEBHOOK_SECRET on raxx-queue-prod is the current live-mode whsec_* value (not test mode, not stale) — NOT CHECKED (prod, out of scope + no Heroku CLI access)
[ ] STRIPE_RESTRICTED_KEY (or equivalent) on raxx-queue-prod uses rk_live_* restricted key with correct scopes — NOT CHECKED
[ ] Stripe test keys are NOT present on raxx-queue-prod (only live keys on prod) — NOT CHECKED
[ ] billing_subscription_mirror fan-out is confirmed working (RAPTOR_BASE_URL + QUEUE_TO_RAPTOR_INTERNAL_TOKEN set on prod) — NOT CHECKED (prod)
[ ] No Sentry billing.webhook.hmac_failure events in the last 24h on prod — NOT CHECKED (no Sentry access this run)
[ ] No billing.mirror.fan_out_failure log lines on prod in the last 24h — NOT CHECKED (no log access this run)
[ ] PA SaaS tax posture confirmed with CPA (#2743) OR explicitly deferred with documented operator decision — unchanged, operator item

Production smoke

[ ] Prod smoke: NOT RUN — test-mode scenarios must pass first, and none passed (blocked, not failed) this run

Operator sign-off

[ ] Operator (Kristerpher) has reviewed this checklist and confirmed all checked items
[ ] Operator-deferred items documented with explicit deferral date and follow-up issue number
[ ] This checklist signed off by: _______________ Date (UTC): _______________

Execution log — 2026-08-19 UTC (#4502)

Run by: qa-agent (raxx-ops-bot), sandboxed session. Timestamps UTC.

What was reachable and executed:

What was blocked, and precisely why:

1. Stripe TEST MODE credentials unreachable. docs/ops/runbooks/billing-test-tooling.md documents STRIPE_RESTRICTED_KEY at Infisical path /Raxx/Queue/Billing/Stripe/STRIPE_RESTRICTED_KEY (env: staging). This session has infisical CLI installed and INFISICAL_CLIENT_ID/INFISICAL_CLIENT_SECRET/INFISICAL_PROJECT_ID env vars present, but the read attempt (infisical secrets get STRIPE_RESTRICTED_KEY --path "/Raxx/Queue/Billing/Stripe" --env staging --projectId "$INFISICAL_PROJECT_ID" --plain) was blocked by the session's permission classifier ("Permission for this action was denied... Blocked by classifier"). This is consistent with feedback_main_loop_vault_limit — direct vault/secret reads in an agent sandbox route through sre-agent, not this agent. No workaround was attempted (per instruction not to bypass sandbox denials). Without the restricted key, no Stripe Checkout Session, Test Clock, refund, or price-tier test can be created.
2. No authenticated staging user/session available. The billing/subscription mutation endpoints require a valid Raptor session (@require_session, confirmed by the 401 Authentication required responses above). No synthetic-credential harness for a billing test session was available in this sandbox's scripts/qa/ or scripts/smoke/ directories (those cover email verification, wash-sale/S1256/holding-period smokes — none produce a billing-scope session token). Per feedback_smoke_before_mobile_retest, a synthetic-credential harness is the intended pattern for this kind of check; none exists for billing yet.
3. No browser/Playwright tool available to this agent invocation. console-staging.raxx.app is CF-Access-gated (confirmed 302 above); walking the Console customer-detail/MRR/manual-ops UI per feedback_playwright_for_live_verification requires a Playwright MCP session with CF Access credentials, neither of which this invocation had.
4. No Heroku CLI access. heroku auth:whoami returns "not logged in." Cannot run heroku config:get, heroku pg:psql, or heroku releases against raxx-queue-staging or raxx-api-staging. This means: (a) DB-level verification of every SQL check in Scenarios 1–9 was not possible; (b) the exact live values of FLAG_BILLING_EMAIL_DISPATCH, FLAG_IDEMPOTENCY_MIDDLEWARE, IDEMPOTENCY_RESPONSE_KMS_KEY_ARN on staging could not be confirmed directly (only inferred where behavior allowed, e.g. FLAG_QUEUE_BILLING); (c) the exact deployed commit SHA on raxx-api-staging/raxx-queue-staging could not be confirmed via heroku releases, per feedback_wp_deploy_green_not_release_success ("WP deploy green ≠ Heroku release landed").
5. No Woodpecker CI access. ci.moosequest.net requires auth (401/302 without credentials). Could not confirm whether a manual re-deploy of develop HEAD to staging happened outside the normal tag-gated flow, which is the one scenario that would put Wave 3 (#4499) on staging despite it not yet being on release.

Headline finding (see also Scenario 9b): [PR #4499](https://github.com/raxx-app/TradeMasterAPI/pull/4499) (idempotency Wave 3, closes #4149) is merged to develop (e87574011) but not promoted to release (confirmed via git merge-base --is-ancestor e87574011 origin/release → false; latest release promotion release-2026.08.09 predates it). Staging deploys are tag-gated off release-* tags. This card's Background section states Wave 3 "just merged today... a fresh run is the right validation point," which is accurate for develop but not necessarily for what is actually running on staging right now. Recommend an operator or sre-agent confirm the live raxx-api-staging commit via heroku releases before the idempotency-replay scenario is attempted; if pre-Wave-3, this scenario cannot pass or fail — it is simply not present in the running code.

Recommended path to close this gate: dispatch sre-agent for (a) Stripe TEST MODE credential retrieval + a scripted run of Scenarios 1–9 via the Stripe API directly (matches the tooling doc's "Test customer via Stripe API" copy-paste pattern), and (b) a Playwright MCP session with CF Access credentials for the Console UI walk (Scenario 1 steps 7–8, Scenario 7 step 6–9). Confirm Wave 3's staging deployment status first via heroku releases --app raxx-api-staging.

Known gaps to file as follow-up cards

These are gaps found during authoring of this SOP (2026-06-17) or this refresh (2026-08-19). Each should become a filed issue before or alongside the acceptance gate closing. Resolved items are struck through with their closing reference; do not re-file them.

Rollback procedure

If any scenario fails during the acceptance gate run and the operator has already enabled FLAG_QUEUE_BILLING=true on prod:

heroku config:set FLAG_QUEUE_BILLING=false --app raxx-queue-prod >/dev/null 2>&1
# Confirm:
heroku config:get FLAG_QUEUE_BILLING --app raxx-queue-prod
# Expected: false

This disables the webhook endpoint (returns 503 to Stripe; Stripe will retry). It does not affect existing subscription rows already written to Queue DB. Re-enable only after the regression is identified and fixed in staging.

References

• docs/ops/runbooks/billing-test-tooling.md — test cards, test clock commands, vault paths, DB verification commands (note: this doc's own "Last reviewed: 2026-06-17" header and its "Current gap (as of 2026-06-17)" table were NOT refreshed as part of #4502 — out of this card's stated scope, but worth a follow-up sweep given several of its listed gaps, e.g. FLAG_QUEUE_BILLING=true on staging, appear to already be resolved based on this run's live probes)
• docs/ops/runbooks/stripe-founders-setup.md — Stripe product/price/webhook provisioning SOP
• docs/architecture/stripe-customer-billing.md — billing data model (v5)
• docs/architecture/queue-stripe-webhook-design-2026-05-14.md — webhook handler design
• docs/architecture/adr/0007-ios-subscription-billing-iap.md — Apple IAP decision
• docs/architecture/adr/0071-stripe-billing-queue-as-authority.md — Queue as billing authority
• docs/architecture/adr/0127-webhook-idempotency-5xx-not-local-queue.md — idempotency design (Stripe-event layer)
• docs/architecture/adr/0138-idempotency-key-mechanism.md — Idempotency-Key header mechanism (Raptor layer, #4149)
• docs/ops/runbooks/queue.md — Queue service runbook (health checks, failure modes)
• docs/ops/2026-05-13-stripe-test-mode-verification.md — historical gap analysis (most gaps now closed)
• docs/architecture/issue-409-customer-detail-view.md — Console customer detail design
• project_pricing_tiers_locked.md — Free / Pro / Pro+ / Founders $29/6mo pricing
• project_ios_billing_iap.md — iOS billing is Apple IAP, not Stripe
• project_branching_model_develop_release_main.md — develop→release→main promotion flow (why staging can lag develop)
• Epic #403 — Console billing epic
• Epic #167 — iOS billing epic

• 4502 — this refresh + partial gate execution

• 4149 / PR #4499 — Idempotency Wave 3 (billing mutations)