Apple IAP sandbox E2E verification

Card: #3688
Epic: #167 — Raxx iOS companion app
Owner: operator (Kristerpher) + ios-agent
System: Raptor staging (api-staging.raxx.app) + App Store Connect sandbox
Prerequisites: Sub-cards #3685 (JWS verify), #3686 (persist), #3687 (entitlement) merged; FLAG_APPLE_IAP flipped ON in staging
Last updated: 2026-06-24

Pre-req checklist — confirm before doing anything else

[ ] [PR #3685](https://github.com/raxx-app/TradeMasterAPI/pull/3685) merged to main (JWS verify)
[ ] [PR #3686](https://github.com/raxx-app/TradeMasterAPI/pull/3686) merged to main (persistence — apple_iap_subscriptions table exists in staging DB)
[ ] [PR #3687](https://github.com/raxx-app/TradeMasterAPI/pull/3687) merged to main (entitlement + reconciliation)
[ ] Alembic migration 0042 applied to staging:
      heroku run flask db upgrade --app raxx-api-staging >/dev/null 2>&1
    Verify: heroku pg:psql --app raxx-api-staging -c "\d apple_iap_subscriptions"
[ ] FLAG_APPLE_IAP turned ON in staging:
      heroku config:set FLAG_APPLE_IAP=1 --app raxx-api-staging >/dev/null 2>&1
    Verify: curl -sf https://api-staging.raxx.app/api/subscriptions/apple/notifications \
              -H "Content-Type: application/json" -d '{"signedPayload":"bad"}' \
            | grep -v '"error":"not_found"'
    (A 400 "bad_request" or 401 is the expected response — not 404 — confirming the flag is ON.)
[ ] Synthetic mock tests in #3685 / #3686 / #3687 are green in CI
    (per feedback_smoke_before_mobile_retest — never burn device cycles for bugs a unit test would catch)
[ ] CF Access WAF skip rule in place for the notification endpoint
    (see Section 4 below — Apple's servers carry no CF-Access token)
[ ] Sandbox tester accounts created in ASC and email-verified (see Section 2)
[ ] ASC sandbox notification URL set (see Section 3)

1. IAP product record spec

Source of truth

Product IDs are defined in ios/Raxx/Resources/RaxxProducts.storekit and must match ios/Raxx/Features/Billing/StoreKitManager.swift ProductID constants exactly.

Products that exist in ASC (provisioned 2026-06-17)

All four records were created via the ASC REST API. ASC IDs are stored in vault at /Raxx/iOS/Sandbox/. ASC App ID: 6778650155.

Current state: All four products are in MISSING_METADATA state. This is expected — Apple requires a review screenshot before a product can move to READY_TO_SUBMIT. Products in MISSING_METADATA are fully usable in the sandbox even before they are submitted for review. The sandbox E2E in this runbook can proceed without resolving MISSING_METADATA.

Price decision required (operator/BLR): The three recurring-subscription products have no price set. Do not invent a price. Once BLR delivers the Pro / Pro+ price recommendation, set prices in ASC: Apps → Raxx → Subscriptions → Raxx Pro Subscriptions → [product] → Pricing → Set Base Price.

Screenshot upload required (to reach READY_TO_SUBMIT): Build the billing screen on device or Simulator, screenshot it, and upload per product at: ASC → Apps → Raxx → In-App Purchases → [product] → Review Information → Screenshot. This step is not required for the sandbox E2E in this runbook.

Verifying the app resolves products

With FLAG_APPLE_IAP ON and the StoreKit configuration file DISABLED in the scheme (see §7 of ios-iap-sandbox-testing.md), open Raxx on a sandbox-tester device and navigate to the billing screen. If BillingView shows product rows, the ASC records resolved. If it shows "Subscription products are not yet available," verify the product IDs in StoreKitManager.swift match the table above character-for-character.

2. Sandbox tester setup (operator dashboard step — no API path)

The ASC REST API explicitly disallows CREATE for sandbox testers. This is a dashboard-only step requiring the operator's Apple ID and 2FA.

Credentials for all three testers are pre-staged in Infisical at /Raxx/iOS/Sandbox/.

Steps

1. Go to appstoreconnect.apple.com — sign in with kris@moosequest.net + 2FA.
2. Navigate to Users and Access → Sandbox → Testers.
3. Click + (Add Sandbox Tester).
4. Fill in the form using vault credentials at /Raxx/iOS/Sandbox/:

5. Click Save. Apple sends a verification email to the tester address.
6. Check ops@raxx.app (Google Workspace) for the verification email. Complete it. The tester account is not usable until the email link is clicked.
7. Repeat for the iap-test-pro-monthly@raxx.app and iap-test-founders@raxx.app accounts.

Minimum tester accounts for this E2E

Reset purchase history (between test runs)

ASC → Users and Access → Sandbox → Testers → [tester] → Manage → Clear Purchase History. This resets sandbox renewal eligibility and lets the same account start fresh.

3. App Store Server Notifications V2 — staging endpoint config

Endpoint

The Raptor route that receives Apple's callbacks (implemented in #3685 / #3686 / #3687):

POST https://api-staging.raxx.app/api/subscriptions/apple/notifications

Apple posts a JSON body { "signedPayload": "<compact JWS string>" }. The endpoint: - Verifies the JWS signature back to Apple Root CA G3 - Decodes originalTransactionId, notificationType, productId, expiresDate - Upserts apple_iap_subscriptions (one row per originalTransactionId) - Appends apple_iap_notification_log (idempotency gate — deduped on SHA-256 of raw payload) - Returns 200 { "status": "ok", ... } — Apple requires 200; it retries on non-200

How to set the sandbox notification URL in ASC

1. appstoreconnect.apple.com → My Apps → Raxx
2. Left sidebar → App Information
3. Scroll to App Store Server Notifications
4. Sandbox server URL field — enter exactly: https://api-staging.raxx.app/api/subscriptions/apple/notifications
5. Leave Production server URL blank until the production E2E is authorized.
6. Click Save.

ASC stores one URL per environment. To disable delivery, clear the field and save.

WAF skip rule requirement (CRITICAL — without this Apple's POSTs will be blocked)

api-staging.raxx.app is behind Cloudflare. CF Access requires a service token on every request. Apple's notification servers carry no such token — their POSTs will receive 403 and Apple will retry indefinitely without any notification landing.

Per feedback_cf_access_does_not_bypass_bot_fight_mode: CF Access runs after the WAF/bot layer. A WAF skip rule keyed on the request path is the correct fix (not IP allowlisting — Apple does not publish stable CIDR ranges for notification delivery).

SRE step required (dispatch sre-agent or do this in the Cloudflare dashboard):

1. Cloudflare dashboard → raxx.app zone → Security → WAF → Custom Rules
2. Add a rule before the CF Access rule: - Name: Apple IAP notifications staging — skip CF Access - Expression: (http.host eq "api-staging.raxx.app" and http.request.uri.path eq "/api/subscriptions/apple/notifications" and http.request.method eq "POST") - Action: Skip → check CF Access (and Bot Fight Mode if BFM is on for staging)
3. Place this rule above any CF Access enforcement rule.
4. Save and deploy.

Verify the rule is working by clicking Request a Test Notification in ASC (§4 Step 1). If the endpoint logs the delivery, the rule is active.

4. Live-fire test procedure

Work through these steps in order. Each step has a verification checkpoint.

Step 0 — Environment health check

# Confirm the endpoint is reachable and the flag is ON
# Expect 400 or 401, not 404
curl -sf -X POST https://api-staging.raxx.app/api/subscriptions/apple/notifications \
  -H "Content-Type: application/json" \
  -d '{"signedPayload":"not.a.real.jws"}' | python3 -m json.tool

# Expected: {"error": "bad_request", "message": "signedPayload is malformed"} HTTP 400
# NOT: {"error": "not_found"} HTTP 404 (means FLAG_APPLE_IAP is still OFF)

Step 1 — On-demand test notification (first live-fire check)

ASC's "Request a Test Notification" fires a real signed sandbox TEST notification to the registered URL without needing a device purchase. Use this to verify the pipeline before touching a phone.

1. appstoreconnect.apple.com → My Apps → Raxx → App Information → App Store Server Notifications
2. Confirm the Sandbox URL is set.
3. Click Request a Test Notification.
4. Apple delivers the notification within 30 seconds to 5 minutes.

Verify:

# Watch logs in real time (Ctrl-C to stop)
heroku logs --app raxx-api-staging --tail | grep -i "apple\|iap\|notification"

# After delivery, check the notification log table
heroku pg:psql --app raxx-api-staging -c \
  "SELECT notification_type, subtype, received_at
   FROM apple_iap_notification_log
   ORDER BY received_at DESC LIMIT 5;"

A TEST notification type row confirms: - Apple's servers reached the endpoint (WAF skip rule is working) - JWS verification passed (otherwise 401 is returned and nothing is written) - Persistence is working

If no row appears after 5 minutes: check Heroku logs for any HTTP 4xx / 5xx, confirm the WAF skip rule is deployed (§3), and confirm FLAG_APPLE_IAP=1.

Step 2 — Sandbox purchase on device (SUBSCRIBED notification)

1. Install Raxx on device via Xcode USB (Cmd+R, target = your device). - Disable the StoreKit config first: Edit Scheme → Run → Options → StoreKit Configuration → None - With None, the app talks to the real Apple sandbox instead of the local file.
2. Sign in to Raxx with the operator passkey.
3. Navigate to the billing screen. Product rows should appear.
4. Tap Subscribe on Raxx Pro Monthly.
5. When Apple's payment sheet appears, enter sandbox tester credentials: - Email: iap-test-fresh@raxx.app (vault key: IAP_TESTER_FRESH_EMAIL) - Password: from vault IAP_TESTER_FRESH_PASSWORD
6. The sheet shows [Environment: Sandbox]. Complete the purchase. No real charge occurs.

Verify:

# Check for SUBSCRIBED notification
heroku pg:psql --app raxx-api-staging -c \
  "SELECT notification_type, subtype, received_at
   FROM apple_iap_notification_log
   WHERE notification_type = 'SUBSCRIBED'
   ORDER BY received_at DESC LIMIT 3;"

# Check the subscription row
heroku pg:psql --app raxx-api-staging -c \
  "SELECT original_transaction_id, product_id, subscription_status,
          app_account_token, environment, created_at
   FROM apple_iap_subscriptions
   ORDER BY created_at DESC LIMIT 3;"

Expected: - apple_iap_notification_log: one row, notification_type = 'SUBSCRIBED', environment = 'Sandbox' - apple_iap_subscriptions: one row, subscription_status = 'active', product_id = 'app.raxx.ios.pro.monthly' - app_account_token is non-null (UUID stamped by StoreKit at purchase time)

Step 3 — Entitlement check

# Requires a valid Raxx session cookie — obtain by signing in via the app or curl the auth flow
curl -sf "https://api-staging.raxx.app/api/subscriptions/apple/entitlement-check" \
  -H "Cookie: session=<your-session-cookie>" | python3 -m json.tool

# Expected (active iOS subscription):
# {
#   "has_active_web_subscription": false,
#   "current_source": "apple",
#   "current_tier": "pro"
# }

Step 4 — Renewal notification

Sandbox subscriptions renew approximately every 5 minutes (1-month period compressed). Wait ~5 minutes after the initial purchase.

heroku pg:psql --app raxx-api-staging -c \
  "SELECT notification_type, subtype, received_at
   FROM apple_iap_notification_log
   ORDER BY received_at DESC LIMIT 5;"

Expected: a DID_RENEW row appears. The apple_iap_subscriptions row subscription_status remains 'active' and expires_date_ms is updated to the new period end.

To accelerate without waiting: ASC → Users and Access → Sandbox → Testers → [tester] → Manage allows manual billing event triggers.

Step 5 — Cancellation and EXPIRED notification

Cancel the sandbox subscription on device: - Settings → [sandbox tester name] → Subscriptions → Raxx Pro → Cancel

The subscription stays active until the current period end (~5 minutes in sandbox). After it lapses, Apple fires an EXPIRED notification.

To accelerate expiry without waiting: in Xcode with device connected, Debug → StoreKit → Manage Transactions → [transaction] → Expire (Xcode 15+).

heroku pg:psql --app raxx-api-staging -c \
  "SELECT notification_type FROM apple_iap_notification_log
   WHERE notification_type IN ('EXPIRED', 'DID_FAIL_TO_RENEW')
   ORDER BY received_at DESC LIMIT 3;"

heroku pg:psql --app raxx-api-staging -c \
  "SELECT subscription_status FROM apple_iap_subscriptions
   WHERE product_id = 'app.raxx.ios.pro.monthly'
   ORDER BY updated_at DESC LIMIT 1;"

Expected: EXPIRED row in log; subscription_status = 'expired'.

Step 6 — Refund notification

In Xcode Transaction Manager (Debug → StoreKit → Manage Transactions with device connected): right-click the transaction → Request Refund. Apple fires a REFUND notification.

heroku pg:psql --app raxx-api-staging -c \
  "SELECT notification_type FROM apple_iap_notification_log
   WHERE notification_type = 'REFUND'
   ORDER BY received_at DESC LIMIT 1;"

heroku pg:psql --app raxx-api-staging -c \
  "SELECT subscription_status FROM apple_iap_subscriptions
   ORDER BY updated_at DESC LIMIT 1;"

Expected: REFUND row in log; subscription_status = 'refunded'.

Step 7 — Forged payload rejected (security gate)

# Manually-crafted payload not signed by Apple — must return 401
curl -s -w "\nHTTP %{http_code}" -X POST \
  https://api-staging.raxx.app/api/subscriptions/apple/notifications \
  -H "Content-Type: application/json" \
  -d '{"signedPayload":"eyJhbGciOiJFUzI1NiJ9.eyJub3QiOiJyZWFsIn0.invalidsig"}'
# Expected: HTTP 401
# {"error": "unauthorized", "message": "Notification signature verification failed"}

# Missing field — must return 400
curl -s -w "\nHTTP %{http_code}" -X POST \
  https://api-staging.raxx.app/api/subscriptions/apple/notifications \
  -H "Content-Type: application/json" \
  -d '{}'
# Expected: HTTP 400
# {"error": "bad_request", "message": "Request body must include 'signedPayload' (string)"}

Step 8 — Reconciliation: no double-count against a web subscription

With the subscription row in active state (from Step 2), confirm the web checkout is blocked:

curl -sf -X POST \
  https://api-staging.raxx.app/api/subscriptions/apple/checkout-block-check \
  -H "Content-Type: application/json" \
  -H "Cookie: session=<your-session-cookie>"
# Expected: {"blocked": true, "reason": "active_ios_subscription", "message": "..."}

After expiry (Step 5), repeat and confirm "blocked": false.

5. Idempotency verification (replay attack resistance)

Click "Request a Test Notification" in ASC twice within a few seconds of each other. Apple will send two deliveries with the same notification payload.

heroku pg:psql --app raxx-api-staging -c \
  "SELECT raw_payload_hash, received_at FROM apple_iap_notification_log
   WHERE notification_type = 'TEST'
   ORDER BY received_at DESC LIMIT 5;"

Expected: only one row per unique test notification (the UNIQUE constraint on raw_payload_hash prevents a second insert; the dispatcher returns ALREADY_PROCESSED and logs at DEBUG without mutating apple_iap_subscriptions).

6. E2E acceptance checklist

Check every box before commenting on #3688 that the E2E is complete:

[ ] Staging endpoint responds 400/401 (not 404) — FLAG_APPLE_IAP confirmed ON
[ ] Alembic migration 0042 applied — apple_iap_subscriptions table exists
[ ] WAF skip rule deployed — Apple notifications reach the endpoint (not 403)
[ ] ASC sandbox URL set to https://api-staging.raxx.app/api/subscriptions/apple/notifications
[ ] All 3 sandbox tester accounts created and email-verified in ASC
[ ] "Request a Test Notification" delivered — TEST row in apple_iap_notification_log
[ ] Sandbox purchase on device — SUBSCRIBED notification received
[ ] apple_iap_subscriptions row: subscription_status = active, environment = Sandbox
[ ] apple_iap_notification_log row: notification_type = SUBSCRIBED
[ ] DID_RENEW notification received after sandbox renewal (~5 min)
[ ] EXPIRED notification received after cancellation; subscription_status = expired
[ ] REFUND notification received; subscription_status = refunded
[ ] Forged payload returns 401
[ ] Missing signedPayload field returns 400
[ ] checkout-block-check returns blocked: true with active sub; false after expiry
[ ] Duplicate notification delivery is a no-op (idempotency confirmed)
[ ] FLAG_APPLE_IAP remains OFF on raxx-api-prod (do not flip until post-launch authorization)

7. Operator decisions still outstanding

8. What not to do

• Do not flip FLAG_APPLE_IAP=1 on production (raxx-api-prod). That is a separate post-launch authorization step.
• Do not set the Production server URL in ASC until production E2E is separately authorized.
• Do not invent price points for Pro or Pro+. Blocked on BLR research.
• Do not create IAP product records with different IDs than those in StoreKitManager.swift. A product ID mismatch means Product.products(for:) returns zero products and the billing screen is blank.
• Do not use the Stripe checkout path to test iOS billing. Stripe is the web/desktop billing path. iOS billing goes through Apple IAP exclusively (project memory: project_ios_billing_iap).

References

• Issue #3688 — this card
• Epic #167 — Raxx iOS companion app
• PR #3685 — JWS verify
• PR #3686 — persist (apple_iap_subscriptions + apple_iap_notification_log)
• PR #3687 — entitlement + reconciliation
• ios/Raxx/Resources/RaxxProducts.storekit — StoreKit product catalog (source of truth for product IDs)
• ios/Raxx/Features/Billing/StoreKitManager.swift — Swift product ID constants
• backend_v2/api/routes/apple_iap.py — notification endpoint implementation
• backend_v2/alembic/versions/0042_apple_iap_subscriptions_and_notification_log.py — DB migration
• docs/ops/runbooks/ios-iap-sandbox-testing.md — client-side sandbox testing SOP (§1–5, earlier state)
• ADR-0007 — docs/architecture/adr/0007-ios-subscription-billing-iap.md
• Apple: App Store Server Notifications v2 — https://developer.apple.com/documentation/appstoreservernotifications
• Apple: Sandbox tester management — https://developer.apple.com/documentation/storekit/testing_in_the_sandbox_environment