Raxx · internal docs

internal · gated

CF Pages — Antlers Next.js runbook

System: Cloudflare Pages (Antlers Next.js — raxx-staging-next + raxx-prod-next) Owner: sre-agent Provisioned: 2026-05-27 Last reviewed: 2026-07-31 (staging FLAG_PAPER_TRADING_V1 / FLAG_OPTIONS_PAPER_TRADING enablement — added Failure mode F) Related issues: #2883 (Phase 3 prod DNS cutover), #4071 (Woodpecker deploy pipeline cutover), #4312 (PostHog key delivery)

Current deploy mechanism (2026-07-23): both raxx-staging-next and raxx-prod-next deploy via Woodpecker CI (.woodpecker/deploy-antlers-next-staging.yaml / deploy-antlers-next-prod.yaml), NOT the GH Actions workflows referenced elsewhere in this doc — those were retired 2026-07-10 (.github/workflows-retired/). The "GitHub Environments" and "Deploy approval flow" sections below describe the retired GHA-era mechanism and are kept for history; the Woodpecker pipelines are the sole active deploy path. This section and "Build configuration" below have been corrected; a full pass on the rest of this doc is tracked as a follow-up (see Action items in the card #4312 PR).

Project inventory

Project name CF Pages project ID Default subdomain Custom domain Environment
raxx-staging-next 9932e284-b2bc-48a3-b6f0-4f2e601eccd1 raxx-staging-next.pages.dev staging-nextjs.raxx.app staging
raxx-prod-next 5347a333-6724-4e07-bc1f-83351b6cfe6a raxx-prod-next.pages.dev None (Phase 3 cutover) production

Project IDs are also stored in Infisical at /MooseQuest/cloudflare/ (env: prod):

Token references (values in vault — never inline):

Build configuration

Setting Value
Build command npm run build:cf
Output directory .vercel/output/static
Root directory frontend/raxx-next
Production branch main (prod) / release (staging)

The npm run build:cf script is defined in frontend/raxx-next/package.json. It runs next build with the @cloudflare/next-on-pages adapter, which outputs the static bundle to .vercel/output/static.

Corrected 2026-07-23 (card #4312): this project does NOT read CF Pages dashboard environment variables at build time — like getraxx, it is a build-then-upload pipeline. The Woodpecker build step runs npm run build:cf inside its own container (baking in NEXT_PUBLIC_* values from the step's environment: block and vault-fetched secrets), and a later deploy-cf step uploads the pre-built .vercel/output/static/ via wrangler pages deploy. CF Pages never runs its own build for this project, so any env var set through the CF Pages dashboard/API is never read. See .woodpecker/deploy-antlers-next-prod.yaml and deploy-antlers-next-staging.yaml.

Environment variables

Baked at Woodpecker build time (see build step environment: block in each .woodpecker/deploy-antlers-next-*.yaml), not on the CF Pages project.

raxx-staging-next (.woodpecker/deploy-antlers-next-staging.yaml)

NEXT_PUBLIC_API_URL=https://api-staging.raxx.app
NEXT_PUBLIC_CLARITY_PROJECT_ID=wsw9s57qh9
FLAG_PAPER_TRADING_V1=1                                                         # 2026-07-31, staging only — see Failure mode F
NEXT_PUBLIC_FLAGS=paper_trading_v1,FLAG_OPTIONS_PAPER_TRADING                    # 2026-07-31, staging only — see Failure mode F
NEXT_PUBLIC_POSTHOG_KEY=<vault: /MooseQuest/posthog/PH_PROJECT_API_KEY_RAXX>   # card #4312
NEXT_PUBLIC_POSTHOG_HOST=<vault: /MooseQuest/posthog/PH_HOST_RAXX>              # card #4312

Do not enable FLAG_PAPER_TRADING_V1 on raxx-prod-next until issue #197 (securities-attorney review of MBT profile narrative copy) is resolved — see frontend/raxx-next/app/(authed)/simulate/page.tsx header comment.

raxx-prod-next (.woodpecker/deploy-antlers-next-prod.yaml)

NEXT_PUBLIC_API_URL=https://api.raxx.app
NEXT_PUBLIC_CLARITY_PROJECT_ID=wsw9s57qh9
NEXT_PUBLIC_FLAGS=FLAG_CE_DATE_PICKER,FLAG_HELP_PASSKEYS
NEXT_PUBLIC_POSTHOG_KEY=<vault: /MooseQuest/posthog/PH_PROJECT_API_KEY_RAXX>   # card #4312
NEXT_PUBLIC_POSTHOG_HOST=<vault: /MooseQuest/posthog/PH_HOST_RAXX>              # card #4312

Staging and prod share the same PostHog project/key (ADR-0145 single-project topology — environment is an event property, not a separate project).

Custom domains + DNS

staging-nextjs.raxx.app

raxx-prod-next — NO custom domain

Production project has no custom domain attached. The DNS cutover of raxx.app from the existing raxx-app CF Pages project to raxx-prod-next is Phase 3 work tracked in #2883. Do NOT add a custom domain or modify DNS for raxx.app apex until Phase 3.

GitHub Environments

Environment name Repo Protection rules Deploy trigger
staging-nextjs raxx-app/TradeMasterAPI None Auto on push to main
production-nextjs raxx-app/TradeMasterAPI None (see note) Workflow dispatch

Note — reviewer gate: GitHub required-reviewer protection rules require GitHub Team or Enterprise plan. The production-nextjs environment was provisioned without a reviewer gate. A manual review step must be enforced in the GH Actions workflow YAML itself (using environment: production-nextjs in the deploy job and a separate approval job) until the plan is upgraded. See operator-action card filed as part of this provisioning.

The existing production environment uses a branch policy rather than required reviewers — same pattern applies here until the plan gate is resolved.

Deploy approval flow

Staging (staging-nextjs environment)

Push to main
  └─> GH Actions workflow fires
       └─> deploy job uses environment: staging-nextjs
            └─> deploys to raxx-staging-next CF Pages project
                 └─> staging-nextjs.raxx.app serves the build

No approval required. Deploy fires automatically on every merge to main for paths matching frontend/trademaster_ui/**.

Production (production-nextjs environment)

workflow_dispatch (manual trigger)
  └─> GH Actions workflow fires
       └─> approval-gate job (requires operator confirmation in workflow step)
            └─> deploy job uses environment: production-nextjs
                 └─> deploys to raxx-prod-next CF Pages project
                      └─> raxx-prod-next.pages.dev serves the build
                           └─> (no public traffic until Phase 3 DNS cutover)

Until the Phase 3 DNS cutover (#2883), prod deploys build the project and make it available on raxx-prod-next.pages.dev only. No end-user traffic flows to it.

How to tell it's broken

How to diagnose (in order)

  1. Check CF Pages deploy token is valid:

bash TOKEN=$(infisical secrets get CF_PAGES_DEPLOY_TOKEN \ --path /MooseQuest/cloudflare/ --env prod --plain) curl -sS -H "Authorization: Bearer ${TOKEN}" \ https://api.cloudflare.com/client/v4/user/tokens/verify # Expect: {"success":true,"result":{"status":"active"}}

  1. Check project exists and has correct config:

bash ACCOUNT_ID=$(infisical secrets get CLOUDFLARE_ACCOUNT_ID \ --path /MooseQuest/cloudflare/ --env prod --plain) curl -sS \ -H "Authorization: Bearer ${TOKEN}" \ "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/pages/projects/raxx-staging-next" \ | python3 -m json.tool | grep -E '"name"|"subdomain"|"build_command"|"destination_dir"'

  1. Check custom domain verification status for staging-nextjs.raxx.app:

bash curl -sS \ -H "Authorization: Bearer ${TOKEN}" \ "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/pages/projects/raxx-staging-next/domains" \ | python3 -m json.tool # Look for: "status": "active" (not "initializing" or "blocked")

  1. If domain status is blocked, check the DNS CNAME record is present and proxied:

bash DNS_TOKEN=$(infisical secrets get CLOUDFLARE_EDIT_DNS \ --path /MooseQuest/cloudflare/ --env prod --plain) curl -sS \ -H "Authorization: Bearer ${DNS_TOKEN}" \ "https://api.cloudflare.com/client/v4/zones/f12dbb5cac57d5591a5058874498a6d1/dns_records?name=staging-nextjs.raxx.app" \ | python3 -m json.tool # Expect: type=CNAME, content=raxx-staging-next.pages.dev, proxied=true

  1. If the deploy fails in GH Actions: check the Actions run logs for the exact error. Common: token scope, project name mismatch, output directory empty after build.

Known failure modes

Failure mode A: CF Pages deploy token expired

Symptom: Deploy step returns HTTP 403 or {"success":false,"errors":[{"code":9106,...}]}.

Cause: CF_PAGES_DEPLOY_TOKEN hit its 90-day rotation cadence or was revoked.

Fix: 1. Read CLOUDFLARE_ACCESS_MGMT_TOKEN from vault (/MooseQuest/cloudflare/) and roll CF_PAGES_DEPLOY_TOKEN per docs/ops/runbooks/cloudflare-tokens.md → Failure mode B. 2. Write the new value back to vault at /MooseQuest/cloudflare/CF_PAGES_DEPLOY_TOKEN (and update the alias CF_PAGES_DEPLOY to match). 3. Update the CF_PAGES_DEPLOY_TOKEN__EXPIRES_AT companion. 4. Re-run the failed deploy workflow.

Verification: Deploy workflow completes with exit 0; project appears in the deployments list via CF API.

Failure mode B: staging-nextjs.raxx.app domain stuck in initializing

Symptom: staging-nextjs.raxx.app serves a CF error instead of the Pages content >30 min after domain was attached.

Cause: Domain verification is pending — CF needs the CNAME to propagate and a TXT verification record to resolve.

Fix: 1. Check domain status via API (step 3 in diagnosis). 2. If status is initializing, wait up to 10 min for CF to verify the CNAME (record ID ca04428054c6a0de38f25e19eb663483 is already in place and proxied). 3. If status is blocked, re-attach the domain:

```bash curl -sS -X DELETE \ -H "Authorization: Bearer ${TOKEN}" \ "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/pages/projects/raxx-staging-next/domains/staging-nextjs.raxx.app"

curl -sS -X POST \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/pages/projects/raxx-staging-next/domains" \ -d '{"name":"staging-nextjs.raxx.app"}' ```

Verification: Domain status returns "status":"active" and curl -sI https://staging-nextjs.raxx.app returns HTTP 200 after first deploy.

Failure mode C: wrong NEXT_PUBLIC_API_URL in deployed bundle

Symptom: API calls from staging-nextjs.raxx.app hit the wrong backend (prod instead of staging, or vice versa).

Cause: Env var set on preview deployment config instead of production, or set in the GH Actions step env rather than the CF Pages project config.

Fix: Verify the env var is on the production deployment config:

curl -sS \
  -H "Authorization: Bearer ${TOKEN}" \
  "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/pages/projects/raxx-staging-next" \
  | python3 -c "
import sys,json
p=json.load(sys.stdin)['result']
print(p.get('deployment_configs',{}).get('production',{}).get('env_vars',{}))
"

If missing or wrong, PATCH the project:

curl -sS -X PATCH \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/pages/projects/raxx-staging-next" \
  -d '{
    "deployment_configs": {
      "production": {
        "env_vars": {
          "NEXT_PUBLIC_API_URL": {"value": "https://api-staging.raxx.app"}
        }
      }
    }
  }'

Verification: Redeploy and confirm the NEXT_PUBLIC_API_URL value in the built bundle.

Failure mode E: HTTP 503 — missing nodejs_compat compatibility flag

Symptom: staging-nextjs.raxx.app (or raxx-prod-next) returns HTTP 503 with body title "Error - no nodejs_compat compatibility flag" immediately after a successful wrangler deploy.

Cause: @cloudflare/next-on-pages uses Node.js APIs (AsyncLocalStorage, streams, etc.) that require the nodejs_compat compatibility flag on the CF Pages project. The flag was not set during project provisioning. First observed: Wave B Phase 2 first deploy, 2026-05-27.

Fix: PATCH the project's production deployment config to add the flag:

TOKEN=$(infisical secrets get CF_PAGES_DEPLOY_TOKEN \
  --path /MooseQuest/cloudflare/ --env prod --plain)
ACCOUNT_ID=$(infisical secrets get CLOUDFLARE_ACCOUNT_ID \
  --path /MooseQuest/cloudflare/ --env prod --plain)

# For staging project:
curl -sS -X PATCH \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/pages/projects/raxx-staging-next" \
  -d '{
    "deployment_configs": {
      "production": {
        "compatibility_flags": ["nodejs_compat"],
        "compatibility_date": "2024-09-23"
      }
    }
  }'

# For production project (Phase 3 — run before prod cutover):
curl -sS -X PATCH \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/pages/projects/raxx-prod-next" \
  -d '{
    "deployment_configs": {
      "production": {
        "compatibility_flags": ["nodejs_compat"],
        "compatibility_date": "2024-09-23"
      }
    }
  }'

After patching, re-trigger the deploy workflow — no code change needed:

gh workflow run deploy-antlers-next-staging.yml --repo raxx-app/TradeMasterAPI

Verification: curl -sS -o /dev/null -w "%{http_code}" https://staging-nextjs.raxx.app/ returns 200.

Prevention: As of #2945 (2026-05-27), deploy-antlers-next-staging.yml includes an idempotent PATCH step ("Ensure nodejs_compat compatibility flag") that runs before the wrangler deploy. On future deploys it will log "nodejs_compat already set — no PATCH needed" and skip. The flag must also be set at project provisioning time (Wave A SRE leg) to avoid the first-deploy 503 window. Both raxx-staging-next and raxx-prod-next have the flag set as of 2026-05-27 (applied via CF API during the #2945 resolution).

Failure mode F: CF Pages dashboard/API env vars silently have no effect

Symptom: A feature flag (e.g. FLAG_PAPER_TRADING_V1) or env var appears set correctly via GET .../pages/projects/raxx-staging-next (or the CF dashboard UI), but the deployed site behaves as if it were unset — e.g. /simulate redirects to /dashboard even after PATCHing deployment_configs.production.env_vars via the CF API.

Cause: As documented under "Build configuration" above, this project does NOT run its own build — Woodpecker builds next build inside its own container and wrangler pages deploy-uploads the static output. CF Pages never invokes its build step, so deployment_configs.production.env_vars set via the CF API/dashboard is read by nothing. It is legacy/stale configuration (confirmed 2026-07-31: the stored build_config.root_dir on this project still points at the retired frontend/trademaster_ui path, consistent with it predating the Woodpecker cutover and never having been cleaned up). Setting env vars there gives false confidence that a flag is live when it is not.

Fix: Add the env var to the build step's environment: block in .woodpecker/deploy-antlers-next-staging.yaml (or -prod.yaml for prod) as plain text for non-secrets, or via the same-step vault-load pattern for secrets (see PostHog example above). Land the change through the normal PR path (develop → cut RC → auto-promote to release), then let the push-to- release trigger (or an event: manual re-run) redeploy.

Do NOT PATCH deployment_configs.production.env_vars via the CF API as a "fix" for this project — it will not error, will appear to succeed, and will have zero effect. (The nodejs_compat compatibility_flags PATCH in deploy-cf below is a different, unrelated field that CF Pages DOES read at deploy time — the accepted-but-inert behavior is specific to env_vars.)

Verification: After redeploy, fetch the built page HTML (or use an authenticated probe) and confirm the flag-gated behavior — never trust the CF Pages project's env_vars listing alone as proof a flag is live for this project.

Failure mode D: build command not found (npm run build:cf)

Symptom: Deploy fails with missing script: build:cf error.

Cause: The build:cf script hasn't been added to frontend/trademaster_ui/package.json yet (Phase 2 Wave A feature-dev task).

Fix: This is a feature-dev dependency, not an SRE fix. Check whether the feature-dev PR adding npm run build:cf has landed on main. If not, the workflow cannot deploy until it does. File an escalation to the feature-dev agent with the blocker.

Verification: cat frontend/trademaster_ui/package.json | python3 -c "import sys,json; s=json.load(sys.stdin); print(s.get('scripts',{}).get('build:cf','MISSING'))" returns the build command, not MISSING.

Phase 3 — production DNS cutover

The raxx-prod-next project is provisioned and receives no public traffic. The DNS cutover from the existing raxx-app CF Pages project (serving raxx.app) to raxx-prod-next is tracked in #2883.

Steps at cutover time (do NOT run now — Phase 3 only):

  1. Attach raxx.app as a custom domain to raxx-prod-next.
  2. Remove raxx.app custom domain from the existing raxx-app project.
  3. Verify DNS propagation via dig raxx.app CNAME.
  4. Run smoke tests against https://raxx.app — use scripts/ops/phase3_post_cutover_smoke.py.
  5. Update this runbook with the cutover date.

For the full post-cutover smoke + rollback procedure, see docs/ops/runbooks/antlers-nextjs-rollback.md.

Emergency stop

To take staging-nextjs.raxx.app offline:

# Delete the custom domain from the CF Pages project
TOKEN=$(infisical secrets get CF_PAGES_DEPLOY_TOKEN \
  --path /MooseQuest/cloudflare/ --env prod --plain)
ACCOUNT_ID=$(infisical secrets get CLOUDFLARE_ACCOUNT_ID \
  --path /MooseQuest/cloudflare/ --env prod --plain)

curl -sS -X DELETE \
  -H "Authorization: Bearer ${TOKEN}" \
  "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/pages/projects/raxx-staging-next/domains/staging-nextjs.raxx.app"

This removes the custom domain; raxx-staging-next.pages.dev remains accessible.

To remove a production deployment (before Phase 3 cutover, raxx-prod-next serves no traffic — nothing to stop).

Escalation

Escalate to operator when: - CF_PAGES_DEPLOY or CLOUDFLARE_ACCESS_MGMT_TOKEN needs re-minting (requires 2FA/dashboard access). - The raxx-prod-next project needs to be deleted and recreated. - Phase 3 DNS cutover is being executed (operator must be present). - A new CF Pages project needs to be provisioned for a surface not covered by this runbook.

References