RCA — staging-nextjs.raxx.app 503 due to missing nodejs_compat flag

Incident ID: 2026-05-27-nodejs-compat-flag-missing Date: 2026-05-27 Severity: SEV-3 Duration: ~5m detection (first deploy run) → pending resolution (operator action required) Blast radius: staging-nextjs.raxx.app — staging surface only, no production traffic, no end users Author: sre-agent

Summary

The first deploy of Antlers Next.js to the raxx-staging-next Cloudflare Pages project succeeded at the wrangler level (assets uploaded, deployment created) but the site returned HTTP 503 with error "no nodejs_compat compatibility flag." The CF Pages project was provisioned during Wave A without the nodejs_compat compatibility flag required by @cloudflare/next-on-pages. The deploy workflow's health check caught the 503 and failed the run. Resolution requires a CF API PATCH to add nodejs_compat to the project's production deployment config, then a re-deploy.

Timeline (all times UTC)

• 00:34 — PR #2933 merged to main (merge SHA 9378ae0d)
• 00:34 — deploy-antlers-next-staging.yml triggered automatically by push to main
• 00:36 — Wrangler deploy step completed successfully (assets uploaded to raxx-staging-next)
• 00:36 — Health check step began polling staging-nextjs.raxx.app
• 00:37 — Health check exhausted 6 attempts (60s), all returning HTTP 503
• 00:37 — Run failed; Slack failure notification fired
• 00:37 — Manual curl confirmed 503 with body title "Error - no nodejs_compat compatibility flag"
• 00:37 — Root cause identified: nodejs_compat flag absent from CF Pages project config
• 00:37 — Runbook updated with Failure mode E; RCA written; operator escalation raised

Impact

• Users affected: 0 (staging-only; no end users have access)
• User-visible symptoms: none (staging-nextjs.raxx.app is not publicly advertised)
• Data integrity: ok
• Revenue / billing: ok

What went well

• Health check in the deploy workflow caught the 503 immediately and failed the run — the problem was detected in the same workflow that introduced it, not later by a human.
• Error title was precise: "Error - no nodejs_compat compatibility flag" — CF Pages surfaces this clearly enough to diagnose without any log digging.
• No production impact; staging isolation worked as designed.

What didn't go well

• The nodejs_compat compatibility flag was not set during Wave A project provisioning. The provisioning runbook (and the provisioning workflow/process) did not include this as a required step for @cloudflare/next-on-pages projects.
• The deploy workflow does not include an idempotent PATCH step to ensure project-level config (compatibility flags) before deploying, leaving a gap between "assets uploaded" and "site serves."

Root cause analysis

• Contributing factor 1: Provisioning omission — The raxx-staging-next CF Pages project was provisioned without the nodejs_compat compatibility flag. The Wave A SRE provisioning work focused on project existence, custom domain attachment, and DNS — not the runtime compatibility layer required by the Next.js adapter. The provisioning checklist had no item for this flag.
• Contributing factor 2: No pre-flight config assertion in workflow — The deploy workflow (deploy-antlers-next-staging.yml) does not include a step that asserts or idempotently sets the required compatibility flags before pushing assets. The system allowed a successful asset upload even when the runtime config would cause every request to fail.
• Contributing factor 3: @cloudflare/next-on-pages hard requirement not documented — The runbook's "Build configuration" section described the build command and output directory but omitted the CF Pages project-level requirement (nodejs_compat). The gap allowed the provisioning step to complete without raising this requirement.

Detection

• What alerted us: health check step in deploy-antlers-next-staging.yml (6 × 10s polling, expects HTTP 200)
• How long between cause and detection: ~3m (wrangler deploy completion → health check failure)
• How to detect faster next time: the health check already catches this within 60s of deploy completion. Pre-flight config assertion (see Action item 1) would catch it before the deploy step even runs.

Resolution

Operator action required. SRE agent does not have authorization to PATCH shared CF Pages project configuration without operator authorization.

Operator steps to resolve:

TOKEN=$(infisical secrets get CF_PAGES_DEPLOY_TOKEN \
  --path /MooseQuest/cloudflare/ --env prod --plain)
ACCOUNT_ID=$(infisical secrets get CLOUDFLARE_ACCOUNT_ID \
  --path /MooseQuest/cloudflare/ --env prod --plain)

curl -sS -X PATCH \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/pages/projects/raxx-staging-next" \
  -d '{
    "deployment_configs": {
      "production": {
        "compatibility_flags": ["nodejs_compat"],
        "compatibility_date": "2024-09-23"
      }
    }
  }'

# Then re-trigger the workflow:
gh workflow run deploy-antlers-next-staging.yml --repo raxx-app/TradeMasterAPI

Validation: curl -sS -o /dev/null -w "%{http_code}" https://staging-nextjs.raxx.app/ returns 200.

Also required for Phase 3 before prod deploy: Apply the same PATCH to raxx-prod-next before the first production deploy. See Action item 3 below.

Action items

References

• Runbook: docs/ops/runbooks/cf-pages-antlers-next.md — Failure mode E (added this incident)
• Wave A PR: #2931
• Wave B Phase 2 PR: #2933
• Phase 3 prod cutover: #2883
• CF Pages compatibility flags docs: https://developers.cloudflare.com/workers/configuration/compatibility-dates/#nodejs-compatibility-flag
• @cloudflare/next-on-pages compatibility requirements: https://github.com/cloudflare/next-on-pages