Raxx · internal docs

internal · gated

getraxx.com deploy runbook

System: getraxx (marketing landing site, frontend/getraxx-landing/, CF Pages project getraxx) Owner: sre-agent / operator Last incident: 2026-08-09 (see docs/incidents/2026-08-09-getraxx-blr-compliance-deploy-gap.md) Last reviewed: 2026-08-09

How to tell it's broken

getraxx.com has no dedicated staging environment. Validating "getraxx staging" means checking the live getraxx.com / getraxx.pages.dev bundle directly — there is no lower environment to check instead.

How to diagnose (in order)

  1. Confirm what main actually contains for the content in question: git fetch origin main git show origin/main:frontend/getraxx-landing/src/components/getraxx/<File>.jsx | grep '<expected string>'
  2. Confirm what's actually live: curl -s https://getraxx.com/ | grep '<expected string>' curl -s https://getraxx.com/assets/*.js | grep '<expected string>' index.html serves cache-control: public, max-age=0, must-revalidate / cf-cache-status: DYNAMIC — always fresh from origin, not a CDN-caching artifact. If content is missing from the live bundle but present on main, the deploy pipeline either never ran or failed.
  3. Check the last deploy-getraxx pipeline run via the Woodpecker API (web UI requires CF Access/Google SSO login, not curl-able without a browser session): WP_TOKEN=$(aws ssm get-parameter --name /ci/woodpecker/admin-api-token \ --with-decryption --region us-east-2 --query "Parameter.Value" --output text) curl -sS -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \ "https://ci.moosequest.net/api/repos/1/pipelines?page=1&perPage=5" Find the most recent pipeline with "branch":"main" and a changed_files entry under frontend/getraxx-landing/. Check its deploy-getraxx workflow state.
  4. If a pipeline shows failure, pull step-level logs (note: log entries are base64-encoded, one line per array entry): curl -sS -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \ "https://ci.moosequest.net/api/repos/1/pipelines/<number>" | python3 -c " import json,sys d=json.load(sys.stdin) for wf in d['workflows']: print(wf['name'], wf['state']) for st in wf['children']: print(' ', st['id'], st['pid'], st['name'], st['state']) " # then, for the failing step's internal 'id' (not 'pid'): curl -sS -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \ "https://ci.moosequest.net/api/repos/1/logs/<pipeline-number>/<step-id>" | python3 -c " import json, base64, sys d=json.load(sys.stdin) print(''.join(base64.b64decode(e['data']).decode('utf-8','replace') for e in d)) "
  5. If no pipeline ran at all for the content in question: check whether the content actually landed on main via a push, or only via develop/ release — the push trigger only fires on branch: main. If the content is live but main never received it, suspect a manual trigger fired against a non-main ref (see Failure mode A).

Known failure modes

Failure mode A: manual trigger deployed a non-main ref (content live, main doesn't have it — or vice versa)

Symptom: Live getraxx.com content doesn't match main's git history in either direction — content on the live site that main doesn't have, and/or content on main that the live site doesn't have.

Cause: .woodpecker/deploy-getraxx.yaml's event: manual trigger has no branch/ref restriction, unlike the push trigger (branch: main). A manual run deploys whatever ref Woodpecker has checked out at trigger time — develop, a feature branch, anything. This is the confirmed root cause of the 2026-08-09 incident: a manual trigger fired against develop mid-wave of a 4-commit authorized copy change, shipping commit 1 of 4 to prod while main (and any future main-push deploy) never received any of the four.

Fix: Not yet hardened — see issue #4451 (restrict event: manual to branch: main, or add an explicit ref-check guard step). Until that lands, never trigger a manual deploy-getraxx run against anything other than main's current tip. If you need to preview un-merged content, use the pr-preview workflow (.woodpecker/pr-preview.yaml) instead — it deploys to a PR-scoped preview URL, not production.

Verification: confirm the deployed commit SHA (npx wrangler pages deployment list --project-name=getraxx, or check the pipeline's commit field) is an ancestor of origin/main's current tip.

Failure mode B: ci-pr required check fails on a main-based PR (stale-branch-guard)

Symptom: A PR from a main-based hotfix branch (targeting main directly, per the ADR-0115 emergency-hotfix path) fails ci/woodpecker/pr/ci-pr at the stale-branch-guard step with a large commits-behind / fork-point-age number, even though the branch is fresh off main.

Cause: stale-branch-guard historically hardcoded its comparison base to develop. main is supposed to be far behind develop (that's the entire point of the develop→release→main promotion model), so any main-based PR structurally fails this check unless it's actually comparing against main. This was fixed twice on develop (docs/incidents/2026-07-25-ci-pr-stale-branch-guard-hardcoded-base.md, docs/incidents/2026-07-30-ci-pr-stale-branch-guard-absolute-age.md) but both fixes can silently be absent from main if no release→main promotion has happened recently — main only receives commits via promotion or direct hotfix, and CI-infra fixes on develop don't reach it any other way.

Fix: Before opening any main-targeted hotfix PR, check whether main already has the current stale-branch-guard logic:

git merge-base --is-ancestor <latest stale-branch-guard fix commit on develop> origin/main \
  && echo "main has the fix" || echo "main is missing it — cherry-pick required"

If missing, cherry-pick the relevant CI-infra-only commit(s) from develop onto your hotfix branch alongside your actual content changes (see PR #4442 for a worked example — cherry-picked ed2237f6d and 1e20e57c8 alongside the compliance-content commits).

Verification: GITHUB_BASE_REF=main bash scripts/ci/check_stale_branch.sh locally should report 0 commits behind (for a branch freshly cut from main's tip) and exit 0.

Failure mode C: build-and-deploy fails with externally-managed-environment (PEP 668)

Symptom: build-and-deploy step fails with:

error: externally-managed-environment
× This environment is externally managed
hint: See PEP 668 for the detailed specification.

Cause: The step's base image (node:22-slim) apt-installs python3 python3-pip, then runs a bare pip3 install --quiet requests to bootstrap the vault-secrets loader. Debian bookworm's python3-pip package now marks the environment PEP-668 "externally managed" by default, so an unflagged pip3 install fails. This is base-image drift (the pipeline worked previously; nothing in deploy-getraxx.yaml itself changed) — confirmed present, same pattern, in deploy-support.yaml, deploy-mockups.yaml, pr-preview.yaml (×2), and deploy-status-page.yaml (see #4454).

Fix: Add --break-system-packages to the pip3 install call:

- pip3 install --quiet requests --break-system-packages

This matches the pattern already in production use in .woodpecker/deploy-queue.yaml.

Verification: re-trigger the pipeline (push or manual, against main only — see Failure mode A) and confirm build-and-deploy reaches the CF Pages deploy step without erroring on the pip3 install line.

Emergency stop

To take getraxx.com deploys offline cleanly (stop any further pipeline runs from publishing, without touching the currently-live site):

# Disable the repo's Woodpecker webhook processing for this pipeline file
# temporarily by renaming it out of the trigger path (requires a PR — this
# is not a live/instant kill switch, deploy-getraxx has no feature-flag
# equivalent). For a true emergency stop, use CF Pages directly:
npx wrangler@4.84.0 pages deployment list --project-name=getraxx
# then roll back to the last known-good deployment ID via the CF dashboard
# or `wrangler pages deployment rollback` if available for the CLI version in use.

There is no automated rollback step in deploy-getraxx.yaml itself — a bad deploy must be corrected by shipping a fix-forward commit through the normal main-push path, or by manually rolling back via the CF Pages dashboard/API.

Escalation

Wake the operator when: - A main-push deploy fails twice in a row for different root causes (as happened 2026-08-09 — stale-branch-guard, then PEP-668 — though both were resolved without operator escalation since fixes were already established patterns elsewhere in the repo). - The failure touches Cloudflare DNS/custom-domain attachment steps (not just the build/deploy step) — those touch production DNS and should not be retried blindly. - Content that reached prod via a manual trigger cannot be explained by any ref currently reachable from origin/main or origin/develop (would indicate an even more unusual trigger source than the documented failure modes above).

References