Raxx · internal docs

internal · gated ↑ index

Console deploy — manual break-glass runbook

System: raxx-console-prod (Heroku) Owner: operator / sre-agent Last incident: 2026-04-30 (see #776 — CI deploy broken, manual push used) Last reviewed: 2026-04-30

Context

console/ is a subdirectory of the monorepo. Heroku expects to receive a repo whose root IS the app. The canonical CI path (deploy-console.yml) handles this with git subtree split and a direct git push. This runbook documents the equivalent shell procedure for use when the CI workflow itself is unavailable (e.g. runner outage, GitHub Actions incident, break-glass scenario).

When to use this runbook

Use this only when deploy-console.yml workflow_dispatch is unavailable or has failed in a way that cannot be fixed quickly. Normal deploys go through CI.

Prerequisites

Manual deploy procedure

# 1. Confirm you are on the correct commit.
git log --oneline -5

# 2. Fetch your Heroku token from vault (or export from environment).
#    Never paste the raw token in a script stored in the repo.
HEROKU_EMAIL="kris@moosequest.net"
HEROKU_API_KEY="<token from Infisical /MooseQuest/heroku/HEROKU_API_KEY>"

# 3. Write a .netrc for authentication.
#    umask 077 ensures the file is created with 600 permissions.
umask 077
cat > "$HOME/.netrc" <<EOF
machine git.heroku.com
  login ${HEROKU_EMAIL}
  password ${HEROKU_API_KEY}
EOF
chmod 600 "$HOME/.netrc"

# 4. Produce a synthetic root commit whose tree IS the console/ subtree.
SUBTREE_SHA=$(git subtree split --prefix=console HEAD 2>/dev/null)
echo "Subtree SHA: $SUBTREE_SHA"

# 5. Push to Heroku.
git push --force \
  "https://git.heroku.com/raxx-console-prod.git" \
  "${SUBTREE_SHA}:refs/heads/main"

# 6. Verify the deploy completed.
#    Wait ~60 s for dyno restart, then:
curl -fsS https://console.raxx.app/health | python3 -m json.tool
# Expected: HTTP 200, {"status":"ok"} (or 302 redirect to CF Access login)

How to tell it's broken

Known failure modes

Failure mode A: akhileshns setRawMode crash (Node 20+)

Symptom: CI step Deploy to Heroku fails with process.stdin.setRawMode is not a function Cause: akhileshns/heroku-deploy uses an interactive TTY API that does not exist in CI runners Fix: The CI workflow now uses the netrc + git-push pattern (PR #776). If this runbook is being used it means CI itself is down — proceed with the manual steps above. Verification: curl -fsS https://console.raxx.app/health returns 200

Failure mode B: Heroku git auth rejection

Symptom: git push returns error: authentication failed or Do not authenticate with username and password using git Cause: The .netrc login is not the registered Heroku account email, OR the token is expired/revoked Fix: 1. Confirm the email matches the Heroku account: heroku auth:whoami (if CLI available) 2. Rotate the token: Heroku dashboard -> Account -> API Key -> Regenerate 3. Update the secret in Infisical and in the GitHub Environment secrets Verification: Retry the push; it should proceed without auth errors

Failure mode C: subtree split fails (no output)

Symptom: git subtree split --prefix=console HEAD exits 0 but prints nothing; push target SHA is empty Cause: The console/ directory does not exist at HEAD, or the git history has no commits touching console/ Fix: Verify the working directory and ref:

git log --oneline --follow -- console/app.py | head -3
ls console/

If console/ is present but subtree split still fails, try with --rejoin:

git subtree split --prefix=console --rejoin HEAD

Verification: SUBTREE_SHA is a 40-character SHA

Enabling runtime-dyno-metadata (one-time setup)

Required for HEROKU_SLUG_COMMIT to be available in the dyno environment (used by the version-footer commit link per #775). Run once per app:

heroku labs:enable runtime-dyno-metadata --app raxx-console-prod

Verify after the next deploy:

heroku run "printenv | grep HEROKU_SLUG_COMMIT" --app raxx-console-prod

Emergency stop

To take raxx-console-prod offline cleanly (scale all dynos to 0):

heroku ps:scale web=0 --app raxx-console-prod

To bring it back:

heroku ps:scale web=1 --app raxx-console-prod

Escalation

Wake the operator when: - The Heroku API token is revoked and cannot be rotated without account access - The console/ subtree has a corrupted git history that blocks subtree split - The dyno crashes on boot after a successful push (application-level bug, not deploy-level)

Contact: Kristerpher via Slack DM (D0AJ7K184TV) or kris@moosequest.net

References