Raxx · internal docs

internal · gated

Sentry alert rules runbook

System: Sentry (issue alert rules — trademaster-raptor project) Owner: operator Last incident: 2026-06-10 (2026-06-10-beta-walk-500), 2026-06-12 (2026-06-12-beta-walkthrough-500) Last reviewed: 2026-06-18 UTC

Managed rules

Rule name Fingerprint / filter Condition Action Frequency Created Issue
skipped_no_postmark_token — email silent failure message contains skipped_no_postmark_token >= 1 event in 5m email ops@raxx.app (IssueOwners / AllMembers) 60 min 2026-05-30 #3135
beta-walk-5xx — beta walkthrough 500 class http.url starts-with /beta/walk/ AND http.status_code starts-with 5 >= 1 event in 5m email ops@raxx.app (IssueOwners / AllMembers) 60 min 2026-06-18 #3683

Notification posture: both rules use IssueOwners / AllMembers fallthrough which routes to kris@moosequest.net (the sole org member, who monitors ops@raxx.app). The Sentry email action is out-of-band from the GitHub Actions SLACK_NOTIFY_LEVEL gate; it fires unconditionally regardless of that variable, matching the loud tier posture in docs/ops/runbooks/slack-notification-policy.md.

Auth

Sentry API token lives at vault path /MooseQuest/sentry/SENTRY_API (env: SENTRY_AUTH_TOKEN). Fetch it from vault before running any script below:

export SENTRY_AUTH_TOKEN=$(python3 scripts/ops/mint_secret.py /MooseQuest/sentry/SENTRY_API 2>/dev/null)

Or pass it directly to the script; the script fetches from vault automatically when SENTRY_AUTH_TOKEN is absent and INFISICAL_* + CF_ACCESS_* env vars are present.

How to create or verify an alert rule

The script accepts a --rule flag. Default is skipped-no-postmark-token (backward-compat).

Create / idempotent re-run (original postmark rule)

python3 scripts/ops/create_sentry_alert_rule.py

Create / idempotent re-run (beta-walk 5xx rule — issue #3683)

python3 scripts/ops/create_sentry_alert_rule.py --rule beta-walk-5xx

The script is idempotent for each rule: if the rule already exists (matched by name), it prints the existing rule ID and exits 0 without creating a duplicate.

Dry-run (inspect payload without creating)

python3 scripts/ops/create_sentry_alert_rule.py --rule beta-walk-5xx --dry-run

Target a different org or project

python3 scripts/ops/create_sentry_alert_rule.py \
  --rule beta-walk-5xx \
  --org moosequest \
  --project trademaster-raptor

Verify all rules exist via Sentry API

curl -s \
  -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  -H "User-Agent: raxx-sre-ops/1.0" \
  "https://sentry.io/api/0/projects/moosequest/trademaster-raptor/rules/" \
  | python3 -m json.tool | grep -E '"name"|"id"'

How to tell skipped_no_postmark_token is firing

What it means: POSTMARK_SERVER_TOKEN is absent or empty on the dyno that attempted an email send. Waitlist confirmation emails are being silently skipped.

Remediation for skipped_no_postmark_token

  1. Verify the config var on prod: bash heroku config:get POSTMARK_SERVER_TOKEN --app raxx-api-prod | wc -c # Expect > 5. If 0 or 1: the token is unset.

  2. If unset, set it (silence stdout — token must not appear in terminal history): bash heroku config:set POSTMARK_SERVER_TOKEN="$POSTMARK_SERVER_TOKEN" \ --app raxx-api-prod >/dev/null 2>&1 Full procedure: docs/ops/runbooks/postmark.md → "Setting POSTMARK_SERVER_TOKEN on Heroku".

  3. Verify: bash heroku run --app raxx-api-prod \ python -c "from api.services.postmark_client import test_postmark_token; test_postmark_token()" # Expect: ✓ Postmark token valid server=<ServerName> id=<ServerID>

  4. Resolve the Sentry issue only after verifying the fix is deployed and the smoke passes.

How to tell beta-walk-5xx is firing

What it means: A request to any /beta/walk/<token>/* route returned HTTP 5xx. Common causes (both observed in production): 1. A Raptor DB table required by the route is missing (migration not applied). Check backend_v2/alembic/versions/ — the route's table must have a Raptor-side migration. 2. An unhandled exception in the route handler (any except Exception that returns 500).

Remediation for beta-walk-5xx

  1. Pull the Sentry event to get the stack trace and breadcrumbs: https://sentry.io/organizations/moosequest/issues/?project=trademaster-raptor Filter by beta-walk. Open the issue and read the full stack trace.

  2. If the trace shows a SQLAlchemy Table not found error: - Check backend_v2/alembic/versions/ for a migration that creates the missing table. - If absent, write the migration (see docs/incidents/2026-06-10-beta-walk-500.md and docs/incidents/2026-06-12-beta-walkthrough-500.md for the established pattern). - PR the migration → merge → release phase applies it → retry the walkthrough URL.

  3. If the trace shows a different error: escalate to the operator with the full stack trace.

  4. Resolve the Sentry issue only after the fix is deployed and the route returns 200.

Adding new alert rules

  1. Define the rule parameters (condition, filter, action, frequency).
  2. Add an entry to the "Managed rules" table above with the notification posture note.
  3. Add the rule key to _KNOWN_RULES in scripts/ops/create_sentry_alert_rule.py.
  4. Implement _build_<rule>_payload() in the same file.
  5. Add tests to scripts/ops/tests/test_create_sentry_alert_rule.py.
  6. Run the script with --dry-run to inspect the payload, then without to create the rule.
  7. Record the Sentry rule ID returned by the script in this table.

Escalation

Wake the operator when: - The alert rule is missing (script exits non-zero or GET returns empty rules list). - Sentry API returns 401 — the SENTRY_API token may have expired or been revoked. - The skipped_no_postmark_token issue fires repeatedly after POSTMARK_SERVER_TOKEN has been set correctly (indicates a different skip path not covered by the alert filter). - The beta-walk-5xx issue fires and the stack trace does not match any documented remediation above.