Sentry alert rules runbook

System: Sentry (issue alert rules — trademaster-raptor project) Owner: operator Last incident: — Last reviewed: 2026-05-30 UTC

Managed rules

Auth

Sentry API token lives at vault path /MooseQuest/sentry/SENTRY_API (env: SENTRY_AUTH_TOKEN). Fetch it from vault before running any script below:

export SENTRY_AUTH_TOKEN=$(python3 scripts/ops/mint_secret.py /MooseQuest/sentry/SENTRY_API 2>/dev/null)

Or pass it directly to the script; the script fetches from vault automatically when SENTRY_AUTH_TOKEN is absent and INFISICAL_* + CF_ACCESS_* env vars are present.

How to create or verify an alert rule

Create / idempotent re-run

python3 scripts/ops/create_sentry_alert_rule.py

The script is idempotent: if the rule already exists (matched by name), it prints the existing rule ID and exits 0 without creating a duplicate.

Dry-run (inspect payload without creating)

python3 scripts/ops/create_sentry_alert_rule.py --dry-run

Target a different org or project

python3 scripts/ops/create_sentry_alert_rule.py \
  --org moosequest \
  --project trademaster-raptor

Verify the rule exists via Sentry API

curl -s \
  -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  -H "User-Agent: raxx-sre-ops/1.0" \
  "https://sentry.io/api/0/projects/moosequest/trademaster-raptor/rules/" \
  | python3 -m json.tool | grep -E '"name"|"id"'

How to tell skipped_no_postmark_token is firing

• Sentry issue "skipped_no_postmark_token — email silent failure" appears in trademaster-raptor.
• Email lands at ops@raxx.app with subject containing the rule name.
• Sentry dashboard: https://sentry.io/organizations/moosequest/issues/ → filter project = trademaster-raptor.

What it means: POSTMARK_SERVER_TOKEN is absent or empty on the dyno that attempted an email send. Waitlist confirmation emails are being silently skipped.

Remediation for skipped_no_postmark_token

1. Verify the config var on prod: bash heroku config:get POSTMARK_SERVER_TOKEN --app raxx-api-prod | wc -c # Expect > 5. If 0 or 1: the token is unset.

2. If unset, set it (silence stdout — token must not appear in terminal history): bash heroku config:set POSTMARK_SERVER_TOKEN="$POSTMARK_SERVER_TOKEN" \ --app raxx-api-prod >/dev/null 2>&1 Full procedure: docs/ops/runbooks/postmark.md → "Setting POSTMARK_SERVER_TOKEN on Heroku".

3. Verify: bash heroku run --app raxx-api-prod \ python -c "from api.services.postmark_client import test_postmark_token; test_postmark_token()" # Expect: ✓ Postmark token valid server=<ServerName> id=<ServerID>

4. Resolve the Sentry issue only after verifying the fix is deployed and the smoke passes.

Adding new alert rules

1. Define the rule parameters (condition, filter, action, frequency).
2. Add an entry to the "Managed rules" table above.
3. Extend scripts/ops/create_sentry_alert_rule.py with a new _build_<rule>_payload() function.
4. Add tests to scripts/ops/tests/test_create_sentry_alert_rule.py.
5. Run the script to create the rule; record the Sentry rule ID in this table.

Escalation

Wake the operator when: - The alert rule is missing (script exits non-zero or GET returns empty rules list). - Sentry API returns 401 — the SENTRY_API token may have expired or been revoked. - The skipped_no_postmark_token issue fires repeatedly after POSTMARK_SERVER_TOKEN has been set correctly (indicates a different skip path not covered by the alert filter).