Runbook: Cron Heartbeat Monitor

Issue #2723 (original). Issue #4531 (2026-08-23 rewrite — WP-native data source). RCA: docs/incidents/2026-05-23-ci-gitleaks-dark-crons-2703-2717.md

System: WP cron cron-heartbeat-monitor (WP cron ID 29) Owner: sre-agent / operator Last incident: 2026-08-23 — see docs/ops/runbooks/ci-woodpecker.md §Cron inventory and issue #4531 (permanent false STALE against retired GH Actions history — fixed by moving the data source to the WP API). Last reviewed: 2026-08-23

What this monitor does

The cron-heartbeat-monitor WP pipeline (.woodpecker/cron-heartbeat-monitor.yaml) runs at 06:30 UTC daily and checks, via scripts/ci/check_cron_heartbeats.py, that billing-retention-cron (WP cron 10), trace-integrity-cron (WP cron 9) and the two Queue billing schedulers each have a successful WP pipeline within their SLA window:

WP cron	Schedule	SLA
`billing-retention-cron`	`0 3 * * *` (nightly)	48h
`trace-integrity-cron`	`0 2 * * 0` (weekly, Sundays)	216h (7-day cadence + 2-day buffer)
`billing-trial-reminders-cron`	`0 * * * *` (hourly)	3h (1h cadence + 2h buffer)
`billing-dunning-downgrade-cron`	`0 5 * * *` (daily, 05:00 UTC)	26h (24h cadence + 2h buffer)

The two Queue billing schedulers (#4831, #4892) are registered by an operator (.woodpecker/billing-trial-reminders-cron.yaml and .woodpecker/billing-dunning-downgrade-cron.yaml carry the exact call). Until a cron is registered there is no pipeline to find, so the monitor reports it stale: register both at the same time as the PR that adds them merges. A green heartbeat means the scheduler ran; both jobs are 200 no-ops on the Queue side while their flags are off, so it does not mean emails went out.

If any cron is stale, the monitor: 1. Prints a per-cron summary to stdout (always). 2. Sends a Postmark alert to ops@raxx.app with the cron name, last success timestamp, and age — when POSTMARK_SERVER_TOKEN is provisioned (see "Known gap" below; as of 2026-08-23 it is not, so delivery is log-only). 3. Exits 1, which fails the WP pipeline step. There is no || true wrapper — a real STALE result correctly turns the pipeline red and surfaces via the daily ci-digest-cron rollup (digest-appropriate cadence, no live paging pre-launch — see feedback_pre_launch_digest_notifications.md).

Data source (rewritten 2026-08-23, #4531)

Before: queried the GitHub Actions API (GET /repos/{repo}/actions/ workflows/{file}.yml/runs) for billing-retention-cron.yml and trace-integrity-cron.yml. Both were fully migrated to WP-native crons on 2026-07-08 and their GH Actions counterparts retired to .github/workflows-retired/. The GH Actions API therefore always returned zero runs — the monitor reported permanent false STALE, so a real staleness incident would have been indistinguishable from this permanent false state.

After: queries the Woodpecker API directly — GET /api/repos/1/pipelines?page=N — and filters on each pipeline's cron field (confirmed live, 2026-08-23: the WP server populates this field with the exact name of the cron that triggered the pipeline). Walks pages newest-to-oldest until a success is found or a cost bound (--max-pages, default 10 pages / ~500 pipelines) is reached. See the module docstring in scripts/ci/check_cron_heartbeats.py for the full design rationale, including why trace-integrity-cron's SLA is 216h (cadence-aware) rather than the original blanket 48h.

Known gap: `WOODPECKER_API_TOKEN` not provisioned

As of 2026-08-23, the WOODPECKER_API_TOKEN Infisical secret at /MooseQuest/woodpecker/ is not provisioned. This was discovered while fixing #4531: a live deploy-failure-monitor pipeline log (that cron already reads the same secret, same path) shows the vault-loader's "key not found ... skipping" warning, followed by a WP API 401 Unauthorized and a masked-success exit — a sibling instance of the same detection-gap class this runbook's monitor now avoids.

Effect on cron-heartbeat-monitor: with the token missing, the rewritten check correctly fails loud — check_cron_heartbeats.py exits 2 with MONITOR ERROR: WOODPECKER_API_TOKEN not provisioned ... — instead of silently reporting "all clear." This is the expected, correct behavior of the fix, not a new bug. Do not "fix" this by wrapping the step in || true again; that reintroduces #4531.

Operator action required before this monitor can produce a real (non-error) result: provision WOODPECKER_API_TOKEN in Infisical at /MooseQuest/woodpecker/ — a Woodpecker personal/API token with read access to GET /api/repos/1/pipelines. deploy-failure-monitor.yaml shares the same gap and will start working from the same fix.

Enabling the monitor

The check is gated behind FLAG_CRON_HEARTBEAT_MONITOR, set directly in the WP step's environment: block (.woodpecker/cron-heartbeat-monitor.yaml) — not a GitHub Actions repo variable (that mechanism was retired along with the GH Actions workflow). To disable the check without disabling the cron itself, set FLAG_CRON_HEARTBEAT_MONITOR: "0" in the YAML.

To enable/disable the WP cron itself:

WP_TOKEN="<admin JWT, from SSM /ci/woodpecker/admin-api-token>"
curl -sf -X PATCH -H "Authorization: Bearer $WP_TOKEN" -H "Content-Type: application/json" \
  -d '{"enabled":true}' \
  "https://ci.moosequest.net/api/repos/1/cron/29"

Per Failure Mode K (docs/ops/runbooks/ci-woodpecker.md), always re-verify via a real scoped trigger and read the actual step log before flipping enabled:true — do not trust a stale PR's "fixed and confirmed working" claim alone.

Alert: `[Raxx ops] cron heartbeat STALE`

Subject: [Raxx ops] cron heartbeat STALE: <cron-name>

Immediate steps

Open the linked WP pipeline URL in the alert email (or the check step's own log if delivery is currently log-only per the known gap above).
Check the stale cron's own WP pipeline history: bash WP_TOKEN="<admin JWT, from SSM /ci/woodpecker/admin-api-token>" curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \ "https://ci.moosequest.net/api/repos/1/pipelines?page=1" \ | python3 -c " import json, sys data = json.load(sys.stdin) for p in data: if p.get('cron') in ('billing-retention-cron', 'trace-integrity-cron'): print(p['number'], p['status'], p['cron'], p.get('finished')) "
Look for: - Failed pipelines (real reconciler error — sweeper endpoint returned non-200, or integrity check returned fail_count > 0). - The cron disabled entirely (GET /api/repos/1/cron → enabled: false for that cron's ID) — schedule was turned off. - No matching pipelines at all within the lookback — cron never fired, or was deleted/re-registered under a different name.

If the cron is failing

Check the endpoint health directly:

# billing-retention-cron
curl -s -X POST \
  -H "X-Admin-Service-Token: $ADMIN_SERVICE_TOKEN" \
  -H "CF-Access-Client-Id: $CF_ACCESS_CLIENT_ID" \
  -H "CF-Access-Client-Secret: $CF_ACCESS_CLIENT_SECRET" \
  https://api.raxx.app/api/internal/jobs/billing-retention?dry_run=true

# trace-integrity-cron
curl -s -X POST \
  -H "X-Admin-Service-Token: $ADMIN_SERVICE_TOKEN" \
  -H "CF-Access-Client-Id: $CF_ACCESS_CLIENT_ID" \
  -H "CF-Access-Client-Secret: $CF_ACCESS_CLIENT_SECRET" \
  https://api.raxx.app/api/internal/jobs/trace-integrity-check?dry_run=true&window_hours=24

If the cron is disabled or never fired

WP_TOKEN="<admin JWT, from SSM /ci/woodpecker/admin-api-token>"
curl -sf -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/cron"
# Confirm billing-retention-cron (id 10) / trace-integrity-cron (id 9)
# both show enabled: true and a sane next_exec.

After restoring the cron

Trigger a scoped manual run to confirm it succeeds (never a bare manual dispatch — see Failure Mode K, docs/ops/runbooks/ci-woodpecker.md):

WP_TOKEN="<admin JWT, from SSM /ci/woodpecker/admin-api-token>"
curl -sf -X POST -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/cron/10"   # billing-retention-cron
curl -sf -X POST -H "Authorization: Bearer $WP_TOKEN" -H "User-Agent: sre-agent/1.0" \
  "https://ci.moosequest.net/api/repos/1/cron/9"    # trace-integrity-cron

Then trigger the heartbeat monitor itself (cron 29) and read the check step's log to confirm the stale result clears.

Alert: `[Raxx ops] cron heartbeat MONITOR ERROR — detection gap`

The monitor job itself could not complete its check. This is a detection gap — the health of billing-retention-cron and trace-integrity-cron is UNKNOWN. As of 2026-08-23 the most likely (expected) cause is the WOODPECKER_API_TOKEN gap documented above.

Read the check step's own log — the error message names the exact cause (missing token, WP API HTTP error, network failure).
If WOODPECKER_API_TOKEN is the cause: this is expected until an operator provisions the secret (see "Known gap" above) — not a new bug.
If a WP API HTTP error (401/403/5xx): verify the token is still valid and has not been rotated/revoked; check https://ci.moosequest.net is reachable and CF Access allows the request (CF_ACCESS_CLIENT_ID / CF_ACCESS_CLIENT_SECRET present in the step's vault-loaded env).

Testing the monitor locally

# Simulate a stale-cron scenario (override SLA to something small so any
# real recent success still reads as stale):
FLAG_CRON_HEARTBEAT_MONITOR=1 \
  python3 scripts/ci/check_cron_heartbeats.py \
  --wp-token "$WOODPECKER_API_TOKEN" \
  --sla-hours 1 \
  --output-file /tmp/heartbeat_result.json

cat /tmp/heartbeat_result.json

References

scripts/ci/check_cron_heartbeats.py — module docstring has the full #4531 rewrite rationale, per-cron SLA reasoning, and pagination/cost bound design.
docs/ops/runbooks/ci-woodpecker.md §Cron inventory (crons 9, 10, 29) and §Failure mode K (never trust a stale "fixed and confirmed working" claim — always re-verify via a scoped trigger + real log read before enabled:true).