GitHub Actions Audit — 2026-06-05

Author: sre-agent Date: 2026-06-05 Status: Recommendation-only — no workflow files modified Operator approval required before any Phase execution

§1 Inventory Table

Total workflows on disk: 72 (including __tests__/ subdirectory with one shell test — not a workflow file). Active registered workflows: 71. All are in active state per the GitHub API. Run data sampled from the most recent 500 completed runs as of 2026-06-05T14:00 UTC.

Duration estimates use createdAt → updatedAt delta on successful runs (best available via gh run list — does not include queue wait time, which is typically 5–15s on the hosted runners).

Runs-per-month for cron workflows derived from cron expression parsing. PR/push trigger run-counts are derived from observed activity in the sample window. The sample window covers roughly 7–10 days of activity, so "runs last 30 days" for frequently-triggered workflows is extrapolated (noted where extrapolated).

Minute-cost rounding: GH Actions rounds every job to the nearest minute, minimum 1 minute. A 9-second job that runs 96 times/month still costs 96 minutes.

Estimated total minutes/month (current state): ~3,400–3,700 min/month

The three highest single contributors: 1. queue-zero-dyno-monitor: ~624 min/month (every-15-min schedule, always runs, 13s per run → rounds to 1 min × 2,880 fires) 2. surface-degraded-auto-file: ~1,200 min/month (every-15-min, 25s per run, but currently failing 100%) 3. mbt-resting-orders-cron: ~344 min/month (every-5-min weekday trading hours)

§2 Categorization

Critical path — must run on every PR or push to main

Path-filtered — correct, keep

These workflows trigger only when relevant paths change and behave correctly:

deploy-antlers-next-prod.yml, deploy-antlers-next-staging.yml, deploy-console.yml, deploy-velvet.yml, deploy-queue.yml, deploy-getraxx.yml, deploy-internal-docs.yml, deploy-customer-docs.yml, deploy-flag-docs.yml, deploy-mockups.yml, deploy-status-page.yml, deploy-status-worker.yml, deploy-support.yml, deploy-console-shim.yml, deploy-worker-waf-log-shipper.yml, lint-cf-access-headers.yml, lint-cf-tokens.yml, lint-cf-pages-deploy-uniqueness.yml, lint-workflow-secret-names.yml, review-app-console.yml, test-notify-deploy-status.yml

Scheduled — keep (firing meaningful output)

Dispatch-only — runs on demand, zero recurring cost

These fire zero recurring cycles. Keep all; they are on-demand tools.

billing-collector-cron.yml (explicitly paused), deploy-antlers-cutover.yml, freescout-apply.yml, launch-readiness-check.yml, sc-ident-6-smoke.yml, test-cloudflare-retry-action.yml, test-emit-audit-event-action.yml

Always-skip / dead triggers

Actively broken — 100% failure rate

These have produced zero successes in the entire observed sample window and are consuming minutes without output:

§3 Cycle-Waster Findings

CW-1: surface-degraded-auto-file running every 15 minutes 100% broken

File: console-degraded-auto-file.yml Cron: */15 * * * * — 2,880 fires/month Cost: ~1,200 min/month (each 25-second run rounds to 1 minute × 2,880) Failure cause: DOMException [DataError]: Invalid keyData / Failed to read private key — the GitHub App private key stored as a secret is malformed or expired. Impact: Zero monitoring output for the entire observable window. The workflow is consuming ~1,200 minutes per month for no operational value. Operator action before Phase 1 can fix: Rotate/re-provision the GitHub App private key secret. Once functional, consider reducing frequency to every 30 minutes (halves cost with no meaningful detection latency change).

CW-2: release.yml firing on every push to main, 0% success rate

File: release.yml Trigger: push to main, no path filter Cost: ~54 min/month (72s avg × ~45 pushes/month) Failure cause: release-please failed: illegal pathing characters in path: frontend/trademaster_ui/../../VERSION — the release-please configuration references a path component from the legacy CRA layout that has since been superseded. The release-please-config.json almost certainly contains a reference to frontend/trademaster_ui with a relative ../../VERSION extra-component path. Impact: Every push to main produces a failed workflow run. This is visible noise that trains reviewers to ignore red workflow icons, which is a detection-erosion risk. Fix: Update release-please-config.json to remove or correct the frontend/trademaster_ui component path. This is a one-line config change, not a workflow change.

CW-3: queue-zero-dyno-monitor running every 15 minutes, healthy but high-frequency

File: queue-zero-dyno-monitor.yml Cron: */15 * * * * — 2,880 fires/month Cost: ~624 min/month (13s avg × 2,880, each rounds to 1 min) Output: 100% success, legitimately monitoring Queue dyno count. Issue: 15-minute frequency is fine for production SLO coverage, but this is pre-launch. The Queue service is not yet customer-facing. Detecting zero dynos within 15 minutes vs within 30 minutes makes no difference at current stage. Proposed change: Relax to */30 * * * * — saves ~312 min/month with zero customer-facing impact.

CW-4: flag-drift-check running every 4 hours

File: flag-drift-check.yml Cron: 0 */4 * * * — 180 fires/month Cost: ~96 min/month Output: 100% success, checks flag YAML drift. Issue: Flag changes happen via PRs (a human action). A flag drift state that exists at 04:00 UTC will still exist at 08:00 UTC. Daily detection is sufficient. Proposed change: Change to 0 6 * * * (daily) — saves ~66 min/month.

CW-5: ci.yml (CI — main) has no path filter, runs full suite on every push

File: ci.yml Trigger: push to main, no path filter Cost: ~138 min/month (183s avg × ~45 pushes/month) Issue: Runs backend tests, frontend tests, security scans, OpenAPI drift on every push to main regardless of what changed. The detect-changes job gates individual test jobs internally, but the workflow still starts a runner and the detect-changes job runs on every push. For pushes that only touch docs/** or mockups-site/**, the entire workflow spins up only to skip everything after detect. Note: The detect-changes pattern is correct for what it does. The improvement is adding a top-level paths filter to skip workflow startup entirely for doc-only or design-only changes. The detect-changes internal logic already handles the rest. Proposed change: Add paths filter excluding docs/**, mockups-site/**, *.md changes.

CW-6: deploy-heroku.yml fires on every push to main with no path filter

File: deploy-heroku.yml Trigger: push to main, no path filter Cost: ~120 min/month (159s avg × ~45 pushes/month) Issue: This workflow deploys Raptor (the backend) to Heroku, but it fires on every push to main — including pushes that only touch frontend/raxx-next/**, docs/**, mockups-site/**, etc. A frontend-only push should not trigger a backend deploy. Proposed change: Add path filter for backend_v2/**, requirements*.txt, Procfile, runtime.txt, .github/workflows/deploy-heroku.yml. This is a careful change — verify against the freeze-check and smoke job gates.

CW-7: slack-notify.yml workflow_run trigger name mismatch — never fires

File: slack-notify.yml Trigger: workflow_run on workflows ['CI'] Issue: The CI workflow is named 'CI — main', not 'CI'. GitHub's workflow_run trigger matches on exact workflow name. This workflow has never fired in the observable sample. The CI failure notification posture (per feedback_pre_launch_digest_notifications) relies on this workflow for failure alerts — but it is silently dead. Fix: Change workflows: ['CI'] to workflows: ['CI — main'] in slack-notify.yml.

CW-8: deploy-antlers.yml (Legacy CRA) active but paths point to superseded code

File: deploy-antlers.yml Name: Deploy Antlers (LEGACY CRA — PAUSED 2026-06-01) State: active (GH Actions API reports active despite the name containing PAUSED) Trigger: push/main on frontend/trademaster_ui/** Issue: The CRA has been superseded by Antlers Next.js. The path frontend/trademaster_ui/** still exists (it is the current Antlers app directory per project_antlers_directory). If any commit touches files in that path, this legacy workflow will fire and attempt a CRA deploy. The name says PAUSED but no mechanism enforces the pause except that the path has low traffic. Risk: Not zero-cost in perpetuity. Any touch of frontend/trademaster_ui/** fires this. Recommend disabling (state=disabled_manually) rather than deleting, as the deployment script may be referenced.

CW-9: Multiple monitor/infra workflows broken 100% with no operator visibility

The following workflows have been silently failing for the entire observable window (2–16 consecutive failures, no successes):

• billing-retention-cron.yml — Postgres migration dependency
• trace-integrity-cron.yml — root cause unclear
• waf-synthetic-probe.yml — root cause unclear
• historical-bars-1min-warm-nightly.yml — likely Postgres/Heroku auth
• reminders-second-factor-nightly.yml — likely Postgres/Heroku auth
• e2e-smoke.yml — new workflow, never succeeded
• tickets-e2e-smoke.yml — never succeeded

Because slack-notify.yml is broken (CW-7), none of these failures have generated Slack alerts. The cron-heartbeat-monitor (cron-heartbeat-monitor.yml) may be detecting some of these — but its output is not visible from this audit.

§4 Phased Reduction Plan

Phase 1 — Safe wins (no workflow file changes, low blast radius)

These can be approved individually. None require coordinated changes across multiple files.

Phase 1 total estimated savings if all items land: ~430 min/month (dominated by fixing surface-degraded-auto-file so it runs correctly but at the same frequency, and reducing zero-dyno-monitor).

Phase 1 total if surface-degraded-auto-file also moves to 30-min after key fix: ~630 min/month.

Phase 2 — Consolidation (workflow file changes, needs testing)

These require changes to workflow YAML and should be done in a single PR with a one-day soak on staging.

Phase 2 estimated additional savings: ~90–120 min/month (path filters + getraxx fix + CF lint consolidation).

Phase 3 — Production-lean shape (architectural changes, operator sign-off required on design)

Phase 3 estimated additional savings: ~150–200 min/month (primarily from removing push-triggered prod deploys that become dispatch-only, and fixing the E2E PR trigger).

§5 Staging Autodeploy + Production-Lean Architecture

Current state per surface

Surface          Staging autodeploy?    Prod autodeploy?    Manual gate?
---------------------------------------------------------------------------
Raptor (API)     YES (push to main,     YES (same workflow,  freeze-check
                 no path filter)        no manual confirm)   on dispatch=prod
Antlers Next.js  YES (push to main,     YES (separate        none
                 raxx-next/** paths)    prod workflow,
                                        same push trigger)
Console          YES (push to main,     YES (same workflow,  freeze-check
                 console/** paths)      no manual confirm)   on dispatch=prod
Velvet           YES (push to main,     YES (same workflow,  freeze-check
                 velvet/** paths)       no manual confirm)
Queue            YES (push to main,     YES (same workflow,  confirm-gate job
                 queue/** paths)        confirm-gate-rejected with manual input)
getraxx          YES (push to main)     (CF Pages — same     none
                                        CF Pages deploy)
Antlers Next     YES (staging           SEPARATE prod        none
                 workflow auto)         workflow auto)

Proposed lean-prod shape (after Phase 3)

Push to main
     |
     v
[Path filters route to correct deploy workflow]
     |
     +---> deploy-raptor-staging   (automatic, on push)
     +---> deploy-antlers-staging  (automatic, on push)
     +---> deploy-console-staging  (automatic, on push)
     +---> deploy-velvet-staging   (automatic, on push)
     +---> deploy-queue-staging    (automatic, on push)
     |
     v
[Staging smoke: synthetic-gate + surface health check]
     |
     v
[Soak period — default: next business day (no automated timer)]
     |
     v
[Operator: dispatch promote-to-prod with surface input]
     |
     v
promote-to-prod.yml
     |
     +---> Verify: latest staging deploy = green
     +---> Run: pre-promote smoke (30s)
     +---> Deploy to production
     +---> Post-deploy: synthetic-gate on prod
     +---> Notify: ops@ digest

What stays the same

• Staging: fast, automatic, push-triggered. No friction.
• PR gate: unchanged — comprehensive, path-aware.
• Scheduled crons: unchanged except frequency adjustments in Phase 1.
• CF Pages surfaces (getraxx, mockups, status-page, customer-docs): autodeploy is correct for these; no customer-critical data at risk.

What changes in Phase 3

• Raptor, Console, Antlers, Velvet, Queue production deploys become workflow_dispatch-only.
• A new promote-to-prod.yml consolidates the confirmation UI.
• The deploy-heroku.yml no longer has a push trigger for production.
• The existing freeze-check and synthetic-gate jobs remain as the backbone.

§6 Estimated Minutes/Month Saved Per Phase

At GH Actions Pro rates ($0.008/min for Ubuntu runners), the current estimated burn is $27–30/month in compute alone, before the base Pro plan cost. All three phases landing brings that to roughly $20–23/month — a ~$7–10/month reduction in compute.

Ubicloud trigger threshold is 5,000 min/month per project_ci_billing. Current estimated usage is 3,400–3,700 min/month, which is 68–74% of that threshold. Even in the worst case (all phases fail to reduce), the repo is not at imminent risk of hitting the Ubicloud threshold. If surface-degraded-auto-file is fixed and remains at */15 frequency, it alone accounts for 1,200 min/month of a single workflow. Fixing its root cause is the single highest-leverage action.

§7 Actively Broken Workflows — Surfacing Separately

The following workflows have been producing zero successes for the entire observable window and should be investigated as separate SRE dispatches (not in this audit's scope to fix):

BROKEN-1: release.yml — 18/18 failures

Cause: release-please-config.json contains stale path reference frontend/trademaster_ui/../../VERSION. Fix: update release-please config. Separately: Node.js 20 deprecation warning on googleapis/release-please-action@5c625bfb5d1ff62eadeeb3772007f7f66fdcf071 — this action is pinned by SHA to a version that runs on Node.js 20; GitHub Actions will force Node.js 24 starting 2026-06-16. This is an imminent breaking change, 11 days away.

BROKEN-2: console-degraded-auto-file.yml — 16/16 failures

Cause: Failed to read private key / Invalid keyData. The GitHub App private key secret is malformed or has been rotated without updating the secret. Monitoring is completely blind.

BROKEN-3: heroku-config-health-nightly.yml — real findings (not broken, but alerting correctly)

Note: This workflow is functioning correctly — it is producing failures because it found real issues: raxx-api-staging is missing REDIS_URL (severity=critical) and has an empty POSTMARK_SERVER_TOKEN (severity=warning). These are legitimate infra gaps, not workflow bugs. Operator action needed: provision REDIS_URL on raxx-api-staging.

BROKEN-4: deploy-internal-docs.yml — recurring failures

Cause: emit-audit-event/action.yml uses github.run_id and github.server_url in a composite action template context where the github context is not available. This is a shared composite action bug that may affect other workflows using emit-audit-event.

BROKEN-5: Multiple nightly crons failing (likely Postgres migration dependency)

billing-retention-cron, historical-bars-1min-warm-nightly, reminders-second-factor-nightly, trace-integrity-cron — likely all failing due to the SQLite→Postgres migration in progress (project_raptor_postgres_migration_decision). These should be paused (schedule commented out, dispatch-only) until the migration lands, matching the pattern already established by billing-collector-cron.yml.

BROKEN-6: Node.js 20 deprecation — deadline 2026-06-16 (11 days)

Affects: actions/checkout@v4, actions/setup-python@v5, actions/create-github-app-token@v1, googleapis/release-please-action (pinned SHA). GitHub will force Node.js 24 after 2026-06-16. Actions that haven't published Node.js 24-compatible versions will break. The most urgent is googleapis/release-please-action pinned to a specific SHA — check if a newer SHA that supports Node 24 is available.

§8 Operator-Approval Checklist

For each proposed action, check the box to greenlight:

Phase 1 (safe wins — low risk)

[ ] P1-A  slack-notify.yml: fix workflow_run name 'CI' → 'CI — main'
          Restores Slack CI-failure alerting. Zero behavioral change otherwise.

[ ] P1-B  deploy-antlers.yml: disable (not delete) via GH API
          Prevents accidental CRA deploy. File stays on disk for reference.

[ ] P1-C  release-please-config.json: remove stale frontend/trademaster_ui component
          Stops 100% Release failure rate. Requires review of config to confirm
          which components are still active.

[ ] P1-D  Rotate GitHub App private key for surface-degraded-auto-file
          (OPERATOR ACTION: requires secret provisioning in repo settings)
          Restores console degradation monitoring.

[ ] P1-E  queue-zero-dyno-monitor: relax cron */15 → */30
          Saves ~312 min/month. Detection window: 15 min → 30 min.

[ ] P1-F  flag-drift-check: relax cron every 4h → daily 06:00 UTC
          Saves ~66 min/month. Flag changes are human-driven via PRs.

Phase 2 (consolidation — moderate risk, test before merge)

[ ] P2-A  ci.yml: add paths filter to skip on doc/design-only pushes
          Needs verification that detect-changes still fires for all code paths.

[ ] P2-B  deploy-heroku.yml: add backend-only path filter
          Needs verification all legitimate backend paths are covered.
          Confirm with operator: are there any infra-file paths outside
          backend_v2/ that should trigger a Raptor deploy?

[ ] P2-C  deploy-getraxx.yml: add npx playwright install to smoke job
          Fixes 4/4 failure rate. Low risk — additive step only.

[ ] P2-D  Fix emit-audit-event composite action github context usage
          Review .github/actions/emit-audit-event/action.yml; pass run_id
          and server_url as explicit inputs from calling workflows.

[ ] P2-E  Consolidate 3 CF lint workflows into 1
          New file replaces 3 files. Identical trigger/path logic, 3 jobs.
          Confirm: are any external processes monitoring these specific
          workflow names?

[ ] P2-F  billing-retention-cron: pause schedule (comment out), dispatch-only
          Mirror billing-collector-cron pattern. Re-enable post-Postgres migration.

Phase 3 (architectural — requires separate operator sign-off on design)

[ ] P3-DESIGN  Operator approves staging-autodeploy / prod-dispatch model
               described in §5 before any Phase 3 workflow changes begin.
               Specifically: confirm all 5 Heroku surfaces (Raptor/Console/
               Antlers/Velvet/Queue) should move to dispatch-only prod deploys.

[ ] P3-A  Introduce per-surface staging smoke prerequisite for prod
[ ] P3-B  Convert prod deploy triggers to workflow_dispatch-only
[ ] P3-C  Create promote-to-prod.yml consolidation workflow
[ ] P3-D  Consolidate 06:00 UTC cron cluster into nightly-ops-sweep
[ ] P3-E  Remove e2e-smoke.yml from PR trigger until suite is green

§9 Ubicloud Assessment

Current estimated burn: ~3,400–3,700 min/month. Ubicloud threshold: 5,000 min/month (per project_ci_billing).

Current usage is at ~70–74% of the threshold. Not at imminent risk. The two highest-cost workflows (surface-degraded-auto-file at ~1,200 min/month when broken, queue-zero-dyno-monitor at ~624 min/month, mbt-resting-orders-cron at ~344 min/month) are all running correctly except the degraded-auto-file. Fixing Phase 1 alone brings the estimate down to ~2,800–3,100 min/month (56–62% of threshold).

Recommendation: Ubicloud is not needed in the near term. If a significant new service launches (iOS CI running full Xcode builds, for instance) or if the E2E suite starts running on every PR and takes 10+ minutes, revisit. Do not queue Ubicloud migration until burn sustains above 4,500 min/month for two consecutive months.

References

• docs/ops/runbooks/ci-hygiene.md — existing CI hygiene runbook
• docs/ops/runbooks/ci-runner-posture.md — runner posture notes
• feedback_gh_actions_transitive_skip — needs chain skip propagation
• feedback_pre_launch_digest_notifications — pre-launch digest posture
• feedback_asset_manifest_layer_a — asset manifest guard (keep!)
• project_ci_billing — GH Actions Pro plan + Ubicloud threshold
• project_raptor_postgres_migration_decision — explains several cron failures