Raxx · internal docs

internal · gated

Heroku support ticket — rate-limit cap increase (2026-05-14)

Date filed: 2026-05-14 (UTC) Status: Resolved — Heroku Support doubled cap to 9,000 req/hr same day Context: Follow-on to incident docs/incidents/2026-05-14-review-app-teardown-429.md

Ticket summary

Filed with Heroku Support to request a rate-limit cap increase after the 2026-05-14 review-app teardown incident caused 14 orphaned raxx-console-pr-* apps and a positive-feedback loop of API calls that exhausted the default 4,500 req/hr limit.

Problem description (filed text)

During a high-PR-close window (~8 PRs closing within 15 minutes), our GitHub Actions teardown workflow repeatedly called heroku apps:destroy after receiving HTTP 429 responses. Each retry added to the API call count, exhausting the rate-limit budget and causing all subsequent teardown jobs to fail immediately.

The root bug (missing retry logic in the workflow) has been fixed in our repository, but we request a temporary cap increase to provide headroom while we verify the fix in production. Current cap: 4,500 req/hr. Requested: 9,000 req/hr.

Resolution

Heroku Support confirmed the increase on 2026-05-14 ~12:00 UTC:

Current usage posture (as of 2026-05-14)

At typical CI throughput (15–25 PRs/day, each triggering ~3–5 Heroku API calls), peak API usage is estimated at 100–150 req/hr, which is approximately 1.5–2% of the new 9,000 req/hr ceiling.

If this rate increases significantly (e.g., many concurrent PRs during a sprint), consider:

  1. Batching teardown jobs with a 10s sleep between them via workflow concurrency (already in place: cancel-in-progress: false + group per PR).
  2. Adding exponential back-off to the retry loops (currently linear 30s).
  3. Filing another support ticket to raise the cap further.

References