RCA — console.raxx.app /billing H12 (Postgres connection-pool exhaustion)

Incident ID: 2026-07-24-console-billing-h12-db-pool-exhaustion Date: 2026-07-24 Severity: SEV-2 (user-facing 503 on /billing; short-lived, contained by manual mitigation) Duration: immediate mitigation (WEB_CONCURRENCY 4→2) applied out-of-band by operator; durable code fix landed same day Blast radius: console.raxx.app (Heroku app raxx-console-prod), specifically the /billing blueprint and any other route served by a worker that was mid-request when its DB connection request could not be satisfied Author: feature-developer

Summary

raxx-console-prod runs Gunicorn with WEB_CONCURRENCY=4 worker processes. console/app/__init__.py set no SQLALCHEMY_ENGINE_OPTIONS, so SQLAlchemy's un-tuned defaults applied: pool_size=5 + max_overflow=10, i.e. up to 15 possible connections per worker. Each Gunicorn worker owns its own SQLAlchemy engine/pool (they do not share a pool across processes), so the dyno as a whole could demand up to 4 workers x 15 = 60 connections against heroku-postgresql:essential-0, which hard-caps total connections at 20 -- 3x the plan's ceiling.

A polling/traffic burst pushed enough concurrent DB-bound requests that the pool(s) could not satisfy new connection checkouts. The affected worker process(es) blocked past Gunicorn's 30s WORKER TIMEOUT and were SIGKILL'd mid-request, producing a Heroku router H12 (request timeout) for at least one in-flight /billing request.

Immediate mitigation (operator, out-of-band): WEB_CONCURRENCY dropped 4->2 on raxx-console-prod, roughly halving the worst-case connection demand (4 workers x 15 = 60 -> 2 workers x 15 = 30). This reduces exposure but does not eliminate it -- 30 still exceeds the 20-connection cap, and any future bump back to WEB_CONCURRENCY=4 (or higher, e.g. a Heroku dyno-type change) would silently reopen the same failure mode. It is a mitigation, not a fix.

Durable fix (this PR): console/app/__init__.py now sets explicit, bounded SQLALCHEMY_ENGINE_OPTIONS -- pool_size=2 + max_overflow=3 (5 connections/worker), pool_pre_ping=True (drop dead connections instead of erroring on use), and pool_recycle=300 (recycle idle connections before a Postgres-side idle timeout can silently kill them). At WEB_CONCURRENCY=4 (the pre-mitigation setting), worst case is 4 x 5 = 20 -- exactly at the essential-0 cap, never over it, regardless of how WEB_CONCURRENCY is set in the future. The pool_size/max_overflow values are only applied to the postgres backend (env-tunable via DB_POOL_SIZE / DB_MAX_OVERFLOW) -- sqlite (local/dev/CI) is unaffected, since sqlite's default pool class for :memory: URLs does not accept those kwargs.

This is the same failure class as a prior runbook reference to docs/incidents/2026-05-07-console-prod-worker-thrash.md -- that RCA document was referenced by docs/ops/runbooks/console-prod-h12-alerts.md (as the incident that motivated H12/WORKER TIMEOUT Slack alerting, #1345) but was never actually written; no such file exists in docs/incidents/. Given the identical symptom signature (H12 + WORKER TIMEOUT on raxx-console-prod), it is likely the 2026-05-07 event was the same unbounded-pool failure mode recurring, caught that time by the #1345 alerting before this permanent fix was made. The dead reference has been corrected in the runbook update accompanying this PR.

Timeline (all times UTC)

• 2026-05-07 (suspected, unconfirmed) -- an earlier H12/WORKER TIMEOUT event on raxx-console-prod prompted the #1345 Heroku-log-drain Slack alerting build-out. The RCA for that event was never written, so its root cause was never formally confirmed -- but the symptom signature matches this incident exactly, and no other structural change to the DB pool occurred between the two dates.
• 2026-07-24 -- a polling/traffic burst against console.raxx.app exhausts the available Postgres connections across the 4 Gunicorn workers' independent pools.
• 2026-07-24 -- one or more workers block on connection checkout past Gunicorn's 30s timeout; SIGKILL'd. Heroku router logs H12 for at least one in-flight /billing request (user-visible 503).
• 2026-07-24 -- operator applies immediate mitigation: WEB_CONCURRENCY 4->2 on raxx-console-prod (out-of-band, via heroku config:set).
• 2026-07-24 -- root cause confirmed: no SQLALCHEMY_ENGINE_OPTIONS set, SQLAlchemy defaults (pool_size=5, max_overflow=10) per worker vs. essential-0's 20-connection hard cap.
• 2026-07-24 -- durable fix (bounded, env-tunable engine options) implemented and tested in this PR.

Impact

• Users affected: any operator/authenticated session hitting /billing (or any other DB-bound route on the affected worker) during the burst window.
• User-visible symptoms: HTTP 503 (H12 request timeout at the Heroku router) for at least one /billing request.
• Data integrity: ok -- no partial writes; the failure mode is a connection checkout timeout before any query executes, not a mid-transaction failure.
• Revenue / billing: ok -- /billing here is the Console operator billing dashboard (internal), not the customer-facing payment/subscription path.

What went well

• The immediate mitigation (WEB_CONCURRENCY 4->2) was correctly identified and applied quickly, containing the blast radius while the durable fix was developed.
• Root cause was confirmed with simple arithmetic (workers x per-worker pool ceiling vs. plan's connection cap) -- no extended diagnosis needed once the Gunicorn worker-count x SQLAlchemy-pool-defaults relationship was considered.
• The existing #1345 H12/WORKER TIMEOUT Slack alerting caught this promptly (that infrastructure was already in place from the earlier, undocumented 2026-05-07 event).

What didn't go well

• No SQLALCHEMY_ENGINE_OPTIONS had ever been set on the Console Flask app, so SQLAlchemy's library defaults (tuned for a single-process app, not a multi-worker Gunicorn dyno) silently governed production connection behavior. Nothing enforced or even surfaced the workers x pool-ceiling vs. plan-cap relationship.
• The docs/incidents/2026-05-07-console-prod-worker-thrash.md RCA referenced by the H12 alerting runbook was never actually written. If it had been, this exact root cause (or evidence ruling it out) would likely have been on record 11 weeks earlier, and this recurrence may have been prevented outright.
• There was no leading-indicator alert for Postgres connection-count pressure itself -- only H12 (request timeout, a lagging symptom of pool exhaustion) and WORKER TIMEOUT (Gunicorn-level, also lagging). By the time either fires, requests have already failed.

Root cause analysis

• Contributing factor 1 -- unbounded per-worker connection pool: console/app/__init__.py set SQLALCHEMY_DATABASE_URI but never SQLALCHEMY_ENGINE_OPTIONS, so SQLAlchemy's defaults (pool_size=5, max_overflow=10) applied per Gunicorn worker process. With WEB_CONCURRENCY=4, worst-case demand was 4 x 15 = 60 connections against a plan hard-capped at 20 -- a 3x overshoot that only avoided constant failure because typical concurrent DB-bound request volume stayed well under the theoretical ceiling most of the time.
• Contributing factor 2 -- no leading-indicator monitoring: the existing #1345 alerting fires on H12 and WORKER TIMEOUT counts, both of which are symptoms after requests have already failed. There was no independent Postgres connection-count metric that would have shown the pool approaching saturation before user-visible failures began.
• Contributing factor 3 -- likely-undocumented recurrence: the dead RCA reference (docs/incidents/2026-05-07-console-prod-worker-thrash.md) suggests this failure class may have already occurred once before, 11 weeks prior, without ever being root-caused or durably fixed -- only alerted on.

Detection

• What alerted us: Heroku router H12 + Gunicorn WORKER TIMEOUT log lines, surfaced via the existing #1345 log-drain Slack alerting (docs/ops/runbooks/console-prod-h12-alerts.md).
• How long between cause and detection: effectively immediate -- the alert fires within the existing 5-minute (H12) / 2-minute (WORKER TIMEOUT) windows once the threshold is crossed.
• How to detect faster next time: add a Postgres connection-count leading indicator (see Action items) so the on-call signal is "approaching the cap" rather than "already 503-ing."

Resolution

• What was changed: 1. console/app/__init__.py -- SQLALCHEMY_ENGINE_OPTIONS now explicitly set: pool_pre_ping=True always; for the postgres backend only, pool_size (default 2, env DB_POOL_SIZE), max_overflow (default 3, env DB_MAX_OVERFLOW), pool_recycle (default 300s, env DB_POOL_RECYCLE_SECONDS). 2. Added pytest coverage (console/tests/test_smoke.py) asserting: (a) the sqlite/default path has no pool_size/max_overflow kwargs (avoids the TypeError sqlite's SingletonThreadPool would raise for :memory: URLs), (b) the postgres path gets the bounded defaults, (c) the env-tunable overrides work, and (d) the documented safety-margin math ((pool_size + max_overflow) * 4 <= 20) holds. 3. This RCA, and a runbook update to docs/ops/runbooks/console-prod-h12-alerts.md adding the root-cause + remediation section and fixing the dead 2026-05-07 RCA reference.
• Validation: full console/tests suite green (6572 passed, 3 pre-existing skips, 0 failures) after the change.
• Immediate mitigation (WEB_CONCURRENCY 4->2, applied out-of-band by the operator) remains in place independent of this code fix; the durable fix removes the need for that specific dyno-count constraint going forward (at WEB_CONCURRENCY=4 post-fix, worst case is exactly 20/20, not 60/20), but there was no instruction to revert the concurrency setting as part of this PR.

Action items

References

• Code fix: console/app/__init__.py (this PR)
• Test coverage: console/tests/test_smoke.py (this PR)
• Runbook: docs/ops/runbooks/console-prod-h12-alerts.md (updated, this PR)
• Referenced-but-never-written prior RCA (dead link, now corrected): docs/incidents/2026-05-07-console-prod-worker-thrash.md
• Heroku Postgres plan limits: https://devcenter.heroku.com/articles/heroku-postgres-plans
• SQLAlchemy engine/pool configuration: https://docs.sqlalchemy.org/en/20/core/pooling.html