RCA — Woodpecker "Login to CI" buttons dead; GitHub OAuth flow unreachable

Incident ID: 2026-07-04-woodpecker-login-buttons-dead Date: 2026-07-04 Severity: SEV-2 Duration: Unknown start → 15:45 UTC resolution (operator-blocked for at least the duration of the shift) Blast radius: Single operator. No pipeline builds affected (agents healthy). No data at risk. Author: sre-agent

Summary

The Woodpecker CI login screen at https://ci.moosequest.net rendered correctly but both "Login to CI" and "Abort" buttons were completely unresponsive after clicking. The operator could pass the ALB Google-OIDC gate but could not initiate the Woodpecker GitHub OAuth login. Root cause: the ALB listener had no bypass rule for Woodpecker's own auth paths (/api/*, /authorize, /login, /logout), so the default OIDC rule intercepted the SPA's API calls and the GitHub OAuth callback, returning Google redirects instead of Woodpecker responses. Fix: added ALB listener rule at priority 25 forwarding those four path patterns directly to Woodpecker without OIDC; also reconciled the fix into infra/ci/alb.tf.

Timeline (all times UTC)

• ~07:00 — Operator attempts to log into ci.moosequest.net. Google OIDC passes. Woodpecker login screen appears but buttons are unresponsive.
• 15:27 — sre-agent engaged. Acknowledged SEV-2 (staging outage / CI platform login blocked).
• 15:27 — Captured ALB listener rules: confirmed default OIDC rule with no bypass for /api/*, /authorize, /login, /logout.
• 15:28 — curl probe confirmed: GET /api/version, /authorize, /login all 302 to accounts.google.com.
• 15:29 — SSM: pulled container env. WOODPECKER_HOST=https://ci.moosequest.net confirmed correct; root cause 2 (host mismatch) ruled out.
• 15:29 — CloudWatch log pull confirmed: zero non-health-check requests ever reached Woodpecker; HTTP access logs are at debug level (suppressed at info log level), confirming the OIDC gate was absorbing all browser traffic.
• 15:32 — DB read-only check: users table contains 1 row for login MooseQuest — a prior GitHub OAuth did complete; Woodpecker session state is intact.
• 15:38 — Applied live fix: aws elbv2 create-rule adding priority-25 rule for /api/*, /authorize, /login, /logout → forward without OIDC.
• 15:43 — Verification curls: /api/user → 401 JSON from Woodpecker; /authorize → 303 to github.com; /login → 200 from Woodpecker; /logout → 303 to /. All paths confirmed reaching Woodpecker.
• 15:45 — IaC patch committed to infra/ci-woodpecker-alb-oidc-bypass branch; PR opened against develop.

Impact

• Users affected: 1 (operator only; Woodpecker is not customer-facing)
• User-visible symptoms: CI login stuck; could not authenticate to Woodpecker
• Pipeline impact: none (builds that were queued before the incident continued; new builds blocked only because no authenticated user could trigger them)
• Data integrity: ok
• Revenue / billing: ok

What went well

• Root cause was fast to isolate (curl probe of four paths, all confirmed Google-redirecting, took under 3 minutes)
• Container env inspection via SSM confirmed WOODPECKER_HOST was correct, narrowing scope immediately
• CloudWatch log drain confirmed zero browser traffic reached Woodpecker, corroborating the ALB-level block
• Fix was a single, reversible aws elbv2 create-rule call with no downtime and immediate propagation

What didn't go well

• The Phase 1 IaC review did not include a test of the full Woodpecker GitHub OAuth login flow end-to-end; only health check + webhook delivery were validated
• The smoke checklist in the runbook did not include a step to verify that clicking "Login to CI" actually redirects to GitHub
• No monitoring exists on the ALB rule set — a destructive terraform apply that drops the priority-25 rule would silently break login again with no alert

Root cause analysis

• Contributing factor 1: Missing ALB bypass rules for Woodpecker's own auth endpoints. The Phase 1 ALB IaC correctly exempted /hook* (webhooks) and /api/badge* (public badges) from OIDC, but did not exempt Woodpecker's GitHub OAuth flow endpoints. When a user clicks "Login to CI", the Woodpecker SPA calls /api/* endpoints and navigates to /login → GitHub → /authorize (callback). All three paths hit the default authenticate-oidc ALB action. Because the OIDC session cookie was already present (the operator had just passed Google), the ALB did forward these requests — but the JavaScript fetch() call to initiate the GitHub OAuth flow received a cross-origin redirect chain that the browser's Fetch API cannot resolve to JSON, leaving loginUrl empty and the click handler a no-op. Additionally, without the bypass rule, any user whose OIDC session expires mid-flow would have the GitHub code and state discarded by a Google re-auth cycle.

• Contributing factor 2: No end-to-end login smoke test in the Phase 1 smoke checklist. The smoke checklist in the runbook verified that the Woodpecker dashboard loads after Google OIDC, but did not require a full Woodpecker GitHub OAuth login (click "Login to CI" → GitHub → authorize → verify Woodpecker session created). This class of failure — outer OIDC intercepting inner OAuth — would have been caught immediately by a headless or manual login smoke.

• Contributing factor 3: HTTP access logs suppressed by default log level. WOODPECKER_LOG_LEVEL=info suppresses Woodpecker's HTTP router debug logs. This is correct for steady-state operation, but it meant CloudWatch logs showed only health checks — no evidence of browser requests. A responder without curl access would have had no signal that requests were not reaching Woodpecker at all. Diagnosis depended on active curl probing rather than passive log inspection.

Detection

• What alerted us: operator reported directly
• How long between cause and detection: approximately 8 hours (container restarted 05:52 UTC; operator engaged sre-agent ~15:27 UTC)
• How to detect faster next time: synthetic login smoke on a schedule; alert if ALB rules drop below 4 non-default entries; alert if /api/user through the ALB returns a non-4xx/non-2xx

Resolution

• What was changed: added ALB HTTPS listener rule at priority 25, path patterns [/api/*, /authorize, /login, /logout], action forward to ci-woodpecker-server-tg, no OIDC action. Rule ARN: arn:aws:elasticloadbalancing:us-east-2:521228113048:listener-rule/app/ci-woodpecker-alb/2bf78b6ff4a0780f/2b0157eb96f64e49/7dbc47a59f1be899.
• IaC: infra/ci/alb.tf patched with aws_lb_listener_rule.woodpecker_own_auth resource; runbook updated with failure mode F and new architecture diagram.
• Validation: curl probes confirmed all four paths return Woodpecker responses (not Google redirects); MooseQuest user row confirmed present in DB.

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md
• IaC change: infra/ci/alb.tf (resource aws_lb_listener_rule.woodpecker_own_auth)
• Live rule ARN: arn:aws:elasticloadbalancing:us-east-2:521228113048:listener-rule/app/ci-woodpecker-alb/2bf78b6ff4a0780f/2b0157eb96f64e49/7dbc47a59f1be899
• Related: ADR-0134, PR #4012 (Phase 1 baseline)