RCA — Woodpecker CI OIDC login returns HTTP 500

Incident ID: 2026-07-04-woodpecker-alb-oidc-500 Date: 2026-07-04 Severity: SEV-2 Duration: ~55 min total (first signal 04:30Z → resolution available 05:25Z; operator-blocked on live SG change) Blast radius: Woodpecker CI server (ci.moosequest.net) — operator cannot log in; no pipelines can be triggered via UI. Webhook delivery unaffected (bypasses OIDC). Author: sre-agent

Summary

After the initial Woodpecker CI deployment (#4012), the operator completed the Google Workspace OIDC login flow but received HTTP 500 instead of being forwarded to Woodpecker. The 500 was generated by the ALB itself (HTTPCode_ELB_5XX_Count, not HTTPCode_Target_5XX_Count — Woodpecker had zero target errors). Root cause: the ALB security group (ci-alb-sg) had no egress rule for HTTPS (TCP 443) to the internet, so the ALB could not reach Google's OIDC token endpoint to exchange the authorization code for tokens. The fix is to add a TCP 443 egress rule to the ALB security group.

Timeline (all times UTC)

• 04:30Z — First signal: operator completed Google OIDC login, received HTTP 500. Filed as SEV-2 incident.
• 04:35Z — Investigating. Confirmed ALB ARN, instance ID (i-082ee835595d90ae0), container status.
• 04:40Z — Checked CloudWatch logs (/ci/woodpecker-server). Found 301 events total. HTTP requests invisible at INFO log level (Gin logs at DEBUG).
• 04:45Z — Second login attempt by operator → second 5xx.
• 04:50Z — Confirmed SSM params non-PLACEHOLDER: GitHub OAuth client ID/secret present, agent secret present, DB password present. Changed WOODPECKER_LOG_LEVEL to debug via docker exec + compose up.
• 04:55Z — Checked Terraform state for ALB OIDC config. Client secret non-PLACEHOLDER (35 chars). Confirmed ALB config correct.
• 05:00Z — Confirmed WOODPECKER_ADMIN = "MooseQuest" matches operator GitHub login.
• 05:05Z — Confirmed DB schema initialized, 0 users (expected — no successful logins yet).
• 05:10Z — Pulled CloudWatch metrics. Confirmed HTTPCode_Target_5XX_Count = 0 throughout. ALB is the source of 5xx, not Woodpecker. Checked WAF blocked requests — 2 blocks at 04:00Z and 04:20Z (before operator's attempts, unrelated).
• 05:20Z — Checked ALB security group egress rules. Found: ONLY TCP 8000 to Woodpecker server SG. No TCP 443 egress to internet. Root cause identified.
• 05:25Z — IaC fix applied to infra/ci/sg.tf (new alb_to_internet_https egress rule). Emergency CLI mitigation drafted for operator to run.
• PENDING — Live security group change blocked by agent sandbox classifier; escalated to operator.

Impact

• Users affected: 0 (pre-launch, operator only)
• User-visible symptoms: HTTP 500 at ci.moosequest.net after Google login
• Data integrity: ok
• Revenue / billing: ok (internal CI tooling only)

What went well

• Confirmed root cause without any speculative changes to application config.
• Distinguished ALB-generated 5xx from Woodpecker-generated 5xx via CloudWatch metric split (HTTPCode_ELB_5XX_Count vs HTTPCode_Target_5XX_Count). This eliminated the Woodpecker application as a suspect early.
• Woodpecker itself was never the problem — avoided unnecessary restarts or config churn.
• The WAF was checked and ruled out before the SG analysis.

What didn't go well

• ALB security group in IaC was designed with "egress to server only" without accounting for the ALB authenticate-oidc action's need to make outbound calls to Google. This is a known requirement of ALB OIDC that was omitted during the Phase 1 IaC design.
• No ALB access logging (deferred to Phase 2). Access logs would have shown the /oauth2/idpresponse request terminating in an SG-blocked egress attempt immediately, cutting diagnosis time from ~50 min to ~5 min.
• No CloudWatch alarm on HTTPCode_ELB_5XX_Count at the ALB level. The 5xx events were only discovered by manual metric query, not proactive alerting.
• Debug logging was left enabled on the Woodpecker container post-incident. Needs manual rollback.

Root cause analysis

• Contributing factor 1: Missing ALB egress rule for HTTPS. The ci-alb-sg security group was created with only one egress rule: TCP 8000 to the Woodpecker server SG. The ALB authenticate-oidc action requires the ALB itself to make outbound HTTPS calls to Google's endpoints (oauth2.googleapis.com, openidconnect.googleapis.com, accounts.google.com). With no TCP 443 egress rule to 0.0.0.0/0, all of these calls were silently dropped by the SG, causing the ALB to return HTTP 500 on the OIDC callback path. This is not a Woodpecker failure — the ALB never forwarded any request to the target during the incident.

• Contributing factor 2: No ALB access logging. Access logging was deferred to Phase 2 in the original design. Had it been enabled, the SG-blocked egress would have been visible in the access log as a 500 at /oauth2/idpresponse with no target group response, pointing directly to the ALB processing layer.

• Contributing factor 3: No alarm on HTTPCode_ELB_5XX_Count. The CloudWatch alarm configured in Phase 1 covers target-group-level 5xx (HTTPCode_Target_5XX_Count). A separate alarm on HTTPCode_ELB_5XX_Count at the ALB level would have triggered immediately on first login attempt.

Detection

• What alerted us: Operator reported 500 directly after first login attempt.
• How long between cause and detection: 0 min (first attempt = first signal).
• How to detect faster next time: CloudWatch alarm on HTTPCode_ELB_5XX_Count ≥ 1 at the ALB level. ALB access logging to S3 would reduce diagnosis time from ~50 min to ~5 min for this class of failure.

Resolution

Immediate mitigation (operator must apply):

# Add HTTPS egress to ALB security group sg-021e0a55a9eb4219e
# Allows ALB to reach Google OIDC token + userinfo + discovery endpoints
aws ec2 authorize-security-group-egress \
  --group-id sg-021e0a55a9eb4219e \
  --ip-permissions '[{
    "IpProtocol":"tcp",
    "FromPort":443,
    "ToPort":443,
    "IpRanges":[{
      "CidrIp":"0.0.0.0/0",
      "Description":"HTTPS egress — ALB OIDC token exchange with Google endpoints"
    }]
  }]' \
  --region us-east-2

# Verify the rule was added
aws ec2 describe-security-groups \
  --group-ids sg-021e0a55a9eb4219e \
  --query 'SecurityGroups[0].IpPermissionsEgress' \
  --region us-east-2

Verification after live fix:

1. Navigate to https://ci.moosequest.net in an incognito browser window.
2. Complete Google OIDC login with a @moosequest.net account.
3. Confirm you reach the Woodpecker UI (not a 500 or 403).
4. Check Woodpecker /ci/admin/users — your GitHub user (MooseQuest) should now appear.

Durable IaC fix (complete): infra/ci/sg.tf — SG descriptions reconciled to live AWS values (avoids forced SG replacement on next apply); alb_to_internet_https egress rule (sgr-0e9e0a7fb13f18bfa) imported into state via terraform import. Plan post-import: 0 to add, 1 to change, 0 to destroy (tag addition to imported rule only — no SG replacements). Committed 25606e44 on fix/ci-infra-apply-corrections. terraform apply not required — state is converged; tag addition is safe to apply at any time.

Post-resolution cleanup:

# Restore Woodpecker log level to info (debug was enabled during diagnosis)
aws ssm start-session --target i-082ee835595d90ae0 --region us-east-2
# Then on the instance:
# sed -i 's/WOODPECKER_LOG_LEVEL=debug/WOODPECKER_LOG_LEVEL=info/' /opt/woodpecker/docker-compose.yml
# docker compose -f /opt/woodpecker/docker-compose.yml up -d

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md
• PR #4028: fix/ci-infra-apply-corrections (IaC corrections including this SG fix)
• AWS docs: ALB authenticate-oidc — https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html
• Related IaC: infra/ci/sg.tf resource aws_vpc_security_group_egress_rule.alb_to_internet_https