RCA — Woodpecker CI v2.8.3 nil-pointer panic at repo activation; resolved by v3.16.0 upgrade

Incident ID: 2026-07-04-woodpecker-v3-upgrade Date: 2026-07-04 Severity: SEV-2 Duration: Ongoing from Phase 1 deploy until 2026-07-04T18:29Z (approx. 12h from first trigger attempt) Blast radius: Woodpecker CI only — raxx-app/TradeMasterAPI could not be activated; no pipelines could run. GitHub Actions CI continued unaffected. No user-facing impact (pre-launch). Author: sre-agent

Summary

The Woodpecker CI v2.8.3 server panicked at server/api/repo.go:149 whenever the operator attempted to activate raxx-app/TradeMasterAPI. The panic was a nil pointer dereference on the forge OAuth token associated with the admin user — a known v2.8.x bug where a freshly-provisioned admin user's forge token record can be nil. The fix was to upgrade the server and agent from v2.8.3 to v3.16.0, where the bug is fixed. The upgrade was executed in-place via SSM without downtime, with an RDS snapshot taken first as the rollback path. DB schema migrated automatically on v3.16.0 server start. After the upgrade, repo activation succeeded (HTTP 200, webhook ID 649361254) and all three Phase 2 pilot crons were registered and enabled via the API.

Timeline (all times UTC)

• 05:52Z — Server v2.8.3 started during Phase 1 deployment (#4012).
• 17:51Z — First panic trace logged: runtime error: invalid memory address or nil pointer dereference at server/api/repo.go:149 during attempted repo activation.
• 17:55Z — Second panic on retry.
• 18:00Z — Operator authorized upgrade to v3.16.0 (MAJOR version migration).
• 18:10Z — Migration guide reviewed (woodpecker-ci.org/migrations). Breaking change analysis complete — no env var renames required for current config.
• 18:16Z — RDS snapshot ci-woodpecker-pre-3x-upgrade initiated.
• 18:22Z — RDS snapshot status: available. Rollback path confirmed.
• 18:24Z — Server compose updated to v3.16.0 via SSM. docker compose up -d. DB migrations ran automatically (no errors).
• 18:24Z — Server confirmed running v3.16.0 (X-Woodpecker-Version: 3.16.0, container healthy).
• 18:29Z — Agent compose updated to v3.16.0 via SSM. Container force-recreated.
• 18:30Z — Agent confirmed connected to server: GET /api/agents returns agent v3.16.0 with recent last_contact.
• 18:31Z — Repo activation: POST /api/repos?forge_remote_id=1172174585 → HTTP 200, "active":true. No panic.
• 18:31Z — GitHub webhook confirmed: ID 649361254, active, events: push, pull_request, pull_request_review, deployment.
• 18:32Z — Crons created via POST /api/repos/1/cron (v3.x path — note: /cron singular, not /crons).
• 18:32Z — Crons enabled via PATCH /api/repos/1/cron/{id} with {"enabled":true}.
• 18:33Z — Manual pipeline triggered: POST /api/repos/1/pipelines {"branch":"develop"} → pipeline 1 created. Compiler error at L50 of .woodpecker/*.yaml — pipeline YAML needs v3.x migration (filed as action item 4). Infrastructure confirmed working; error is in application-level pipeline config.
• 18:34Z — ASG scaled to desired=0 (soak period). Soak start: 2026-07-04T18:34Z.
• 18:34Z — IaC PR initiated: infra/ci/variables.tf bumped to v3.16.0, runbook updated.

Impact

• Users affected: 0 (pre-launch)
• User-visible symptoms: none
• Data integrity: ok
• Revenue / billing: ok

What went well

• RDS snapshot taken before any change. If DB migration had failed, rollback was one restore command.
• Breaking change analysis was thorough — confirmed no env var renames needed before touching the server. No config drift introduced.
• SSM-only access worked as designed. No SSH required.
• The v2.8.3 server continued running during analysis. The upgrade window was under 10 minutes from snapshot-ready to v3.16.0 healthy.
• The v3.x agent container reconnects automatically after a brief gRPC disconnect — expected behavior, not an error.

What didn't go well

• The repo activation panic existed in v2.8.3 from the start of Phase 1. There was no pre-activation smoke test that would have caught this before Phase 2.
• The v3.x cron API path change (/crons → /cron) was not in the official migration guide — it required direct source code inspection to find. API compatibility was not verified before Phase 2 cron registration.
• The v3.x enabled: false default for new crons was not documented. Crons were silently inactive after creation until PATCH'd.
• SSM command JSON parameter escaping caused two failed attempts (sed | pipe operator misparse; Python quote nesting). Added toil to an otherwise straightforward tag bump.

Root cause analysis

• Contributing factor 1: v2.8.x nil-pointer bug in forge OAuth token handling (server/api/repo.go:149). The Woodpecker v2.8.3 repo-activation handler assumed the admin user's forge token record was non-nil. For a freshly-provisioned server where the admin has never completed the GitHub OAuth login flow via the Woodpecker UI, the forge token is nil. The code did not nil-check before dereferencing, causing a panic on every activation attempt. The system allowed Phase 1 to deploy without a pre-activation smoke test that would have caught this immediately.

• Contributing factor 2: No pre-activation smoke test in Phase 1. The Phase 1 smoke checklist included checking health and UI access, but not a test of repo activation via API. A single curl against /api/repos?forge_remote_id=... post-deploy would have surfaced the panic in minutes during Phase 1 rather than blocking Phase 2.

• Contributing factor 3: API path and behavior changes in v3.x not fully documented. The cron endpoint changed from /crons to /cron (singular), and new crons default to enabled: false. Neither change was in the official migration guide — they required source code inspection. The system (official docs) did not adequately communicate all API-level breaking changes.

Detection

• What alerted us: Operator triggered Phase 2 cron registration and encountered the panic.
• How long between cause and detection: ~12h (Phase 1 completed at ~05:52Z; first activation attempt at 17:51Z).
• How to detect faster next time: add a post-deploy smoke step to the runbook that calls POST /api/repos?forge_remote_id=<test-id> within 5 minutes of server start.

Resolution

• Upgraded server and agent from v2.8.3 to v3.16.0 in-place via SSM.
• RDS snapshot ci-woodpecker-pre-3x-upgrade taken before upgrade (status: available, no restore needed).
• DB schema auto-migrated cleanly on v3.16.0 startup.
• Repo raxx-app/TradeMasterAPI activated (repo ID = 1), webhook installed (GitHub webhook ID 649361254).
• Crons created and enabled: security-scan-nightly (id=1), alembic-version-check (id=2), maxmind-asn-refresh (id=3).
• IaC updated: infra/ci/variables.tf versions bumped to v3.16.0.
• Validation: /api/agents returns agent connected on v3.16.0. /api/version header = 3.16.0. /healthz = 200. ALB bypass paths intact.

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md
• RDS snapshot: ci-woodpecker-pre-3x-upgrade (available, us-east-2)
• GitHub webhook: ID 649361254 on raxx-app/TradeMasterAPI
• Woodpecker migration guide: https://woodpecker-ci.org/migrations
• Woodpecker v3.16.0 router source: github.com/woodpecker-ci/woodpecker@b75ee661 server/router/api.go