Woodpecker CI HA runbook

System: Woodpecker CI server + RDS (Option D hardening, ADR-0136)
Owner: operator / sre-agent
Last reviewed: 2026-07-14
ADR: docs/architecture/adr/0136-woodpecker-ha-option-d.md
Related runbook: docs/ops/runbooks/ci-woodpecker.md

Architecture summary

Internet
  │
  ▼
ALB (ci-woodpecker-alb) — us-east-2, public subnets a+b
  │  /healthz:8000 health check, 30s interval
  ▼
TG (ci-woodpecker-server-tg) — port 8000
  │
  ├─ [Static mode, server_use_asg=false] aws_instance.woodpecker_server (us-east-2a)
  │
  └─ [ASG mode, server_use_asg=true]    aws_autoscaling_group.woodpecker_server
                                          min=1 max=1, spans us-east-2a + us-east-2b
                                          ELB health check, 180s grace period
  │
  ▼
RDS PostgreSQL 15 (ci-woodpecker, db.t4g.micro)
  ├─ [db_multi_az=false] single AZ (current)
  └─ [db_multi_az=true]  Multi-AZ, synchronous standby in second AZ

Agents: ci-woodpecker-agents-asg — min=1 (on-demand floor), max=10
          x86_64 spot + on-demand, private subnets a+b
          Agent-to-server gRPC on port 9000 (direct private IP or SSM discovery)

Failure mode guarantee table

What is NOT guaranteed

• Zero-downtime during ASG server replacement (in-flight builds re-run; ~3-5min).
• Zero-downtime during RDS Multi-AZ failover (~60-120s blip).
• Self-hosted SLA — no vendor SLA covers the CI orchestration layer.
• Cross-region DR — us-east-2 region failure takes down all CI.

How to tell it's broken

Diagnosis steps (in order)

1. Check ALB TG health: aws elbv2 describe-target-health \ --target-group-arn $(aws elbv2 describe-target-groups \ --names ci-woodpecker-server-tg \ --query 'TargetGroups[0].TargetGroupArn' --output text \ --region us-east-2) \ --region us-east-2 Expected: at least one target with State.State = "healthy".

2. Check server instance / ASG:

Static mode (server_use_asg=false): aws ec2 describe-instance-status \ --instance-ids i-082ee835595d90ae0 \ --region us-east-2

ASG mode (server_use_asg=true): aws autoscaling describe-auto-scaling-groups \ --auto-scaling-group-names ci-woodpecker-server-asg \ --region us-east-2 \ --query 'AutoScalingGroups[0].{Desired:DesiredCapacity,InService:Instances[?LifecycleState==`InService`]|length(@)}'

3. Check CloudWatch logs: aws logs filter-log-events \ --log-group-name /ci/woodpecker-server \ --log-stream-name-prefix server \ --start-time $(($(date +%s) - 1800))000 \ --region us-east-2 \ --filter-pattern "ERROR" \ | jq '.events[].message'

4. Shell into server via SSM Session Manager: INSTANCE_ID=$(aws autoscaling describe-auto-scaling-groups \ --auto-scaling-group-names ci-woodpecker-server-asg \ --region us-east-2 \ --query 'AutoScalingGroups[0].Instances[?LifecycleState==`InService`].InstanceId' \ --output text) aws ssm start-session --target "$INSTANCE_ID" --region us-east-2 Then: cd /opt/woodpecker && docker compose ps && docker compose logs --tail=100

5. Check RDS status: aws rds describe-db-instances \ --db-instance-identifier ci-woodpecker \ --region us-east-2 \ --query 'DBInstances[0].{Status:DBInstanceStatus,MultiAZ:MultiAZ,SecondaryAZ:SecondaryAvailabilityZone}' Expected: "Status": "available". During failover: "Status": "failing-over" (normal; wait ~60-120s).

Known failure modes

Failure mode A: Server container crashed, instance healthy

Symptom: ALB TG unhealthy, CloudWatch alarm ci-alb-no-healthy-targets. EC2 instance is Running/2-2. docker compose ps shows woodpecker-server Exited.

Cause: OOM, unhandled panic, or bad config. The restart: unless-stopped policy restarts automatically; if it keeps crashing, the container enters a restart-backoff loop.

Fix:

# On server via SSM session
cd /opt/woodpecker
docker compose logs woodpecker-server --tail=200
docker compose restart woodpecker-server
# If crash-looping: check .env for missing/empty vars
docker compose config

Verification: curl -sf http://localhost:8000/healthz returns HTTP 200 on server; ALB TG health check recovers.

Failure mode B: Server EC2 instance terminated / hardware failure (ASG mode)

Symptom: ci-server-asg-no-healthy-instance CloudWatch alarm ALARM. ALB TG healthy host count = 0. ci-alb-no-healthy-targets fires.

Expected behavior (ASG-of-1 enabled): ASG detects failed ELB health check, terminates instance, launches replacement. Expected RTO: ~3-5min.

How to verify recovery:

# Watch ASG activity
aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name ci-woodpecker-server-asg \
  --region us-east-2 \
  --max-items 5

# Watch TG health
watch -n 10 'aws elbv2 describe-target-health \
  --target-group-arn <TG_ARN> \
  --region us-east-2'

After recovery: Update SSM /ci/woodpecker/server-host with the new instance's private IP:

NEW_IP=$(aws ec2 describe-instances \
  --filters "Name=tag:aws:autoscaling:groupName,Values=ci-woodpecker-server-asg" \
            "Name=instance-state-name,Values=running" \
  --query 'Reservations[0].Instances[0].PrivateIpAddress' \
  --output text --region us-east-2)

aws ssm put-parameter \
  --name /ci/woodpecker/server-host \
  --type SecureString \
  --value "$NEW_IP" \
  --overwrite \
  --region us-east-2
echo "Updated server-host SSM to: $NEW_IP"

Agents pick up the new IP on next boot (spot replacement) or next connection attempt.

Failure mode C: RDS Multi-AZ failover in progress

Symptom: SNS email "RDS FAILOVER ALERT — DB: ci-woodpecker". Woodpecker server logs show database connection errors. ALB TG may briefly show unhealthy if server can't reconnect fast enough.

Expected behavior: RDS promotes the standby replica in the second AZ. Woodpecker server reconnects automatically via the CNAME endpoint (does not change). Total blip: ~60-120s.

How to verify:

aws rds describe-events \
  --source-identifier ci-woodpecker \
  --source-type db-instance \
  --duration 60 \
  --region us-east-2 \
  --query 'Events[].{Time:Date,Message:Message}'

Action: Wait 120s. If the server does not recover, run docker compose restart woodpecker-server via SSM session to force a reconnect cycle.

Failure mode D: Agents not connecting after server replacement

Symptom: WoodpeckerNoWorkersWithBacklog alert fires. Pipelines pending, zero workers on WP UI.

Cause (ASG mode): Server's private IP changed after ASG replacement. SSM /ci/woodpecker/server-host has stale IP. New agent instances boot, fail to resolve server, exit (boot script exits with error).

Fix: 1. Update SSM param (see Failure mode B, "After recovery" step above). 2. Manually terminate stale agent instances; ASG replaces them with the correct config: bash aws autoscaling set-desired-capacity \ --auto-scaling-group-name ci-woodpecker-agents-asg \ --desired-capacity 0 \ --region us-east-2 # Wait 60s for instances to terminate aws autoscaling set-desired-capacity \ --auto-scaling-group-name ci-woodpecker-agents-asg \ --desired-capacity 1 \ --region us-east-2

Failure mode E: Self-inflicted config error

Symptom: Server container fails to start after a compose or env change. docker compose logs woodpecker-server shows config/startup error.

Fix — rollback via SSM Run Command:

INSTANCE_ID=$(aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=ci-woodpecker-server" \
            "Name=instance-state-name,Values=running" \
  --query 'Reservations[0].Instances[0].InstanceId' \
  --output text --region us-east-2)

aws ssm send-command \
  --instance-ids "$INSTANCE_ID" \
  --document-name "AWS-RunShellScript" \
  --parameters '{"commands":["cd /opt/woodpecker && git diff -- docker-compose.yml && git checkout -- docker-compose.yml && docker compose up -d"]}' \
  --region us-east-2

Staged rollout procedures

RDS Multi-AZ cutover

Prerequisite: Announce CI maintenance window (~5 min) at ops@raxx.app.

Duration: ~60-120s RDS restart. CI will be down while server loses DB connection.

Steps:

1. Set variable in infra/ci/terraform.tfvars: db_multi_az = true

2. Plan and review: bash cd infra/ci terraform plan -var-file=terraform.tfvars | grep -E "^(~|#|-|Plan)" Expected: ~ aws_db_instance.woodpecker: multi_az = false -> true. No destroy.

3. Apply during maintenance window: bash terraform apply -var-file=terraform.tfvars -auto-approve

4. Monitor RDS status: bash watch -n 10 'aws rds describe-db-instances \ --db-instance-identifier ci-woodpecker \ --region us-east-2 \ --query "DBInstances[0].{Status:DBInstanceStatus,MultiAZ:MultiAZ,SecondaryAZ:SecondaryAvailabilityZone}"' Expected: Status transitions modifying → available. MultiAZ: true. SecondaryAvailabilityZone: us-east-2b.

5. Verify CI health: curl -sf https://ci.moosequest.net/healthz

6. Post-apply: Confirm ci-rds-failover-event EventBridge rule is ENABLED in AWS console.

Server ASG-of-1 cutover

Prerequisite: Announce CI maintenance window (~5 min). All in-flight builds will be re-run.

Duration: ~3-5min CI downtime.

Steps:

1. Scale agents to floor only (avoid builds during transition): bash aws autoscaling set-desired-capacity \ --auto-scaling-group-name ci-woodpecker-agents-asg \ --desired-capacity 1 \ --region us-east-2

2. Set variable in infra/ci/terraform.tfvars: server_use_asg = true

3. Plan: bash cd infra/ci terraform plan -var-file=terraform.tfvars Expected: + aws_launch_template.woodpecker_server_asg[0], + aws_autoscaling_group.woodpecker_server[0], + aws_autoscaling_attachment.woodpecker_server[0]. No destroy of existing aws_instance.woodpecker_server.

4. Apply (creates ASG resources alongside existing instance): bash terraform apply -var-file=terraform.tfvars -auto-approve

5. Watch ASG spin up: bash watch -n 10 'aws autoscaling describe-auto-scaling-groups \ --auto-scaling-group-names ci-woodpecker-server-asg \ --region us-east-2 \ --query "AutoScalingGroups[0].{Desired:DesiredCapacity,InService:Instances[?LifecycleState==\`InService\`]|length(@)}"' Wait until InService=1.

6. Get new server IP and update SSM: bash NEW_IP=$(aws ec2 describe-instances \ --filters "Name=tag:aws:autoscaling:groupName,Values=ci-woodpecker-server-asg" \ "Name=instance-state-name,Values=running" \ --query 'Reservations[0].Instances[0].PrivateIpAddress' \ --output text --region us-east-2) aws ssm put-parameter \ --name /ci/woodpecker/server-host \ --type SecureString \ --value "$NEW_IP" \ --overwrite --region us-east-2 echo "SSM /ci/woodpecker/server-host = $NEW_IP"

7. Remove old standalone instance from ALB TG (prevents dual-server split-brain): bash # Remove the direct TG attachment from Terraform state terraform state rm aws_lb_target_group_attachment.woodpecker_server # Deregister from TG directly (immediate effect) OLD_TG_ARN=$(aws elbv2 describe-target-groups \ --names ci-woodpecker-server-tg \ --query 'TargetGroups[0].TargetGroupArn' --output text --region us-east-2) aws elbv2 deregister-targets \ --target-group-arn "$OLD_TG_ARN" \ --targets Id=i-082ee835595d90ae0 \ --region us-east-2

8. Terminate old standalone instance and remove from Terraform state: bash terraform state rm aws_instance.woodpecker_server aws ec2 terminate-instances --instance-ids i-082ee835595d90ae0 --region us-east-2

9. Remove old standalone instance resource from infra/ci/server.tf and outputs.tf in a follow-up PR (cleanup after cutover is verified).

10. Verify CI health: bash curl -sf https://ci.moosequest.net/healthz # Trigger a test pipeline push or manual pipeline

11. Verify agents connect: bash # Check Woodpecker UI: https://ci.moosequest.net/woodpecker (after OIDC auth) # Workers should show > 0 connected agents

Failover drill procedure

Run this drill quarterly or after any major infra change to verify the auto-healing path is operational.

Drill A: Server instance termination drill

Goal: Verify ASG-of-1 replaces the server within 5min and CI recovers.

Prerequisites: server_use_asg=true applied. No builds running (check WP UI first).

Steps:

1. Note current server instance ID: bash CURRENT_INSTANCE=$(aws autoscaling describe-auto-scaling-groups \ --auto-scaling-group-names ci-woodpecker-server-asg \ --region us-east-2 \ --query 'AutoScalingGroups[0].Instances[0].InstanceId' \ --output text) echo "Terminating: $CURRENT_INSTANCE"

2. Record start time (UTC): bash START=$(date -u +%Y-%m-%dT%H:%M:%SZ) echo "Drill start: $START"

3. Terminate the instance: bash aws ec2 terminate-instances --instance-ids "$CURRENT_INSTANCE" --region us-east-2

4. Watch ASG replace and ALB TG recover: bash # Watch in one terminal watch -n 5 'aws elbv2 describe-target-health \ --target-group-arn <TG_ARN> --region us-east-2 \ | jq ".TargetHealthDescriptions[].TargetHealth.State"'

5. Record recovery time (when TG shows healthy): bash END=$(date -u +%Y-%m-%dT%H:%M:%SZ) echo "Drill end: $END (recovery)"

6. Update SSM server-host param with new instance IP (see Failure mode B above).

7. Verify agents reconnect and pipeline can run.

Pass criteria: TG recovers to healthy within 5min. Agents reconnect. Test pipeline succeeds.

Record: Document drill date, actual RTO, and any deviations in docs/incidents/ as a drill report (SEV-4).

Drill B: RDS failover drill

Goal: Verify RDS Multi-AZ failover completes within 2min and server reconnects.

Prerequisites: db_multi_az=true applied. Check aws rds describe-db-instances shows MultiAZ: true.

Steps:

1. Initiate failover (AWS-managed; no data lost): bash aws rds reboot-db-instance \ --db-instance-identifier ci-woodpecker \ --force-failover \ --region us-east-2

2. Watch RDS events: bash watch -n 10 'aws rds describe-events \ --source-identifier ci-woodpecker \ --source-type db-instance \ --duration 5 \ --region us-east-2 \ --query "Events[-5:].{Time:Date,Message:Message}"'

3. Confirm ALB TG recovers (server reconnects after ~60-120s blip): bash watch -n 10 'aws elbv2 describe-target-health \ --target-group-arn <TG_ARN> --region us-east-2'

Pass criteria: RDS status transitions failing-over → available within 2min. ALB TG returns to healthy. SNS email received. No data loss confirmed (run test pipeline after).

Emergency stop

To take Woodpecker CI offline cleanly (e.g., security incident, runaway agent spend):

# Scale agents to zero (immediate: terminates spot burst, drains floor)
aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --min-size 0 --desired-capacity 0 \
  --region us-east-2

# Stop server container without terminating instance (faster restart)
INSTANCE_ID=<current_server_instance_id>
aws ssm send-command \
  --instance-ids "$INSTANCE_ID" \
  --document-name "AWS-RunShellScript" \
  --parameters '{"commands":["cd /opt/woodpecker && docker compose down"]}' \
  --region us-east-2

To restart:

# Restart server
aws ssm send-command \
  --instance-ids "$INSTANCE_ID" \
  --document-name "AWS-RunShellScript" \
  --parameters '{"commands":["cd /opt/woodpecker && docker compose up -d"]}' \
  --region us-east-2

# Restore agent floor
aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name ci-woodpecker-agents-asg \
  --min-size 1 --desired-capacity 1 \
  --region us-east-2

Escalation

Escalate to operator when: - RDS Multi-AZ failover does not complete within 5min. - Server ASG-of-1 does not replace within 10min (LT or AMI issue may require operator to update variables and re-apply). - Any data integrity concern with the Woodpecker database. - Cost anomaly from excessive ASG scaling.

Contact: ops@raxx.app

References

• ADR-0136: docs/architecture/adr/0136-woodpecker-ha-option-d.md
• CI Woodpecker runbook: docs/ops/runbooks/ci-woodpecker.md
• Memory: feedback_wp_config_service_needs_allowed_hosts
• Grafana HA dashboard: ci-ha-health (uid) — accessible via SSM port-forward on port 3000