Raxx · internal docs

internal · gated

ADR-0142 — Woodpecker CI High Availability: Option D

Status: Accepted
Date: 2026-07-14
Author: sre-agent
Operator authorization: Kristerpher Henderson, sole superadmin, MooseQuest LLC / Raxx
Cards: #4141 (IaC), #4142 (config-svc cutover)
Implements: Option D from the HA option analysis below
Runbook: docs/ops/runbooks/woodpecker-ha.md

Context

Woodpecker CI server (i-082ee835595d90ae0) ran as a standalone EC2 instance (t4g.small, ARM64 Graviton3) in a single private subnet (us-east-2a). It had no auto-healing, no automated failover, and no standby. Two availability gaps were observed:

  1. No server auto-healing. If the EC2 instance crashed or was terminated by AWS (spot? unlikely for on-demand t4g.small, but hardware failure remains), CI would remain down until an operator manually launched a replacement and reconfigured it.

  2. Single-AZ RDS. The PostgreSQL database (ci-woodpecker, db.t4g.micro) ran in a single AZ. An AZ-level AWS failure would take down both the server and the database simultaneously with no automated recovery path.

In addition, two reliability issues needed hardening:

  1. WOODPECKER_EXTENSIONS_ALLOWED_HOSTS not set. Setting WOODPECKER_CONFIG_SERVICE_ENDPOINT (required for ADR-0137 active mode) without allowlisting the host hard-fails every pipeline at config-load with no forge-fallback. The config-svc is co-located at localhost:8095; the allowlist must be baked in before the cutover.

  2. No Prometheus metrics endpoint. WOODPECKER_METRICS_SERVER_ADDR was not set, so the ci-monitoring Prometheus instance had no scrape target for WP-native metrics (woodpecker_pending_pipelines, woodpecker_workers, etc.), making WoodpeckerNoWorkersWithBacklog inert.

HA Options Considered

Option A — NLB in front of two EC2 servers

Two active Woodpecker servers behind an NLB. Woodpecker does not support multi-server active-active; this would require an upstream patch or a session-affinity workaround. Cost: ~$16/mo NLB + second EC2 ~$14/mo = +$30/mo.

Rejected: Woodpecker v3 does not support distributed server mode. Multi-server deployments cause split-brain on pipeline state.

Option B — litestream SQLite + S3 replication, single server

Replace RDS with SQLite + litestream (streaming WAL to S3). Lower cost; RPO is the S3 WAL lag (typically <1s). RTO for restore is minutes (litestream restore from S3). Server is still a single point of failure.

Rejected: Downgrade from managed RDS to operator-managed backup/restore loop. Does not address server single-point-of-failure. Current RDS cost ($13/mo) is already inside the $107/mo CI budget envelope.

Option C — Managed Kubernetes (EKS Fargate)

Move Woodpecker server to EKS Fargate with pod auto-restart. Eliminates the EC2 management layer.

Rejected: Overengineered for a single-server CI workload. EKS control plane adds ~$72/mo. Fargate pricing for a 0.5 vCPU / 1 GB pod ≈ $15/mo. Total delta: +$87/mo, nearly doubling the CI budget. ADR-0134 explicitly chose EC2 for cost and operational simplicity.

Option D — Harden Woodpecker + add HA (SELECTED)

Four targeted improvements on the existing architecture:

  1. Bake WOODPECKER_EXTENSIONS_ALLOWED_HOSTS into the server user_data template (non-disruptive; takes effect on next server restart/replacement).
  2. Enable WOODPECKER_METRICS_SERVER_ADDR=:9001 in the server compose config (non-disruptive; activates the Prometheus scrape target for ci-monitoring).
  3. RDS Multi-AZ — flip multi_az = var.db_multi_az (staged; ~60-120s RDS restart to provision standby, RPO≈0 after).
  4. Server ASG-of-1 — convert standalone EC2 to ASG-of-1 spanning us-east-2a + us-east-2b, ELB health check, automatic instance replacement on EC2 failure (~3-5min RTO). Staged: resources provisioned at count=0 initially; operator runs cutover procedure.

Selected: Additive, non-disruptive where possible, staged where disruptive. Stays within existing $107/mo budget. No architecture changes to the agent fleet or the RDS schema.

Decision

Implement Option D. Two changes are non-disruptive (apply immediately); two are staged (operator-gated apply with maintenance window announcement).

Immediately applicable

Change File Effect
WOODPECKER_EXTENSIONS_ALLOWED_HOSTS=localhost infra/ci/templates/user_data_server.sh.tpl Takes effect on next server restart; prerequisite for ADR-0137 #4142 active cutover
WOODPECKER_METRICS_SERVER_ADDR=:9001 infra/ci/templates/user_data_server.sh.tpl Activates Prometheus scrape endpoint
CloudWatch alarms (ALB TG healthy hosts, server ASG, RDS failover EventBridge) infra/ci/cloudwatch.tf Immediate alerting improvement
SSM /ci/woodpecker/server-host parameter infra/ci/ssm.tf Placeholder provisioned; populated during ASG cutover
Agent IAM: add server-host SSM read infra/ci/iam.tf Harmless until ASG mode; required for agent boot in ASG mode
Monitoring CloudWatch IAM + datasource terraform/ci-monitoring/iam.tf, cloud-init.sh.tpl Grafana CloudWatch panels for HA dashboard
HA Grafana dashboard terraform/ci-monitoring/templates/cloud-init.sh.tpl Visibility into ALB TG, ASG-of-1, RDS health

Staged (operator-gated)

Change Variable How to apply Expected disruption
RDS Multi-AZ db_multi_az = true Announce CI maintenance window (~5 min). Set var, terraform apply. ~60-120s RDS restart during standby provisioning
Server ASG-of-1 server_use_asg = true Full cutover procedure in runbook §Server ASG-of-1 cutover ~3-5min CI downtime during TG attachment cutover

Failure mode guarantee table

Failure Auto-heal? RTO (time to recovery) RPO (data loss)
RDS AZ failure (Multi-AZ enabled) Yes ~60-120s automated failover ~0 (synchronous standby replication)
Server EC2 crash / AWS hardware failure (ASG-of-1 enabled) Yes ~3-5min (ASG launches replacement, waits for health check) 0 (state in RDS)
AZ outage (both server + RDS in same AZ) Yes — server ASG places in other AZ; RDS Multi-AZ promotes standby ~2-4min ~0
Spot agent reclaim Yes — ASG replaces agent Near-zero (in-flight build re-runs after reconnect; ~1-2min) 0 (pipeline state in RDS)
Self-inflicted config error (bad compose env) No — server replaced but bad config re-applied ~5min to detect + rollback via SSM Run Command 0
RDS AZ failure (Multi-AZ NOT yet enabled) No Operator-initiated failover; minutes to hours depending on page time 0 to last backup window
AWS region outage No Out of scope; no cross-region DR Variable

What is NOT guaranteed

Consequences

Positive

Negative / constraints

References