Raxx · internal docs

internal · gated

ADR-0137 — Woodpecker external config service: O(1) pipeline config fetch with SHA cache

Status: Proposed Date: 2026-07-10 UTC Deciders: Kristerpher (operator); software-architect (design) Scope: Woodpecker CI server on EC2 i-082ee835595d90ae0 (us-east-2, acct 521228113048); new wp-config-svc process co-located on the same host Refs: ADR-0134 (self-hosted EC2 decision); ADR-0135 (WP migration phases 3–5); docs/incidents/2026-07-09-wp-forge-context-deadline.md (incident that drove this); Failure mode B-2 in docs/ops/runbooks/ci-woodpecker.md

Context

On 2026-07-09, the Woodpecker CI server's Go HTTP connection pool accumulated stale half-open TCP connections after ~5 days of uptime. This caused context deadline exceeded errors when the server tried to fetch pipeline config from GitHub, blocking all pipeline creation for ~45 minutes and delaying the v1.10.3 production deploy.

The mitigation — raising WOODPECKER_FORGE_TIMEOUT to 120s and scheduling a weekly server restart — is runway, not a cure. The underlying problem is structural:

On every push or pull_request webhook event, Woodpecker performs a live, sequential GitHub Contents API fetch to load pipeline config: - 1 call: list the .woodpecker/ directory - 79 calls: fetch each file's content (one per pipeline file) - Total: 80 sequential HTTPS round-trips per webhook event - Typical serial latency: 80 × ~100ms = ~8 seconds against GitHub API from EC2 us-east-2

This is O(N) with pipeline-file count. Every feature PR pays it. Any GitHub API jitter compounds serially. The original 10s WOODPECKER_FORGE_TIMEOUT had no safety margin; the raised 120s limit is generous but does nothing to reduce the number of API calls.

Woodpecker supports a first-class escape hatch: WOODPECKER_CONFIG_SERVICE_ENDPOINT. When set, WP POSTs the repo + SHA + netrc context to an HTTP endpoint instead of fetching config from the forge. The endpoint returns the resolved pipeline configs in one response, replacing the per-file forge fetch entirely.

The permanent fix is to build and operate this external config service (wp-config-svc).

Invariants

Decision

Build and co-locate wp-config-svc on the WP EC2 host as a docker-compose service, using the Git Trees API with parallel blob fetching and an in-process SHA-keyed immutable cache. Auth to GitHub via a dedicated GitHub App installation token stored in SSM. Roll out in shadow mode (HTTP 204 pass-through) first, then cut over.

Each component of this decision is detailed below.

Data flow

GitHub webhook POST → ci.moosequest.net/hook (ALB → WP server :8000)
       │
       ▼
WP server processes webhook, needs pipeline config
       │
       ├── [WOODPECKER_CONFIG_SERVICE_ENDPOINT set]
       │       │
       │       ▼
       │   POST http://localhost:8095/config
       │   Body: {repo, sha, netrc, configs: []}
       │   Auth: Authorization: Bearer <JWT signed by WP Ed25519 key>
       │       │
       │       ▼
       │   wp-config-svc (on same host, :8095)
       │       │
       │       ├── [SHA cache hit] ──────────────────────────────┐
       │       │                                                  │
       │       └── [SHA cache miss]                              │
       │               │                                         │
       │               ▼                                         │
       │   1. GET /repos/{owner}/{repo}/git/trees/{sha}?recursive=1
       │      (GitHub API — one call, lists full tree with blob SHAs)
       │               │                                         │
       │               ▼                                         │
       │   2. Filter: entries where path starts with .woodpecker/│
       │               │                                         │
       │               ▼                                         │
       │   3. N parallel GET /repos/{owner}/{repo}/git/blobs/{sha}
       │      (GitHub API — N concurrent async HTTP requests)    │
       │               │                                         │
       │               ▼                                         │
       │   4. Decode base64 blob content, parse YAML             │
       │               │                                         │
       │   5. Cache result keyed on commit SHA (immutable)       │
       │               │                                         ▼
       │   HTTP 200 → {"configs": [{name, data}, ...]}  ←───────┘
       │       │
       │       ▼ (on non-200 / timeout from ECS)
       └── WP native forge fetch (fallback — O(N) sequential, 120s timeout)

Config retrieval strategy

Decision: Git Trees API (1 call) + parallel blob fetches (N concurrent) + SHA cache

Three options evaluated:

Option A — Directory listing + sequential blob fetch (current WP native behavior) 80 sequential HTTPS calls, ~8s at 100ms/call. Eliminated — this is the problem.

Option B — Tarball endpoint (GET /repos/{owner}/{repo}/tarball/{sha}) One HTTP call downloads the entire repo as a .tar.gz archive, then extract only .woodpecker/* files. One API call consumed; tarball size for this repo is estimated at 5–15 MB (large: Python + migrations + frontend). Download time ~400ms–1.2s plus decompression overhead. Wastes bandwidth on non-pipeline files (ratio: ~400 KB wanted vs ~10 MB transferred). Not recommended.

Option C — Git Trees API + parallel blob fetches (RECOMMENDED) - GET /repos/{owner}/{repo}/git/trees/{sha}?recursive=1 returns the full tree in one API call (~30 KB JSON response). Parses blob SHAs for all .woodpecker/*.yaml entries. - N parallel GET /repos/{owner}/{repo}/git/blobs/{blob_sha} requests fired concurrently via async HTTP (aiohttp). Effective latency = max(single blob roundtrip) ≈ 100–200ms regardless of N. - Total uncached: 1 tree call (~100ms) + parallel blobs (~100–200ms) = ~200–300ms total. - 79 blobs × ~5 KB each = ~395 KB transferred. Efficient.

SHA-keyed immutable cache: A commit SHA maps to exactly one set of .woodpecker/ file contents, forever. Cache structure: dict[str, list[Config]] in-process, keyed on {repo_full_name}:{commit_sha}. Entries never expire (SHA is immutable). Process restart clears cache; cold-start recovery is trivial (next cache miss re-fetches). Cache size: 79 files × ~5 KB × N cached SHAs. For 100 distinct SHAs cached: ~40 MB. Acceptable for the t4g.small host (4 GB RAM). Apply an LRU cap at 500 entries if memory becomes a concern.

API rate-limit accounting: - Per uncached event: 1 tree + 79 blob calls = 80 API calls - GitHub App installation token: 15,000 req/hr (vs 5,000/hr OAuth App — see Auth section) - Budget for cold events (GitHub App): 15,000 / 80 = 187 cold events/hr - Expected cache hit rate: >90% (most branch pushes share a recently-cached SHA or branch tip moves one commit at a time — prior SHA already cached from the preceding push event) - Practical cold-event rate: <20/hr in normal development — well within any rate limit

Scale math

Metric Before After (cache miss) After (cache hit)
GitHub API calls per webhook event 80 (serial) 80 (1 tree + 79 parallel) 0
Wall-clock latency ~8,000 ms ~200–300 ms ~1 ms
Speedup vs baseline — ~27× ~8,000×
FORGE_TIMEOUT headroom 2 s (10s limit, 8s cost) 119.7 s (120s limit, 0.3s cost) 119.999 s
GitHub API calls/hr ceiling (GitHub App) 62 events/hr at 80 calls 187 cold events/hr unlimited
O(N) growth with pipeline file count yes — serial × latency yes — parallel, 1× latency no

The O(N) growth on the cold path remains, but parallel vs serial changes the effective cost from N × latency to 1 × latency. Adding pipeline files no longer degrades wall-clock time materially.

Where it runs

Co-located on the same EC2 instance as Woodpecker server (i-082ee835595d90ae0), as a new service in the existing docker-compose at /opt/woodpecker/docker-compose.yml.

Why co-located, not cross-box: - WP→ECS call latency: 0 ms (loopback) vs ~1 ms (same VPC) vs ~10 ms (cross-AZ). - ECS downtime is not a new single point of failure. WP natively falls back to forge fetch if ECS returns non-200. The slow forge path (with 120s timeout) remains the safety net. ECS dying = CI returns to pre-fix behavior (slow), not to broken CI. - No extra EC2 cost: additional docker-compose service on a host with available headroom. - Operational simplicity: one deployment unit, one SSM-backed config, one restart sequence. - The "ECS dies when box dies" concern is moot: if the EC2 box dies, WP also dies and CI is down regardless of ECS. Co-location adds zero marginal outage risk.

Network binding: wp-config-svc binds to 127.0.0.1:8095 only. WP connects to http://localhost:8095/config. No ALB rule changes, no security group changes, no public exposure, no DNS entry.

Auth

WP → ECS: Ed25519 JWT signature verification

Woodpecker signs each POST to the config service endpoint with a JWT using an Ed25519 private key held internally by the WP server. The corresponding public key is exposed at: GET https://ci.moosequest.net/api/signature/public-key

On startup, wp-config-svc fetches and caches this public key (WP admin token required; see Open question 2). Each incoming POST is rejected (HTTP 400) unless the JWT in the Authorization: Bearer <token> header is valid against this key. This prevents config injection from any process that happens to reach localhost:8095.

No new WP configuration is required for the signature — it is automatic when WOODPECKER_CONFIG_SERVICE_ENDPOINT is set.

ECS → GitHub: dedicated GitHub App installation token

Provision a GitHub App named raxx-ci-config with a single permission: Repository contents: read-only. Install it on raxx-app/TradeMasterAPI.

SSM parameters to create under /ci/config-svc/ (SecureString): - /ci/config-svc/github-app-id - /ci/config-svc/github-app-installation-id - /ci/config-svc/github-app-private-key (PEM, multi-line)

The service mints an installation access token (POST /app/installations/{id}/access_tokens) at startup and refreshes it 5 minutes before the 1-hour expiry. The token is held in memory only; never written to disk or logs.

Why a dedicated App vs reusing the WP OAuth App token: GitHub App installation tokens provide 15,000 req/hr per installation vs the OAuth App's 5,000/hr. The WP forge OAuth App (/ci/woodpecker/github-client-id) is also used by WP for webhook delivery, status checks, and user authentication — sharing its rate limit bucket with the config service creates contention. A dedicated App with contents:read scopes narrowly and has an independent limit.

This is complementary to the bot App token consolidation tracked separately. The raxx-ci-config App is a server-side infra identity, not an agent pipeline identity.

Fallback if App provisioning is delayed: Use the existing WP OAuth App credentials (/ci/woodpecker/github-client-id + github-client-secret) in the interim. Acceptable for Phase 0 shadow mode where the ECS makes no production-path API calls that matter. Upgrade to the dedicated App before Phase 1 cutover.

Signature verification detail

# On startup — fetch WP's Ed25519 public key:
# (ALB bypass rule for /api/signature/* required — see Open question 2)
resp = httpx.get(
    "https://ci.moosequest.net/api/signature/public-key",
    headers={"Authorization": f"Bearer {WP_ADMIN_TOKEN}"}
)
WP_VERIFY_KEY = load_pem_public_key(resp.text.encode())

# On each POST /config:
def verify_wp_request(authorization: str) -> bool:
    token = authorization.removeprefix("Bearer ")
    try:
        jwt.decode(token, WP_VERIFY_KEY, algorithms=["EdDSA"])
        return True
    except jwt.InvalidTokenError:
        return False
# Reject with HTTP 400 if verification fails; log source host + rejection reason.

No new SSM parameters needed for the WP public key — it is read from the live WP server. The key changes only if WP is fully re-provisioned (rare); cache refresh on service restart is sufficient.

Rollout plan

Phase 0 — Shadow mode (soak before cutover)
  wp-config-svc starts and binds to 127.0.0.1:8095
  WOODPECKER_CONFIG_SERVICE_ENDPOINT=http://localhost:8095/config set in WP docker-compose
  ECS returns HTTP 204 on every request → WP falls back to native forge fetch
  ECS simultaneously runs the full GitHub API path (tree + parallel blobs + cache)
  Logs emitted: "shadow: computed N configs for {sha}, cache_hit={bool}, latency={ms}"
  WP behavior: unchanged; production unaffected; forge fetch still happens
  Success criteria: 20+ successful shadow runs across push and pull_request events;
  ECS output (config names + content SHA) matches WP forge fetch output for each event

Phase 1 — Active mode cutover
  ECS returns HTTP 200 with computed configs instead of 204
  WP uses ECS output for pipeline config; forge fetch no longer runs per event
  WOODPECKER_FORGE_TIMEOUT can be lowered to 30s (ECS path ~300ms; fallback still needed)
  Monitor for 48 hours: ECS p99 latency, cache hit rate, GitHub API error rate, fallback rate
  Success criteria: zero forge fallbacks, ECS latency p99 < 500ms, no pipeline creation errors

Phase 2 — Steady state
  Remove weekly server restart workaround (mitigated connection pool accumulation; ECS
  reduces per-event WP→GitHub API connections by ~99%, eliminating the root cause)
  Update docs/ops/runbooks/ci-woodpecker.md: retire Failure mode B-2 mitigations, add
  ECS restart and health check runbook entries
  Lower WOODPECKER_FORGE_TIMEOUT to 30s in SSM + restart WP

Rollback at any phase: Remove WOODPECKER_CONFIG_SERVICE_ENDPOINT from WP docker-compose env, restart WP container. WP immediately reverts to native forge fetch. 5-minute operation. The ECS can remain running with no effect on WP.

Migrations

No database schema changes. No data migrations. Changes are: - New docker-compose service entry in /opt/woodpecker/docker-compose.yml (ECS container). - New SSM parameters under /ci/config-svc/ (GitHub App credentials). - One new WP environment variable (WOODPECKER_CONFIG_SERVICE_ENDPOINT) in docker-compose. - New GitHub App raxx-ci-config (operator browser action).

All changes are reversible. Rollback is described above.

Security considerations

Security / GDPR checklist

Language choice rationale

Service: wp-config-svc

Language tier: Tier 2 — Python

Rationale: No Tier 1 criterion applies. The service is not on a latency-critical auth hot path (p99 target is ~300ms for cache miss, not sub-5ms). It handles no customer PII, no payment credentials, no audit hash chain, and no cryptographic key material beyond verifying a standard JWT. Throughput ceiling is ~200 cold events/hr — no Tier 1 volume threshold. Per docs/architecture/language-tier-policy.md §1, Tier 2 (Python) is the correct default. aiohttp provides sufficient parallelism for the blob fetch pattern.

API contract portability (Tier 2): The single endpoint POST /config follows the Woodpecker external config service protocol (stable, Woodpecker-documented JSON schema). This protocol is language-agnostic; a future Rust or Go rewrite would implement the same HTTP interface with zero redesign. No portability risk.

Alternatives considered

WP native forge fetch with tuned timeouts (current mitigation)

WOODPECKER_FORGE_TIMEOUT=120s + weekly restart. Addresses symptoms (timeout), not root cause (O(N) sequential API calls). O(N) cost grows with pipeline count. This stays as the fallback path; not as the primary path.

Tarball endpoint (GET /repos/{owner}/{repo}/tarball/{sha})

One API call replaces 80. Rejected: downloads the entire repo (~5–15 MB) to retrieve ~400 KB of pipeline files. Wastes bandwidth; decompression adds latency. Git Trees + parallel blobs is marginally more complex but significantly more efficient.

Lambda behind the ALB as the config service endpoint

Serverless, auto-scales, no idle cost. Rejected: Lambda cold start adds 100–500ms; adds a cross-VPC network hop; requires ALB listener rule changes and a new IAM trust policy. The co-located process has equivalent reliability (WP falls back on failure regardless of service location) with lower operational complexity.

Fork or patch Woodpecker to cache internally

WP's forge client in server/forge/github/github.go could be patched to add a SHA cache internally. Rejected: couples cache lifecycle to the WP process (same restart behavior); diverges from upstream; no advantage over the external config service which is exactly what Woodpecker provides for this use case. Staying on the upstream-supported path reduces maintenance burden.

Open questions

  1. GitHub App approval: GitHub may require App review for new App registrations under certain org policies. If review blocks timely provisioning, fall back to the WP OAuth App credentials for Phase 0 shadow mode and upgrade before Phase 1. sre-agent to confirm at Card A provisioning time.

  2. ALB bypass for /api/signature/public-key: The WP public key endpoint at https://ci.moosequest.net/api/signature/public-key is an /api/* path, which is covered by the existing ALB priority-25 rule (bypasses Google OIDC, forwards directly to WP). Confirm this is accessible with a WP admin Bearer token only — no OIDC required. sre-agent to verify before Card B implementation.

  3. WP admin token for public key fetch: The ECS startup needs to fetch the WP Ed25519 public key. Confirm whether this endpoint requires Bearer auth or is unauthenticated. If auth required, add /ci/config-svc/wp-admin-token to SSM parameter set in Card A.

Consequences

Positive

Negative / risks

Neutral

Revisit when