RCA — Two independent, non-blocking CI findings surfaced during release-2026.08.19 staging shepherd

Incident ID: 2026-08-19-ci-boundary-velvet-deploy-false-positive-refspec Date: 2026-08-19 Severity: SEV-3 (non-blocking; discovered proactively, not user-reported) Duration: N/A — no outage; both findings were latent/undetected until this shepherd run exercised the paths Blast radius: CI infrastructure only. deploy-staging (the tag-scoped pipeline gating actual staging deploys) was unaffected and completed 8/8 green. No customer-facing impact. Author: sre-agent

Summary

While shepherding the tag-scoped deploy-staging pipeline for release-2026.08.19 (develop HEAD 3506b3e02), the companion push-triggered pipeline (ci-boundary, ci-console, deploy-velvet, deploy-queue, deploy-antlers-next-staging, infra-ci-validate, terraform-validate — all fired by the same gatekeeper push to release) surfaced two independent, non-blocking findings: (1) a gitleaks false positive on a brand-new Wave 3 idempotency test file that follows an already-allowlisted convention but was never added to the allowlist, compounded by a separately-broken report-upload step that has silently failed to publish itemized findings for an unknown period; and (2) a deploy-velvet.yaml bug — introduced 2026-07-08, latent for ~6 weeks — that silently corrupts the Heroku push refspec to empty, which Heroku's pre-receive hook correctly rejects. Both are CI-only; deploy-staging and the actual staging deploy were unaffected.

Timeline (all times UTC)

• 19:18 — deploy-staging (Woodpecker pipeline 6540, tag release-2026.08.19) completes 8/8 steps green; staging release v1062 live, RELEASE_VERSION=0.0.112-6540.
• 19:18 — Gatekeeper's git merge --no-ff push to release (commit 923dd9f4) fans out to a second, push-triggered Woodpecker pipeline (6541) covering ci-boundary, ci-console, deploy-antlers-next-staging, deploy-queue, deploy-velvet, infra-ci-validate, terraform-validate.
• ~19:25 — deploy-antlers-next-staging, infra-ci-validate, terraform-validate complete success. ci-boundary fails at the gitleaks step (leaks found: 1); gitleaks-report-upload also fails (vault 404).
• ~19:33 — deploy-velvet fails at its deploy step: Heroku rejects the push ("Push rejected, cannot delete main branch") 5x.
• 19:33–19:40 — sre-agent reproduces the gitleaks finding locally (matching WP's exact 2330-commit scan via --log-opts=origin/release and the release branch's own .gitleaks.toml), confirms it is a test-fixture false positive ("SECRET_KEY": "test-sc-ik-5-never-prod" in backend_v2/tests/test_idempotency_billing_wave3_4149.py, same shape as three prior allowlisted entries), and separately diagnoses the deploy-velvet.yaml refspec bug from the step log + git blame.
• 19:36 — Wave 3 route probe on staging confirms the five idempotency-gated billing routes are live (401 auth-required, not 404), independently confirming deploy-staging's success and giving a code-signature anchor for the release.
• 19:45 — Two type:reliability issues filed (#4513 deploy-velvet fix, #4514 gitleaks allowlist + vault-path gap); this RCA and a ci-woodpecker.md runbook update written. ci-console (test-only failure, node binary missing in the WP test image — out of sre-agent scope, flagged for routing) and deploy-queue status noted as unresolved at report time.

Impact

• Users affected: none.
• User-visible symptoms: none.
• Data integrity: ok.
• Revenue / billing: ok — Wave 3 idempotency routes confirmed live and auth-gated as expected; no billing-path impact from either finding.
• CI-visible impact: ci-boundary and deploy-velvet show red on every release/main push until fixed, training the team to treat that as noise. deploy-velvet has had zero successful deploys since 2026-07-08 (six weeks), though no velvet/** code change has actually needed deploying in that window — so no service is currently running stale code as a result of this bug, but the next real Velvet change will silently fail to ship (loud CI red, but the running dyno stays on old code) until #4513 is fixed.

What went well

• The tag-scoped deploy-staging pipeline itself is well-isolated: none of these companion-pipeline failures touched it, and the shepherd bar (8 steps, release-phase migration, alembic heads, code-signature probe, RELEASE_VERSION, no-ff promote, CF Pages) all passed cleanly and independently.
• Both root causes were fully diagnosable from Woodpecker step logs + local git blame/git show reproduction without needing write access or guessing — the WP ${VAR} vs $VAR interpolation gotcha was already documented (Failure mode E-4 in ci-woodpecker.md) with an exact matching precedent, and the gitleaks chronic-FP allowlist pattern was already established (#4385 → #4445 → #4465) with a clear file-naming convention to follow.
• --redact on the gitleaks report meant local reproduction could confirm the false positive without ever exposing a real secret value in this transcript.

What didn't go well

• Failure mode E-4 (ci-woodpecker.md) was documented with an exact "Lint check" grep command after its first occurrence (2026-07-05), but that check is manual-only — nothing in CI runs it automatically. deploy-velvet.yaml reintroduced the identical anti-pattern three days later (2026-07-08) and it went undetected for six weeks because the buggy step's path filter never matched a push until now.
• The gitleaks chronic-FP allowlist (#4385/#4445/#4465) is a per-file exact-path list, not a pattern covering the whole test_idempotency_*.py family it was established for — so each new wave repeats the same triage cycle instead of being covered automatically.
• The gitleaks-report-upload step's vault path (/MooseQuest/aws/ci-boundary/gitleaks-report-writer/) has apparently never been provisioned (404, not "secret rotated/expired"), meaning itemized gitleaks findings have never successfully reached S3 — every prior finding (true or false positive) had to be triaged from the truncated step-log summary alone, same as this one, unless someone reproduced it manually like this run did.

Root cause analysis

• Contributing factor 1 (deploy-velvet): The system allows a previously-fixed anti-pattern class (WP ${VAR} interpolation stripping shell-local variables to empty) to be reintroduced in new files with no automated check — only a runbook-documented manual grep. A documented fix that isn't enforced is not a fix, it's a hope.
• Contributing factor 2 (deploy-velvet latency): Path-filtered pipelines can carry a broken step silently for arbitrarily long if the path filter itself rarely matches — there is no periodic/synthetic exercise of low-traffic deploy paths, so a break is only discovered the next time a real change (or, as here, an unrelated workflow-file edit) happens to trigger it.
• Contributing factor 3 (gitleaks allowlist): The allowlist mechanism requires exact per-file additions rather than a pattern covering an already-identified recurring class, so the same triage cost repeats on every new instance of an already-understood false positive.
• Contributing factor 4 (report-upload vault gap): No provisioning verification step confirms the gitleaks-report-writer IAM credential and its vault path exist together before the step is relied upon — the gap was only visible by reading the step's own log output during an unrelated diagnosis.

Detection

• What alerted us: none of these were alerted — discovered by manually reviewing the companion pipeline's step statuses while shepherding the primary deploy-staging pipeline, because the task's own standing bar ("fix forward anything broken") prompted a look past the immediate success signal.
• How long between cause and detection: deploy-velvet — ~6 weeks (introduced 2026-07-08, detected 2026-08-19). Gitleaks allowlist gap — hours (Wave 3 test file merged 2026-08-19, ci-boundary ran within the hour on promotion).
• How to detect faster next time: see action items — CI-enforced lint for the ${VAR} class, and either a broader allowlist regex or a type:reliability auto-file on any new backend_v2/tests/test_idempotency_* file lacking the allowlist entry.

Resolution

• Not yet applied — both are config-only fixes filed as type:reliability issues with exact diffs for the operator/next write-capable agent to apply (sre-agent does not commit; see docs/ops/runbooks/velvet-operator.md and docs/ops/runbooks/gitleaks.md for the relevant systems).
• ci-woodpecker.md Failure mode E-4 updated with this occurrence as a documented precedent for the next diagnosis.

Action items

References

• Runbook: docs/ops/runbooks/ci-woodpecker.md (Failure mode E-4, updated this run)
• Runbook: docs/ops/runbooks/gitleaks.md
• Runbook: docs/ops/runbooks/velvet-operator.md
• Issues: #4513 (deploy-velvet), #4514 (gitleaks allowlist + vault path)
• Woodpecker pipelines: 6540 (deploy-staging, success, 8/8), 6541 (push-to-release companion pipeline)
• Prior precedent: docs/ops/runbooks/ci-woodpecker.md Failure mode E-4, fixed 2026-07-05 across vcpkg-manifest-check.yaml, migration-collision-check.yaml, maxmind-asn-refresh.yaml, alembic-version-check.yaml
• Gitleaks chronic-FP precedent: #4385, #4445, #4465