RCA — ci-boundary gitleaks step scans only 1 commit despite depth: 0; no persisted findings

Incident ID: 2026-08-12-ci-boundary-gitleaks-scan-scope Date: 2026-08-12 Severity: SEV-3 (security-scanning control produced unreliable signal; ci-boundary is a post-merge advisory check, not a deploy gate — no user-facing or deploy-blocking impact, but the scan's trustworthiness was compromised since introduction) Duration: Latent defect — present since ci-boundary.yaml's Wave A port (issue #4016); first observed release-2026.08.09 (WP pipeline #5095), root-caused and fixed 2026-08-12. Blast radius: ci-boundary WP pipeline (push to release/main only — advisory security scan, not a deploy-staging/deploy-prod gate). No production traffic, no deploy blocked. Confidence in every historical ci-boundary gitleaks scan result is reduced retroactively (each was scanning ~1 commit, not the intended full history). Author: sre-agent

Summary

.woodpecker/ci-boundary.yaml's gitleaks step logged INF 1 commits scanned on the release-2026.08.09 push (WP pipeline #5095) despite the clone step declaring depth: 0 (full history, per the file's own comment). Root cause: WP's woodpeckerci/plugin-git defaults partial: true on every non-tag event, and per the plugin's own documentation, partial "overwrite[s] depth with 1" — silently downgrading the declared full-history clone to a single-commit fetch. Additionally, no itemized finding detail (file/rule/ commit) was persisted anywhere once the WP step log's summary line was written — a "leaks found: N" result was unactionable without a manual local re-run. Both gaps are fixed in the same change: partial: false added to the clone config, and the gitleaks step now writes a redacted JSON report persisted as an S3 build artifact on every run (pass or fail).

Timeline (all times UTC)

• 2026-08-09 ~night — release-2026.08.09 pushed to release. WP pipeline #5095's ci-boundary gitleaks step logs INF 1 commits scanned, WRN leaks found: 6, no itemized detail before exit non-zero.
• 2026-08-10 — Issue #4445 filed: anomalous scan scope flagged, not blindly fixed (security-control change without visibility into what's flagged). Local re-run with WP's exact command (--no-git=false, full history) found 16 pre-existing matches across ~4755 commits — different from WP's reported 6-on-1-commit result, confirming the scope mismatch was real.
• 2026-08-10 05:19 — card-groomer grooming pass on #4445: FAILs atomicity (three bundled deliverables — scan-scope root-cause, findings persistence, allowlist triage). Proposes split into three cards.
• 2026-08-10 05:28 — #4445 closed as split-superseded; #4463 (scan-scope fix), #4464 (JSON artifact persistence), #4465 (allowlist triage, owner security-agent) filed.
• 2026-08-12 — card-groomer marks #4463/#4464 ready-for-dev, flags same-file/same-step overlap (.woodpecker/ci-boundary.yaml's gitleaks step), recommends one combined PR — owner sre-agent.
• 2026-08-12 — sre-agent branches fresh off origin/develop@4391670746. Root-caused via woodpeckerci/plugin-git's published docs (docs.md): partial row states default true except when tags are fetched, and "overwrite[s] depth with 1"; depth row confirms "overwritten by partial." ci-boundary.yaml's when.event: push (not tag) means partial defaulted true, silently downgrading depth: 0.
• 2026-08-12 — Local reproduction: gitleaks detect --source=. --config=.gitleaks.toml --no-git=false against this repo's full local history (non-shallow clone) scanned 4792 commits, found 16 leaks — consistent with the operator's original full-history local run on #4445 (4755 commits / 16 leaks; count delta is organic repo growth between 2026-08-10 and 2026-08-12, not a discrepancy). Verified --redact produces a JSON report with Secret/Match fields replaced by REDACTED while RuleID/File/Commit/Fingerprint stay intact — safe to persist as a build artifact.
• 2026-08-12 — Fix implemented: partial: false added to ci-boundary.yaml's clone settings; gitleaks step writes reports/gitleaks.json (--report-format=json --report-path=... --redact); new gitleaks-report-upload step (runs on success AND failure) publishes the report to s3://raxx-ci-artifacts/gitleaks-reports/<pipeline>/gitleaks.json using a new least-privilege IAM user (raxx-ci-boundary-gitleaks-report-writer-ci, infra/ci/ci-artifacts.tf, s3:PutObject on gitleaks-reports/* only, 14-day bucket lifecycle expiry). terraform fmt/validate pass locally; live terraform apply + AWS key mint deliberately deferred to the operator (see Resolution and Action items — new AWS credential mints escalate per current credentials-lockdown posture).
• 2026-08-12 — RCA + runbook update authored; PR opened.

Impact

• Users affected: none (internal CI/security-tooling only).
• User-visible symptoms: none.
• Data integrity: ok.
• Revenue / billing: ok.
• Security-signal integrity: degraded, now corrected. Every historical ci-boundary gitleaks run since the pipeline's introduction (#4016) was scanning only the single pushed commit, not full history. No evidence any push introduced a NEW secret that a full-history scan would have caught and a 1-commit scan missed (the 1-commit scan still covers the commit(s) actually landing in that push, which is the highest-value range for catching new leaks) — but the "full history" pre-existing-drift detection the pipeline's own header comment claims as its purpose was not happening.

What went well

• The original diagnosis (#4445) refused to blindly "fix" a security control without visibility into what it was flagging — correctly escalated instead of guessing.
• card-groomer caught the bundled-deliverable atomicity problem and split the card before dispatch, which kept the scan-scope fix and the artifact-persistence fix independently reviewable while still flagging the same-file overlap so they could land together.
• A near-identical shallow-clone class of bug had already been diagnosed and fixed in deploy-staging.yaml/gatekeeper (git fetch --unshallow pattern) — the existing runbook precedent shortened the search space, though the actual mechanism (partial vs. plain shallow clone) turned out to be different enough that the same fix (git fetch --unshallow) would not have applied here; the clone-plugin-level partial: false setting was the correct fix for this pipeline's config-only clone block.
• Local reproduction against a full non-shallow clone was fast and cheap (~13s for gitleaks itself) — full-history scope is not expensive enough at this repo's current size to justify a bounded-range design instead.

What didn't go well

• depth: 0's own documentation ("overwritten by partial") was not discoverable from .woodpecker/ci-boundary.yaml itself — the file's comment asserted "Full history required... (depth: 0)" without knowing that depth: 0 alone doesn't achieve that on a non-tag event. This is a gap in how WP pipeline authors verify clone-plugin defaults, not a one-off typo.
• The same latent misconfiguration (depth: 0 without partial: false) is present in at least two other pipelines (migration-collision-check.yaml, security-scan-nightly.yaml) and was not caught until this incident forced a read of plugin-git's actual default-value table. Neither of those pipelines has been independently verified against a live run as part of this fix (see Non-goals / Action items).
• No itemized findings were ever retrievable from a WP ci-boundary build without a manual local re-run — the pipeline's only signal was a summary count, which is not actionable and (as this incident showed) not even reliably scoped.

Root cause analysis

• Contributing factor 1 — plugin-git's partial default silently overrides depth. woodpeckerci/plugin-git defaults partial: true on every event except tag events (or where tags are explicitly fetched). When partial is true, it does not merely constrain the checkout — it explicitly overwrites the configured depth value with 1, regardless of what depth was set to. ci-boundary.yaml's clone block set depth: 0 but never set partial, so the plugin's default (true) applied and silently produced a 1-commit clone on every push-triggered run. This is a config-completeness gap: the system allowed a clone config to declare "full history" via one setting (depth) while a second, undocumented-in-this-repo setting (partial) silently contradicted it, with no validation step that would have caught the mismatch before a real pipeline run.
• Contributing factor 2 — no persisted itemized findings behind the step's summary line. The gitleaks step wrote only a human-readable summary (leaks found: N) to the WP step log, with no --report-format/ --report-path flags. WP's log capture does not reliably retain everything printed before a non-zero exit (per the original #4445 report, the log ended at the WARN line with no itemized table). This meant the only way to know what a scan actually flagged was a manual local re-run — and that re-run, run against the wrong scope, would not even match what WP had scanned. The system allowed a security-scan step to exist for months without a durable, retrievable record of its findings.

Detection

• What alerted us: manual operator/agent diagnosis of the release-2026.08.09 push in #4445 — not an automated alert. ci-boundary is advisory and does not page or block; a scoped-wrong scan would not have surfaced on its own until this manual review.
• How long between cause and detection: latent since #4016 (pipeline introduction); first observed 2026-08-09/10, ~months of undetected incorrect scope.
• How to detect faster next time: none available today — ci-boundary has no monitoring beyond "pipeline failed/passed" in the WP UI, and a 1-commit scan that finds fewer leaks than expected does not fail loudly (it still "passes" or "fails" based on whatever it did scan, with no cross-check against the intended range). Action item below proposes a lightweight commit-count sanity check in the pipeline itself.

Resolution

• What was changed:
• .woodpecker/ci-boundary.yaml: added partial: false to clone.settings (root cause fix, #4463); gitleaks step now runs with --report-format=json --report-path=reports/gitleaks.json --redact (#4464); new gitleaks-report-upload step (when.status: [success, failure]) publishes the report to s3://raxx-ci-artifacts/gitleaks-reports/<pipeline>/gitleaks.json.
• infra/ci/ci-artifacts.tf: new IAM user raxx-ci-boundary-gitleaks-report-writer-ci (write-only, gitleaks-reports/* prefix), new S3 lifecycle rule (14-day expiry), bucket-policy statements granting/denying accordingly, new Terraform output for the user ARN.
• docs/ops/runbooks/gitleaks.md: new Failure mode E documenting the partial-vs-depth root cause and fix pattern; new section on retrieving the S3-persisted findings artifact; escalation entry for the pending IAM-provisioning step.
• Validation performed pre-merge (all local, no live WP run required for these):
• gitleaks detect --source=. --config=.gitleaks.toml --no-git=false against this repo's full local (non-shallow) history: 4792 commits scanned, 16 leaks found — confirms the corrected scan scope and reproduces the operator's original full-history count (16) from #4445, not the anomalous WP-reported 6-on-1-commit result.
• gitleaks detect ... --redact locally confirmed to redact Secret/ Match fields in the JSON report while leaving RuleID/File/ Commit/Fingerprint intact — safe to persist per #4464's AC5 risk mitigation. No raw secret values were printed to any log, PR, or this RCA at any point during this investigation.
• python3 -c "import yaml; yaml.safe_load(open('.woodpecker/ci-boundary.yaml'))" — YAML parses.
• terraform fmt -check and terraform validate (via -backend=false, matching infra-ci-validate.yaml's exact CI check) both pass clean on infra/ci/ci-artifacts.tf.
• Validation NOT YET performed (requires a live push-triggered ci-boundary run, which only happens post-merge on a real release/main push — see Action items):
• Confirming the corrected N commits scanned on an actual WP pipeline run (AC5 of #4463).
• Confirming the gitleaks-report-upload step successfully authenticates and uploads (blocked on the operator running terraform apply + mint_ci_aws_keys.sh + vault write for the new IAM user — deliberately not done autonomously by sre-agent; see Action items).

Action items

References

• Runbook: docs/ops/runbooks/gitleaks.md (Failure mode E)
• Related incidents/issues: #4016 (ci-boundary pipeline introduction), #4385 (prior chronic-allowlist discussion, closed fix-don't-file), #4445 (original anomaly report, closed split-superseded), #4463, #4464, #4465 (parallel security-agent triage of the 16 pre-existing matches)
• Precedent: deploy-staging.yaml/gatekeeper shallow-clone fix (git fetch --unshallow) — same failure class, different plugin-level mechanism; documented for context, not directly reused here.
• Vendor doc: woodpeckerci/plugin-git — docs.md (depth/partial/tags settings table), https://github.com/woodpecker-ci/plugin-git