Gitleaks runbook

System: gitleaks nightly security scan + ci-boundary post-merge scan + ci-develop push scan Owner: sre-agent Last incident: 2026-11-04 (no outage; #4576 replaced the never-provisioned report-writer IAM user and vault keys with the agent instance role — see Failure mode F). Before that 2026-10-30 (ci-develop security-secrets red on every develop push because its clone was partial, so .gitleaksignore could never match — see Failure mode G). Same day: ci-boundary red on every release/main push (unprovisioned report-upload credentials plus two false positives — see Failure mode F). Before that 2026-08-10 (ci-boundary gitleaks step scanned only 1 commit on release push despite depth: 0 — see #4463/#4464, RCA docs/incidents/2026-08-12-ci-boundary-gitleaks-scan-scope.md) Last reviewed: 2026-08-12 (#4483 — added verify-full-history regression gate to ci-boundary.yaml and security-scan-nightly.yaml, following #4481's class sweep of the same depth: 0 without partial: false gap across the rest of .woodpecker/)

How to tell it's broken

• GitHub issue filed by security_file_issues.py with label type:security and title matching gitleaks: — check if the flagged value is a real secret or a false positive.
• Nightly scan failing with findings that were previously suppressed (allowlist regression).
• CI gitleaks step exits non-zero on a PR that should be clean.
• ci-boundary (release/main push) gitleaks step log shows a suspiciously low N commits scanned count relative to the pushed range — see Failure mode E.

Where to find findings for a ci-boundary run

Since #4464, the gitleaks-report-upload step publishes the itemized JSON report (secret values redacted via --redact; file/rule/commit metadata intact) to s3://raxx-ci-artifacts/gitleaks-reports/<CI_PIPELINE_NUMBER>/gitleaks.json on every ci-boundary run — pass or fail. Retention is 14 days. Pull it with:

aws s3 cp s3://raxx-ci-artifacts/gitleaks-reports/<pipeline-number>/gitleaks.json - \
  | python3 -m json.tool

The WP step log's "leaks found: N" summary line alone is not enough to triage a finding — always pull the artifact for file/rule/commit detail rather than re-running the scan locally.

The upload uses the Woodpecker agent's EC2 instance role (s3:PutObject on gitleaks-reports/* only; no static key, #4576), so it works as soon as the infra/ci Terraform for #4576 is applied. Before that apply there is no artifact (Failure mode F; the gitleaks-reports/ prefix has never held an object). Reproduce the pushed range locally instead. This is the exact pipeline command, bounded to what the push clone contains (history reachable from the pushed commit):

gitleaks detect --source=. --config=.gitleaks.toml --no-git=false \
  --log-opts="<pushed-commit-sha>" --report-format=json --report-path=/tmp/gl.json --redact
python3 -c "import json; [print(f['Fingerprint']) for f in json.load(open('/tmp/gl.json'))]"

The fingerprint (<sha>:<file>:<rule>:<line>) is the exact string a .gitleaksignore entry needs, once you have confirmed the finding is not a credential.

How to diagnose (in order)

1. Read the issue title — it includes the file path and line number. Read that line in the repo.
2. Classify the finding: is the value a real credential (API token, private key, password) or a public identifier / placeholder?
3. Run gitleaks locally on the full history to reproduce: gitleaks detect --source=. --config=.gitleaks.toml --no-git=false --report-path=/tmp/gl.json --report-format=json python3 -c "import json; [print(f['RuleID'], f['File'], f['StartLine'], repr(f['Match'])) for f in json.load(open('/tmp/gl.json'))]"
4. If the finding is a false positive and an allowlist entry exists, check why it isn't suppressing: - Is regexTarget = "match" set? Without it, regexes test against the extracted Secret (bare capture group), not the full Match string. - Does the regex actually match the Match string? Test with Python: re.search(pattern, match_string). - Is the path allowlist entry correct? Check the path prefix in the finding's File field.

Known failure modes

Failure mode A: allowlist regex not suppressing — missing regexTarget = "match"

Symptom: An allowlist regex entry exists for the pattern, but gitleaks still flags it in full-history scans. The regex is written to match a variable-assignment expression (e.g. cf_access_account_id\s*=\s*"..."), not just the bare secret value.

Cause: gitleaks [allowlist] regexes default to matching against the extracted Secret field (just the capture group from the rule's regex). Without regexTarget = "match", a regex like varname\s*=\s*"value" tests against value — which doesn't match the full expression.

Fix: 1. Add regexTarget = "match" to the [allowlist] block in .gitleaks.toml. This applies to all entries in the block. 2. Verify all other regex entries still work (they should — substring match against the longer Match string is backward-compatible for most patterns). 3. Run gitleaks detect --source=. --config=.gitleaks.toml --no-git=false and confirm findings drop to 0.

Verification: Local full-history scan exits 0. File PR, confirm CI scan passes.

Failure mode B: new false positive — public identifier flagged as generic-api-key

Symptom: A hex string, UUID, or high-entropy public identifier committed to a .tf, .json, or similar config file is flagged under generic-api-key or generic-api-key.

Cause: gitleaks generic-api-key rule fires on Shannon entropy thresholds, not on pattern context. Public identifiers (Cloudflare account IDs, zone IDs, GCP project IDs, etc.) often have enough entropy to trigger it.

Decision criteria — add to allowlist only if ALL of these are true: - The value is publicly visible (e.g. appears in dashboard URLs, API response metadata, or documentation) - It cannot be used to authenticate — possession of the value alone grants no access - The variable name documents the public nature (e.g. _account_id, _zone_id, not _token, _key, _secret)

Fix: 1. Add a regex entry with regexTarget = "match" (already set if the block has it) matching variable_name\s*=\s*"value_pattern". 2. Add a comment citing the issue number and explaining why it's safe. 3. Run full-history scan to verify. 4. Do NOT use a bare hex pattern as the regex — it will over-suppress any 32-char hex value.

Verification: gitleaks detect --source=. --config=.gitleaks.toml --no-git=false exits 0.

Failure mode C: real secret committed — must rotate

Symptom: gitleaks flags a value that is a live credential (API token, private key, password). The variable name or context confirms it (_token, _key, _secret, password, BEGIN PRIVATE KEY, etc.).

Fix: This is a security incident, not a false positive. Follow the security incident response protocol: 1. Classify severity (SEV-1 if token is for a production system with blast radius, SEV-2 if staging/internal). 2. Rotate the credential immediately at the vendor dashboard or API. Do not wait for history rewrite. 3. Update vault at the canonical path (/MooseQuest/<vendor>/). 4. Verify all consumers of the rotated key still work. 5. File a rewrite plan (BFG or git filter-repo) in the incident issue. Do NOT execute the rewrite without explicit operator authorization (force-push to main is destructive). 6. Write an RCA.

Verification: Vendor confirms old token is revoked. New token passes smoke test. Vault has new value.

Failure mode D: path allowlist not suppressing — wrong path prefix

Symptom: A file at console/app/sops/rotation/foo.md is flagged, but the allowlist only covers ^docs/ops/runbooks/rotation/.

Cause: The same file content lives at multiple paths (e.g. synced copies). Each distinct path prefix needs its own entry.

Fix: Add the missing path regex to the paths array in .gitleaks.toml. Keep entries narrow — prefer directory-level paths over single-file entries.

Verification: Full-history scan exits 0.

Failure mode E: ci-boundary scans anomalously few commits despite depth: 0

Symptom: WP ci-boundary gitleaks step log shows INF N commits scanned where N is small (often 1) on a push-triggered release/main build, even though .woodpecker/ci-boundary.yaml's clone.settings.depth is 0 (full history, per the file's own comment).

Cause: WP's woodpeckerci/plugin-git defaults partial: true on every event that is not a tag event. Per plugin-git's own docs (docs.md, partial row): partial clone "only fetch[es] the one commit and it's blob objects... overwrite[s] depth with 1." depth's own doc row confirms it is "overwritten by partial." So a push-triggered clone silently ignores depth: 0 unless partial: false is also set — depth: 0 alone is not sufficient. Confirmed against woodpeckerci/plugin-git main-branch docs 2026-08-12; see docs/incidents/2026-08-12-ci-boundary-gitleaks-scan-scope.md.

Fix: Add partial: false alongside depth: 0 in the clone.settings block:

clone:
  - name: clone
    image: woodpeckerci/plugin-git
    settings:
      depth: 0
      partial: false

This is already applied in .woodpecker/ci-boundary.yaml as of #4463. If a new WP pipeline is added with a full-history clone requirement (any depth: 0 push/pull_request/cron/manual-triggered pipeline — NOT a tag event), it needs the same partial: false pairing.

Class-sweep status (#4481, closed 2026-08-12): a repo-wide grep of .woodpecker/*.yaml for clone.settings.depth: 0 found exactly three matches. All three now carry partial: false:

A repo-wide grep for git clone/git fetch --depth outside clone.settings (ci-pr.yaml, queue-docker-smoke.yaml's deliberate --depth=1 shallow fetches; deploy-prod.yaml/deploy-staging.yaml/cut-release-candidate.yaml/ review-app-console.yaml's git fetch --unshallow pattern; vcpkg-manifest-check.yaml's in-step clone of an external repo, not this one) confirmed none of those are instances of this class — they either intend a shallow fetch, already use the --unshallow workaround from an earlier incident, or aren't cloning this repo at all.

Verification: Trigger a real push-triggered ci-boundary build (next release/main push) and confirm the gitleaks step log's N commits scanned matches git rev-list --count <ref> for the pushed ref, not 1. Local reproduction (does not require a live WP run, useful for pre-merge confidence): gitleaks detect --source=. --config=.gitleaks.toml --no-git=false against a full (non-shallow) local clone should report commit counts in the thousands for this repo, not single digits. For security-scan-nightly.yaml, the same local reproduction applies; live confirmation requires either waiting for the next 08:07 UTC cron run or a manual trigger via the WP API (POST /api/repos/.../pipelines) and reading the gitleaks step log — not yet performed as of #4481's merge (config fix landed; live-run confirmation is a natural follow-up on the next scheduled run, same deferral pattern as

4463's action item 2).

Automated regression gate (#4483): Both ci-boundary.yaml and security-scan-nightly.yaml now carry a verify-full-history step, inserted immediately after clone and before the step that consumes full history (gitleaks). It is a one-line boolean assertion:

  - name: verify-full-history
    image: alpine/git:2.45.2
    commands:
      - |
        if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then
          echo "FATAL: clone is shallow despite depth:0 — plugin-git 'partial' default likely overrode depth. See docs/ops/runbooks/gitleaks.md Failure mode E." >&2
          exit 1
        fi
        echo "Full-history clone confirmed (not shallow)."

In security-scan-nightly.yaml the step also carries failure: ignore, matching its sibling scanner steps in that file during the A/B dual-run pilot (ADR-0134) — the step still exits non-zero and fails loud in the WP step log, it just doesn't escalate beyond what the rest of that pipeline already does. ci-boundary.yaml's copy has no failure: ignore; it is a genuine hard gate on that pipeline.

Why a boolean assertion, not a commit-count comparison. The #4463 RCA's action item #4 originally floated comparing gitleaks' own "N commits scanned" against git rev-list --count HEAD. That comparison is a null check: both numbers are computed inside the same clone, so when partial: true silently truncates the clone to 1 commit, git rev-list --count HEAD inside that same shallow clone also reports 1 — the two sides agree, and the check would not have caught #4463. git rev-parse --is-shallow-repository instead reads .git/shallow's presence directly, which is set by the clone itself regardless of how many total commits the repo has. This eliminates the false-positive risk on a genuinely small repo/branch history (fresh branch, single commit) without any tunable threshold. Ruling: PM+QA+architect adjudication on #4483, 2026-08-12.

Locally-verified edge cases (required before merge, not just asserted): - A full (non-shallow) clone of a fresh 1-commit repo: git rev-parse --is-shallow-repository → false → step passes. - A --depth 1 clone of the same repo (git clone --depth 1 file://..., note plain local-path clones ignore --depth — use a file:// URL to reproduce): git rev-parse --is-shallow-repository → true → assertion trips, step exits 1.

migration-collision-check.yaml intentionally does not get this step (or any shallow-clone check). Its check step resolves the PR base SHA via gh api, not git fetch, and its git diff/git show calls only need the two named commits present as objects — not a full ancestor walk. A shallowness assertion there would test a precondition the pipeline's logic never consumes. See the migration-collision-check.yaml row in the class- sweep table above for the evidence trail (#4294, #4297, #4322, #4369 passed under the pre-existing partial: true default).

Failure mode F: ci-boundary red on every release/main push

Symptom: ci/woodpecker/push/ci-boundary is failure on every push to release and main (every recorded run back to 2026-10-02 at least, including pushes where gitleaks found nothing). A real finding is indistinguishable from the standing failure.

Cause (two independent layers, both found 2026-10-30): 1. gitleaks-report-upload fails on every run. It loads AWS keys from vault /MooseQuest/aws/ci-boundary/gitleaks-report-writer/ with VAULT_ALLOW_MISSING=false, but the IAM user raxx-ci-boundary-gitleaks-report-writer-ci has never been created (aws iam list-users shows only the three original CI users) and s3://raxx-ci-artifacts/gitleaks-reports/ has never held an object. gitleaks writes [] when clean, which is non-empty, so the step's "nothing to upload" early exit never applies. The step is evidence publishing, not a check, but it counted toward the pipeline status. 2. From 2026-10-29, gitleaks itself failed on pushes whose history contained e8fce6168 (#4814), a synthetic secret-looking string in a unit test (a false positive). A second one, 83b6fa655 (#4811, a localStorage entry name constant called STORAGE_KEY), was on develop and would have failed the next promotion.

Fix applied: (1) the upload step now has failure: ignore, so the pipeline status reflects the scanners only; none of the scanner steps may carry it (pinned by scripts/ci/tests/test_ci_boundary_pipeline.py). (2) The two false positives are suppressed by exact fingerprint in .gitleaksignore, each with the reasoning; a before/after local run over origin/develop origin/release origin/main went from 2 findings to 0. The PII scan passes on the main-tip and develop-tip trees, so it was not a cause.

Superseded (#4576, operator decision 2026-11-04): no static key. The writer IAM user and the vault keys are gone from the design. gitleaks-report-upload now uses the agent EC2 instance role through the AWS CLI default credential chain (agents set http_put_response_hop_limit = 2, so the step container reaches IMDS). The role policy ci-woodpecker-agent-gitleaks-reports-policy grants s3:PutObject on raxx-ci-artifacts/gitleaks-reports/* only (infra/ci/iam.tf), and the bucket policy names the role (infra/ci/ci-artifacts.tf). The step unsets AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN and AWS_PROFILE first, because any of them, even empty, would shadow the instance role. failure: ignore stays. The step is pinned by scripts/ci/tests/test_ci_boundary_pipeline.py (no vault loader, no from_secret, no AWS credential variables).

Still operator-gated: the terraform apply of infra/ci that attaches the policy (the plan also carries unrelated pending changes; see docs/ops/runbooks/ci-woodpecker.md §Scale-in drain hook, post-apply checks). Until it lands the upload step fails with AccessDenied or Unable to locate credentials, shows red in the Woodpecker UI by design (it cannot fail the pipeline), and no report is published; use the local reproduction in "Where to find findings".

Verification: after the apply, the next release/main push runs ci-boundary green with gitleaks-report-upload green, and aws s3 ls s3://raxx-ci-artifacts/gitleaks-reports/ --recursive --region us-east-2 lists <pipeline-number>/gitleaks.json. Before the apply, gitleaks-report-upload is the only red step.

Triage rule that applies to every new finding: read the flagged line first. If it is a credential, it is Failure mode C (rotate), never an ignore entry. Only a value that is demonstrably not a credential gets a fingerprint, and the entry must say why. Do not repeat the flagged literal in the comment: the comment is itself a new line that gitleaks scans.

Failure mode G: ci-develop security-secrets red on findings that are already triaged

Symptom: ci/woodpecker/push/ci-develop fails at security-secrets on every push to develop; the log shows 1 commits scanned and leaks found: N (N was 4 on 2026-10-30) with no itemized findings, while ci-boundary and the nightly scan are clean for the same code. Every later ci-develop step shows skipped (a skip propagates through the step order).

Cause: ci-develop.yaml had no clone: block, so woodpeckerci/plugin-git made a partial depth-1 clone (Failure mode E). gitleaks detect --no-git=false then walks one synthetic commit that contains the whole tree. A .gitleaksignore fingerprint is <introducing-commit-sha>:<file>:<rule>:<line>, so none could match the tip SHA, and paths that only exist after a rename (here .github/workflows-retired/deploy-status-worker.yml) were flagged too. This is the Failure mode E class in a pipeline the #4481 sweep could not find, because it had no depth: 0 to grep for. Any pipeline that runs gitleaks detect --no-git=false needs the clone: block.

Fix: ci-develop.yaml now has the same clone: block as ci-boundary.yaml (depth: 0, partial: false), and the security-secrets step fails loudly if the clone is shallow. A full-history scan of origin/develop with the pipeline command returned no leaks found (2528 commits, 14s locally).

Reproduce the shallow case locally:

git clone --depth 1 file://$PWD /tmp/shallow && cd /tmp/shallow
gitleaks detect --source=. --config=.gitleaks.toml --no-git=false --redact \
  --report-format=json --report-path=/tmp/gl.json

The step log never itemizes findings; the report file does (--redact keeps secret values out of it).

Verification: the next ci-develop run shows N commits scanned with N in the thousands and no leaks found.

Testing an allowlist change

Always test allowlist changes against the full-history scan, not just --no-git:

gitleaks detect --source=. --config=.gitleaks.toml --no-git=false
# Expected: "no leaks found" (exit 0)

For CI v8.18.4 (the pinned version in nightly-security-scan.yml), verify with:

curl -sSL https://github.com/gitleaks/gitleaks/releases/download/v8.18.4/gitleaks_8.18.4_darwin_arm64.tar.gz \
  | tar -xz -C /tmp gitleaks
/tmp/gitleaks detect --source=. --config=.gitleaks.toml --no-git=false

Emergency stop

To disable the nightly scan temporarily (e.g. while a real incident is being triaged):

1. Edit .github/workflows/nightly-security-scan.yml — comment out the schedule: trigger.
2. Open a PR with the change. Do not merge to main without a paired "re-enable" commit ready.
3. Log the disable in the incident issue with a stated re-enable time.

Escalation

Escalate to Kristerpher when: - A confirmed real secret is found in history and rotation has been completed but force-push authorization is needed for history rewrite - The CI scanner version (v8.18.4) needs to be bumped — check gitleaks CHANGELOG for breaking changes to allowlist behavior first - A new finding cannot be classified as real vs false positive within 30 minutes of investigation - The gitleaks-report-upload step in ci-boundary.yaml is failing to authenticate to AWS even after the infra/ci apply for #4576: it uses the agent instance role, so check that ci-woodpecker-agent-gitleaks-reports-policy is attached to ci-woodpecker-agent-role and that the step container can reach IMDS (agent launch template http_put_response_hop_limit = 2). Do not reintroduce a static key or the retired raxx-ci-boundary-gitleaks-report-writer-ci user; a new AWS key mint escalates per the credentials-lockdown posture. The terraform apply itself is operator-gated.