Gitleaks runbook

System: gitleaks nightly security scan Owner: sre-agent Last incident: 2026-05-01 (see docs/incidents/2026-05-01-gitleaks-cf-account-id-false-positive.md) Last reviewed: 2026-05-01

How to tell it's broken

• GitHub issue filed by security_file_issues.py with label type:security and title matching gitleaks: — check if the flagged value is a real secret or a false positive.
• Nightly scan failing with findings that were previously suppressed (allowlist regression).
• CI gitleaks step exits non-zero on a PR that should be clean.

How to diagnose (in order)

1. Read the issue title — it includes the file path and line number. Read that line in the repo.
2. Classify the finding: is the value a real credential (API token, private key, password) or a public identifier / placeholder?
3. Run gitleaks locally on the full history to reproduce: gitleaks detect --source=. --config=.gitleaks.toml --no-git=false --report-path=/tmp/gl.json --report-format=json python3 -c "import json; [print(f['RuleID'], f['File'], f['StartLine'], repr(f['Match'])) for f in json.load(open('/tmp/gl.json'))]"
4. If the finding is a false positive and an allowlist entry exists, check why it isn't suppressing: - Is regexTarget = "match" set? Without it, regexes test against the extracted Secret (bare capture group), not the full Match string. - Does the regex actually match the Match string? Test with Python: re.search(pattern, match_string). - Is the path allowlist entry correct? Check the path prefix in the finding's File field.

Known failure modes

Failure mode A: allowlist regex not suppressing — missing regexTarget = "match"

Symptom: An allowlist regex entry exists for the pattern, but gitleaks still flags it in full-history scans. The regex is written to match a variable-assignment expression (e.g. cf_access_account_id\s*=\s*"..."), not just the bare secret value.

Cause: gitleaks [allowlist] regexes default to matching against the extracted Secret field (just the capture group from the rule's regex). Without regexTarget = "match", a regex like varname\s*=\s*"value" tests against value — which doesn't match the full expression.

Fix: 1. Add regexTarget = "match" to the [allowlist] block in .gitleaks.toml. This applies to all entries in the block. 2. Verify all other regex entries still work (they should — substring match against the longer Match string is backward-compatible for most patterns). 3. Run gitleaks detect --source=. --config=.gitleaks.toml --no-git=false and confirm findings drop to 0.

Verification: Local full-history scan exits 0. File PR, confirm CI scan passes.

Failure mode B: new false positive — public identifier flagged as generic-api-key

Symptom: A hex string, UUID, or high-entropy public identifier committed to a .tf, .json, or similar config file is flagged under generic-api-key or generic-api-key.

Cause: gitleaks generic-api-key rule fires on Shannon entropy thresholds, not on pattern context. Public identifiers (Cloudflare account IDs, zone IDs, GCP project IDs, etc.) often have enough entropy to trigger it.

Decision criteria — add to allowlist only if ALL of these are true: - The value is publicly visible (e.g. appears in dashboard URLs, API response metadata, or documentation) - It cannot be used to authenticate — possession of the value alone grants no access - The variable name documents the public nature (e.g. _account_id, _zone_id, not _token, _key, _secret)

Fix: 1. Add a regex entry with regexTarget = "match" (already set if the block has it) matching variable_name\s*=\s*"value_pattern". 2. Add a comment citing the issue number and explaining why it's safe. 3. Run full-history scan to verify. 4. Do NOT use a bare hex pattern as the regex — it will over-suppress any 32-char hex value.

Verification: gitleaks detect --source=. --config=.gitleaks.toml --no-git=false exits 0.

Failure mode C: real secret committed — must rotate

Symptom: gitleaks flags a value that is a live credential (API token, private key, password). The variable name or context confirms it (_token, _key, _secret, password, BEGIN PRIVATE KEY, etc.).

Fix: This is a security incident, not a false positive. Follow the security incident response protocol: 1. Classify severity (SEV-1 if token is for a production system with blast radius, SEV-2 if staging/internal). 2. Rotate the credential immediately at the vendor dashboard or API. Do not wait for history rewrite. 3. Update vault at the canonical path (/MooseQuest/<vendor>/). 4. Verify all consumers of the rotated key still work. 5. File a rewrite plan (BFG or git filter-repo) in the incident issue. Do NOT execute the rewrite without explicit operator authorization (force-push to main is destructive). 6. Write an RCA.

Verification: Vendor confirms old token is revoked. New token passes smoke test. Vault has new value.

Failure mode D: path allowlist not suppressing — wrong path prefix

Symptom: A file at console/app/sops/rotation/foo.md is flagged, but the allowlist only covers ^docs/ops/runbooks/rotation/.

Cause: The same file content lives at multiple paths (e.g. synced copies). Each distinct path prefix needs its own entry.

Fix: Add the missing path regex to the paths array in .gitleaks.toml. Keep entries narrow — prefer directory-level paths over single-file entries.

Verification: Full-history scan exits 0.

Testing an allowlist change

Always test allowlist changes against the full-history scan, not just --no-git:

gitleaks detect --source=. --config=.gitleaks.toml --no-git=false
# Expected: "no leaks found" (exit 0)

For CI v8.18.4 (the pinned version in nightly-security-scan.yml), verify with:

curl -sSL https://github.com/gitleaks/gitleaks/releases/download/v8.18.4/gitleaks_8.18.4_darwin_arm64.tar.gz \
  | tar -xz -C /tmp gitleaks
/tmp/gitleaks detect --source=. --config=.gitleaks.toml --no-git=false

Emergency stop

To disable the nightly scan temporarily (e.g. while a real incident is being triaged):

1. Edit .github/workflows/nightly-security-scan.yml — comment out the schedule: trigger.
2. Open a PR with the change. Do not merge to main without a paired "re-enable" commit ready.
3. Log the disable in the incident issue with a stated re-enable time.

Escalation

Escalate to Kristerpher when: - A confirmed real secret is found in history and rotation has been completed but force-push authorization is needed for history rewrite - The CI scanner version (v8.18.4) needs to be bumped — check gitleaks CHANGELOG for breaking changes to allowlist behavior first - A new finding cannot be classified as real vs false positive within 30 minutes of investigation