RCA — BCP vault backup pipeline produced zero backups since inception (silent no-op from 2026-06-30)

Incident ID: 2026-08-29-bcp-vault-backup-silent-noop Date discovered: 2026-08-29 (via prior read-only cross-repo audit, TradeMasterAPI issue #4535) Severity: SEV-3 Duration: From the original GitHub Actions workflow (2026-05-29, #3049) until at least 2026-10-30, the daily vault snapshot has never produced a backup object in s3://raxx-iac-state-prod/vault-exports/. It reported green (exit 0) from 2026-06-30 (soft-skip added to the GHA workflow) until 2026-09-11 (#4598, hard-fail on develop); before 2026-06-30 it failed red at the GPG step with a cryptic error, and the key is still not confirmed provisioned (see Action items). Blast radius: vault.raxx.app (self-hosted Infisical) had zero off-host encrypted backups the entire time. 22 sibling repos/services depend on this vault for secrets. No customer-facing surface affected directly, but a total vault loss during this window (host failure, corruption, compromise) would have had no BCP recovery path beyond the separate Lightsail AutoSnapshot (7-day retention, whole-disk, not app-level export). Author: sre-agent

Timeline of the preflight behaviour (verified against the commit history, UTC)

Summary

The vault-export step of .woodpecker/bcp-vault-snapshot-daily.yaml has a preflight check for GPG_BACKUP_PUBLIC_KEY (required to encrypt the export before it leaves the host). GPG_BACKUP_PUBLIC_KEY was never provisioned at Infisical path /MooseQuest/bcp/ (originally it was expected as a GitHub Actions repo secret). When the preflight found the key empty, it printed a warning and executed exit 0 — a soft skip that reports pipeline success. Every subsequent step (GPG import, export, upload, S3-size verification) was simply never reached, and the notify-on-failure step (Slack/Postmark page) never fired because the pipeline never registered as failed. The result: a backup pipeline that ran daily, reported green from 2026-06-30 onward, and never once wrote a backup — an invisible BCP gap that would only have been discovered at restore time.

Root cause analysis

• Primary cause — success/failure signaling was inverted for the one condition that matters most. A missing encryption key is not a benign, skippable condition for a backup pipeline; it is the definition of backup failure (no artifact can be produced). The preflight treated it as the former (exit 0) instead of the latter (exit 1), which suppressed the pageable failure path entirely.
• Contributing factor — the key was never provisioned in the first place. GPG_BACKUP_PUBLIC_KEY at /MooseQuest/bcp/ was never set when the pipeline was built or ported (issue #4016 was the Woodpecker port; the original workflow is #3049). Because the pipeline didn't hard-fail, this gap was never surfaced as an alert, a failed CI run, or anything else an on-call engineer would see.
• Contributing factor — the documented runbook remediation (docs/ops/runbooks/vault-disaster-recovery.md, "BCP backup failure" section) was written for the pipeline's prior GitHub Actions incarnation (gh secret set, workflow_dispatch) and had not been updated for the Woodpecker/Infisical port (ADR-0135, Wave A batch 2). A responder following the runbook literally would have run commands that don't apply to the current tooling.
• No "zero backups produced" alerting existed independent of the pipeline's own (broken) exit code — the only signal of pipeline health was the pipeline's own status, which is exactly the signal that was compromised.

Detection

• What alerted us: a prior read-only cross-repo SRE audit inferred the gap from Infisical path enumeration (no GPG_BACKUP_PUBLIC_KEY at /MooseQuest/bcp/) and flagged it as TradeMasterAPI issue #4535. There was no automated detection from the pipeline itself at any point.
• Time between cause and detection: about three months (2026-05-29 to 2026-08-29), of which the last two months reported green (2026-06-30 onward).
• How to detect faster next time: (1) the preflight now hard-fails and pages instead of silently succeeding (this fix); (2) recommend an independent, pipeline-external check that alerts if no new object has landed in s3://raxx-iac-state-prod/vault-exports/ within the last 25 hours, so a different failure mode in the pipeline itself (e.g. the whole step being skipped by a Woodpecker config error) can't reproduce this same blind spot (see action items).

Resolution

• What was changed (this fix, sre-agent, fix/vault-backup-pipeline): 1. .woodpecker/bcp-vault-snapshot-daily.yaml preflight step changed from exit 0 (silent skip) to exit 1 (hard fail) when GPG_BACKUP_PUBLIC_KEY is empty, so the existing notify-on-failure step (Slack + Postmark, already correctly wired to fire when: status: failure) actually pages as originally intended. This landed on develop through #4598 (2026-09-11), not through the PR that carries this RCA; that PR now contains only this RCA and the runbook refresh. 2. docs/ops/runbooks/vault-disaster-recovery.md "BCP backup failure" section updated to match the current Woodpecker/Infisical pipeline (was still describing the retired GitHub Actions/gh secret set procedure), and now documents the rvault set provisioning path (with the manifest caveat: rvault set has no --path flag), the base64 encoding the pipeline expects, a warning that scripts/ops/secrets/mint_gpg_backup_key.sh still targets a GitHub Actions secret, and the bootstrap-circularity reason the private key must never be stored in Infisical itself.
• What was not completed by this agent (see Follow-ups / blocker below): actually generating the GPG backup keypair and provisioning GPG_BACKUP_PUBLIC_KEY into /MooseQuest/bcp/, and the corresponding end-to-end verification of a real object landing in S3. This requires an Infisical admin/write identity (rvault:admin in macOS Keychain, ~/.config/rvault/admin.env, or RVAULT_ADMIN_CLIENT_ID/RVAULT_ADMIN_CLIENT_SECRET), all of which this agent's sandboxed session was denied read access to by its own permission controls. This is being reported as a hard blocker rather than worked around.

Action items

Status as of 2026-10-30 UTC (from the repo and issue #4535): #4535 is still open and labelled operator-blocked. Nothing on develop shows the key provisioned or a real object verified, so items 1 and 2 are open and past due. Item 3 is only partly covered: bcp-smoke-monthly Check 5 (.woodpecker/bcp-smoke-monthly.yaml) looks for yesterday's vault-exports/ object, but only monthly, not within 25 hours. There is no evidence of items 4 and 5 on develop.

References

• Issue: #4535 (prior read-only cross-repo audit; not closed by this fix — closing requires action items 1-2 to complete, which are blocked pending operator vault-admin action)
• Pipeline: .woodpecker/bcp-vault-snapshot-daily.yaml
• Runbook: docs/ops/runbooks/vault-disaster-recovery.md (updated as part of this fix)
• Vault tooling: vault-kit (bin/rvault, scripts/mint-identity.sh)
• Original pipeline issue: #4016