Vault disaster recovery runbook

System: vault.raxx.app — self-hosted Infisical CE on AWS Lightsail (raxx-vault, small_3_0, 2 vCPU / 2 GB RAM, us-east-1a) Owner: operator (Kristerpher) Last incident: docs/incidents/2026-07-23-vault-origin-cert-expiry.md (2026-07-23 — SEV-2, origin TLS cert expiry / CI 526s) Last reviewed: 2026-10-30 UTC (BCP backup failure: key provisioning via mint_gpg_backup_key.sh)

TLS / origin certificate (Caddy)

vault.raxx.app terminates TLS on the origin via Caddy (/etc/caddy/Caddyfile), which reverse-proxies to the local Infisical backend at 127.0.0.1:8080. As of 2026-07-24 the site uses a static Cloudflare Origin CA certificate — Caddy's automatic/on-demand Let's Encrypt ACME is disabled for this site and must stay disabled.

Why ACME cannot work here: the raxx.app zone proxies vault.raxx.app (orange cloud) and gates it with Cloudflare Access. Both standard ACME challenge types are blocked as a result: tls-alpn-01 never reaches Caddy because Cloudflare terminates client TLS at the edge, and http-01 never reaches Caddy's challenge handler because Cloudflare Access intercepts unauthenticated requests to any path (including the ACME challenge path) with a login redirect. This is architectural, not a misconfiguration — do not re-enable ACME for this site. See docs/incidents/2026-07-23-vault-origin-cert-expiry.md.

Current cert (installed 2026-07-24):

Caddyfile site block (for reference — do not hand-edit without updating cert paths to match):

vault.raxx.app {
    reverse_proxy 127.0.0.1:8080
    tls /etc/caddy/tls/cf-origin-vault.pem /etc/caddy/tls/cf-origin-vault.key
    ...
}

How to tell the origin cert needs renewal

ssh -i ~/.ssh/claude_infisical_ed25519 ubuntu@<raxx-vault-ip> \
  'sudo openssl s_client -connect localhost:443 -servername vault.raxx.app \
    </dev/null 2>/dev/null | openssl x509 -noout -issuer -dates'

Expected: issuer=...CloudFlare Origin SSL Certificate Authority..., notAfter in the future. Fleet cert-expiry monitoring (TLSOriginProbeDown{instance="vault.raxx.app"}, delivered in PR #4332) should alert well before notAfter — do not rely solely on this manual check.

Renewal procedure (before 2029-07-23, or immediately if compromised)

This requires a Cloudflare token with SSL-and-Certificates-write scope, which is minted from CLOUDFLARE_ACCESS_MGMT_TOKEN (/MooseQuest/cloudflare). See docs/ops/runbooks/cloudflare-tokens.md for the current minting procedure and docs/ops/runbooks/freescout-cert-renewal.md Failure Mode E for the general CSR → issue → install pattern. Key differences for this host vs. the FreeScout/Apache pattern:

1. Regenerate the CSR from the existing key (openssl req -new -key /etc/caddy/tls/cf-origin-vault.key -subj '/CN=vault.raxx.app') or generate a fresh key if rotating.
2. Request requested_validity: 1095 (3 years) — not 5475 (15 years).
3. Install cert + key under /etc/caddy/tls/ (create if missing), not /etc/ssl/private/ — Caddy runs as the unprivileged caddy user, which cannot traverse the root-only /etc/ssl/private/ directory the way Apache can.
4. Set ownership root:caddy, directory 750, files 640.
5. Confirm the Caddyfile tls directive points at the new paths, then sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile followed by sudo systemctl reload caddy.
6. Revoke the short-lived minting token immediately after issuance.

Break-glass: reaching Cloudflare/Infisical credentials when the vault edge itself is down

If vault.raxx.app's TLS is broken (as in this incident), every Cloudflare credential capable of fixing it normally lives inside the down vault — a bootstrap circularity. The canonical recovery path is to bypass the broken Cloudflare edge entirely and reach the Infisical backend directly on the origin's loopback interface over SSH:

# 1. SSH to the origin (Infisical listens on 127.0.0.1:8080; Caddy's
#    reverse_proxy target — confirm via `cat /etc/caddy/Caddyfile`).
ssh -i ~/.ssh/claude_infisical_ed25519 ubuntu@<raxx-vault-ip>

# 2. From a second local shell (or piped via stdin to avoid secrets in remote
#    argv/history — never echo secret values):
printf '%s\n%s\n' "$INFISICAL_CLIENT_ID" "$INFISICAL_CLIENT_SECRET" \
  | ssh -i ~/.ssh/claude_infisical_ed25519 ubuntu@<raxx-vault-ip> '
    read -r CID; read -r CSEC
    curl -sS -X POST -H "Content-Type: application/json" \
      --data-binary "{\"clientId\":\"${CID}\",\"clientSecret\":\"${CSEC}\"}" \
      http://127.0.0.1:8080/api/v1/auth/universal-auth/login
  '
# Returns an accessToken usable against http://127.0.0.1:8080/api/v3/secrets/raw/<NAME>
# with the usual workspaceId/environment/secretPath query params. Fetch secrets BY
# NAME only — do not enumerate/list the full path.

This works because the Infisical container itself is healthy even when only the Cloudflare-facing TLS termination in front of it is broken — SSH to the host is a completely independent access path from the CF edge. Treat this as the documented, rehearsed procedure, not an improvisation, whenever the vault edge is unreachable and a Cloudflare-capable credential is needed to fix it.

Snapshot schedule and retention

AutoSnapshot was enabled on 2026-05-21 via:

aws lightsail enable-add-on \
  --resource-name raxx-vault \
  --add-on-request 'addOnType=AutoSnapshot,autoSnapshotAddOnRequest={snapshotTimeOfDay=05:00}' \
  --region us-east-1

Future hardening (post-launch): migrate snapshot management to the Terraform root that will manage raxx-vault (no Lightsail TF root exists as of 2026-05-21). See #2652 action item.

BCP backup failure — pageable event

The daily vault snapshot (.woodpecker/bcp-vault-snapshot-daily.yaml, Win 3) failure is a pageable event. A failed backup means no encrypted off-Infisical copy of secrets was created for that day.

Alert channels (both fire on failure): - Slack: ops webhook channel (via SLACK_WEBHOOK_URL) - Email: ops@raxx.app via Postmark (via POSTMARK_OPS_ALERT_TOKEN)

How to investigate a backup failure:

1. Open the failing run in Woodpecker (.woodpecker/bcp-vault-snapshot-daily.yaml, vault-export step). Look at which command failed. Since #4598 (2026-09-11), a missing GPG_BACKUP_PUBLIC_KEY hard-fails this step (exit 1) instead of silently exiting 0 — do not reintroduce a soft-skip here.
2. Most common cause: GPG_BACKUP_PUBLIC_KEY missing or empty at Infisical path /MooseQuest/bcp/. Fix: operator provisions a dedicated backup GPG keypair, keeps the private key offline (never in Infisical, never in this repo — see bootstrap-circularity note below), and provisions only the public half. The pipeline runs base64 --decode on the value, so it must be stored base64-encoded, and the key must have an encryption-capable key or subkey (a sign-only key makes the pipeline's gpg --encrypt fail with Unusable public key).

a) Mint a new keypair (preferred): scripts/ops/secrets/mint_gpg_backup_key.sh. It generates an ed25519 key with a cv25519 encryption subkey in a throwaway keyring, writes the public key to the vault as base64 at the exact place the pipeline reads it (Infisical env prod, path /MooseQuest/bcp, secret GPG_BACKUP_PUBLIC_KEY), writes the private key to ~/raxx-bcp-private-key.asc (mode 600), reads the vault value back the way the pipeline does and checks the fingerprint. It never prints key material. Run it from an interactive terminal (gpg asks for a passphrase through pinentry; keep that passphrase with the offsite copy, because restoring a snapshot needs it) with the vault write credentials in the environment: bash export INFISICAL_CLIENT_ID=... INFISICAL_CLIENT_SECRET=... INFISICAL_PROJECT_ID=... export CF_ACCESS_CLIENT_ID=... CF_ACCESS_CLIENT_SECRET=... # vault.raxx.app is CF Access gated bash scripts/ops/secrets/mint_gpg_backup_key.sh --dry-run # shows the plan, touches nothing bash scripts/ops/secrets/mint_gpg_backup_key.sh Use a non-empty passphrase at the pinentry prompt: with an empty one, GnuPG 2.5.x drops the secret encryption subkey on export and import, and restore fails.

INFISICAL_PROJECT_ID must be the project behind the INFISICAL_PROJECT_ID Woodpecker org secret. If GPG_BACKUP_PUBLIC_KEY already exists the script shows its fingerprint and asks before replacing it: snapshots already in S3 are encrypted to that key, so keep its private key. Afterwards copy ~/raxx-bcp-private-key.asc to offsite storage and delete the local file; it is the only copy. (Earlier versions of this script, tracked in #4535, published the armored key, unencoded, as a GitHub Actions repo secret, which this pipeline never reads. Do not use an old checkout.)

b) Re-using an existing keypair: provision the public half by hand. bash gpg --armor --export YOUR_BACKUP_KEY_FINGERPRINT | base64 | tr -d '\n' > /tmp/gpg_backup_public_key.b64 rvault set GPG_BACKUP_PUBLIC_KEY --file /tmp/gpg_backup_public_key.b64 rm -f /tmp/gpg_backup_public_key.b64 The secret must land at /MooseQuest/bcp/, env prod, in the Infisical project the pipeline reads (the project behind the INFISICAL_PROJECT_ID Woodpecker org secret). rvault set has no --path flag: it takes project, env and folder from the nearest .vault.toml, and it also appends the key name to that manifest. Do not run it from the repo root — this repo's .vault.toml targets /Raxx/Raptor (env dev), so the secret would land in the wrong place. Run it from a scratch directory whose .vault.toml has env = "prod" and secret_path = "/MooseQuest/bcp". It needs the rvault admin identity — see vault-kit README.md / SECURITY.md for how admin credentials are resolved; per-repo read-only identities cannot write. 3. Second most common: CF Access service token expired (CF_ACCESS_CLIENT_ID / CF_ACCESS_CLIENT_SECRET). Fix: rotate per docs/ops/runbooks/rotation/cloudflare-access-service-token.md. 4. Infisical universal auth credentials expired (INFISICAL_CLIENT_ID / INFISICAL_CLIENT_SECRET). Fix: rotate in the Infisical dashboard and update the Woodpecker org secret. 5. AWS IAM key permissions issue (AWS_BACKUP_ACCESS_KEY_ID). Fix: verify key has s3:PutObject + s3:HeadObject on raxx-iac-state-prod bucket.

Why the private key can't live in the same vault it backs up: if the vault itself is ever lost and the only copy of the decryption key was stored inside it, the backup is unrecoverable — a bootstrap circularity. The private key must be held by the operator outside Infisical (offline/local keychain), separate from the system it's meant to be the recovery path for.

After fixing: trigger bcp-vault-snapshot-daily manually in Woodpecker (event: manual) and confirm a .json.gpg object > 1KB appears in s3://raxx-iac-state-prod/vault-exports/$(date -u +%Y-%m-%d).json.gpg (the pipeline's own verify step checks this, but confirm independently too).

How to tell the vault needs recovery

• Symptom 1: vault.raxx.app is unreachable (CF Access login page never loads; curl returns connection refused or timeout).
• Symptom 2: Lightsail console shows raxx-vault in Stopped or Error state.
• Symptom 3: Agent sessions fail with vault-read errors even after confirming the CF WAF skip rule is live (see docs/ops/runbooks/vault-access.md).
• Symptom 4: AWS Lightsail support event for us-east-1a AZ affecting instance.

How to diagnose (in order)

1. Check Lightsail instance state:

aws lightsail get-instance \
  --instance-name raxx-vault \
  --region us-east-1 \
  --query 'instance.state'

Expected: {"code": 16, "name": "running"}.

2. Attempt a direct health probe (if instance is running but vault is not serving):

# Requires CF Access service-token headers or operator browser session
# From an agent session:
curl -sf \
  -H "CF-Access-Client-Id: $CF_ACCESS_CLIENT_ID" \
  -H "CF-Access-Client-Secret: $CF_ACCESS_CLIENT_SECRET" \
  -H "User-Agent: raxx-sre/1.0" \
  "https://vault.raxx.app/api/v1/health" \
  | python3 -m json.tool

Expected: {"status": "ok"} or equivalent Infisical health response. A 502/504 with the instance running usually means the Infisical Docker container has crashed — SSH in and check.

3. SSH into the instance to check container state:

aws lightsail download-default-key-pair --region us-east-1 \
  --output text --query 'privateKeyBase64' \
  | base64 --decode > /tmp/ls-key.pem
chmod 600 /tmp/ls-key.pem

# Get the public IP
aws lightsail get-instance \
  --instance-name raxx-vault \
  --region us-east-1 \
  --query 'instance.publicIpAddress' \
  --output text

ssh -i /tmp/ls-key.pem bitnami@<PUBLIC_IP> "sudo docker ps -a; sudo docker logs infisical --tail 50"

Restore procedure

Step 1 — List available snapshots

aws lightsail get-auto-snapshots \
  --resource-name raxx-vault \
  --region us-east-1

AutoSnapshot names follow the format raxx-vault-auto-YYYY-MM-DD. Pick the most recent healthy snapshot.

For any manually-created snapshots:

aws lightsail get-instance-snapshots \
  --region us-east-1 \
  --query 'instanceSnapshots[?fromInstanceName==`raxx-vault`].[name,state,createdAt]'

Step 2 — Create replacement instance from snapshot

aws lightsail create-instance-from-snapshot \
  --instance-name raxx-vault-restore \
  --instance-snapshot-name <snapshot-name> \
  --availability-zone us-east-1a \
  --bundle-id small_3_0 \
  --region us-east-1

Wait for the instance to reach running state:

aws lightsail get-instance \
  --instance-name raxx-vault-restore \
  --region us-east-1 \
  --query 'instance.state'

Typical boot time: 3–5 minutes.

Step 3 — Verify the restored instance is healthy

Get the new public IP:

aws lightsail get-instance \
  --instance-name raxx-vault-restore \
  --region us-east-1 \
  --query 'instance.publicIpAddress' \
  --output text

Probe directly (bypassing CF Access) to confirm Infisical is serving:

curl -sf "http://<NEW_IP>:80/api/v1/health"

If healthy, proceed to DNS swap. If unhealthy, SSH in and check container logs (Step 3 of diagnose section above).

Step 4 — DNS swap

The vault hostname (vault.raxx.app) is a Cloudflare-proxied DNS record pointing to the Lightsail static IP.

Assign a new static IP to the restored instance (recommended — avoids DHCP churn):

# Allocate a new static IP
aws lightsail allocate-static-ip \
  --static-ip-name raxx-vault-restore-static \
  --region us-east-1

# Attach it
aws lightsail attach-static-ip \
  --static-ip-name raxx-vault-restore-static \
  --instance-name raxx-vault-restore \
  --region us-east-1

# Get the address
aws lightsail get-static-ip \
  --static-ip-name raxx-vault-restore-static \
  --region us-east-1 \
  --query 'staticIp.ipAddress' \
  --output text

Update the Cloudflare DNS A record for vault.raxx.app to point at the new static IP. Use the CLOUDFLARE_EDIT_DNS token (not CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN — wrong scope; see docs/ops/runbooks/cloudflare-tokens.md and memory reference_cloudflare_tokens.md):

# Zone ID for raxx.app
ZONE_ID=$(infisical secrets get CF_ZONE_ID_RAXX_APP \
  --path /MooseQuest/cloudflare/ --plain)

CF_DNS_TOKEN=$(infisical secrets get CLOUDFLARE_EDIT_DNS \
  --path /MooseQuest/cloudflare/ --plain)

# Get the record ID for vault.raxx.app
RECORD_ID=$(curl -sS \
  -H "Authorization: Bearer $CF_DNS_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?name=vault.raxx.app&type=A" \
  | python3 -c "import sys,json; r=json.load(sys.stdin)['result']; print(r[0]['id'])")

# Update to new IP
NEW_IP=<new static IP from above>

curl -sS -X PUT \
  -H "Authorization: Bearer $CF_DNS_TOKEN" \
  -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
  --data "{\"type\":\"A\",\"name\":\"vault.raxx.app\",\"content\":\"$NEW_IP\",\"ttl\":60,\"proxied\":true}"

CF propagation with proxy enabled: near-instant (CF caches the origin IP).

Step 5 — Verify end-to-end

curl -sf \
  -H "CF-Access-Client-Id: $CF_ACCESS_CLIENT_ID" \
  -H "CF-Access-Client-Secret: $CF_ACCESS_CLIENT_SECRET" \
  -H "User-Agent: raxx-sre/1.0" \
  "https://vault.raxx.app/api/v1/health"

Then confirm a secret read works (agent session):

infisical secrets get SENTRY_INTERNAL_INTEGRATION_TOKEN \
  --env=prod \
  --path=/MooseQuest/sentry \
  --plain | head -c 8

Expected: 8 non-empty characters.

Step 6 — Rename + clean up

Once raxx-vault-restore is confirmed healthy and DNS is pointing at it:

# There is no Lightsail rename-instance command — stop the old instance
# and detach its static IP before re-allocating if needed.
# Retain the old instance for 24h before deletion as a rollback option.
aws lightsail stop-instance \
  --instance-name raxx-vault \
  --region us-east-1

# After 24h clean-up (operator decision):
aws lightsail delete-instance \
  --instance-name raxx-vault \
  --region us-east-1

# Re-enable AutoSnapshot on the restored instance:
aws lightsail enable-add-on \
  --resource-name raxx-vault-restore \
  --add-on-request 'addOnType=AutoSnapshot,autoSnapshotAddOnRequest={snapshotTimeOfDay=05:00}' \
  --region us-east-1

RTO target

1–2 hours from first alert to vault serving traffic, assuming: - A healthy auto-snapshot is available (7-day window). - The operator has AWS CLI access and CF DNS edit scope. - No AZ-level AWS outage (if us-east-1a is affected, restore to us-east-1b by changing --availability-zone in Step 2).

Manual snapshot (on-demand, before risky ops)

Take a manual snapshot before any maintenance that modifies the vault host (OS updates, Docker upgrades, config changes):

aws lightsail create-instance-snapshot \
  --instance-name raxx-vault \
  --instance-snapshot-name "raxx-vault-pre-maintenance-$(date -u +%Y%m%d-%H%M)" \
  --region us-east-1

Manual snapshots are NOT subject to the 7-day auto-rotation policy — they persist until explicitly deleted.

Verify auto-snapshot is still enabled

Include this in the weekly SRE sweep:

aws lightsail get-instance \
  --instance-name raxx-vault \
  --region us-east-1 \
  --query 'instance.addOns'

Expected:

[
    {
        "name": "AutoSnapshot",
        "status": "Enabled",
        "snapshotTimeOfDay": "05:00"
    }
]

If status is Disabled or the add-on is missing, re-enable using the command in §Snapshot schedule and retention above.

Emergency stop (clean shutdown)

To stop the vault instance without data loss (e.g., if the host is compromised and needs isolation):

aws lightsail stop-instance \
  --instance-name raxx-vault \
  --region us-east-1

Note: stopping the instance will break all agent sessions, CI pipelines, Velvet rotation, and any service that reads from Infisical. Coordinate with the operator before executing outside an active security incident.

Escalation

Wake the operator when: - The restore procedure fails at any step and no healthy snapshot is available. - The AZ us-east-1a is degraded per AWS status (cross-AZ restore changes the --availability-zone value but requires operator decision). - The Infisical master encryption key or database is corrupt (snapshot may not help — escalate to Infisical support). - Estimated recovery time exceeds the 2-hour RTO target.

AWS status page: https://health.aws.amazon.com/health/status Lightsail service health: https://health.aws.amazon.com/health/status (filter: Lightsail, us-east-1)

Cross-references

• Vault access + WAF runbook: docs/ops/runbooks/vault-access.md
• Secrets source of truth: all vault credentials live in Infisical — never inline in repo files (memory: feedback_no_inline_secrets_in_repo.md, feedback_secrets_in_vault_sop.md)
• DNS token scope: memory reference_cloudflare_tokens.md — use CLOUDFLARE_EDIT_DNS, not the automation token
• Burr SSO gateway pattern (CF Access as OIDC provider): memory project_burr_sso_gateway.md — vault is the first Burr consumer; CF Access policy for vault.raxx.app must be preserved during DNS swap
• AWS workload secrets in SSM (not Infisical): memory feedback_aws_workloads_use_ssm_not_vault.md
• Issue #2652 (this work): Lightsail AutoSnapshot enabling + TF hardening backlog
• Issue #2650 (SEV-4 diagnostic): instance name + bundle confirmation
• RCA: docs/incidents/2026-07-23-vault-origin-cert-expiry.md — origin TLS cert expiry, static Origin CA cert migration, break-glass credential-access procedure
• Cloudflare tokens runbook: docs/ops/runbooks/cloudflare-tokens.md
• FreeScout cert renewal runbook (prior art for Cloudflare Origin CA issuance): docs/ops/runbooks/freescout-cert-renewal.md