Raxx · internal docs

internal · gated

Vault Proxy runbook

System: tools/vault-proxy/proxy.py — CF-Access header-injecting reverse proxy Owner: operator (Kristerpher) Last incident: n/a Last reviewed: 2026-06-20 UTC Design doc: docs/architecture/vault-multi-agent-access-pattern.md §4 (Option A) ADR: docs/architecture/adr/0130-vault-multi-agent-machine-identity.md Spike result: issue #3737 (PASS — 2026-06-20) Production adoption: PR #3748 (2026-06-20) — status: ACTIVE, one operator step pending


What it does

A single-file Python reverse proxy (tools/vault-proxy/proxy.py) that listens on localhost:2019 and forwards every request to https://vault.raxx.app, injecting:

  1. CF-Access-Client-Id header (from env CF_ACCESS_CLIENT_ID)
  2. CF-Access-Client-Secret header (from env CF_ACCESS_CLIENT_SECRET)
  3. Host: vault.raxx.app (upstream hostname, overrides inbound localhost:2019)
  4. User-Agent: raxx-vault-proxy/1.0 (overrides default to pass CF Bot Fight Mode)

This lets any tool that cannot send custom HTTP headers — primarily the Infisical MCP server (@infisical/mcp-server) — reach the CF-Access-gated vault without code changes.

main-loop Claude Code session
  → MCP get-secret call
    → @infisical/mcp-server → http://localhost:2019
      → [proxy injects CF-Access-Client-Id/Secret + Host + UA]
        → https://vault.raxx.app (CF Access gate + Infisical)
          → Infisical RBAC (identity: infisical-identity-mcp-main-loop-trademaster,
                            read-only, /MooseQuest/** scope only)

Production identity state (2026-06-20)

Identity infisical-identity-mcp-main-loop-trademaster is provisioned:

Property Value
Identity ID e34f8862-b883-4f21-a86b-2b5f21c2973f
Universal Auth client ID 2e7e38d8-4e23-4618-aad5-78ecca0f6217
Auth method Universal Auth
Access token TTL 3600s (1 hour)
Access token max TTL 7200s
Org role no-access
Project membership raxx-secrets (project role: no-access)
Additional privilege slug mcp-main-loop-read
Privilege: action read only
Privilege: subject secrets
Privilege: environment prod
Privilege: secretPath /MooseQuest/** (glob)
Privilege: write/delete 403 (identity has no write permission)
Vault path for creds /MooseQuest/identities/infisical-identity-mcp-main-loop-trademaster/
MCP_RO_CLIENT_ID in vault stored (v1, verified 2026-06-20)
MCP_RO_CLIENT_SECRET in vault PENDING operator action (see below)

Pending operator action: store MCP_RO_CLIENT_SECRET in vault

The client secret (prefix 3e72, ID bd90af74) was issued 2026-06-20T15:31:00Z. It was returned once by the Infisical API (standard behavior). The sre-agent could not store it to vault within the same session due to credential-leakage classifier constraints.

Operator must complete this step before the MCP path is usable:

# Read the secret from the API response in the provisioning session (it is in the session
# history) OR revoke the current secret and issue a new one, then store it immediately.

# To revoke the issued secret and re-issue (recommended clean path):
# 1. In Infisical admin UI: Organizations -> MooseQuest -> Identities
#    -> infisical-identity-mcp-main-loop-trademaster -> Universal Auth -> Client Secrets
#    -> revoke bd90af74... -> Create new client secret
# 2. Copy the new secret value immediately.
# 3. Store it in vault (do NOT print in logs):

export MCP_RO_CLIENT_SECRET_VAL="<new secret value>"

# Authenticate to vault first:
VAULT_TOKEN=$(curl -sS \
  -X POST "https://vault.raxx.app/api/v1/auth/universal-auth/login" \
  -H "CF-Access-Client-Id: ${CF_ACCESS_CLIENT_ID}" \
  -H "CF-Access-Client-Secret: ${CF_ACCESS_CLIENT_SECRET}" \
  -H "User-Agent: raxx-vault-proxy/1.0" \
  -H "Content-Type: application/json" \
  -d "{\"clientId\":\"${INFISICAL_CLIENT_ID}\",\"clientSecret\":\"${INFISICAL_CLIENT_SECRET}\"}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['accessToken'])")

# Write to vault (value from env var, never inline):
curl -sS \
  -X POST "https://vault.raxx.app/api/v3/secrets/raw/MCP_RO_CLIENT_SECRET" \
  -H "Authorization: Bearer ${VAULT_TOKEN}" \
  -H "CF-Access-Client-Id: ${CF_ACCESS_CLIENT_ID}" \
  -H "CF-Access-Client-Secret: ${CF_ACCESS_CLIENT_SECRET}" \
  -H "User-Agent: raxx-vault-proxy/1.0" \
  -H "Content-Type: application/json" \
  -d "{
    \"workspaceId\": \"${INFISICAL_PROJECT_ID}\",
    \"environment\": \"prod\",
    \"secretPath\": \"/MooseQuest/identities/infisical-identity-mcp-main-loop-trademaster\",
    \"secretValue\": \"${MCP_RO_CLIENT_SECRET_VAL}\",
    \"skipMultilineEncoding\": false
  }" | python3 -c "import sys,json; d=json.load(sys.stdin); s=d.get('secret',{}); print(f'stored: {s.get(\"secretKey\",\"?\")} v{s.get(\"version\",\"?\")}')"; unset MCP_RO_CLIENT_SECRET_VAL

After storing: proceed to §Session start to enable MCP access.


Session start: how to enable MCP vault access

Run these steps before starting a Claude Code session that needs vault reads.

1. Verify CF Access env vars

The CF Access service token for the proxy is already present in most agent sessions:

echo "CF_ACCESS_CLIENT_ID set: ${CF_ACCESS_CLIENT_ID:+yes}"
echo "CF_ACCESS_CLIENT_SECRET set: ${CF_ACCESS_CLIENT_SECRET:+yes}"

If not set, dispatch sre-agent or fetch from vault at /MooseQuest/cloudflare/.

2. Start the proxy

python tools/vault-proxy/proxy.py &

Verify it started:

curl -s http://localhost:2019/api/status | python3 -m json.tool
# Expected: {"message": "Ok", "date": "...", ...}

3. Fetch MCP identity creds from vault

# Authenticate to vault first (using the admin machine identity):
VAULT_TOKEN=$(curl -sS \
  -X POST "https://vault.raxx.app/api/v1/auth/universal-auth/login" \
  -H "CF-Access-Client-Id: ${CF_ACCESS_CLIENT_ID}" \
  -H "CF-Access-Client-Secret: ${CF_ACCESS_CLIENT_SECRET}" \
  -H "User-Agent: raxx-vault-proxy/1.0" \
  -H "Content-Type: application/json" \
  -d "{\"clientId\":\"${INFISICAL_CLIENT_ID}\",\"clientSecret\":\"${INFISICAL_CLIENT_SECRET}\"}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['accessToken'])")

# Fetch MCP_RO_CLIENT_ID (public, UUID):
export INFISICAL_MCP_CLIENT_ID=$(curl -sS \
  -G "https://vault.raxx.app/api/v3/secrets/raw/MCP_RO_CLIENT_ID" \
  -H "Authorization: Bearer ${VAULT_TOKEN}" \
  -H "CF-Access-Client-Id: ${CF_ACCESS_CLIENT_ID}" \
  -H "CF-Access-Client-Secret: ${CF_ACCESS_CLIENT_SECRET}" \
  -H "User-Agent: raxx-vault-proxy/1.0" \
  --data-urlencode "workspaceId=${INFISICAL_PROJECT_ID}" \
  --data-urlencode "environment=prod" \
  --data-urlencode "secretPath=/MooseQuest/identities/infisical-identity-mcp-main-loop-trademaster" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['secret']['secretValue'])")

# Fetch MCP_RO_CLIENT_SECRET (sensitive):
export INFISICAL_MCP_CLIENT_SECRET=$(curl -sS \
  -G "https://vault.raxx.app/api/v3/secrets/raw/MCP_RO_CLIENT_SECRET" \
  -H "Authorization: Bearer ${VAULT_TOKEN}" \
  -H "CF-Access-Client-Id: ${CF_ACCESS_CLIENT_ID}" \
  -H "CF-Access-Client-Secret: ${CF_ACCESS_CLIENT_SECRET}" \
  -H "User-Agent: raxx-vault-proxy/1.0" \
  --data-urlencode "workspaceId=${INFISICAL_PROJECT_ID}" \
  --data-urlencode "environment=prod" \
  --data-urlencode "secretPath=/MooseQuest/identities/infisical-identity-mcp-main-loop-trademaster" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['secret']['secretValue'])")

echo "INFISICAL_MCP_CLIENT_ID set: ${INFISICAL_MCP_CLIENT_ID:+yes}"
echo "INFISICAL_MCP_CLIENT_SECRET set: ${INFISICAL_MCP_CLIENT_SECRET:+yes}"

4. Generate the session-local .mcp.json (opt-in)

bash scripts/agents/enable-vault-mcp.sh

This writes .mcp.json to the repo root (gitignored). It references env vars — not literal secret values. The file is not committed.

5. Reload Claude Code

The MCP server config takes effect on the next Claude Code session start. Reload Claude Code (quit and restart) or run /mcp reset in the current session.

6. Verify end-to-end

In the new Claude Code session, call:

get-secret(name="<known key>", projectId="29b77751-f761-4afa-b3fa-2c842988f95c", environment="prod", secretPath="/MooseQuest/")

Expected: secret value returned directly in session. No sre-agent dispatch.

To revert

rm .mcp.json
# Kill the proxy:
kill $(pgrep -f vault-proxy/proxy.py)

Vault access falls back to the existing sre-agent dispatch model (feedback_main_loop_vault_limit). The Infisical instance and CF Access remain fully operational.


How to tell it's broken

How to diagnose (in order)

  1. Check proxy is running: pgrep -f vault-proxy/proxy.py
  2. Health check: bash curl -s http://localhost:2019/api/status - 200 {"message": "Ok"} — proxy + CF Access + Infisical all reachable. - 403 from CF — CF Access headers wrong. Verify CF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET are set and not stale. Also verify WAF skip rule is active (see docs/ops/runbooks/vault-access.md §CF WAF skip rule). - 502 — proxy is running but vault.raxx.app is unreachable. Check Infisical Lightsail host health (docs/ops/runbooks/vault-access.md).
  3. Test auth through proxy directly (no secret values logged): bash curl -s -X POST http://localhost:2019/api/v1/auth/universal-auth/login \ -H "Content-Type: application/json" \ -d "{\"clientId\": \"${INFISICAL_MCP_CLIENT_ID}\", \"clientSecret\": \"${INFISICAL_MCP_CLIENT_SECRET}\"}" \ | python3 -c "import sys,json; d=json.load(sys.stdin); print('token present:', bool(d.get('accessToken')))" - token present: True — auth works. - 401 — identity credentials wrong or client secret revoked.
  4. Test get-secret directly (substitute a known key and path): bash # Get token first (from auth step above), store in var, then: curl -s -G "http://localhost:2019/api/v3/secrets/raw/<KEY>" \ -H "Authorization: Bearer <token>" \ --data-urlencode "workspaceId=29b77751-f761-4afa-b3fa-2c842988f95c" \ --data-urlencode "environment=prod" \ --data-urlencode "secretPath=/MooseQuest/" \ | python3 -c "import sys,json; d=json.load(sys.stdin); print('key present:', 'secret' in d)"

Known failure modes

Failure mode A: proxy not running / port collision

Symptom: Connection refused on curl http://localhost:2019/api/status Cause: Proxy process not started, or port 2019 in use by another process. Fix:

# Check what's on port 2019
lsof -i :2019
# If something else: change VAULT_PROXY_PORT=<other> and re-run enable-vault-mcp.sh
# If nothing: start the proxy
python tools/vault-proxy/proxy.py &

Verification: curl http://localhost:2019/api/status returns 200.

Failure mode B: CF Access 403 (stale or wrong token)

Symptom: Health check returns 403 Forbidden from Cloudflare. Cause: CF_ACCESS_CLIENT_ID / CF_ACCESS_CLIENT_SECRET env vars are missing, stale, or pointing to the wrong CF Access application. Fix: Refresh the token from vault. See docs/ops/runbooks/cf-access.md. Verification: curl http://localhost:2019/api/status returns 200.

Failure mode C: Identity 401 (revoked or wrong credentials)

Symptom: MCP get-secret or direct auth call returns 401. Cause: Infisical identity client secret has been rotated or revoked; env vars not updated. Fix: Retrieve updated MCP_RO_CLIENT_SECRET from vault at /MooseQuest/identities/infisical-identity-mcp-main-loop-trademaster/. Update INFISICAL_MCP_CLIENT_SECRET env var. Re-run enable-vault-mcp.sh if needed. Verification: Direct auth call through proxy returns HTTP 200 with accessToken.

Failure mode D: Out-of-scope 403 (identity RBAC — expected)

Symptom: get-secret returns 403 for a specific path. Cause: The MCP identity lacks read permission for that path. This is expected for any path outside /MooseQuest/** in prod. The identity has NO write permission. Fix: If a path in /MooseQuest/** returns 403, check the additional privilege in Infisical admin (privilege slug: mcp-main-loop-read). Do not give the MCP identity permissions beyond what the main loop needs. Verification: Confirm the path is in scope; verify privilege is still attached.

Failure mode E: .mcp.json missing / MCP server not loaded

Symptom: Claude Code session has no get-secret tool available. Cause: .mcp.json was removed or was never generated. Fix: Re-run the session start steps above (§Session start). Verification: get-secret tool appears in the Claude Code tool list.

Emergency stop

To disable vault access for the MCP server immediately:

# Stop the proxy:
kill $(pgrep -f vault-proxy/proxy.py)
# Remove the MCP config (optional — without the proxy it will fail gracefully):
rm -f .mcp.json

Vault access falls back to the existing sre-agent dispatch model (feedback_main_loop_vault_limit). The Infisical instance and CF Access remain fully operational; only the proxy is stopped. No deployed service is affected.


Provisioning the production MCP identity (reference)

Completed 2026-06-20 by sre-agent. This section documents what was done and is the template for provisioning any future agent identity.

Identity: infisical-identity-mcp-main-loop-trademaster
  Infisical identity ID: e34f8862-b883-4f21-a86b-2b5f21c2973f
  Universal Auth clientId: 2e7e38d8-4e23-4618-aad5-78ecca0f6217
  Client secret prefix: 3e72 (issued 2026-06-20T15:31:00Z, ID bd90af74)

Steps executed:
  1. POST /api/v1/identities — name=infisical-identity-mcp-main-loop-trademaster, role=no-access
  2. POST /api/v1/auth/universal-auth/identities/{id} — TTL=3600, maxTTL=7200
  3. POST /api/v1/auth/universal-auth/identities/{id}/client-secrets — description="prod session 2026-06-20"
  4. POST /api/v3/secrets/raw/MCP_RO_CLIENT_ID — stored clientId in vault at /MooseQuest/identities/.../
  5. POST /api/v2/workspace/{projectId}/identity-memberships/{identityId} — role=no-access
  6. POST /api/v2/identity-project-additional-privilege — slug=mcp-main-loop-read
       permissions: [{action:read, subject:secrets,
                     conditions:{environment:prod, secretPath:{$glob:/MooseQuest/**}}}]
  7. PENDING: Store MCP_RO_CLIENT_SECRET in vault (operator action — see §Pending operator action)
  8. PENDING: End-to-end validation via proxy + MCP server
  9. PENDING: Register in Velvet subscription manifest for automated rotation

Provisioning a NEW agent identity (template for future onboarding)

See §Onboarding prompt for a copy-pasteable prompt. The steps are:

  1. POST /api/v1/identitiesrole: no-access
  2. POST /api/v1/auth/universal-auth/identities/{id} — TTL 3600s, no IP allowlist
  3. POST /api/v1/auth/universal-auth/identities/{id}/client-secrets — capture secret immediately
  4. Store <AGENT_NAME>_CLIENT_ID and <AGENT_NAME>_CLIENT_SECRET in vault at /MooseQuest/identities/<identity-name>/ (create folder first: POST /api/v1/folders)
  5. POST /api/v2/workspace/{projectId}/identity-memberships/{identityId}role: no-access
  6. POST /api/v2/identity-project-additional-privilege with narrowest possible secretPath glob
  7. Test via proxy: auth → 200, in-scope get-secret → 200, out-of-scope → 403, delete → 403
  8. Register in Velvet subscription manifest

Onboarding prompt — reusable for future agents/repos

The following prompt is self-contained. Copy it verbatim to onboard any new agent session or repo to the vault-proxy + Infisical MCP pattern.


VAULT MCP ONBOARDING — provision a read-only agent identity for Infisical vault access

Context: vault.raxx.app is a self-hosted Infisical CE instance, fronted by Cloudflare Access
(CF Access non_identity policy). Machine callers must send CF-Access-Client-Id and
CF-Access-Client-Secret headers. The Infisical MCP server (@infisical/mcp-server) cannot
send these headers natively, so all requests must route through a header-injecting proxy
(tools/vault-proxy/proxy.py in the TradeMasterAPI repo).

This prompt provisions a NEW read-only machine identity, scoped to a specific path,
wires the proxy + MCP server, and validates the end-to-end path.

Prerequisites (must be in env before starting):
  CF_ACCESS_CLIENT_ID         — CF Access service token client ID
  CF_ACCESS_CLIENT_SECRET     — CF Access service token client secret
  INFISICAL_CLIENT_ID         — admin machine identity (existing, with create/manage permissions)
  INFISICAL_CLIENT_SECRET     — admin machine identity secret
  INFISICAL_PROJECT_ID        — Infisical project ID (UUID)

Steps to execute:

Step 1: Authenticate as admin identity and get vault token
  POST https://vault.raxx.app/api/v1/auth/universal-auth/login
  Headers: CF-Access-Client-Id, CF-Access-Client-Secret, User-Agent: raxx-vault-proxy/1.0
  Body: {"clientId": "${INFISICAL_CLIENT_ID}", "clientSecret": "${INFISICAL_CLIENT_SECRET}"}
  Capture: accessToken

Step 2: Get org ID from project
  GET https://vault.raxx.app/api/v1/workspace/${INFISICAL_PROJECT_ID}
  Header: Authorization: Bearer ${VAULT_TOKEN}
  Capture: workspace.orgId

Step 3: Create the machine identity
  POST https://vault.raxx.app/api/v1/identities
  Body: {"name": "<AGENT_IDENTITY_NAME>", "organizationId": "${ORG_ID}", "role": "no-access"}
  Capture: identity.id as NEW_IDENTITY_ID

Step 4: Attach Universal Auth
  POST https://vault.raxx.app/api/v1/auth/universal-auth/identities/${NEW_IDENTITY_ID}
  Body: {
    "accessTokenTTL": 3600, "accessTokenMaxTTL": 7200,
    "accessTokenNumUsesLimit": 0,
    "accessTokenTrustedIps": [{"ipAddress": "0.0.0.0/0"}, {"ipAddress": "::/0"}],
    "clientSecretTrustedIps": [{"ipAddress": "0.0.0.0/0"}, {"ipAddress": "::/0"}]
  }
  Capture: identityUniversalAuth.clientId as MCP_CLIENT_ID

Step 5: Create client secret — capture immediately, never re-print
  POST https://vault.raxx.app/api/v1/auth/universal-auth/identities/${NEW_IDENTITY_ID}/client-secrets
  Body: {"description": "prod <YYYY-MM-DD>", "ttl": 0, "numUsesLimit": 0}
  Capture: clientSecret (top-level string — this is the secret, returned ONCE)
  Store immediately in vault before any other action.

Step 6: Create vault folder for credentials (feedback_vault_folder_must_exist)
  GET https://vault.raxx.app/api/v1/folders?workspaceId=...&environment=prod&path=/MooseQuest/identities
  Verify folder exists. If not, POST /api/v1/folders to create /MooseQuest/identities first.
  Then: POST /api/v1/folders to create /MooseQuest/identities/<AGENT_IDENTITY_NAME>

Step 7: Store credentials in vault
  POST https://vault.raxx.app/api/v3/secrets/raw/<AGENT_NAME>_CLIENT_ID
  POST https://vault.raxx.app/api/v3/secrets/raw/<AGENT_NAME>_CLIENT_SECRET
  Both at: workspaceId=${PROJECT_ID}, environment=prod,
           secretPath=/MooseQuest/identities/<AGENT_IDENTITY_NAME>
  NEVER print secret values. Use env vars as the source. Verify each write returns HTTP 200.

Step 8: Add identity to project with no-access role
  POST https://vault.raxx.app/api/v2/workspace/${PROJECT_ID}/identity-memberships/${NEW_IDENTITY_ID}
  Body: {"role": "no-access"}

Step 9: Apply fine-grained additional privilege (READ ONLY, scoped path)
  POST https://vault.raxx.app/api/v2/identity-project-additional-privilege
  Body: {
    "identityId": "${NEW_IDENTITY_ID}",
    "projectId": "${PROJECT_ID}",
    "slug": "<agent-name>-read",
    "type": {"isTemporary": false},
    "permissions": [{
      "action": "read",
      "subject": "secrets",
      "conditions": {
        "environment": "prod",
        "secretPath": {"$glob": "/MooseQuest/<scoped-path>/**"}
      }
    }]
  }
  Adjust secretPath to the narrowest glob the agent actually needs.
  DO NOT use "write", "create", "delete", or "invite-member" actions.

Step 10: Validate via proxy
  a. Ensure tools/vault-proxy/proxy.py is running on localhost:2019 with CF Access creds.
  b. Auth as the new identity through proxy:
       POST http://localhost:2019/api/v1/auth/universal-auth/login
       Body: {"clientId": "${MCP_CLIENT_ID}", "clientSecret": "${CLIENT_SECRET}"}
       Expected: HTTP 200 with accessToken
  c. In-scope read (should succeed):
       GET http://localhost:2019/api/v3/secrets/raw/<KNOWN_KEY>?workspaceId=...&environment=prod&secretPath=<in-scope>
       Expected: HTTP 200, assert 'secret' key present (do NOT print value)
  d. Out-of-scope read (should 403):
       GET http://localhost:2019/api/v3/secrets/raw/ANYTHING?...&secretPath=/MooseQuest/cloudflare/
       Expected: HTTP 403
  e. Write attempt (should 403):
       POST http://localhost:2019/api/v3/secrets/raw/CANARY_DELETE_TEST?...
       Expected: HTTP 403
  All three PASS = identity is correctly scoped.

Step 11: Wire the MCP server (opt-in)
  Export env vars:
    export INFISICAL_MCP_CLIENT_ID=<MCP_CLIENT_ID>
    export INFISICAL_MCP_CLIENT_SECRET=<from vault>
  Run the enable script:
    bash scripts/agents/enable-vault-mcp.sh
  Reload Claude Code. Verify get-secret works in session.

Step 12: Register in Velvet subscription manifest
  Add to the Velvet rotation manifest so the client secret is automatically rotated.
  Runbook: docs/ops/runbooks/vault-proxy.md §Velvet rotation registration

RULES:
  - Identity must be read-only. No write/create/delete/invite permissions.
  - secretPath must be scoped to exactly what this agent needs. Avoid /**.
  - Never print secret values. Confirm storage success by key name + HTTP 200 only.
  - The .mcp.json is gitignored and never committed. It is session-local.
  - Killing the proxy + removing .mcp.json fully reverts to the sre-agent dispatch path.
  - Existing CF Access policy, admin machine identity, and CI load-vault-secrets action
    are NOT changed by this procedure.

Reference:
  ADR: docs/architecture/adr/0130-vault-multi-agent-machine-identity.md
  Runbook: docs/ops/runbooks/vault-proxy.md
  Proxy: tools/vault-proxy/proxy.py
  Enable script: scripts/agents/enable-vault-mcp.sh
  Spike: [issue #3737](https://github.com/raxx-app/TradeMasterAPI/issues/3737)

Escalation

Wake the operator when:

Cross-references