Raxx · internal docs

internal · gated

Break-glass admin provisioning runbook

System: agent-breakglass-admin identity + scripts/ops/breakglass-admin.sh Owner: operator only (Kristerpher) — every step in this runbook is a Tier C action per its own classification policy (policy/breakglass-classification.yaml lists edits-to-this-file-adjacent operations, but identity minting itself is not something any agent session can perform at all, at any tier — see ADR-0151 §6). ADR: ADR-0151 (graduated agent write-authority) Last reviewed: 2026-08-31 (initial) Not this runbook: the console's break-glass RBAC group (time-limited elevation minted via POST /api/rbac/grants) is documented in console-rbac-break-glass.md.

What this identity is for

agent-breakglass-admin is the single, dedicated credential that scripts/ops/breakglass-admin.sh uses to perform Tier B (pre-approved) and Tier C (per-request-approved) operations. It is deliberately:

Step 1: mint the identity (operator-only, cannot be delegated to any session)

This is an admin-level write to Infisical (org-admin-scoped role, per ADR-0151's brief) — the same category of action /onboard-vault's own docs describe as requiring an existing admin identity that only the operator holds. No agent session can bootstrap this, by design (ADR-0149's core rule: an admin identity cannot be minted by anything that isn't already an admin identity).

  1. Sign in to the Infisical org (vault.raxx.app) as the operator.
  2. Create a new machine identity named agent-breakglass-admin.
  3. Attach Universal Auth to it.
  4. Scope the role as narrowly as the break-glass operations actually need — this runbook does not prescribe a specific role name because the exact Infisical role model may have changed since ADR-0151 was written; the requirement is: enough access to perform the Tier B/C operations breakglass-admin.sh implements (see the script's case "$OPERATION" block for the current, authoritative list), and no more. Do not default to a blanket org-admin role if a narrower one satisfies the actual operations — re-evaluate this each time a new Tier B/C executor is added to the script.
  5. Set the Universal Auth access-token TTL to the shortest value Infisical allows that's still ≥30 minutes (ADR-0151 §5.4's target range is 30–60 minutes) — this bounds how long a minted token can be used even if something goes wrong downstream of minting.
  6. Mint exactly one client secret. Record the clientId:clientSecret pair — you'll store it in the next step, not anywhere else (not a password manager note, not a chat message, not a file on disk outside the SSM parameter).

Step 2: store the credential where the wrapper reads it

aws ssm put-parameter \
  --name "/ci/breakglass/agent-breakglass-admin-credential" \
  --value "<clientId>:<clientSecret>" \
  --type SecureString \
  --region us-east-2

Do not run this with the value visible in shell history if avoidable — prefer piping from a source that doesn't land in .bash_history/.zsh_history, or clear history immediately after. This is the same SSM-SecureString pattern already used for every other credential in infra/ci/ — see docs/ops/runbooks/ci-woodpecker.md's secrets-fill-in checklist for the house convention if you want a second reference point.

Step 3: verify provisioning succeeded

From a shell with your own AWS credentials (not an agent session):

scripts/ops/breakglass-admin.sh wp-pipeline-cancel 1 999999 --reason "provisioning dry-run — nonexistent pipeline, expected to fail at the WP API layer, not at the provisioning check"

Expect: the script gets PAST the "not provisioned" refusal (you should NOT see the not_provisioned_or_killswitched audit line) and instead fails later, at whatever point the underlying wp-pipeline.sh cancel call fails for pipeline 999999 (which doesn't exist) — that failure is expected and fine; it's proof the provisioning check itself now passes. Check /tmp/raxx-ops-audit.log for the full record.

Step 4: the kill switch (use any time)

aws ssm delete-parameter \
  --name "/ci/breakglass/agent-breakglass-admin-credential" \
  --region us-east-2

Immediately and completely disables breakglass-admin.sh — every subsequent invocation, Tier B or Tier C, hits the "not provisioned" refusal path again. Re-provision by repeating Steps 1–2 with a freshly-minted client secret (don't reuse an old one if you have any doubt about why you needed to kill-switch it).

Separately, you can also revoke just the client secret from the Infisical UI without deleting the SSM parameter, if you want a faster single-action revocation without needing AWS access at that moment — either action achieves the same outcome (the identity can no longer authenticate), so use whichever is faster to reach at the time.

breakglass-admin.sh's GitHub-comment approval verification for Tier C is not yet implemented (see the script's inline comment at the relevant case branch) — it currently refuses every non-interactive Tier C request outright rather than accepting an unverified one. Until that verification path is completed and adversarially reviewed, Tier C is only usable from an interactive session (the operator answering an AskUserQuestion directly). Do not consider Tier C available for unattended/cloud sessions before that follow-up lands — this is a known, deliberate gap, not an oversight to route around.

Dry-run: exercising the refusal paths without ever completing a privileged operation

Useful for confirming the script's classification/refusal logic works as documented, without needing the identity provisioned at all:

# Missing --reason
scripts/ops/breakglass-admin.sh wp-pipeline-cancel 1 1

# Unclassified operation -> safe default to Tier C, refuses without --approval-ref
scripts/ops/breakglass-admin.sh some-made-up-operation --reason "testing"

# Tier A operation -> refused, told to use the direct [ADR-0149](https://internal-docs.raxx.app/architecture/adr/0149-session-agent-credential-model.html) wrapper
scripts/ops/breakglass-admin.sh vault-secret-write-scoped --reason "testing"

Each should produce a clear refusal message and (for the latter two, which reach the point where TIER is known) an audit line in /tmp/raxx-ops-audit.log — check that file after running these to confirm the audit mechanism itself is working, independent of whether the identity is provisioned.