Raxx · internal docs

internal · gated

ADR 0151 — Graduated agent write-authority: autonomous work, pre-approved escalation, operator-gated break-glass for consequential actions

Status: Accepted Date: 2026-08-31 UTC Deciders: Kristerpher Henderson (operator directive, direct AskUserQuestion confirmation in the control-session, 2026-08-31); software-architect (design/authorship) Scope: scripts/ops/breakglass-admin.sh, policy/breakglass-classification.yaml, docs/ops/runbooks/breakglass-admin-provisioning.md, the agent-breakglass-admin Infisical/Woodpecker identity (operator-minted), the audit sink for all Tier B/C operations. Refs: ADR-0149 (session/agent credential model — bounded scripts, no shared admin, <slug>-rw scoped write identities), ADR-0131 (handoff invariants — HANDOFF-INVARIANT-1), #4544, #4545, #4567, #3738, project_kms_audit_chain_approved (pre-approved KMS HMAC hash-chain pattern, alias/raxx-audit-hmac, ~$2/mo, currently dormant elsewhere pending SC-A11/SC-A14).

1. Context

ADR-0149 established the house rule for agent credential access: bounded, narrowly- scoped wrapper scripts hold their own least-privilege credential internally; no session ever holds a standing admin credential; a caller-settable flag is never trusted as a security boundary. That model works well for routine mutating work — vault secret writes scoped to one repo's path, CI pipeline cancel/restart, recycling a wedged agent instance — where the credential the wrapper holds is (or, per ADR-0149 §9, will become) itself narrowly scoped.

It does not, by design, say anything about consequential actions: deleting a customer account, a destructive prod data migration, terminating a prod-serving instance, rotating a credential that gates prod access, changing IAM/SG/WAF/ CloudFront/DNS, or deleting a protected branch/tag. Those actions are currently simply unreachable by any agent session — there is no wrapper, no identity, no path. That is the correct default, and ADR-0149 leaves it there deliberately.

Tonight's incident chain (the CI fleet OOM cascade, #4545 and its follow-ups) repeatedly hit a real gap in that all-or-nothing model: routine, low-consequence, fully reversible ops (cancel a stuck pipeline, recycle one dead agent instance) needed an admin-equivalent Woodpecker token because no scoped alternative exists yet (the verb-scoped WP token and dedicated recycle IAM role from ADR-0149 §9 are not built). The operator's resolution, decided directly (not inferred, not relayed): the line should be drawn on consequence, not on capability. An agent being technically able to reach a destructive API is not itself the problem — the operator not being in the loop for a decision with real consequence is. In the operator's own words: "the agent is able to delete an account or edit a prod service, but I don't want it making that decision on its own."

This ADR defines three tiers of write-authority to operationalize that line, and is a deliberate, explicit exception to ADR-0149's "no admin identity ever reaches a session" posture — narrowly, for break-glass tiers only, under the guardrails in §5–§8.

2. Decision

Three tiers of agent write-authority, classified by consequence and reversibility, not by which system the action touches:

Tier A — Autonomous

Routine mutating work through ADR-0149's scoped wrapper scripts and <slug>-rw per-repo write identities. No approval, no elevated credential, no break-glass identity involved. This tier is unchanged by this ADR — it is the steady state for the vast majority of agent work and stays exactly as ADR-0149 defines it.

Tier B — Pre-approved break-glass (no per-request human gate)

A named, closed list of operations that are: - Low-consequence — bounded blast radius, no customer/financial impact, no prod outage risk if done correctly. - Reversible — the action itself, or its effect, can be undone without data loss (a cancelled pipeline can be re-run; a recycled instance is replaced by design; a non-prod/staging-only write doesn't touch anything customer-facing).

Initial Tier B list (see policy/breakglass-classification.yaml for the authoritative, versioned copy — this ADR's list is illustrative, the YAML is canonical): - Woodpecker pipeline cancel/restart (the operations scripts/ops/wp-pipeline.sh already implements — this ADR does not replace that script; breakglass-admin.sh invokes the same underlying operations under the classification/audit framework defined here, using the break-glass identity in place of the raw admin token wp-pipeline.sh currently reads directly, once that identity exists). - Recycling a single wedged CI agent instance (same relationship to scripts/ops/ci-agent-recycle.sh). - A non-prod (dev/staging) secret write. - A staging-only config or deploy change.

An agent session that needs a Tier B operation mints a short-TTL access token for the dedicated agent-breakglass-admin identity (see §6), performs the ONE named operation, audits it (§7), and lets the token expire. No human is asked — the policy classification itself is the pre-approval, decided once by the operator when the classification file is authored/amended (and that file is itself Tier C to edit — see §4).

Tier C — Operator-gated break-glass (per-request human gate)

Everything with real consequence: account/customer deletion, destructive data operations or destructive prod migrations, terminating or rebooting a prod-serving instance, prod config/deploy/rollback outside the normal gated pipeline, credential rotation or revocation that affects prod access, IAM/security-group/WAF/CloudFront/DNS changes, deleting a protected branch or tag, and any prod write whose failure mode includes outage or data loss.

Flow: the agent requests (states the operation, the target, and the reason); the operator approves that specific request; only then does the wrapper mint a token scoped to that one named operation, execute it, audit it, and let the token expire. The approval is per-request — a standing "yes you can do Tier C things" is not a valid approval for this tier; each request needs its own.

Safe default

Anything unclassified, or ambiguous between tiers, is Tier C. There is no "assume it's fine" fallback. The classification file (policy/breakglass-classification.yaml) that maps operations to tiers is itself a Tier C artifact to edit — reclassifying something down from C to B, or adding a new B-tier operation, requires the same per-request operator approval as any other Tier C action. This closes the obvious end-run (an agent quietly widening its own pre-approved list).

3. Relationship to ADR-0149

ADR-0149 §5 (anti-laundering) and its core "no standing admin credential reaches a session" principle are not weakened by this ADR. Specifically:

4. Approval channel

Interactive session (a human is actively driving the conversation): the in-session AskUserQuestion tool, answered directly by the operator in that session. This is the only valid Tier C approval mechanism for an interactive session — not a summary of what the operator supposedly said elsewhere, not a peer's paraphrase.

Async / cloud / unattended session (no human actively present): a signed operator approval posted as a comment on the operation's tracking issue, under the operator's own GitHub identity — the same standard already established for other operator-gated actions in this codebase (#4443). The wrapper must verify the comment's author is the operator's real GitHub account, not infer approval from a paraphrase, a linked screenshot, or a peer session's assertion.

In both cases: a peer session relaying "the operator approved this" is not an approval, regardless of how specific, confident, or well-cited the relay is. This is the same rule that already governed every credential decision made tonight (control-session and TradeMasterAPI's sre-agents both independently refused exactly this pattern several times before this ADR existed) — this ADR writes that pattern down as policy rather than leaving it as emergent behavior.

5. Hard rules

  1. Safe default to Tier C for anything unclassified or ambiguous (§2).
  2. The classification policy is itself Tier C to edit (§2) — no self-widening.
  3. Single, dedicated identity: agent-breakglass-admin, distinct from every other identity in the estate (the shared CI read identity, any <slug>-rw write identity, the WP admin token used directly by the current wp-pipeline.sh/ci-agent-recycle.sh). Never upgrade an existing "everyday" identity to break-glass scope — a compromise or bug in routine tooling must not automatically grant break-glass reach.
  4. Short TTL: Universal Auth access tokens minted on demand, TTL 30–60 minutes maximum, never longer. Never stored in an environment variable between invocations — minted, used, discarded within one script run.
  5. Every escalation — Tier B and Tier C both — is audited, via: - The KMS HMAC hash-chain pattern already approved for this purpose (project_kms_audit_chain_approved, alias/raxx-audit-hmac, ~$2/mo, currently dormant elsewhere pending SC-A11/SC-A14 — this ADR is a second consumer of the same pre-approved mechanism, not a new spend decision), and - A durable local file sink (same pattern as scripts/ops/wp-pipeline.sh's /tmp/raxx-ops-audit.log, but this wrapper's sink additionally records: actor (id -un), tier (B or C), operation name, target, reason (free text, supplied by the caller), the Tier-C approval reference (issue comment URL, or "interactive-session" + timestamp for an in-session AskUserQuestion), and result).
  6. One-command kill switch: revoking/disabling the agent-breakglass-admin identity (documented in the provisioning runbook, §-level operator action) immediately and completely removes all break-glass capability, Tier B and C both, with no other code path around it.
  7. Wrapper hardening — the #4566 standard, exactly: this script is the most security-critical piece of tooling in the estate (broader reach than wp-pipeline.sh/ci-agent-recycle.sh, which only reach one system each). It must be built and adversarially reviewed to at least the same standard those two scripts reached after four rounds of review: environment-state validation before any privileged call (not a caller-settable sentinel — see wp-pipeline.sh header for why), unconditional env -i re-exec under a curated PATH with AWS_*/relevant credential-file env fully scrubbed on every code path (not just the re-exec branch — round 3 of that review found exactly this gap), token delivered to any API call via stdin/config-file mechanisms never argv, mktemp with a mode-600 file and an EXIT/INT/ TERM/HUP trap for cleanup, and no secret value ever reaching stdout, stderr, or a log line.

6. The agent-breakglass-admin identity

Minted and stored only by the operator — no session, at any tier, can create or rotate this identity itself (that would be a Tier C operation performed on itself, which is exactly the kind of self-widening §5.2 forbids). Provisioning steps are documented in docs/ops/runbooks/breakglass-admin-provisioning.md and are operator-only, matching the existing convention that admin-identity minting in this estate (see ADR-0149, /onboard-vault) is never something an agent does unassisted.

7. Residual risk (stated unsoftened)

8. Deliverables

  1. This document.
  2. scripts/ops/breakglass-admin.sh — the bounded wrapper. Merges as reviewed code but functionally INERT until the operator completes §6's provisioning (matches how scripts/ops/wp-pipeline.sh/ci-agent-recycle.sh merged as reviewed-but-not-yet-allow-listed code in #4566/ADR-0149). Adversarial review to the #4566 standard is a merge-blocking follow-up, owned by TradeMasterAPI's session per its own commitment.
  3. docs/ops/runbooks/breakglass-admin-provisioning.md — operator-only steps to mint the identity, where its credential is stored, the kill switch, and a dry-run procedure that exercises the wrapper's refusal paths without ever completing a real privileged operation.
  4. policy/breakglass-classification.yaml — the canonical, versioned Tier A/B/C operation list.

9. Consequences