Velvet v2 — Rotation Flows Design

Status: Design locked 2026-05-03
Author: architect-agent
Parent epic: #907
Refs: Kristerpher directive 2026-05-03 ~06:00 UTC; incidents rot_2b1e (2026-05-02 20:50 UTC), PR #906 handler gap, drift recovery runbook
ADRs: 0037-velvet-service-bus-subscription-model, 0038-velvet-three-stage-operational-flow, 0039-velvet-revocation-401-criterion, 0040-velvet-consumer-registration-static-manifest
Operator runbook: docs/ops/runbooks/velvet-operator.md
Handler-author guide: docs/architecture/velvet-handler-author-guide.md

1. Context

What failed in v1

The v1 handler design (PR #906, cards #914–#916) modelled rotation as a single handler function with four sequential steps: mint → validate → distribute → revoke. The 2026-05-02 incident (rot_2b1e) exposed the critical flaw in this model:

• The handler used the old token both to authenticate the mint call and to authenticate the distribute calls.
• When the old token had already expired or drifted, the very first step failed with a misleading "distribute incomplete" error — but distribute never started. The error message enumerated all five distribution destinations as if it had attempted them.
• Recovery required hand-syncing vault to four Heroku apps outside the rotation pipeline entirely.

Independently: the v1 model hard-codes distribution destinations inside each handler. Adding a new consumer (GitHub Actions secret, Heroku config var, or a new service) requires a handler code change rather than a consumer registration.

What v2 fixes

Three changes address root causes, not symptoms:

1. Explicit Stage 1 (Verify) uses a fresh auth probe before any state change occurs. If Stage 1 fails, nothing is touched. The operator sees a clear gate failure rather than a misleading distribution error.
2. Service bus: every consumer declares what token it needs and how to reach it. Velvet enumerates registered subscribers at rotation time — no hard-coded destination lists in handlers.
3. Three distinct flows: Testing (non-destructive), Operational (mint + distribute + validate + revoke), and Revocation (emergency termination). Each is separately designed with its own state machine and UI.

2. Invariants

These constraints are non-negotiable. Any implementation that routes around them requires explicit operator approval.

3. Data Model

3a. rotation_jobs table (revised from V3 #910)

The existing schema is extended to support the three-stage state machine and per-subscriber distribution tracking.

CREATE TABLE rotation_jobs (
    id                  UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    credential_name     TEXT NOT NULL,
    env                 TEXT NOT NULL CHECK (env IN ('prod', 'staging')),
    flow_type           TEXT NOT NULL CHECK (flow_type IN ('testing', 'operational', 'revocation')),
    status              TEXT NOT NULL CHECK (status IN (
                            'init',
                            'verifying', 'verify_failed',
                            'verified',
                            'minting', 'mint_failed',
                            'minted',
                            'distributing', 'distribute_partial', 'distribute_failed',
                            'distributed',
                            'validating', 'validate_partial', 'validate_failed',
                            'validated',
                            'revoking', 'revoke_failed',
                            'done',
                            'aborted'
                        )),
    idempotency_key     TEXT UNIQUE,
    operator_id         TEXT NOT NULL,
    operator_abort      BOOLEAN NOT NULL DEFAULT false,
    force_revoke        BOOLEAN NOT NULL DEFAULT false,
    created_at          TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at          TIMESTAMPTZ NOT NULL DEFAULT now(),
    verified_at         TIMESTAMPTZ,
    minted_at           TIMESTAMPTZ,
    distributed_at      TIMESTAMPTZ,
    validated_at        TIMESTAMPTZ,
    revoked_at          TIMESTAMPTZ,
    completed_at        TIMESTAMPTZ,
    error_stage         TEXT,
    error_message       TEXT,
    new_token_hash      TEXT,          -- SHA-256 of new token, never the value
    old_token_hash      TEXT,          -- SHA-256 of old token at job creation time
    vendor_revoke_ref   TEXT,          -- vendor-returned revocation ID/txn, if any
    metadata            JSONB          -- handler-specific: vendor auth IDs, scope list, etc.
);

CREATE INDEX idx_rotjobs_credential  ON rotation_jobs (credential_name, env);
CREATE INDEX idx_rotjobs_status      ON rotation_jobs (status);
CREATE INDEX idx_rotjobs_created     ON rotation_jobs (created_at DESC);
CREATE INDEX idx_rotjobs_operator    ON rotation_jobs (operator_id);

3b. rotation_job_consumers table (new)

One row per registered subscriber per rotation job. Tracks per-consumer distribute and validate state independently.

CREATE TABLE rotation_job_consumers (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    job_id          UUID NOT NULL REFERENCES rotation_jobs(id) ON DELETE CASCADE,
    consumer_id     TEXT NOT NULL,          -- matches subscription manifest consumer_id
    env             TEXT NOT NULL,
    distribute_status   TEXT NOT NULL DEFAULT 'pending'
                            CHECK (distribute_status IN ('pending','in_progress','succeeded','failed','skipped')),
    validate_status     TEXT NOT NULL DEFAULT 'pending'
                            CHECK (validate_status IN ('pending','in_progress','succeeded','failed','skipped')),
    distribute_attempt_count    INT NOT NULL DEFAULT 0,
    validate_attempt_count      INT NOT NULL DEFAULT 0,
    distribute_error            TEXT,
    validate_error              TEXT,
    distribute_completed_at     TIMESTAMPTZ,
    validate_completed_at       TIMESTAMPTZ,
    healthcheck_http_status     INT,        -- last HTTP status from healthcheck endpoint
    created_at                  TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at                  TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_rotjob_consumers_job   ON rotation_job_consumers (job_id);
CREATE INDEX idx_rotjob_consumers_cid   ON rotation_job_consumers (consumer_id);

3c. subscription_manifest.yml (versioned config — not a database table)

Location: docs/architecture/velvet/subscription-manifest.yml
Loaded at Velvet app startup; re-loaded on SIGHUP without restart.

# Velvet service-bus subscription manifest
# Format version: 2
# Every consumer that holds a copy of a rotatable token must register here.

subscriptions:
  - token_name: HK_PLATFORM_FULL
    consumer_id: heroku-config-console-prod
    env: prod
    update_endpoint: "https://api.heroku.com/apps/raxx-console-prod/config-vars"
    update_method: PATCH
    update_auth_token_name: HK_PLATFORM_FULL   # Velvet reads THIS from vault to auth the update
    healthcheck_endpoint: "https://api.heroku.com/account"
    healthcheck_method: GET
    healthcheck_auth_header: "Authorization: Bearer {token}"
    healthcheck_success_status: 200
    capabilities:
      - update
      - healthcheck
    description: "Heroku config var on raxx-console-prod"

  - token_name: HK_PLATFORM_FULL
    consumer_id: github-actions-secret
    env: prod
    update_endpoint: "https://api.github.com/repos/raxx-app/TradeMasterAPI/actions/secrets/HEROKU_API_KEY"
    update_method: PUT_ENCRYPTED          # Velvet handles NaCl encryption for GH Secrets API
    update_auth_token_name: GH_APP_OPS_BOT
    healthcheck_endpoint: null            # GH secrets have no direct healthcheck; use a special verify flow
    capabilities:
      - update
    description: "GitHub Actions secret HEROKU_API_KEY"

  - token_name: PM_SERVER_MAIL
    consumer_id: infisical-postmark-server
    env: prod
    update_endpoint: "velvet://infisical/postmark/PM_SERVER_MAIL"   # internal; Velvet handles directly
    update_method: INFISICAL_WRITE
    healthcheck_endpoint: "https://api.postmarkapp.com/server"
    healthcheck_method: GET
    healthcheck_auth_header: "X-Postmark-Server-Token: {token}"
    healthcheck_success_status: 200
    capabilities:
      - update
      - healthcheck
    description: "Infisical vault entry for Postmark server token"

Schema fields:

4. APIs / Contracts

4a. Velvet REST API additions

All existing v1 endpoints remain. New endpoints:

POST   /tokens/{name}/rotate
       Body: { "flow_type": "operational" | "testing" | "revocation",
               "idempotency_key": "<uuid>",
               "force_revoke": false }
       Returns 202: { "job_id": "<uuid>", "status": "init" }

GET    /tokens/{name}/rotations/{job_id}
       Returns rotation_jobs row + consumers array

POST   /tokens/{name}/rotations/{job_id}/stage
       Body: { "action": "verify" | "proceed_mint" | "proceed_revoke" |
                         "abort" | "force_revoke" }
       Returns 200: { "job_id", "status", "consumers": [...] }

GET    /tokens/{name}/rotations/{job_id}/stream
       SSE endpoint; emits { "event": "state_change", "data": {...} }
       on every status transition

4b. Consumer update endpoint contract

When Velvet PATCHes/POSTs a consumer's update_endpoint, the body is:

{
  "velvet_job_id": "<uuid>",
  "token_name": "HK_PLATFORM_FULL",
  "token_value": "<new_value>",
  "rotate_timestamp": "2026-05-03T06:00:00Z"
}

For HEROKU_CONFIG_PATCH: Velvet wraps this into the Heroku Platform API shape ({"HEROKU_API_KEY": "<new_value>"}) before sending. The manifest update_method determines the adapter.

For PUT_ENCRYPTED (GitHub Secrets): Velvet fetches the repo public key, NaCl-seals the value, and constructs the GitHub Secrets API payload — no raw value leaves Velvet's process.

For INFISICAL_WRITE: Velvet calls Infisical directly using its own machine identity. No outbound HTTP to a third party; the update is internal.

Expected consumer response (for HTTPS consumers with their own endpoints):

{ "accepted": true, "consumer_id": "my-service" }

HTTP 200 or 204 is treated as success. Any 4xx/5xx marks that consumer's distribute_status = 'failed'.

4c. Healthcheck contract

Velvet GETs healthcheck_endpoint with the new token substituted into healthcheck_auth_header. It checks only the HTTP status code against healthcheck_success_status.

For revocation validation: Velvet GETs with the old (now-revoked) token. Success criterion is HTTP 401.

Consumers with healthcheck_endpoint: null must be manually confirmed by the operator via the UI before the validation stage can advance. The console surfaces a "Manual confirm" button for these rows.

5. State Machines and Sequence Diagrams

5a. Operational Flow (manual rotation and expiry-driven)

stateDiagram-v2
    [*] --> init : POST /rotate (flow_type=operational)

    init --> verifying : operator clicks "Verify"
    verifying --> verify_failed : auth probe fails / no permissions
    verifying --> verified : auth probe OK + permission confirmed

    verify_failed --> aborted : operator acknowledges
    verify_failed --> verifying : operator retries

    verified --> minting : operator clicks "Proceed"
    minting --> mint_failed : vendor mint API error
    minting --> minted : new token value in hand

    mint_failed --> aborted : operator aborts (old token still valid)

    minted --> distributing : fan-out to all registered consumers
    distributing --> distribute_partial : some consumers failed
    distributing --> distribute_failed : all consumers failed
    distributing --> distributed : all consumers succeeded

    distribute_partial --> distributing : operator retries failed rows
    distribute_partial --> aborted : operator aborts (new token minted but not fully distributed — see cleanup)
    distribute_failed --> aborted : operator aborts

    distributed --> validating : Velvet calls all consumer healthchecks with new token
    validating --> validate_partial : some healthchecks failed
    validating --> validate_failed : all healthchecks failed
    validating --> validated : all consumers return healthcheck_success_status

    validate_partial --> validating : operator retries failed rows
    validate_partial --> aborted : operator force-aborts
    validate_failed --> aborted : operator aborts

    validated --> revoking : operator clicks "Revoke old token" (type-to-confirm)
    revoking --> revoke_failed : vendor revoke API error
    revoking --> done : vendor confirms revocation

    revoke_failed --> revoking : operator retries
    revoke_failed --> done : operator marks "manually revoked" (with required ticket ID)

    done --> [*]
    aborted --> [*]

Abort semantics by state:

Abort does NOT automatically revoke the new token. The operator is shown the residual state and must take manual action per the runbook.

5b. Operational Flow — Sequence Diagram

sequenceDiagram
    actor Op as Operator (console)
    participant C as Console UI
    participant V as Velvet API
    participant DB as Postgres (rotation_jobs)
    participant Vault as Infisical Vault
    participant Vendor as Vendor API (e.g. Heroku)
    participant Bus as Service Bus (consumer adapters)

    Op->>C: Click "Rotate HK_PLATFORM_FULL"
    C->>V: POST /tokens/HK_PLATFORM_FULL/rotate {flow_type: operational}
    V->>DB: INSERT rotation_jobs {status: init}
    V-->>C: 202 {job_id}
    C-->>Op: Show Stage 1 modal

    Op->>C: Click "Verify"
    C->>V: POST /rotations/{job_id}/stage {action: verify}
    V->>Vendor: GET /account (old token auth probe)
    V->>Vendor: GET /account/authorizations (check rotate permission)
    alt Verify OK
        V->>DB: UPDATE status=verified
        V-->>C: {status: verified}
        C-->>Op: Show "Verified — proceed?" gate
    else Verify failed
        V->>DB: UPDATE status=verify_failed
        V-->>C: {status: verify_failed, error: ...}
        C-->>Op: Show failure — nothing changed
    end

    Op->>C: Click "Proceed to mint"
    C->>V: POST /rotations/{job_id}/stage {action: proceed_mint}
    V->>Vendor: POST /oauth/authorizations (mint new token)
    V->>DB: UPDATE status=minted, new_token_hash=sha256(new)
    V->>Vault: Write new_token to vault (non-active slot)
    V-->>C: {status: minted}

    Note over V,Bus: Fan-out to all registered consumers for HK_PLATFORM_FULL
    V->>DB: INSERT rotation_job_consumers rows {status: pending}
    V->>DB: UPDATE rotation_jobs status=distributing

    loop Each consumer (parallel)
        V->>Bus: Adapter: PATCH /apps/{app}/config-vars {HEROKU_API_KEY: new}
        Bus-->>V: 200 OK
        V->>DB: UPDATE rotation_job_consumers {distribute_status: succeeded}
    end

    V->>DB: UPDATE rotation_jobs status=distributed
    V-->>C: SSE event: state_change {status: distributed, consumers: [...]}
    C-->>Op: Show Stage 3 panel: all green / partial

    Note over V,Bus: Validate — healthcheck each consumer with new token
    loop Each consumer (parallel)
        V->>Bus: GET healthcheck_endpoint (new token in auth header)
        Bus-->>V: 200 OK (new token valid at vendor)
        V->>DB: UPDATE rotation_job_consumers {validate_status: succeeded}
    end

    V->>DB: UPDATE rotation_jobs status=validated
    V-->>C: SSE {status: validated}
    C-->>Op: Show "Revoke old token?" gate (type-to-confirm)

    Op->>C: Type confirm phrase + click Revoke
    C->>V: POST /rotations/{job_id}/stage {action: proceed_revoke}
    V->>Vendor: DELETE /oauth/authorizations/{old_token_id}
    V->>Vault: Mark old token slot inactive
    V->>DB: UPDATE rotation_jobs status=done
    V-->>C: SSE {status: done}
    C-->>Op: Show audit summary

5c. Revocation Flow — State Machine

stateDiagram-v2
    [*] --> rev_init : POST /rotate {flow_type: revocation}

    rev_init --> rev_revoking : operator clicks "Revoke" (type-to-confirm)
    rev_revoking --> rev_revoke_failed : vendor returns non-2xx
    rev_revoking --> rev_revoked : vendor confirms revocation

    rev_revoke_failed --> rev_revoking : operator retries
    rev_revoke_failed --> [*] : operator aborts (token still valid)

    rev_revoked --> rev_validating : Velvet enumerates subscribers; healthchecks with old (revoked) token
    rev_validating --> rev_leaked : any consumer returns non-401
    rev_validating --> rev_done : all consumers return 401

    rev_leaked --> rev_done : operator acknowledges leak + files incident ticket
    rev_done --> [*]

5d. Revocation Flow — Sequence Diagram

sequenceDiagram
    actor Op as Operator
    participant C as Console UI
    participant V as Velvet API
    participant DB as Postgres
    participant Vendor as Vendor API
    participant Bus as Consumer healthchecks

    Op->>C: Click "Revoke HK_PLATFORM_FULL" (no replacement)
    C->>V: POST /tokens/HK_PLATFORM_FULL/rotate {flow_type: revocation}
    V->>DB: INSERT rotation_jobs {status: rev_init, flow_type: revocation}
    V-->>C: 202 {job_id}
    C-->>Op: Show revocation modal: type-to-confirm gate

    Op->>C: Type confirm phrase + click "Revoke now"
    C->>V: POST /rotations/{job_id}/stage {action: proceed_revoke}
    V->>Vendor: DELETE /oauth/authorizations/{token_id}
    alt Vendor confirms revocation
        V->>DB: UPDATE status=rev_revoked
        Note over V,Bus: Walk registered subscribers; healthcheck with revoked token
        loop Each subscriber
            V->>Bus: GET healthcheck_endpoint (revoked token in auth header)
            alt Returns 401
                V->>DB: validate_status=succeeded (401 IS the success criterion)
            else Returns non-401
                V->>DB: validate_status=failed, healthcheck_http_status=<code>
                Note over V: Flag as rotation_leaked
            end
        end
        alt All 401
            V->>DB: UPDATE rotation_jobs status=rev_done
            C-->>Op: "Revocation confirmed — all consumers locked out"
        else Any non-401
            V->>DB: UPDATE rotation_jobs status=rev_leaked
            C-->>Op: Show leaked consumer list + "Investigate" button
        end
    else Vendor revocation fails
        V->>DB: UPDATE status=rev_revoke_failed
        V-->>C: {error: vendor_error_detail}
        C-->>Op: Retry or abort
    end

5e. Testing Flow

The testing flow runs the full operational state machine against a throwaway token provisioned for the test. It never touches live credentials. Key differences:

• flow_type = testing in rotation_jobs
• Velvet creates a test token at the vendor (a token with a 1-hour TTL, no real scopes), runs the full pipeline, then auto-revokes at the end regardless of validation outcome.
• Consumers are exercised against the throwaway token so the bus + adapters prove out.
• Test flow does not require a FreeScout ticket ID (friction gate removed).
• Audit log marks the job as flow_type=testing; it does not appear in operational rotation history by default.

Testing flow state machine is identical to operational but transitions automatically (no operator gates) and the terminal state is always revoking → done (auto-revoke of the test token).

6. UI/UX Modal Design

The rotation modal is replaced with a multi-panel stage wizard. Each stage is a distinct panel within the same modal container. The operator cannot jump forward; they can abort at any point.

Panel layout

+------------------------------------------+
| Rotate: HK_PLATFORM_FULL         [Abort] |
+------------------------------------------+
| [1. Verify] [2. Mint+Dist] [3. Val+Rev] |   <- stage progress bar (current highlighted)
+------------------------------------------+
| <stage-specific content>                 |
+------------------------------------------+

Stage 1 — Verify panel: - Token name, last-rotated date, rotation cadence, registered consumer count - "Verify credentials" button (triggers the auth probe) - On success: green check + "Credentials verified — 4 consumers registered. Proceed?" - On failure: red banner with the exact probe error (e.g., "Auth probe returned 401 — current token invalid") - Abort button available throughout

Stage 2 — Mint + Distribute panel: - Shows "Minting new token..." spinner while Velvet calls vendor mint - After mint: consumer table

Consumer                  | Environment | Status
--------------------------+-------------+---------
heroku-config-console-prod| prod        | [green] Succeeded
heroku-config-api-prod    | prod        | [green] Succeeded
github-actions-secret     | prod        | [amber] In progress...
heroku-config-staging     | staging     | [red]   Failed — 401 on PATCH

• Per-row retry button for failed rows
• "Retry all failed" bulk action
• Proceed to Stage 3 button (active only when all rows are green or skipped)

Stage 3 — Validate + Revoke panel: - Same consumer table, now showing validation (healthcheck) status - "Manual confirm" button for consumers with healthcheck_endpoint: null - After all validation green: "Revoke old token?" confirmation gate - Type-to-confirm field: revoke HK_PLATFORM_FULL - FreeScout ticket ID field (required) - Revoke button triggers vendor DELETE + vault mark-inactive - Final audit summary panel after done: - Job ID, duration, consumers updated, operator, UTC timestamp

Revocation modal (simpler, separate entry point): - Single panel: "This permanently revokes HK_PLATFORM_FULL. No replacement will be minted." - Type-to-confirm: revoke HK_PLATFORM_FULL permanently - FreeScout ticket ID - After revocation: consumer table showing 401 status per row (green = confirmed locked out, red = investigation needed)

Visual parity notes: The modal shell established in _rotate_modal_v2.html (Slate/rose palette, type-to-confirm, TOTP gate, FreeScout ticket field) is retained. The TOTP gate is preserved on the Stage 3 revoke gate. The danger banner ("This will invalidate the current value") becomes per-stage contextual copy.

7. Migrations

7a. Schema migration: rotation_jobs revision

The existing rotation_jobs schema (card #910, not yet applied to any live database — app not yet shipped) is replaced in its entirety by the v2 schema. Because raxx-velvet-prod and raxx-velvet-staging do not yet exist, this is a fresh schema, not a live migration.

Migration order: 1. 001_create_rotation_jobs_v2.sql — full table with v2 status enum 2. 002_create_rotation_job_consumers.sql — join table 3. 003_indexes.sql — all indexes

All migrations run via the Velvet release Procfile command (Alembic). Idempotent (CREATE TABLE IF NOT EXISTS + DO $$ BEGIN ... EXCEPTION WHEN duplicate_object THEN NULL; END $$).

7b. v1 handler cards (#914–#916) retirement

Cards #914 (Postmark handler), #915 (Heroku handler), #916 (Cloudflare handler) are replaced by per-vendor adapter cards in the v2 slate. They are not closed yet — they stay open with a comment pointing to their replacement cards. They are removed from the sprint backlog.

The existing PR #906 handler (Heroku Mode A rotation) remains operational in the console app until the Velvet v2 Heroku adapter is live and smoke-tested. After that, PR #906's handler is deprecated and gated behind a feature flag set to off.

7c. Subscription manifest bootstrap

docs/architecture/velvet/subscription-manifest.yml is checked in as part of the design PR. It is initially populated with the three highest-priority consumers: 1. Postmark server token → Infisical write (lowest risk, proves the bus) 2. Heroku platform key → four Heroku config-var PATCH destinations + one GitHub Actions secret 3. Cloudflare DNS token → one Infisical write (CF tokens cannot be rotated via API — Velvet delegates to operator with clear instructions)

8. Rollout Plan

dark        Velvet v2 app deployed; subscription manifest loaded; no rotation traffic
            Smoke: GET /tokens/{name} works; subscription manifest parses cleanly
            Card: B1 (scaffold + manifest loader)

flag        feature flag velvet_v2_rotation gates POST /rotate endpoint
            Postmark adapter live; test rotation run against throwaway PM token
            Cards: B2 (schema), B3 (service bus core), B4 (Postmark adapter)

beta        Heroku adapter live; one operational rotation of HK_PLATFORM_FULL in staging
            Console UI v2 modal behind flag velvet_v2_rotation_ui
            Cards: B5 (Heroku adapter), B6 (GH Actions adapter), B7 (console UI)

            B5 Status: implemented in PR #947 — velvet/adapters/heroku_config_var.py
              HerokuConfigVarAdapter: PATCH /apps/{app}/config-vars via HTTP (no CLI),
              auth via HK_PLATFORM_FULL env var, flag FLAG_VELVET_HEROKU_CONFIG_VAR_ADAPTER,
              response body always <REDACTED> in logs, 429 retry once after 5s.

            B6 Status: implemented in PR #947 — velvet/adapters/github_actions_secret.py
              GithubActionsSecretAdapter: GET public-key + NaCl SealedBox + PUT secret,
              auth via GH_APP_OPS_BOT env var (depends on #925 — degrades gracefully),
              flag FLAG_VELVET_GH_ACTIONS_SECRET_ADAPTER, raw value never leaves process
              unencrypted.

ga          All v1 handler cards deprecated; PR #906 handler behind off-flag
            Rotation history visible in console for all token types
            Cards: B8 (CF adapter), B9 (revocation flow), B10 (runbook), B11 (v1 retirement)

9. Security Considerations

PII collected: None. The audit log stores operator_id (a console account identifier — an opaque UUID, not an email address), job IDs, and token name. Token values never appear.

Retention: Rotation job rows retained for 2 years (per audit requirements for credential-change events). Purge job runs monthly; deletes rotation_jobs rows older than 730 days; cascades to rotation_job_consumers via FK.

Credential values at rest: New token value is held in Velvet process memory only during distribute. It is written to Infisical via the app's own machine identity. The value is never written to the rotation_jobs table. new_token_hash (SHA-256, hex) is the only stored form.

Credential values in transit: All Velvet → vendor calls are HTTPS. Consumer update_endpoint URIs must be HTTPS (manifest validator rejects http://). The token_value field in the update request body is transmitted once per consumer over TLS. It is never logged by Velvet; the consumer's logging posture is the consumer's responsibility.

Credential replay risk: Velvet holds no credential in a form that can be replayed beyond the current rotation job's lifetime. Once a job reaches done, the new token value is no longer accessible via Velvet (only via vault, with a fresh read). This satisfies invariant I1 and the system-wide no-stored-credentials constraint.

Breach notification: If the rotation_leaked status is reached (revocation confirmed at vendor but one or more consumers still respond non-401), an ops alert is emitted to the operator's Slack channel (SL_BOT_NOTIFY) within 30 seconds. The alert message includes: job_id, credential_name, list of consumer_ids that did not return 401, and a link to the console investigation panel.

Service-bus auth: When Velvet calls a consumer's update_endpoint, it signs the request with an HMAC-SHA256 request signature using a shared secret (VELVET_CONSUMER_SHARED_SECRET) stored in vault. The consumer verifies the signature before applying the new token value. Consumers without HMAC verification are classified as capabilities: [update_no_verify] in the manifest and flagged in the console as degraded trust.

Secrets rotation of Velvet's own credentials: Velvet's Infisical machine identity secret and Heroku bootstrap config vars are rotated manually per a dedicated runbook (card B10). The subscription manifest itself is rotated by re-deploying the app with an updated manifest file.

Kill-switch: Feature flag velvet_v2_rotation set to off disables all POST /rotate endpoints and returns HTTP 503 with body {"error": "rotation_disabled"}. The console surfaces this as "Rotation temporarily disabled" in the UI.

10. Open Questions

These block specific sub-cards and require Kristerpher's explicit decision before implementation begins.

Appendix A: Sub-card Slate (pending Kristerpher review before filing)

See section-level heading "Sub-card slate" in the accompanying briefing.

Dropped from v1 slate: - V7 #914 (Postmark handler) — replaced by B4 - V8 #915 (Heroku handler) — replaced by B5 + B6 + B14 - V9 #916 (CF handler) — replaced by B8

Retained from v1 slate (with revision comments filed): - V1 #908 (scaffold) — merged into B1; close #908 when B1 ships - V2 #909 (GET proxy) — still valid, no change needed - V3 #910 (rotation_jobs schema) — superseded by B2; comment filed on #910 - V4 #911 (POST /rotate) — superseded by B3/B9; comment filed on #911 - V5 #912 (auth middleware) — superseded by B10; comment filed on #912 - V6 #913 (SSM integration) — becomes B15; retained pending OQ resolution - V10 #917 (first console callsite) — still valid; now depends on B7 - V11 #918 (runbook) — superseded by B11; comment filed on #918