Raxx · internal docs

internal · gated

AWS KMS runbook

System: AWS KMS (account 521228113048, shared between Raxx/TradeMasterAPI and the "Avenue" product) Owner: sre-agent / Kristerpher Terraform root: terraform/ci-monitoring/ (kms-cost-tracking.tf) Card: AWS KMS cost-attribution task, 2026-07-24 Last incident: none (this is a monitoring/attribution deliverable, not an incident response) Last reviewed: 2026-07-24

How to tell there's a problem

How to diagnose (in order)

  1. Check the live dashboard first. Grafana → AWS KMS Request Volume & Cost Attribution (uid: kms-cost-tracking). Panel 0 (KMS Actual Billed Cost) is live today and reads the native AWS/Billing EstimatedCharges metric (ServiceName=awskms), account-wide, no dependency on any other team's infrastructure. bash aws ssm start-session --region us-east-2 --target <ci-monitoring-instance-id> \ --document-name AWS-StartPortForwardingSession \ --parameters '{"portNumber":["3000"],"localPortNumber":["3000"]}' # then open http://localhost:3000 Note: EstimatedCharges is a month-to-date cumulative counter — it resets to ~0 on the 1st of each month. A sawtooth pattern is normal; a slope change mid-tooth is the signal to chase.

  2. Cross-check with Cost Explorer (source of truth for what's actually billed): bash aws ce get-cost-and-usage --region us-east-1 \ --time-period Start=$(date -u -v-30d +%Y-%m-%d),End=$(date -u +%Y-%m-%d) \ --granularity DAILY --metrics UsageQuantity UnblendedCost \ --filter '{"Dimensions":{"Key":"SERVICE","Values":["AWS Key Management Service"]}}' \ --group-by Type=DIMENSION,Key=USAGE_TYPE Look for *-KMS-Keys (flat $1/mo/CMK — this is currently ~100% of the bill) vs *-KMS-Requests (currently $0, inside free tier).

  3. Enumerate CMKs and their purpose (find what's actually costing money — only customer-managed keys have the $1/mo flat fee; AWS-managed alias/aws/* keys are free): bash for region in us-east-1 us-east-2 us-west-1 us-west-2; do aws kms list-aliases --region "$region" --output json done As of 2026-07-24: 11 CMKs in us-east-1 + 1 DR replica in us-west-2, all aliased avenue-* (a separate product sharing this AWS account, not Raxx). Raxx's own us-east-2 KMS footprint is 6 AWS-managed default keys (alias/aws/ebs, alias/aws/rds, alias/aws/ssm, alias/aws/lambda, alias/aws/dynamodb, alias/aws/acm) — $0/mo flat fee.

  4. For request-volume attribution (who/what is calling KMS, not just how much it costs), see "Known failure modes" below — this requires either enabling the gated avenue-trail metric filters (operator decision) or an ad-hoc CloudWatch Logs Insights query against avenue-trail.

Known failure modes

Failure mode A: New CMK appears, bill grows by ~$1/mo

Symptom: Cost Explorer *-KMS-Keys usage type cost increases by a flat amount that doesn't track any request-volume change. Cause: A new customer-managed key was created somewhere in the account (Terraform apply, console click, another team's workload). Every CMK is $1/mo flat regardless of whether it's ever used. Fix: aws kms list-aliases --region <region> in all 4 regions with history (us-east-1, us-east-2, us-west-1, us-west-2) to find the new key + its alias/description. If it's a Raxx resource, confirm it's intentional (check the PR that created it). If it's not a Raxx resource (e.g., another avenue-* key), this is not actionable by sre-agent — note it for the operator's account-wide awareness only. Verification: New key's alias/description explains its purpose; cost delta matches $1/mo (prorated).

Failure mode B: Request-volume cost appears (*-KMS-Requests line item goes nonzero)

Symptom: Cost Explorer shows a nonzero UnblendedCost for *-KMS-Requests for the first time. Cause: Account-wide KMS request volume crossed the 20,000/month free tier. As of 2026-07-24, CloudTrail-observed volume (avenue-trail, both trails show identical account-wide counts since both are multi-region) was already ~330,000 crypto operations over a 14-day window — see the open discrepancy note below. If Cost Explorer ever starts reflecting that volume as billed, this is the failure mode that will fire. Fix: Enable the gated metric filters in terraform/ci-monitoring/kms-cost-tracking.tf (var.enable_avenue_trail_kms_metrics = true, requires sign-off from whoever owns the Avenue side of the account first — read the file header) to get per-operation attribution on the dashboard, then run the CloudWatch Logs Insights query below to attribute to a specific key/principal. Do not guess; the dominant historical driver (CloudTrail's own S3 log delivery + CloudWatch Logs envelope encryption) is unlikely to be a Raxx workload, but confirm before concluding that. bash aws logs start-query --region us-east-1 \ --log-group-name "/aws/cloudtrail/avenue-trail" \ --start-time <epoch-14d-ago> --end-time <epoch-now> \ --query-string 'fields eventName, requestParameters.keyId, userIdentity.invokedBy | filter eventSource="kms.amazonaws.com" | stats count(*) as cnt by eventName, requestParameters.keyId, userIdentity.invokedBy | sort cnt desc | limit 30' Verification: Attribution names a specific key + calling principal; cost model in docs/ops/aws-kms-cost-model.md updated with the new driver.

Failure mode C: Raptor audit-chain HMAC goes live and grows unexpectedly

Symptom: kms_generate_mac / kms_verify_mac panels (once enabled) show nonzero and growing counts. Cause: backend_v2/api/services/customer_audit_writer_service.py calls kms.generate_mac once per audit-log write (compute_event_hash) and kms.verify_mac once per event during the nightly 24h-window integrity check (backend_v2/jobs/audit_integrity_check.py, default mode) — this is the approved ~$2/mo baseline. Additionally, a quarterly full-chain re-verification (--mode monthly, actually run quarterly per docs/ops/runbooks/soc2-quarterly-attestation.md Step 3) re-verifies every audit event since genesis, for every customer — this call volume scales with total historical audit-event count, not event rate, and will grow every quarter as the audit log accumulates. Fix: This is expected growth, not a bug — but if it grows faster than expected, check AUDIT_KMS_KEY_ARN is pointed at a single HMAC key (not accidentally re-created per deploy) and that the nightly job's window is still bounded to 24h (window_start = now_utc - timedelta(hours=24) in audit_integrity_check.py) and hasn't regressed to a full-history re-verify on every run. Verification: kms_generate_mac daily count ≈ daily audit-event write volume; kms_verify_mac daily count ≈ same (nightly) plus a predictable quarterly spike (full history size) around each SOC-2 attestation window.

Failure mode D: CI spot-runner churn drives CreateGrant spikes

Symptom: kms_create_grant panel (once enabled) spikes. Cause: Each new EC2 instance launched from an encrypted AMI/EBS snapshot creates a fresh KMS grant to decrypt the source snapshot. Correlates with Woodpecker CI spot-runner cycling (see docs/incidents/2026-07-13-ci-spot-wedge-silent-outage.md). Fix: Not a cost problem today (grant operations are not billed per-request) — but a useful correlate. If it spikes far beyond normal CI cadence, check ci-woodpecker-agents-asg for a scale-flap loop (see docs/ops/runbooks/woodpecker-ha.md / docs/ops/runbooks/ci-runner-posture.md). Verification: Grant spike timing matches ASG launch events in the same window.

Open discrepancy (not yet resolved — do not paper over)

CloudTrail (avenue-trail, both regions covered via multi-region trail) shows ~330,000 KMS crypto operations (GenerateDataKey + Decrypt, dominated by fas.s3.amazonaws.com and logs.amazonaws.com invokers) over a 14-day window ending 2026-07-24. Cost Explorer, for the same account, shows the *-KMS-Requests usage type at $0 cost and a much lower quantity (~20,000-21,000/month across all regions) than the CloudTrail-observed rate would imply if every logged event were a separately-billed request. This has NOT been reconciled. Do not assume either number is "the real one" without further evidence. See docs/ops/aws-kms-cost-model.md §Open Discrepancy for the leading hypotheses (S3 Bucket Key caching, CloudTrail delivery/billing-lag artifacts). The billed-dollar panel (Panel 0, AWS/Billing EstimatedCharges) is authoritative for cost regardless of how this resolves — it already reflects whatever AWS actually charges.

Emergency stop

There is no "stop" action for KMS itself — you cannot disable KMS without breaking every service that depends on it (RDS, EBS, SSM SecureString, S3 SSE-KMS, the Raptor audit chain). If a specific CMK is suspected of runaway cost or a security issue:

# Schedule deletion (7-30 day waiting period, reversible during the window):
aws kms schedule-key-deletion --key-id <key-id> --pending-window-in-days 30 --region <region>
# To disable immediately without scheduling deletion (reversible):
aws kms disable-key --key-id <key-id> --region <region>

Do not run either command without confirming what depends on the key first (aws kms list-grants and check the service that owns the alias). Disabling a key in active use will break that service immediately.

Escalation

Wake the operator when: - Cost Explorer's *-KMS-Requests line item goes from $0 to nonzero (free tier exceeded) — this needs an operator decision on whether to cap/throttle usage. - Attribution points to a Raxx-owned workload as the driver of unexpected volume (as opposed to the currently-identified "Avenue" / CloudTrail self-overhead pattern). - Enabling var.enable_avenue_trail_kms_metrics requires a decision this runbook cannot make unilaterally (cross-team resource sign-off). - The audit-chain HMAC (GenerateMac/VerifyMac) goes live and the operator wants to confirm the ~$2/mo baseline still holds before it's load-bearing for SOC-2 attestation.

References