AWS KMS cost attribution + model

Date: 2026-07-24 Author: sre-agent Type: Investigation + monitoring + cost-model deliverable (not an incident — no RCA format used) Account: 521228113048 · Regions checked: us-east-1, us-east-2, us-west-1, us-west-2 Related: docs/ops/runbooks/aws-kms.md · Grafana dashboard kms-cost-tracking (terraform/ci-monitoring/)

1. Attribution — what's actually driving the bill

1.1 Headline finding

The current KMS dollar cost is driven entirely by a different product ("Avenue"), not by Raxx/TradeMasterAPI, and not by request volume.

AWS account 521228113048 is shared between Raxx/TradeMasterAPI (us-east-2 workloads: Lightsail vault host, SSM, RDS, Lambda, EBS, ACM — all using AWS-managed default keys, alias/aws/*, which are free) and a separate product called "Avenue" (dev + staging environments, us-east-1 + a DR replica in us-west-2), which owns 11 customer-managed keys (CMKs) plus 1 DR replica:

Each CMK costs a flat $1/month regardless of usage. 11 CMKs in us-east-1 + 1 in us-west-2 = ~$11.68/month in flat key fees (Cost Explorer shows $9.71 for us-east-1 + $0.97 for us-west-2 over the trailing 30 days, prorated for keys created mid-period — total $10.68/month).

Cost Explorer, grouped by USAGE_TYPE for SERVICE = AWS Key Management Service, trailing 30 days:

Request-volume cost is zero — both regions' request counts are within the 20,000/month account-wide free tier. 100% of the current KMS bill is the flat per-CMK fee on "Avenue" keys. Raxx's own KMS footprint (us-east-2) uses only free AWS-managed keys and negligible request volume.

1.2 The one Raxx-specific approved use case is not live yet

The only Raxx-specific, operator-approved KMS use case is the audit-chain HMAC (~$2/mo approved, KMS.GenerateMac/VerifyMac, implemented in backend_v2/api/services/customer_audit_writer_service.py). Checked directly:

• No HMAC-spec KMS key exists in the account in any of the 4 regions checked (all keys found are SYMMETRIC_DEFAULT / ENCRYPT_DECRYPT, not HMAC_256 / GENERATE_VERIFY_MAC).
• Zero GenerateMac or VerifyMac events appear in CloudTrail over the last 14 days.
• The code path (compute_event_hash) raises CustomerAuditKmsError in production if AUDIT_KMS_KEY_ARN isn't set — it does not silently fall back (that fallback is dev/test-only).

Conclusion: the approved ~$2/mo audit-chain baseline is contributing $0 today because the feature has not been deployed to a live KMS key yet. This is worth flagging on its own — when it does go live, watch it against this baseline (dashboard panel included, currently at zero — see §2).

1.3 Why request-volume (CloudTrail) counts are so high, and who's calling

CloudTrail attribution (avenue-trail, a multi-region trail that captures every region's KMS calls, including Raxx's us-east-2 activity) over a 14-day window (2026-07-10 → 2026-07-24):

Digging into the fas.s3.amazonaws.com-invoked GenerateDataKey calls by target key ARN:

• 104,091 calls → avenue-dev-logs CMK
• 81,881 calls → avenue-staging-logs CMK

Both of these CMKs are the exact keys CloudTrail uses to SSE-KMS-encrypt each trail's own delivered log files in S3 (avenue-trail's KmsKeyId = avenue-dev-logs; avenue-staging-trail's KmsKeyId = avenue-staging-logs, per aws cloudtrail describe-trails). CloudTrail is, in large part, encrypting its own output and that shows up as the single biggest KMS caller in the account. The second-largest attributed invoker, logs.amazonaws.com (Decrypt 25,159 + GenerateDataKey 18,386 = 43,545 over 14 days), is CloudWatch Logs group encryption on the same two CMKs — again Avenue-side infrastructure, not Raxx.

Anomaly flagged, not acted on: the account has two separate multi-region CloudTrail trails (avenue-trail and avenue-staging-trail), both owned by Avenue, both capturing the identical account-wide event stream (verified: identical per-eventName counts in both trails' log groups for the same window). Running two full multi-region trails when one would capture the same data is redundant — it roughly doubles CloudTrail's own S3-delivery/KMS overhead. This is an Avenue-side infrastructure decision, out of Raxx/sre-agent's scope to change, but is surfaced here for the operator's account-wide cost awareness, since it's the single largest contributor to raw KMS call volume in the account (though, per §1.1, not to current cost, since that volume is still inside the free tier).

CreateGrant/RetireGrant/GenerateDataKeyWithoutPlaintext from autoscaling.amazonaws.com in us-east-2 (Raxx's region) correlate with the Woodpecker CI spot-runner ASG cycling encrypted EBS boot volumes on instance launch/terminate (see docs/incidents/2026-07-13-ci-spot-wedge-silent-outage.md). This IS Raxx-adjacent infrastructure, but grant operations aren't currently billed per-request, so the dollar impact is $0; it's included in the dashboard purely as an operational-churn signal.

1.4 Volume trend (2026-07-10 → 2026-07-24, GenerateDataKey + Decrypt + Encrypt + GenerateDataKeyWithoutPlaintext, daily)

07-10:  5,975   07-15: 19,491   07-20: 29,889
07-11: 14,932   07-16: 21,058   07-21: 30,498
07-12: 15,171   07-17: 29,605   07-22: 27,669
07-13: 15,477   07-18: 28,943   07-23: 23,421
07-14: 15,450   07-19: 28,932   07-24: 19,493 (partial day)

Raw daily call volume roughly 5x'd over the two-week window (day 1 partial ~6k → peak ~30.5k on 07-21), then pulled back slightly in the last 2 days. This is a genuine growth trend worth watching even though it isn't costing money today — it consumed most of the account's request free tier by mid-window on a naive daily-rate basis.

1.5 Open discrepancy — flagged explicitly, not resolved

CloudTrail shows ~330,000 crypto operations over 14 days (§1.3). Cost Explorer's billed *-KMS-Requests quantity for the trailing 30 days is only ~20,872 total across all regions, and its cost is $0. These numbers don't reconcile, and this document does not paper over that. Leading (unverified) hypotheses:

1. S3 Bucket Keys may be enabled on the CloudTrail delivery buckets and/or Avenue's S3 buckets, which would mean most individual object operations reuse a cached data key (no new billed KMS request) even though CloudTrail may still log a management-event record per underlying S3 request in some configurations. Not yet confirmed either way.
2. Billing-metric lag/aggregation: Cost Explorer daily data can lag 24-48h and aggregates differently than CloudTrail's per-event stream; a systematic under-count this large (15-20x) would be unusual but hasn't been ruled out.
3. Free-operation misclassification: some fraction of the counted events may not be separately-billed "requests" under AWS's current KMS pricing model (only cryptographic operations are billed; some administrative/list operations are not — but GenerateDataKey/Decrypt are unambiguously billable operations per AWS's own pricing page, so this alone doesn't fully explain the gap).

What resolves this going forward: the live dashboard (§2, Panel 0) tracks the actual billed dollar figure directly from AWS/Billing, so the operator doesn't need this discrepancy resolved to trust the cost number — it's authoritative regardless. The CloudTrail-volume panels (once enabled, see §2) will let us watch whether raw volume growth ever translates into billed cost, which would itself be informative data toward resolving the discrepancy.

2. Tracking now live

Chosen approach: native CloudWatch dashboard (Grafana, on the existing terraform/ci-monitoring/ stack), using two data sources:

1. Live today, zero new AWS resources: the native AWS/Billing EstimatedCharges metric (ServiceName=awskms), which this account already emits (billing alerts were already enabled — confirmed via aws cloudwatch list-metrics --namespace AWS/Billing). This is the authoritative $ figure and requires no dependency on any other team's infrastructure.
2. Proposed, gated off pending operator/Avenue-team sign-off: CloudWatch Logs metric filters on avenue-trail (the account's only source of account-wide KMS request-volume data) emitting 6 custom metrics (kms_requests_total, kms_generate_data_key, kms_decrypt, kms_generate_mac, kms_verify_mac, kms_create_grant) into namespace MooseQuest/KMS. Code is written and reviewed-ready in terraform/ci-monitoring/kms-cost-tracking.tf, gated behind var.enable_avenue_trail_kms_metrics (default false).

Why gated, not applied

Attaching a CloudWatch Logs metric filter to avenue-trail is read-only and does not modify the trail's own behavior, retention, or cost — but it does attach a Raxx-owned monitoring resource to another product's shared infrastructure, which is a cross-team resource decision, not a routine "extend our own stack" action. sre-agent did not apply this without sign-off. The IAM policy needed for Grafana to read any CloudWatch metric (ci-monitoring-cloudwatch-read, pre-existing in terraform/ci-monitoring/iam.tf but not yet applied — this was found as pre-existing drift, unrelated to this task, and applied as part of this work since it's Raxx's own IAM role) has been applied, so Panel 0 is fully live.

Why not a brand-new Raxx-owned CloudTrail trail instead? Considered and rejected for now: the account already has two trails providing full multi-region coverage; adding a third (even management-events-only) would be a genuinely new recurring cost (additional-trail pricing, ~$2/100k events, plus S3 storage) and a new spend decision, which is explicitly out of scope for this deliverable ("no unilateral spend increase... surface as an operator decision"). If the operator wants Raxx to own its own trail independent of Avenue, that's a clean option but needs an explicit go-ahead given the cost.

Where to view it

Grafana dashboard AWS KMS Request Volume & Cost Attribution (uid: kms-cost-tracking) on the ci-monitoring host:

aws ssm start-session --region us-east-2 --target <ci-monitoring-instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["3000"],"localPortNumber":["3000"]}'
# then http://localhost:3000 -> AWS KMS Request Volume & Cost Attribution

• Panel 0 (KMS Actual Billed Cost) — live now, shows the $/day trend (month-to-date cumulative, resets on the 1st).
• Panels 1-4 (per-operation request counts, audit-chain HMAC watch, CI churn signal) — will show "No data" until var.enable_avenue_trail_kms_metrics is flipped to true (operator decision, see runbook). Left in place now so no further dashboard work is needed once that decision is made.
• Panel 5 — text panel summarizing pricing + the open discrepancy, linking back to this doc.

Note on deployment: the dashboard JSON is delivered via the cloud-init template in this PR. Because the ci-monitoring EC2 instance's user_data is set to ignore_changes (to avoid instance replacement on every template edit — see main.tf), the new dashboard file needs one manual push to the live instance after merge, per the existing "Updating configs" procedure in docs/ops/runbooks/ci-monitoring.md (place the JSON in /opt/monitoring/grafana/dashboards/; Grafana's provisioning watcher picks it up within 30s). sre-agent did not perform this push directly — that action goes through the reviewed commit/PR path per the operator's own instruction, not an ad hoc live-host edit.

Cost of what's live: $0 (the billing metric is free; the one IAM policy applied has no cost). Cost of what's proposed-but-gated: ~$1.80/month if enabled (6 custom metrics × ~$0.30/month each).

3. Cost model

3.1 Pricing used (confirmed current, 2026)

• $1/month per customer-managed key (CMK), prorated hourly. Applies to symmetric, asymmetric, HMAC, multi-Region, and imported-material keys alike. AWS-managed keys (alias/aws/*) are free.
• $0.03 per 10,000 requests for symmetric operations: Encrypt, Decrypt, GenerateDataKey/GenerateDataKeyWithoutPlaintext, ReEncrypt, GenerateRandom, and — same rate — GenerateMac/VerifyMac (HMAC).
• 20,000 requests/month free tier, calculated account-wide across all regions (not per-key, not per-region).
• Asymmetric (RSA/ECC) sign/verify operations are priced higher per-request; not currently in use in this account (no asymmetric keys found).
• Source: https://aws.amazon.com/kms/pricing/

3.2 Current monthly run-rate, by driver

3.3 How each driver scales

• Avenue CMKs: scales with key count, not usage. Flat +$1/mo per new CMK created (dev vs staging vs prod environments each typically add a full set — the dev/staging pattern already shows 5 CMKs each). If Avenue adds a "prod" environment mirroring dev/staging, expect +5 CMKs (+$5/mo).
• CloudTrail self-overhead (dominant volume driver, currently $0 cost): scales with account-wide CloudTrail event rate × 2 (two redundant multi-region trails) × delivery frequency. Grows as EITHER trail's source account activity grows (Avenue's own deploys/API calls) OR Raxx's own us-east-2 activity grows (since both trails are multi-region and capture Raxx too). At current volume (~20-30k crypto ops/day) this is nowhere near a request-cost problem (would need to exceed ~667/day sustained, i.e., already ~30-45x over that on a raw-count basis per §1.5's unreconciled numbers — watch this closely once the discrepancy is understood).
• Audit-chain HMAC (Raxx): will scale ~linearly with audited customer events once deployed — 1 GenerateMac per audit-log write, plus 1 VerifyMac per event during the nightly 24h-window integrity check (roughly 2x the write rate). Additionally, a quarterly full-history re-verification (SOC-2 attestation, --mode monthly run quarterly) re-verifies every event since genesis — this call volume grows with total cumulative audit-event count, not event rate, so each quarter's attestation run gets more expensive than the last as the audit log grows. This detail was not visible in the original "~$2/mo" estimate and is worth the operator's awareness before the feature goes live.
• SSM SecureString decrypts: not currently a measurable cost driver in this account (us-east-2 KMS-Requests = 6,457/month, well within free tier) despite the pattern being flagged as a plausible risk in the task brief. If dyno count, boot frequency, or cron frequency of Raxx's Heroku apps increase substantially, revisit this — each SecureString read on the AWS side (/ci/woodpecker/*, /ci/config-svc/*, cron secrets) is a Decrypt call.
• CI spot-runner churn (CreateGrant/RetireGrant): scales with Woodpecker CI job/instance-cycle frequency. Not billed today (grant ops are free) but would become relevant if AWS ever changes grant-operation pricing, or as a leading indicator of infra churn worth investigating on its own (see runbook Failure Mode D).

3.4 Scaling projection (illustrative, from current measured volumes)

Caveat: the 10x/100x request-cost columns use the unreconciled CloudTrail volume (§1.5), not the currently-billed Cost Explorer volume, as the base rate — deliberately conservative/pessimistic, since we don't yet know why they disagree. If the discrepancy resolves toward the lower Cost Explorer number, these projections would be ~15-20x too high (worst case, not best guess).

3.5 Quick wins (informational — none require sre-agent spend authority beyond what's already applied)

1. Consolidate the two Avenue multi-region CloudTrail trails into one. This is the single largest lever on raw KMS call volume (halves the CloudTrail-self-inflicted GenerateDataKey/Decrypt overhead). Out of scope for Raxx/sre-agent — this is Avenue-owned infrastructure; flagged here for the operator's account-wide awareness only.
2. Enable S3 Bucket Keys on any Avenue S3 buckets using SSE-KMS with a CMK, if not already enabled — dramatically reduces GenerateDataKey call volume for high-frequency object writes by caching data keys at the bucket level (AWS default TTL ~5 min). Same ownership caveat as above.
3. Data-key caching in the Raxx audit-chain HMAC service, when it goes live: compute_event_hash already does one GenerateMac call per write with no batching — this is appropriate for HMAC (there's no data-key caching pattern for GenerateMac/VerifyMac, unlike Encrypt/Decrypt+GenerateDataKey), so no change needed there. The one real lever is bounding the nightly window (already done, 24h) and being deliberate about quarterly full-history re-verify cost as the audit log grows — consider whether quarterly cadence should become less frequent, or windowed/incremental with periodic checkpoints, once audit-event volume is large enough that quarterly full-history verification becomes materially expensive. Not urgent today (feature isn't live).
4. The audit-chain HMAC's approved ~$2/mo baseline should be revisited once deployed to include the quarterly full-history re-verify cost, which the original estimate likely didn't account for (see §3.3).

None of the above were acted on unilaterally — items 1-2 are outside Raxx's ownership boundary, and items 3-4 are forward-looking notes for when a not-yet-deployed feature ships, not current-state changes.

4. Summary for the operator

• Source of the KMS bill: 11+1 customer-managed keys belonging to "Avenue" (a different product sharing this AWS account), at the flat $1/mo/CMK rate. Not Raxx. Not request volume (that's $0, within free tier per Cost Explorer). ~$10.68/month total today.
• The one Raxx KMS feature you approved (~$2/mo audit-chain HMAC) is not live — contributing $0 currently. Worth knowing before you see it appear on a future bill and wonder if it's misbehaving; when it goes live, watch the dashboard's audit-chain panel against this baseline.
• Tracking is live: Grafana kms-cost-tracking dashboard, Panel 0 shows real $/day today with zero new infrastructure. Deeper per-operation attribution is written and ready (kms-cost-tracking.tf) but gated pending your decision on whether to attach a read-only metric filter to the Avenue team's CloudTrail trail (their infrastructure, not touched without sign-off).
• Nothing was spent or changed beyond: one pre-existing (already-in-repo, not-yet-applied) IAM policy for Grafana's CloudWatch read access — $0 cost, Raxx-owned resource only.
• Decision needed from you: (a) sign off on enabling the gated avenue-trail metric filters (~$1.80/mo) for deeper attribution, or decline and rely on the billing-metric panel alone; (b) whether you want Raxx to eventually own an independent CloudTrail trail instead of depending on Avenue's, given the small ongoing cost that would add.