AWS cost breakdown & attribution — shared account 521228113048

Prepared: 2026-08-02 UTC Author: sre-agent Scope: Infrastructure-budget Phase 1 — attribution, tagging, ranked cut list (analysis only) Account: 521228113048 (shared: Raxx CI infra + Avenue project, us-east-2 primary + us-east-1 Lightsail, region-swept) Credentials used: session AWS creds resolved to IAM user aws-developer-alpha (AdministratorAccess) — this is the account's actual active session identity; it is not claude-infisical-bootstrap as referenced in the task brief. Flagging so the operator can confirm which IAM user is the intended standing identity for this account (see Findings below). No resource changes, no deletions, no instance modifications were made. Cost-allocation tags were applied to previously-untagged resources and activated in Billing — both are additive, non-destructive, pre-approved Phase 1 actions.

TL;DR

• June 2026 total: $61.20. July 2026 total: $502.06. Aug MTD (2 days): $16.31. The "~$500/mo" bill is not a steady-state run rate — it is a July step-change driven by two independent build-outs that both went live in the same window: the Raxx CI runner fleet (ADR-0134, live 2026-07-03/04) and a new "Avenue" project's AWS footprint (dev + staging, built out 2026-06-17 through 2026-07-05).
• Attribution of the July $502.06: Raxx CI infra $229.38 (46%), Avenue dev+staging $261.31 (52%), unattributed $1.00 (two unrelated Route 53 hosted zones — see Findings), mixed/shared/immaterial (S3, CloudWatch, Bedrock, tax) $10.37 (2%). This reconciles to $502.06 within a cent.
• Headline finding: the Raxx CI stack (ADR-0134) is running at a full-month-equivalent run rate of ~$254/mo against its own operator-accepted budget of ~$107/mo — 137% over. The overage is almost entirely NAT Gateway data processing ($66.89) and VPC interface-endpoint hourly charges ($53.76), not compute (spot agents cost $19.12). This is flagged as the top action item, not a general "trim compute" item.
• Tags applied: all previously-untagged Raxx-owned resources (Lightsail raxx-vault instance + its static IP + snapshot, 7 raxx-tickets backup snapshots, raxx-tickets instance, and 3 untagged S3 buckets — raxx-iac-state-prod, raxx-support-attachments, raxx-waf-logs-prod) now carry project/service/env tags. Terraform-managed resources (all ci-woodpecker-* and all avenue-* resources) were already fully tagged by their Terraform modules — see Findings for a tag-key-casing issue found in that existing scheme.
• Cost-allocation tags activated in AWS Billing: project, Project, env, Environment, Component, component, managed-by, ManagedBy, Purpose, purpose, Name — all were previously inactive (zero cost-allocation tags were active before this pass). Activation can take up to 24h to populate in Cost Explorer's tag-filter UI per AWS's documented propagation delay.
• Projected savings if the ranked cut list below is executed in Phase 3: roughly $180–230/mo, which would bring the account from the current $502/mo run rate to ~$270–320/mo on its own — short of the stated $200–250 target unless Avenue's dev/staging scheduling cut (the single largest lever, and explicitly Avenue's call, not Raxx's) is also approved. See "Path to $200–250" at the end of the cut list.

1. Cost Explorer breakdown — last 2 full months + MTD

All figures are UnblendedCost, USD, pulled via aws ce get-cost-and-usage on 2026-08-02.

By SERVICE

The daily trend inside July (top-5 services: EC2-Other, VPC, ELB, RDS, ECS) shows a clean step-change from ~$6/day (Jul 1–3) to ~$11–17/day starting 2026-07-04 — the exact date the ci-woodpecker (Raxx CI, ADR-0134) stack went live in us-east-2.

Top-3 drivers (July, by $ and by usage type)

1. EC2 - Other, $187.17 — dominated by NAT Gateway: hourly charges ($30.29 raxx / $66.96 avenue) and data processing ($66.89 raxx / $0.21 avenue). The raxx NAT gateway processed 1.49 TB in ~28 days on a CI fleet that has no image/dependency cache — this is the single biggest line item in the entire bill.
2. Amazon VPC, $86.19 — VpcEndpoint-Hours ($53.76, raxx: 4 interface endpoints × 2 AZ, ec2messages/ssm/ssmmessages/logs) + public IPv4 hourly charges ($22.33 avenue / $10.09 raxx — the AWS-wide Feb-2024 $0.005/hr public-IPv4 charge, unavoidable while these ALBs/NAT gateways are internet-facing).
3. Amazon ELB, $48.83 — 3 ALBs total: ci-woodpecker-alb (raxx, us-east-2, $15.17) + avenue-dev-api + avenue-staging-api (us-east-1, $33.65 combined).

Usage-type drill-down, other top services (July)

• RDS ($41.86): 3× db.t4g.micro Postgres — ci-woodpecker (raxx, $13.15), avenue-dev-postgres + avenue-staging-postgres (avenue, $28.72).
• ECS ($36.80): 100% avenue — Fargate tasks in avenue-dev and avenue-staging clusters (no Raxx ECS usage; Raxx CI uses EC2, not Fargate).
• CloudTrail ($27.02): 100% avenue — two multi-region trails (avenue-trail, avenue-staging-trail), both IsMultiRegionTrail: true and IncludeGlobalServiceEvents: true. Both trails independently log 100% of account-wide API activity regardless of their env tag — CloudTrail doesn't filter by tag, so this is genuinely duplicate event capture, not two trails each covering half the account. This matches the "two redundant multi-region trails" flagged during the earlier KMS pass.
• KMS ($11.84): 11 keys, all avenue-* (dev/staging × feedback/media/logs/rds/secrets, plus tf-state + tf-state-dr).
• Secrets Manager ($8.48): 100% avenue — all secrets are avenue/* or RDS-managed rotation secrets for the two avenue Postgres instances.
• WAF ($8.34): 100% raxx — ci-woodpecker-waf (the only Web ACL in the account; avenue has none).
• Lightsail ($6.82): 100% raxx — raxx-tickets (FreeScout) + raxx-vault (Infisical).
• Route 53 ($5.80): Health-Check-* $4.79 (avenue — 2 non-AWS HTTPS health checks against bdca7f-{dev,staging}-api.moosequest.app) + HostedZone $1.00 (2 zones — unattributed, see Findings) + $0.01 query cost.
• S3 ($1.97): 57 GB-month, 100% S3 Standard (no IA/Glacier tiering in use anywhere in the account), ~173k requests. Spread across 21 buckets; no single bucket is cost-material.

2. Resource → project attribution

Raxx (CI infra, ADR-0134, ci-woodpecker naming, us-east-2)

Avenue (avenue-* naming, us-east-1, dev + staging)

All Avenue resources are already fully tagged by their Terraform modules (project=avenue, env=dev\|staging, managed-by=terraform) — no gaps found.

Third tenant found — flagged, not tagged, not attributed

prototype-site-{dev,staging,prod}-{site,cflogs}-521228113048 (6 S3 buckets, us-west-1, CloudFormation-managed stacks named prototype-site-{dev,staging,prod}) and two Route 53 hosted zones (apl-project-monkey.net, boingboing-inc.net, $0.50/mo each = $1.00/mo) do not match Raxx or Avenue naming/tagging conventions anywhere in this account. Cost impact is immaterial (S3 buckets are near-empty; the zones are $1.00/mo total), but I did not guess an owner and did not tag these — misattributing a third project's resources as Raxx or Avenue would corrupt the cost-allocation data this whole exercise exists to produce. Operator: confirm ownership of these 8 resources before the next cost sweep folds them into either bucket.

3. Tags applied and cost-allocation activation

Applied via lightsail tag-resource / s3api put-bucket-tagging (additive, non-destructive — no existing tags were touched or removed):

Deliberately NOT re-tagged: every ci-woodpecker-* and avenue-* resource, because they are Terraform-managed. Adding CLI-set tags on top of a Terraform-owned resource creates drift that the next terraform apply will silently revert, undoing the tagging work without anyone noticing. These resources are already comprehensively tagged by their modules; the gap is a key-casing inconsistency (see Finding below), not missing coverage.

Cost-allocation tags activated in AWS Billing (aws ce update-cost-allocation-tags-status, zero errors): project, Project, env, Environment, Component, component, managed-by, ManagedBy, Purpose, purpose, Name. All were inactive before this pass — Cost Explorer's tag-based cost views have never worked in this account until now. AWS documents up to 24h propagation before these appear queryable in Cost Explorer's tag filter.

Finding — tag key casing splits cost-allocation views: the ci-woodpecker Terraform module tags with Project (capital P); the avenue Terraform module tags with project (lowercase). AWS cost-allocation tags are case-sensitive keys — activating both (done above) works around it for this account, but any Cost Explorer report grouped by "the project tag" will silently miss all Raxx CI resources unless the operator also groups by Project. Recommend a follow-up ticket to standardize the CI Terraform module's tag key to lowercase project to match Avenue's convention (and the AWS-recommended lowercase convention) — not done here since it's a Terraform-file change, out of scope for this read/tag-only pass.

4. Ranked cut list — Phase 3 draft, nothing executed

Ordered by projected $/mo saved. No cut in this list weakens a security control without the risk explicitly stated. All are drafts for operator review; none were applied.

Not on this list, deliberately: downsizing the Raxx CI spot agent fleet or the t4g.small server/monitoring boxes. Compute is $19.12/mo of the $502.06 total — the cheapest line item in the whole stack. Cutting compute would save single-digit dollars while risking CI throughput; the actual overspend is 100% networking (NAT + VPC endpoints), which items #1 and #3 already address.

Path to $200–250/mo

Items #1–3 and #6–7 (the ones sre-agent can execute without needing another team's sign-off) total ~$95–105/mo in projected savings, bringing the account to roughly $395–405/mo — real progress, but not the stated $200–250 target on their own. Item #4 (Avenue dev/staging scheduling) is the only lever large enough to close the remaining gap, at ~$115–135/mo, and it is explicitly Avenue's infrastructure and Avenue's decision. The operator should treat #4 as the deciding factor for whether $200–250/mo is reachable this cycle, or whether the target needs to move to ~$300–320/mo pending Avenue's own environment-scheduling decision.

5. Findings for operator attention (not cuts — process/attribution gaps)

1. Session credential mismatch. This pass ran as IAM user aws-developer-alpha (AdministratorAccess), not claude-infisical-bootstrap as referenced in the task brief and in project_aws_iam_state.md. Both have full admin. Confirm which is the intended standing identity — if aws-developer-alpha is a leftover/unexpected user, that's worth a separate IAM hygiene look.
2. Third-tenant resources in the shared account (prototype-site-*, apl-project-monkey.net, boingboing-inc.net) — see section 2. Immaterial cost, but unattributed; confirm ownership.
3. Tag key casing split (Project vs project) between the Raxx CI and Avenue Terraform modules — see section 3.
4. ADR-0134's own "revisit when" trigger reads "Monthly EC2 cost exceeds $150/month." Taken literally (EC2 compute only), that trigger has NOT fired — compute is $19.12/mo. The actual overrun is against the ADR's stated whole-stack budget of ~$107/mo (VPC+ALB+WAF+IAM+RDS+agents), which the full-month-equivalent run rate of ~$254/mo breaches by 137%. Recommend the ADR's revisit trigger be reworded to cover total stack cost, not EC2-compute-only, so this class of overrun trips the trigger automatically next time.

References

• ADR: docs/architecture/adr/0134-ci-ec2-self-hosted-runner-fleet.md
• Card: #4012 (infra(ci): Terraform baseline — VPC, ALB, WAF, IAM, SSM, runner fleet (Phase 1))
• Prior related run-rate work: docs/business/bookkeeper/2026-06-04-monthly-runrate-estimate.md