Raxx · internal docs

internal · gated

Logging + Performance Observability Comparison

Status: research-only. This document does NOT constitute legal or tax advice. Before signing any annual contract, consult a CPA. Before selecting a log storage vendor that processes user-activity data, consult an attorney about data-retention obligations in your operating jurisdictions. Last updated: 2026-06-10. Sources as of that date — verify freshness.

TL;DR

Splunk Cloud is enterprise-grade pricing (entry ~$675/year for 5 GB/day) and is architecturally mismatched for a pre-launch single-operator SaaS generating well under 1 GB/day of log data. The operator's S3 instinct is correct for compliance cold-store, but S3 alone gives zero real-time search or alerting. The best v1 path is: Heroku log drain → Better Stack (3 GB free, real-time search + alerts) + S3 archive via lifecycle policy for long-term retention. Cloudflare Pages analytics are free and already on — nothing to buy there.

Log Volume Estimate — Raxx Pre-Launch

Surface Dyno / Instance Count Estimated log volume/mo Notes
raxx-api-prod (Raptor) 1–2 dynos 1–5 GB Python Flask/Gunicorn — moderate verbosity
raxx-console-prod 1 dyno 0.5–2 GB Lighter traffic surface
raxx-velvet-prod 1 dyno 0.5–1 GB Token rotation service — low traffic
Cloudflare Pages (all surfaces) Serverless ~0.1–0.5 GB CF handles routing; app logs minimal
GitHub Actions CI runs ~0.1–0.5 GB Build/test logs; ephemeral
Total pre-launch estimate — ~2–10 GB/mo Wide band due to verbosity config
Total at 10x (post-launch) — ~20–100 GB/mo More dynos + user-event logs

Note: Heroku Logplex retains only the last 1,500 lines per app in memory; anything beyond that requires a log drain. No volume billing from Heroku for log drain egress — it is purely a push-over-syslog/HTTPS mechanism. Source: https://devcenter.heroku.com/articles/logging

Options Compared

Option A — Splunk Cloud

What it is: Enterprise SIEM + log analytics platform. Ingest-based pricing.

Pricing: - 5 GB/day ingest: ~$675/year (~$56/month) - 20 GB/day ingest: ~$2,000/year (~$167/month) - 1–10 GB/day range: $1,800–$18,000/year depending on features and support tier - 90 days indexed storage included. Extended retention billed extra. - Splunk ES add-on (SIEM security): $25–$45/GB/day on top of base pricing.

At Raxx pre-launch volume (2–10 GB/mo ≈ 0.07–0.33 GB/day): Raxx's volume falls below Splunk's minimum commercially listed tier (5 GB/day). Splunk Cloud does not publish a sub-5-GB/day tier. This means either (a) you pay for more than you use or (b) you are in a free trial that eventually bills at enterprise minimums. Splunk is the wrong tool for this scale. This is the operator's instinct to confirm, not act on.

At 10x (20–100 GB/mo ≈ 0.7–3.3 GB/day): Still below the 5 GB/day minimum tier. Splunk only becomes cost-appropriate at sustained multi-GB-per-day ingest, which Raxx will not reach until significant traffic growth well beyond 100 customers.

What you LOSE by not choosing Splunk: Full-text search with SPL, out-of-box dashboards for Heroku/Python, SIEM alerting, compliance-grade audit trail. These are valid post-Series-A features. Not pre-launch needs.

Setup complexity: High. 8–16+ hours for production-grade Splunk Cloud setup. Lock-in risk: Very high. SPL queries are proprietary. Splunk data format and dashboards do not export cleanly.

Sources: - https://www.splunk.com/en_us/products/pricing/ingest-pricing.html - https://costbench.com/software/log-management/splunk-cloud/ - https://expanso.io/blog/splunk-pricing-guide/

Option B — S3 + Athena (Operator's "just collect logs in S3" instinct)

What it is: Heroku log drain pushes syslog to an HTTPS endpoint (a small Lambda or Fluent Bit sidecar writes to S3). Athena queries S3 with SQL.

Cost at pre-launch (2–10 GB/mo): - S3 Standard storage: $0.023/GB/month × 10 GB = $0.23/month - S3 Intelligent-Tiering for older logs (30+ days): transitions to $0.004/GB for archive → negligible - Athena queries: $5.00/TB scanned. At 10 GB, one full-table scan = $0.05. With Parquet + partitioning, effectively $0.001–$0.005/query. - Lambda log-drain receiver: < $1/month at pre-launch volume (well within free tier). - Total pre-launch: ~$1–5/month all-in.

Cost at 10x (20–100 GB/mo): - S3 storage: 100 GB × $0.023 = $2.30/month - Athena: depends on query frequency. 10 queries/day × $0.05/query = $15/month (unoptimized). With Parquet compression: ~$0.50–$2/month. - Total at 10x: ~$5–20/month.

What you LOSE vs a SaaS log viewer: - No real-time log streaming or tailing (Athena is batch-query, not live-tail) - No alerting on log patterns (Athena does not push; you'd need CloudWatch Logs Insights or a scheduled Lambda) - No pre-built dashboard for Heroku/Python/Flask errors - No full-text search UI (you write SQL; Parquet column-skip means you need to know your schema) - Engineering setup time: 4–8 hours to build a reliable drain-to-S3 pipeline + schema

What you KEEP / GAIN: - True long-term retention at near-zero cost (archive logs indefinitely) - Full SQL access to every log line - No vendor lock-in — open formats (JSON/Parquet), standard AWS tooling - Compliance archive with immutable S3 object lock available

Setup complexity: Medium-high. Heroku syslog drain → Lambda → S3 → Glue crawler → Athena. Or use a managed forwarder like Vector/Fluent Bit (open source). Documented patterns exist. Lock-in risk: Very low — open data formats, standard AWS.

Sources: - https://aws.amazon.com/athena/pricing/ - https://cloudburn.io/blog/amazon-athena-pricing - https://aws.amazon.com/s3/pricing/ - https://costimizer.ai/blogs/aws-s3-storage - https://devcenter.heroku.com/articles/log-drains

Option C — Papertrail (Heroku SolarWinds addon)

What it is: Syslog-based SaaS log management. Native Heroku add-on (one-click install from Heroku dashboard). Acquired by SolarWinds.

Pricing (2026): - Free: 50 MB/month, 48-hour search retention - Small: $7/month for 1 GB/month, 7-day retention - Medium: $40/month for 10 GB/month, 7-day retention - Large: $230/month for 100 GB/month, 7-day retention - Overage: up to 200% of plan volume at ~30% premium

At pre-launch (2–10 GB/mo): - Small plan ($7/month): adequate for minimal verbosity. One busy deploy week could hit the 1 GB cap. - Medium plan ($40/month): comfortable headroom. Likely right-sized. - Archive to S3: available (Papertrail can upload daily archives to your S3 bucket — hybrid path).

At 10x (20–100 GB/mo): - Large plan ($230/month) or 2× Medium. Search retention still only 7 days — older logs hit S3 archive.

What you LOSE: - 7-day retention is short for debugging production issues that surface days after the event - SolarWinds ownership adds enterprise-sales dynamic and potential price trajectory risk - Performance metrics / APM are not included — log search only

What you KEEP: - Fastest Heroku setup (< 30 min, native add-on) - Live log tailing from CLI and web UI - Regex log search - Alert on log patterns (basic) + email/PagerDuty/Slack webhooks

Setup complexity: Very low — Heroku one-click add-on + syslog drain is automatic. Lock-in risk: Low for data (S3 archive export); medium for alerting rules.

Sources: - https://www.papertrail.com/plans/ - https://devcenter.heroku.com/articles/papertrail - https://betterstack.com/community/comparisons/heroku-logging-tools/

Option D — Better Stack Logtail (Combined Uptime + Logs)

What it is: Better Stack's log management module (formerly Logtail). Works as a Heroku log drain target. Integrates with Better Stack on-call paging.

Pricing: - Free: 3 GB/month, 3-day retention - Paid bundles: ~$0.15/GB ingestion, ~$0.08/GB/month retention. Practical starting point ~$24/month for 10 GB/30-day retention. - If already paying the $29/month Responder plan for paging, log ingestion is on the same platform — operational simplicity benefit.

At pre-launch (2–10 GB/mo): - Free tier (3 GB) may cover minimal-verbosity pre-launch. Upgrade if logs exceed that. - With the $29/month Responder plan already running: log ingestion on same bill.

At 10x (20–100 GB/mo): - ~$15–$20/month for ingestion + retention at 10× scale. More cost-efficient than Papertrail Large.

Key differentiator: Single vendor for uptime monitoring, on-call paging, and log management. Alert-on-log-pattern directly tied to the same on-call schedule — no cross-tool webhook wiring needed.

Setup complexity: Low — Heroku log drain or OpenTelemetry agent. 1–2 hours. Lock-in risk: Low — standard log ingestion formats; data exportable.

Sources: - https://betterstack.com/pricing - https://betterstack.com/log-management - https://technology.toolsinfo.com/tool/better-stack-formerly-logtail

Option E — Grafana Loki + S3 Backend (Self-Hosted)

What it is: Open-source log aggregation system designed to work like Prometheus but for logs. S3 as the storage backend.

Cost: - S3 storage: same as Option B above — $0.023/GB/month for Standard; lifecycle rules to Glacier ($0.004/GB) for cold. - Loki itself: free software. - Infrastructure to run Loki: minimum 1 EC2 instance or ECS task. At t3.small ($15–$20/month), that's the baseline infra cost. - Kubernetes (full microservices mode): 6+ Pods — not appropriate at this scale. Use single-binary mode. - Total pre-launch: ~$20–30/month (infra) + negligible S3. - At 10×: $20–50/month depending on instance size and S3 volume.

Important: Grafana OnCall OSS was archived March 24, 2026. If you self-host Loki for logs, you cannot pair it with self-hosted OnCall for paging — you would need Grafana Cloud or a separate paging tool.

What you LOSE vs SaaS: - Operational burden: upgrades, capacity management, cardinality explosions if developers add high-cardinality labels - No managed on-call integration without Grafana Cloud Pro tier

What you KEEP / GAIN: - True cost floor near $20–30/month all-in - No per-GB SaaS markup - Full data control / sovereignty

Setup complexity: Medium-high. 4–8 hours for single-binary Loki + Promtail agent + Heroku drain. Higher for Kubernetes mode. Lock-in risk: Very low — open source, open format.

Sources: - https://grafana.com/docs/loki/latest/operations/storage/ - https://www.modern-datatools.com/tools/grafana-loki/pricing - https://oneuptime.com/blog/post/2026-02-09-grafana-loki-object-storage/view - https://www.siriusopensource.com/en-us/blog/true-cost-grafana

Option F — Cloudflare Logpush + Workers Analytics (What's Already Free)

What is already free and active on your CF plan:

  1. Cloudflare Web Analytics: built into Pages, free on all plans. Tracks requests, data transfer, page views, visits, API requests per Pages project. 10% traffic sample with extrapolation. No code changes required — enable via Pages dashboard Metrics tab. Source: https://developers.cloudflare.com/pages/how-to/web-analytics/

  2. Cloudflare Zone Analytics: request counts, bandwidth, cache hit rates, threat summaries — available on free CF plan at zone level. Real-time but sampled.

What requires a paid plan:

  1. Cloudflare Logpush: sends raw HTTP request logs, firewall logs, Workers logs to an external destination (S3, Datadog, Splunk, etc.). Requires CF Enterprise plan for zone-level log push. Workers Trace Events Logpush requires Workers Paid plan ($5/month). - Workers Paid Logpush: 10 million requests included free, then $0.05/million. - Zone-level Logpush: Enterprise only — not available at current CF plan level (unsourced — verify your specific CF plan at https://www.cloudflare.com/plans/).

Implication for Raxx: The CF Pages surfaces (getraxx, docs, mockups, status, raxx-app) generate aggregate analytics already visible in the CF dashboard for free. For raw log export from CF, either accept the sampled analytics or upgrade to CF Pro/Business/Enterprise for more granular Logpush access.

Sources: - https://developers.cloudflare.com/logs/logpush/ - https://developers.cloudflare.com/pages/how-to/web-analytics/ - https://developers.cloudflare.com/workers/platform/pricing/

Cost Comparison Matrix

Option Pre-launch cost/mo (2–10 GB) 10x scale cost/mo (20–100 GB) Real-time search Alert on log patterns Setup hours Lock-in risk
Splunk Cloud Not viable (below minimum tier) ~$56–167 Yes (SPL) Yes 8–16 hrs Very high
S3 + Athena only ~$1–5 ~$5–20 No (batch SQL) No (needs extra Lambda) 4–8 hrs Very low
Papertrail Small/Medium $7–40 $40–230 Yes (live tail) Basic (email/webhook) < 1 hr Low
Better Stack Logtail $0 (free tier 3 GB) ~$15–20 Yes Yes (tied to on-call) 1–2 hrs Low
Grafana Loki self-hosted ~$20–30 (infra) ~$20–50 Yes (LogQL) Yes (Grafana Alerting) 4–8 hrs Very low
CF Logpush (free tier) $0 (analytics only, sampled) $0 No (dashboard only) No 0 (already on) None
S3 + Better Stack hybrid ~$29 (Responder) + ~$1 S3 ~$29–50 Yes (Better Stack) + cold SQL Yes 2–4 hrs Low

Heroku log drain → Better Stack (hot storage, real-time) + S3 archive (cold storage, compliance)

Total hybrid cost pre-launch: ~$29–35/month (Better Stack Responder + S3 storage rounding to $1–6/month).

What Splunk-in-S3 Actually Means

The operator's framing "we should just collect all logs in S3" is architecturally sound for compliance retention. S3 is the right long-term store. The gap is the hot layer: S3 + Athena alone has no real-time search, no alerting, and requires SQL query fluency to investigate incidents. The hybrid model addresses this: hot tier (Better Stack or Papertrail) for real-time + cold tier (S3) for archive.

Splunk is the right answer for a company with a dedicated SRE team, 50+ GB/day ingest, compliance reporting requirements (SOC 2 Type II, FedRAMP), and budget for a $10k+/year contract. That is not Raxx v1.

Jurisdiction Flags

Timing / Deadlines

Better Stack Responder ($29/month annual) as the hot layer + S3 + lifecycle rule as the cold archive. Single vendor for uptime, paging, and log ingestion. Heroku drain setup takes under 2 hours. Free tier evaluates the product before committing. S3 archive ensures you never lose logs for compliance/debugging. Total spend: $29–35/month.

If the $29/month Responder is not yet justified pre-launch: use Better Stack free tier (3 GB/3-day retention) for real-time search, and add Papertrail Small ($7/month) if you need longer retention. This is a stopgap — consolidate to the Responder plan at launch.

Recommended Path — v2 (Post-Launch / First 100 Customers)

As log volume grows and the team expands: evaluate Grafana Cloud (unified metrics + logs + tracing + on-call) as a consolidation play. At 10× volume, Grafana Cloud Pro + Loki (~$20–50/month for logs + $20/active IRM user) competes well with Better Stack. The switch is low-friction because both use standard log drain / OpenTelemetry ingestion. Re-evaluate at 50 GB/month sustained.

Questions for Your CPA

  1. S3 + Athena involves AWS infrastructure costs. If Raxx is structured as MooseQuest LLC dba Raxx, do AWS bills accrue as a business expense in the LLC or at the operator level? Does it matter for pass-through deductibility?
  2. If you prepay an annual SaaS log contract (Better Stack $348/year), is that deducted in the payment year or amortized across the service year?
  3. For log data that includes customer trading-session metadata: does long-term S3 retention trigger any state-specific financial-record retention requirements that a CPA or attorney should review?

Sources