Design: public ingress for monitoring.moosequest.net + product-status dashboard coverage

Status: Accepted. Cards A, C and D are in the code; Card B is pending. §§1-10 are the 2026-08-31 proposal; §11 records what was built (verified 2026-10-30 UTC). Author: software-architect Refs: #4547 (Dyn existence-monitor gap), #4545/#4544 (CI incident chain), #4568 (epic), [ADR-0149](https://internal-docs.raxx.app/architecture/adr/0149-session-agent-credential-model.html) (credential model), ADR-0150, ADR-0134 (ci-monitoring box), card #1022 (stack build)

1. Context

Grafana (+ Prometheus/Alertmanager/blackbox-exporter) runs on the ci-monitoring EC2 box, 100% loopback-bound, accessible only via aws ssm start-session --document-name AWS-StartPortForwardingSession. Two asks:

1. Give it a real public hostname, monitoring.moosequest.net, gated the way this org already gates internal tools — not "obscure but reachable."
2. Broaden Prometheus/blackbox coverage from CI-only to a "product status" view (api.raxx.app, console, vault.raxx.app, tickets.raxx.app, Woodpecker) and ship one overview dashboard for it.

This lands the same week as the CI incident chain (#4544/#4545) that produced ADR-0149. The ci-monitoring box is shared, fragile infra right now (build cache + alerting brain for two products) — this design sequences public-ingress work away from the box's existing scrape/alert config so a botched step can't repeat that incident class.

2. Invariants

• No raw internet exposure of an app relying solely on its own login form. (FreeScout port-22-to-0.0.0.0/0 precedent — do not repeat.) Grafana gets an outer auth gate in front of its own login, not instead of it.
• Secrets in SSM/Infisical per workload type, never in files that ship. OIDC client secret is an AWS-workload secret → SSM, per the AWS-workloads-use-SSM rule (same as ci.moosequest.net's google_client_id/google_client_secret).
• No session/dispatched agent holds an admin credential; mutating ops go through reviewed, bounded scripts (ADR-0149). DNS is manual by design (see §4) — no agent attempts it.
• Audit trail for access-affecting change. ALB access logs + CF/Google OIDC sign-in events must be retained; Grafana's own auth log stays on.
• Least blast radius to the shared box. Public-ingress infra must not require editing the box's live scrape/alert config in the same change.
• Grafana exposes real topology (instance IDs, private IPs, alert states across two products) — treat it as sensitive as the vault UI, not as a status page.

3. Assumption check: is moosequest.net still Dyn, or moved to Cloudflare?

Verified live: moosequest.net NS records are still ns[1-4].dns.dyn.com (Oracle Dyn) and ci.moosequest.net still resolves via CNAME to the Woodpecker ALB. The zone has not moved to Cloudflare. This confirms the brief's assumption: the vault.raxx.app (Cloudflare Access) precedent does not apply here — Cloudflare Access requires the hostname's DNS to be proxied through Cloudflare's edge, and moosequest.net isn't delegated to CF at all. The ci.moosequest.net precedent (ALB + authenticate-oidc against Google Workspace) is the correct, and only architecturally consistent, pattern for monitoring.moosequest.net.

4. Architecture decision (see ADR-0150 for the full record)

A new, dedicated public ALB (monitoring-grafana-alb) — not a second hostname bolted onto the existing ci-woodpecker-alb (as built, the operator reversed this on 2026-09-03: see §11) — in the same VPC as ci-monitoring and the Woodpecker box (public subnets already exist there per infra/ci).

• Listener :443 default action: authenticate-oidc (Google Workspace, moosequest.net-domain-restricted via the IdP's own domain scoping — Grafana has no equivalent of Woodpecker's own OAuth, so there is no bypass-path complexity like ci.moosequest.net's /api/* carve-outs) → forward to a target group pointed at the ci-monitoring instance's private IP, port 3000.
• Listener :80: redirect to :443 only (matches ci.moosequest.net).
• No listener-rule bypasses. Grafana has no webhook/badge-equivalent unauthenticated surface to carve out — every path goes through the OIDC gate. (Grafana's own login screen still sits behind that gate as a second factor of "is this really the operator," not as the only gate.)
• New security group (monitoring-alb-sg) with 443/80 open to 0.0.0.0/0, egress only to the target group. The existing ci-monitoring-sg gets one additive ingress rule: TCP 3000 from monitoring-alb-sg only — the SG already documents "port 3000 bound to 127.0.0.1 inside Docker, not reachable from other hosts" as its current invariant, so this is a scoped, explicit exception, not a re-opening of the box.
• Grafana's docker-compose port publish changes from 127.0.0.1:3000:3000 to 3000:3000 (still SG-gated — no host firewall change needed since the box has no other public IP path). This is the one line inside the box's own compose that changes for this to work; it does not touch scrape/alert config.
• Why a new ALB, not a shared one: the WP ALB's OIDC action is wired 1:1 to one certificate/one default target group; a second cert (SAN or separate ACM cert) + a hostname-based listener rule is possible but couples Grafana's availability/blast-radius to Woodpecker's ALB (already carrying webhook-bypass listener rules and an ASG-of-1 cutover in flight per alb.tf). A second, small, independently-deployable ALB costs ~$16/mo and keeps the two failure domains separate — directly serving invariant "least blast radius to the shared box."
• Grafana admin login stays enabled (defense in depth) but the admin password rotates out of "the only gate" role; day-to-day access is via Google OIDC + a to-be-decided Grafana role mapping (open question, §9).

5. DNS — operator action (Kristerpher, manual, Oracle Dyn)

No IaC path exists for the Dyn zone. After the ALB is applied and its DNS name is known (terraform output monitoring_alb_dns_name), Kristerpher adds, by hand, in the Dyn control panel:

monitoring.moosequest.net.  CNAME  <monitoring-alb-dns-name>.us-east-2.elb.amazonaws.com.  TTL 300

Same shape as the existing ci.moosequest.net CNAME. Also needs an ACM certificate for monitoring.moosequest.net validated via DNS — that CNAME validation record is a second manual Dyn entry ACM will emit; add it before the cert issues. Card #4547 (no existence monitor for hand-typed Dyn records) already tracks the class of risk this introduces — flagging here rather than solving it in this pass.

6. Broader scrape/probe coverage

Follows /onboard-o11y exactly — Terraform (cloud-init.sh.tpl) is the source of truth, hot-applied on the box, one target/probe at a time, idempotent:

No new alert severities invented — reuse RaptorProdDown-shaped rules (probe_success == 0, for: 2m, severity: critical) per target, routed through the existing Alertmanager → Postmark → kris@moosequest.net path. Per the skill's liveness-≠-health lesson, each of these gets a follow-up open question (§9) on whether a deeper functional signal exists per surface — do not block this rollout on inventing one for all four at once.

7. Dashboards

One new "Product Status — Overview" dashboard: a single row of up/down stat panels (one per probed surface) + a shared latency graph, sourced from the same blackbox-http/blackbox-tls jobs. Existing CI-specific boards (wp-pipeline-health, ci-ha-health, host-metrics, blackbox-probes, kms-cost-tracking) are untouched. Deliberately not building a per-surface detail board in this pass — that's scope creep into an unmaintained board pile; open a follow-up card per surface only if the overview proves insufficient.

8. Rollout sequence (risk-minimizing — read in order)

1. Card A — ALB + target group + security groups + ACM cert (Terraform only, infra/monitoring-ingress/ new root, does NOT touch terraform/ci-monitoring/). No DNS live yet; verify via ALB's own DNS name + curl -v for TLS/OIDC redirect.
2. Card B — Grafana compose port-publish change (127.0.0.1:3000:3000 → 3000:3000) + the one additive SG rule on ci-monitoring-sg. Applied via the same hot-apply + cloud-init-template discipline as /onboard-o11y. This is the only card that touches the shared box, and it's a one-line diff, isolated from scrape/alert config.
3. Operator action — Kristerpher adds the Dyn CNAME(s) (§5) once Card A's ALB DNS name exists.
4. Card C — Prometheus/blackbox coverage expansion for the four product surfaces (pure /onboard-o11y pattern, additive to prometheus.yml / blackbox.yml / rule files).
5. Card D — "Product Status — Overview" Grafana dashboard.
6. Card E — Grafana OIDC/role-mapping hardening + access review (post-launch, see open questions).

Cards A/B can land and be verified independently of C/D/E — public ingress does not depend on broadened coverage, and vice versa. If Card B's SG/port change causes any issue, rollback is a one-line revert with no coupling to Card C/D.

9. Security considerations

• PII: none. Grafana shows infra metrics/topology, not customer data.
• Retention: ALB access logs → S3, default 90d (match ci-woodpecker-alb's eventual logging posture — currently enable_deletion_protection only, no access logs enabled there either; open question, see below).
• Data-subject requests: N/A (no PII).
• Audit: Google OIDC sign-in is logged by Google Workspace admin console (existing org capability, no new work). ALB access logs give per-request audit if enabled (see open question).
• Credential replay: OIDC client secret in SSM SecureString, IAM-instance-role read only, never printed, matches the existing ci.moosequest.net pattern exactly. Grafana admin password is a fallback local account, not exposed to the ALB layer, rotatable without redeploy via existing SSM param + hot-apply.
• Breach: if the OIDC client secret leaked, revoke in Google Workspace admin console + rotate SSM param + terraform apply (no redeploy of the box needed, ALB-side only). Named in the runbook update (Card F, optional, see open questions).
• Kill-switch: the ALB's default_action can be flipped to a static 503 fixed-response in one terraform apply if the gate needs to be pulled in an incident — cheaper and faster than a DNS change.

10. Open questions (operator decision needed before Cards C/D/E are claimed)

1. Should ALB access logging be enabled for monitoring-grafana-alb (and retroactively for ci-woodpecker-alb, which currently has none)? This design assumes "yes, 90d S3" but that's a new recurring cost + a gap in the existing ALB that predates this work — confirm before Card A bakes in a logging bucket.
2. Grafana role mapping post-OIDC: should every moosequest.net Google account get Grafana Editor, or should this stay Admin-only (Kristerpher) with future viewers added explicitly? Affects whether Card E is "map by domain" or "map by explicit allowlist."
3. Per-surface functional-health signal (liveness ≠ health, per /onboard-o11y) — for console/vault.raxx.app/tickets.raxx.app, is blackbox http_2xx alone sufficient for v1, or does any of them already expose a deeper /health payload worth alerting on directly? Recommend deferring to a follow-up card per surface rather than blocking this rollout.
4. Should the ci-woodpecker-alb pattern's WAF rate-limit rule (referenced in alb.tf's webhook/badge comments) be mirrored onto monitoring-grafana-alb, or is the OIDC gate (no anonymous path at all) sufficient without a separate WAF?

11. As built (verified against develop, 2026-10-30 UTC)

Sections 1 to 10 are the design as proposed on 2026-08-31. ADR-0150 is now Accepted and its "As built" table is the authoritative record; the points that change this document:

• §4 and §8 Card A: shared ALB, not a new one. On 2026-09-03 the operator chose a host-header listener rule on the existing ci-woodpecker-alb (priority 30, OIDC then forward) instead of the dedicated monitoring-grafana-alb, monitoring-alb-sg and separate infra/monitoring-ingress/ root described above. Code: infra/ci/monitoring-ingress.tf. The ":80 redirect" and "default action" language in §4 therefore refers to the shared listener, and the kill-switch in §9 is the listener rule, not the ALB default action.
• §5 DNS: the Terraform output is monitoring_cname_record (target: the shared ALB's DNS name), not monitoring_alb_dns_name. ACM validation CNAME outputs are monitoring_acm_validation_cname_name and monitoring_acm_validation_cname_value.
• §8 Card B (#4571) is not done. The compose port publish is still 127.0.0.1:3000:3000 and ci-monitoring-sg has no ALB ingress rule, so the path is closed end to end.
• §6 Card C (#4572) coverage in the cloud-init template: blackbox-http probes the console and Raptor Heroku origins (the CF-fronted api.raxx.app is covered by the existing Raptor probe, not a second target) and https://tickets.raxx.app/login; blackbox-http-origin-vault probes the Infisical health endpoint directly at the Lightsail origin; blackbox-tls covers vault.raxx.app, tickets.raxx.app, api.raxx.app, console.raxx.app and others.
• §7 Card D (#4573): dashboard product-status-overview exists in the template.
• §10 open questions 1 and 4 (ALB access logs, WAF) and 2 (Grafana role mapping) are still open: cards #4575 and #4574, waiting on operator decisions. alb.tf still configures no access logs.
• Refs: [ADR-0149](https://internal-docs.raxx.app/architecture/adr/0149-session-agent-credential-model.html) (credential model).