RCA — tickets-e2e-smoke false FAIL: smoke script never updated for #4262 mailto-only contact page

Incident ID: 2026-08-09-tickets-e2e-smoke-mailto-drift Date: 2026-08-09 Severity: SEV-3 (non-blocking — the actual customer-facing golden path was healthy throughout; only the synthetic smoke's assertions were stale) Duration: Smoke test drift present since 2026-07-24 (PR #4335 merged); first detected/filed as a tracked issue 2026-08-07 07:06 UTC (#4429); root-caused and fixed 2026-08-09 Blast radius: CI/ops tooling only — tickets-e2e-smoke daily cron and the auto-filed type:reliability issue queue. No customer impact: tickets.raxx.app/new-conversation served HTTP 200 with a working mailto:support@raxx.app contact page the entire time. Author: sre-agent

Summary

The daily tickets-e2e-smoke cron (and pipeline #4832, a one-off manual trigger fired as collateral of the 2026-08-07 Woodpecker CI server replace — see docs/incidents/2026-08-07-woodpecker-server-replace-pip-cache-activation.md) failed and auto-filed issue #4429. Investigation found the customer-facing system under test was healthy: https://tickets.raxx.app/new-conversation returns HTTP 200 and serves a working "Contact Support" page whose submit button opens the visitor's email client pre-addressed to support@raxx.app. That page is the intended, durable, already-documented design (docs/ops/runbooks/freescout.md Failure mode G, shipped via PR

4335 / issue #4262 on 2026-07-24) — a deliberate interim fix for the

/new-conversation 404 (#4249) that predates the full web-form submission path planned under epic #651.

The smoke script (scripts/ops/tickets_e2e_smoke.py), however, was never updated after #4262 shipped. Step 2 asserted the page contained a method= form attribute (true only for a server-rendered form; the mailto page's <form id="form"> has none because a client-side JS handler intercepts submit). Step 3 POSTed synthetic form data to /new-conversation expecting a server-side ticket-creation response — there has never been one at this path; the static page returns HTTP 200 regardless of method, so Step 3 was producing a false-positive PASS while creating no ticket. Step 4 then failed to find that (nonexistent) ticket via the FreeScout API — masked by a second, independent bug: the API call was missing the CF-Access-Client-Id/ CF-Access-Client-Secret headers required by the /api/* CF Access application (decision=non_identity), so the actual response was an HTTP 403 CF Access block, not a clean "not found."

Both defects are fixed in PR #4446: Step 2 now accepts the documented mailto-only page shape, Step 3 SKIPs cleanly instead of asserting a meaningless POST, and both tickets_e2e_smoke.py and tickets_cleanup_smoke.py now send the CF Access service-token headers on every FreeScout API call.

Timeline (all times UTC)

• 2026-07-24 — PR #4335 (issue #4262) merges: /new-conversation becomes a durably-provisioned static mailto-only contact page. docs/ops/runbooks/freescout.md Failure mode G is corrected to document this. scripts/ops/tickets_e2e_smoke.py is not updated in the same PR — the drift is introduced here, silently (Steps 2-3 continue to assert the pre-#4262 server-rendered-form shape).
• 2026-08-07 06:19–06:42 — Unrelated: Woodpecker CI server replace (docs/incidents/2026-08-07-woodpecker-server-replace-pip-cache-activation.md). The replace's golden-path validation step triggers a full manual pipeline on develop (pipeline #4832), which — because Woodpecker manual triggers fan out to every workflow matching event: manual — incidentally also runs tickets-e2e-smoke for the first time since the #4262 drift was introduced two weeks earlier.
• 2026-08-07 07:05:58–07:07:05 — Pipeline #4832, workflow tickets-e2e-smoke (WP step 253/e2e-smoke) runs and fails:
• Step 1 (CF Access gate) — PASS (302, no CF Access redirect).
• Step 2 (new ticket form loads) — FAIL: HTTP 200 from /new-conversation, but no <form ... method=...> detected. Body snippet in the log confirms this is the real, intended "Contact Support — Raxx" mailto page, not an error page.
• Step 3 (synthetic ticket submitted) — PASS (false positive — POSTed to a page with no backend; HTTP 200 returned regardless).
• Step 4 (ticket visible in FreeScout API) — FAIL: HTTP 403 from /api/conversations, body {"message":"Forbidden...","status_code":403, "aud":"7e22c50e..."} — a CF Access non-identity block, not FreeScout's own "ticket not found" response.
• Step 5 — SKIP (Step 4 found nothing to tag).
• Step 6 (invalid path 404 behaviour) — PASS.
• Overall: FAIL (2 of 6 steps).
• 2026-08-07 07:06:57 — notify-on-failure fires (Slack + Postmark ops@raxx.app); auto-file-issue step opens #4429 with the pipeline link and step-log pointers, per the auto-ticketing pipeline (#4263).
• 2026-08-09 19:16–19:25 — PM grooming pass (#4429 comment) tightens acceptance criteria; card-groomer marks it READY, routes to sre-agent.
• 2026-08-09 (this incident) — Triage:
• Confirmed live: curl -sI https://tickets.raxx.app/new-conversation returns HTTP/2 200; full body fetch shows the documented mailto contact form (<form id="form">, JS handler builds a mailto:support@raxx.app link). Golden path was never actually broken.
• Pulled the WP pipeline #4832 step log directly (GET /api/repos/1/logs/4832/28304 — the v3 log-fetch endpoint is /api/repos/{id}/logs/{pipeline_number}/{step_id}, not /api/repos/{id}/logs/{pipeline_id}/{workflow_pid}, which 200s with SPA HTML instead of erroring) and confirmed the exact Step 2/Step 4 failure text quoted above.
• Traced the page-shape change to PR #4335 (#4262, merged 2026-07-24) via gh api graphql cross-referenced-PR lookup on issue #4249, and confirmed docs/ops/runbooks/freescout.md Failure mode G already documents the mailto-only design as intentional and durable — this is smoke-script drift, not a live regression.
• Confirmed the CF Access /api/* service-token requirement (docs/ops/runbooks/freescout.md §CF Access applications, app ca6fd315-...) and confirmed via grep that tickets_e2e_smoke.py's _api_get/_api_patch never sent CF-Access-Client-Id/-Secret, even though .woodpecker/tickets-e2e-smoke.yaml already loads CF_ACCESS_CLIENT_ID/CF_ACCESS_CLIENT_SECRET into the step's environment for this exact purpose.
• Fixed forward in scripts/ops/tickets_e2e_smoke.py: Step 2 accepts the mailto-only page shape; Step 3 SKIPs (no POST attempted) when Step 2 detects that shape; Step 4/5 gained the CF Access headers. Also fixed the identical, not-yet-triggered CF Access header gap in scripts/ops/tickets_cleanup_smoke.py (same pipeline, weekly cron) to prevent the same root cause resurfacing on the next Sunday run.
• Updated .woodpecker/tickets-e2e-smoke.yaml header comment and docs/ops/runbooks/freescout.md Failure mode G with an addendum documenting the fix.
• python3 -m pytest scripts/ops/tests/test_tickets_e2e_smoke.py scripts/ops/tests/test_tickets_cleanup_smoke.py -q — 65 passed (18 new/ updated cases covering the mailto-only PASS path, the Step 3 SKIP path, and the CF Access header helper in both scripts).
• Opened PR #4446 (fix/4429-tickets-e2e-smoke-mailto-drift → develop). WP ci-pr pipeline (repo pipelines #5105/#5108, one per commit) both completed success.
• Verified via WP cron 22 (tickets-e2e-smoke-daily) — single-workflow scoped trigger (POST /api/repos/1/cron/22 "start a cron job now"), not a bare manual dispatch — per operator instruction to avoid the ~65-workflow fan-out a plain manual trigger causes. See Resolution/Validation below for the specific green-run pipeline number.

Impact

• Users affected: none. tickets.raxx.app/new-conversation was reachable and functional (mailto contact flow) throughout.
• User-visible symptoms: none.
• Data integrity: ok.
• Revenue / billing: ok.
• Operational cost: one open type:reliability issue (#4429) sat unresolved for ~2 days after the daily cron would have (silently, per the dedup rule) kept re-surfacing the same false FAIL every day it ran, and every Step-3 "PASS" was itself misleading (falsely implying the web ticket-submission path was verified end-to-end when no ticket was ever created) — a false sense of coverage is its own toil/risk, independent of the FAIL noise.

What went well

• The auto-ticketing pipeline (#4263) did exactly what it was built for: turned a silent daily-cron failure into a durable, deduped, triage-able GitHub issue instead of depending on someone noticing a Slack alert.
• The WP pipeline step log was sufficient, on its own, to fully diagnose both root causes (mailto-shape mismatch and missing CF Access headers) without needing to reproduce locally first — the smoke script's own detailed per-step detail strings (body snippets, HTTP codes, response bodies) did their job.
• docs/ops/runbooks/freescout.md Failure mode G was already accurate and current (last updated 2026-07-24) — cross-referencing it immediately distinguished "known, accepted, documented design" from "novel failure," preventing a wrong-headed attempt to "fix" the mailto page itself.
• The fix is purely additive to the smoke script/tests — zero product code touched, consistent with sre-agent scope.

What didn't go well

• PR #4335 (#4262) changed the actual shape of /new-conversation and updated the runbook in the same PR, but did not update or even flag scripts/ops/tickets_e2e_smoke.py, which asserts against that exact page. There is no automated linkage between "a PR changes a page the smoke test asserts on" and "the smoke test's assertions get reviewed" — this is a process gap, not a person gap: nothing in CI cross-references application surface changes against the smoke scripts that exercise them.
• Because the daily cron had apparently not run against develop since before 2026-07-24 in a way that got noticed (or had been running and matching against a dedup key some other way) until the 2026-08-07 replace incidentally triggered a full manual run, the drift was invisible for roughly two weeks. A cron that silently fails without alerting until an unrelated manual trigger surfaces it is itself a detection gap.
• Step 3's prior implementation had no way to distinguish "the form POST worked" from "the origin returns 200 for any request regardless of whether anything happened" — the smoke asserted on HTTP status and a handful of substring markers, not on any independently verifiable side effect (until Step 4's API check, which was itself broken by the missing CF Access headers). Two independently-broken verification layers stacked meant the false positive in Step 3 was never caught by Step 4 either.

Root cause analysis

• Contributing factor 1 — no dependency tracking between application surface changes and the smoke assertions that exercise them: PR #4335 correctly updated the runbook's documentation of the new page shape but had no mechanism (lint, checklist item, or test coupling) prompting a corresponding update to tickets_e2e_smoke.py. The smoke script and the page it tests drifted independently for two weeks with no automated signal.
• Contributing factor 2 — the CF Access /api/* service-token requirement and the smoke script's authentication were never cross-checked. .woodpecker/tickets-e2e-smoke.yaml was provisioned with CF_ACCESS_CLIENT_ID/CF_ACCESS_CLIENT_SECRET secrets specifically for this step, but the Python script never read or sent them — the env vars were dead weight from day one. This wasn't caught earlier because Step 4 never got exercised successfully long enough to reveal it (Step 3's false PASS meant Step 4 only ever ran against a nonexistent ticket, so its failure mode — "ticket not found" vs. "CF Access 403" — looked superficially similar in the day-to-day silent-pass reporting model described in feedback_pre_launch_digest_notifications, until this incident's full step log made the distinction visible).
• Contributing factor 3 — the daily cron's silence-on-pass design (per feedback_pre_launch_digest_notifications.md) meant there was no positive signal confirming Steps 3-5 were exercising anything real. A smoke that always reports the same six-line PASS report gives no signal that Step 3 stopped meaning what its name implies the day #4262 shipped.

Detection

• What alerted us: the daily tickets-e2e-smoke cron's failure path (Slack + Postmark alert, auto-filed issue #4429) — but only because an unrelated event (the 2026-08-07 Woodpecker server replace's full manual validation trigger) happened to run it; see What didn't go well.
• How long between cause (2026-07-24 drift introduced) and detection (2026-08-07 07:06 UTC): ~13 days.
• How to detect faster next time: no new monitoring needed — the daily cron already runs unconditionally and would have caught this the very next day (2026-07-25) had it not been for a separate question of whether the daily cron itself was executing reliably in that window. Confirming the daily cron's own execution history (that it fires every day, not just when incidentally triggered by unrelated manual runs) is the actual gap; see action item #3.

Resolution

• What was changed: 1. scripts/ops/tickets_e2e_smoke.py — Step 2 (check_new_ticket_form) now accepts either a server-rendered form (method= present) or the documented mailto-only contact page (mailto:support@raxx.app marker present), returning an is_mailto_only flag. Step 3 (submit_test_ticket) SKIPs cleanly (no POST attempted) when that flag is set, instead of asserting a meaningless POST against a page with no backend. _api_get/_api_patch (Steps 4/5) now send CF-Access-Client-Id/CF-Access-Client-Secret from the CF_ACCESS_CLIENT_ID/CF_ACCESS_CLIENT_SECRET env vars the pipeline already provisions. check_ticket_in_api gained an explicit 403 branch distinguishing a CF Access block from a "ticket not found." 2. scripts/ops/tickets_cleanup_smoke.py — same CF Access header fix applied proactively (weekly cron, same defect class, not yet triggered/observed failing). 3. .woodpecker/tickets-e2e-smoke.yaml — header comment corrected to describe the current expected Step 1/2/6-live, Step 3-5-skip behavior. 4. docs/ops/runbooks/freescout.md — Failure mode G addendum + "Last reviewed" bump documenting this fix and cross-referencing this RCA. 5. scripts/ops/tests/test_tickets_e2e_smoke.py and test_tickets_cleanup_smoke.py — 18 new/updated test cases covering the mailto-only PASS path, the Step 3 SKIP path, the 403-vs-401 distinction, and the CF Access header helper in both scripts (65 total, all passing).
• Validation:
• python3 -m pytest scripts/ops/tests/test_tickets_e2e_smoke.py scripts/ops/tests/test_tickets_cleanup_smoke.py -q — 65 passed.
• python3 scripts/ops/tickets_e2e_smoke.py --dry-run run locally against the live tickets.raxx.app — Steps 1/2/6 PASS; confirms the fixed Step 2 correctly recognizes the current mailto-only page.
• PR #4446 opened against develop; WP ci-pr pipeline green on both commits (#5105 for 9a6a3b1c5, #5108 for 77ff1cc45). Merged 19:48:29 UTC as squash commit 8676595ef.

• Post-merge: WP cron 22 (tickets-e2e-smoke-daily) triggered directly via POST /api/repos/1/cron/22 ("start a cron job now" — scoped to the single tickets-e2e-smoke.yaml workflow only; confirmed via the response body, which listed exactly one workflow, tickets-e2e-smoke, not the ~65-workflow fan-out a bare manual dispatch would trigger). Pipeline #5114, commit 8676595ef (develop tip, includes this fix), completed success in ~40s. e2e-smoke step (id 29445) result: [PASS] HTTP 302 Step 1: portal root not gated by CF Access [PASS] HTTP 200 Step 2: new ticket form loads [SKIP] HTTP --- Step 3: synthetic ticket submitted └─ SKIP — /new-conversation is the documented mailto-only contact page (#4262; freescout.md Failure mode G). Re-enable once epic #651 ships a web-form submit path. [SKIP] HTTP --- Step 4: ticket visible in FreeScout API [SKIP] HTTP --- Step 5: smoke-test tag applied [PASS] HTTP 404 Step 6: invalid path returns proper error (not CF Access) Pipeline step echoed PASS - tickets.raxx.app E2E smoke passed. and exited 0; notify-on-failure and auto-file-issue both correctly skipped (no failure to act on). Confirms both fixes: Step 2 recognizes the live mailto-only page without needing CF Access removed, and Steps 3-5 no longer produce a false PASS or a masked-403 FAIL.

  Note (pre-existing, not introduced by this fix): the smoke script's own plain-text report footer prints a generic RESULT: SKIP — CF Access is still gating tickets.raxx.app... line whenever any step result is skip, regardless of the actual skip reason (_render_text_report's skipped = any(r.status == "skip" ...) check is not reason-specific). This is cosmetically misleading in the human-readable summary — Steps 3-5 are correctly SKIPping for the mailto-only-page reason recorded in each step's own detail text, not because CF Access is on (Step 1 clearly PASSed the CF Access check) — but it does not affect the pipeline's actual pass/fail determination: the YAML wrapper's own grep -q "CF Access still gating" check (an exact-phrase match against the markdown report, not the text-report footer) correctly did not match, so the step echoed PASS and exited 0 as shown above. Not fixed in this PR (out of scope for #4429; left as a minor, non-blocking readability quirk — a future pass could make the footer reason-specific).

Action items

References

• Runbook: docs/ops/runbooks/freescout.md (Failure mode G)
• Issue: #4429 (this incident's tracked ticket, auto-filed)
• PR: #4446 (fix)
• Related PRs: #4335 (issue #4262 — durable mailto page), closed issue #4249 (original 404)
• Related incident: docs/incidents/2026-08-07-woodpecker-server-replace-pip-cache-activation.md (the event that incidentally surfaced this drift)
• Woodpecker API note used during triage: GET /api/repos/{id}/logs/{pipeline_number}/{step_id} is the correct v3 step-log endpoint (pipeline number, not internal pipeline id; step id, not workflow pid) — unrecognized path shapes 200 with SPA HTML rather than erroring, per docs/ops/runbooks/ci-woodpecker.md §v3.x API changes "Unrecognized paths" note.