Raxx · internal docs

internal · gated ↑ index

Auto-Ticketing Pipeline — Overview

System: Console auto-ticketing (FreeScout operations mailbox) Owner: operator / sre-agent Introduced: 2026-05-05 (PRs #1154, #1155, #1156) Last reviewed: 2026-05-05

Related runbooks: - Rollout sequence: docs/ops/runbooks/auto-ticketing-rollout.md - Incident response: docs/ops/runbooks/auto-ticketing-runbook.md - Mailbox provisioning: docs/ops/runbooks/freescout-operations-mailbox-provisioning.md - FreeScout system: docs/ops/runbooks/freescout.md

Mental model

The auto-ticketing pipeline automatically creates FreeScout tickets in the operations mailbox from three distinct entry points. All three write to the same mailbox but carry different tag taxonomies and idempotency rules. The operator does not need to manually create tickets for degraded surfaces or P2+ alerts — the system files them.

Entry point A — Operator-clicks-Investigate (PR #1155, issue #1147)
  console /console/status grid → degraded tile hover → "Investigate" button click
  → POST /console/status/investigate/<surface_id>
  → FreeScout ticket (tag: auto:status, auto:status:<surface_id>)

Entry point B — System-fires-alert (PR #1156, issue #1148)
  status_poller emits P2+ alert → alert_aggregator emit hook
  → POST to FreeScout API
  → FreeScout ticket (tag: auto:alert, auto:alert:<source>)
  + Slack DM (always fires at P2+, additive)

Entry point C — Operator-views-customer (PR #1154, issue #1049)
  console /console/customers/<id>/tickets
  → renders customer's open + recent FreeScout tickets inline
  → click-through links open the ticket in FreeScout directly
  (read-only view; does not create tickets)

Entry point C is a read-only surface. It does not create tickets — it surfaces tickets that already exist in FreeScout for the customer. The two ticket-creating paths are A and B.

Where tickets land

All auto-created tickets land in the operations mailbox. The mailbox is identified by the env var FREESCOUT_OPERATIONS_MAILBOX_ID on the console Heroku app. Retrieving and provisioning this value is documented in docs/ops/runbooks/freescout-operations-mailbox-provisioning.md.

The operations mailbox is separate from the customer-facing support mailbox (FREESCOUT_SUPPORT_MAILBOX_ID). Tickets auto-filed by the pipeline are operator-internal — they are not visible to customers.

Tag taxonomy

Every auto-created ticket carries at least two tags:

Tag	Applied by	Meaning
`auto:status`	Entry point A (Investigate button)	Ticket was filed from the status grid
`auto:status:<surface_id>`	Entry point A	The specific surface that was degraded (e.g. `auto:status:api-prod`)
`auto:alert`	Entry point B (system alert)	Ticket was filed by the alerts pipeline
`auto:alert:<source>`	Entry point B	The alert source (e.g. `auto:alert:status_poller`)

The sub-tags enable FreeScout saved searches and workflow automations scoped to a single surface or alert source.

Idempotency model

The two ticket-creating paths use different idempotency keys and do not dedup against each other.

Entry point A — Investigate button (PR #1155)

Key: (surface_id, current_status, open-ticket-within-60min)

If an open ticket already exists for the same surface with the same status value and was created within the last 60 minutes, clicking Investigate returns the existing ticket URL instead of creating a new one.
After 60 minutes, a new Investigate click creates a fresh ticket (the surface may have cycled through degraded → ok → degraded again).
The 60-minute window is checked against the FreeScout API. The console holds no persistent state for this.

Entry point B — System alert (PR #1156)

Key: alert_key = <source>:<alert_id[:16]>

alert_id[:16] is the first 16 characters of the alert's unique identifier as emitted by the alert aggregator.
An in-process state cache keyed on alert_key prevents the same alert from firing a ticket on every poll cycle.
The cache is process-local (not persisted across dyno restarts). After a dyno restart, the first occurrence of a recurring alert will create a new ticket. This is intentional — the restart itself is a state change worth knowing about.
The alert pipeline does not check whether an Investigate-button ticket already exists for the same surface. The two paths are additive: a degraded surface under active investigation may receive both an Investigate ticket and an alert ticket. The tags allow operators to correlate them.

When to expect tickets vs Slack DMs

Severity	Slack DM	FreeScout ticket
P3	Yes	No
P2	Yes	Yes (Entry point B)
P1	Yes	Yes (Entry point B)
Operator-initiated	No	Yes (Entry point A)

P3 alerts remain Slack-only. The auto-ticket threshold is P2+. If FLAG_CONSOLE_ALERTS_AUTO_TICKET is off, all severities fall back to Slack-only regardless of level.

Audit trail

Every ticket creation is recorded in the console audit_log table.

Event	Source	Payload fields
`console.status.ticket_filed`	Entry point A	`surface_id`, `status`, `ticket_id`, `ticket_url`, `idempotent` (bool)
`console.alerts.ticket_filed`	Entry point B	`alert_key`, `severity`, `ticket_id`, `ticket_url`

To query for recent ticket-filing events:

SELECT created_at, action, payload
FROM audit_log
WHERE action IN ('console.status.ticket_filed', 'console.alerts.ticket_filed')
ORDER BY created_at DESC
LIMIT 50;

All timestamps in the audit log are UTC.

Example entry (ticket flood diagnostic):

created_at: 2026-05-05 14:32:17 UTC
action:     console.alerts.ticket_filed
payload:    {"alert_key": "status_poller:a3f9b812", "severity": "P2", "ticket_id": 412, "ticket_url": "https://tickets.raxx.app/conversation/412"}

If repeated console.alerts.ticket_filed rows appear with the same alert_key across successive poll cycles, the in-process cache is not holding. See the incident response runbook at docs/ops/runbooks/auto-ticketing-runbook.md.

Auto-generated from docs/ in raxx-app/TradeMasterAPI. Gated behind Cloudflare Access. Re-deployed on every push to main.