Auto-Ticketing Pipeline — Staged Rollout

System: Console auto-ticketing flags Owner: operator Introduced: 2026-05-05 (PRs #1155, #1156) Last reviewed: 2026-05-05

Related runbooks: - Pipeline overview: docs/ops/runbooks/auto-ticketing-pipeline-overview.md - Incident response: docs/ops/runbooks/auto-ticketing-runbook.md - Mailbox provisioning: docs/ops/runbooks/freescout-operations-mailbox-provisioning.md - Feature flag ops: docs/ops/feature-flags-runbook.md

Flags controlled by this rollout

The per-customer ticket view (Entry point C, #1154) is not flag-gated. It is available whenever the console is deployed and FREESCOUT_OPERATIONS_MAILBOX_ID is set.

Prerequisites

Complete all prerequisites before flipping either flag. Flipping without them results in tickets failing to create silently (the code degrades gracefully but produces no tickets).

1. FREESCOUT_OPERATIONS_MAILBOX_ID provisioned

The operations mailbox must exist in FreeScout and its ID must be stored in Infisical before the env var can be set on Heroku.

Full procedure: docs/ops/runbooks/freescout-operations-mailbox-provisioning.md

Quick reference: - Retrieve the mailbox ID: GET https://tickets.raxx.app/api/mailboxes with Authorization: Bearer <FREESCOUT_API_KEY> - Vault path: /MooseQuest/freescout/FREESCOUT_OPERATIONS_MAILBOX_ID - Heroku apps: raxx-console-staging and raxx-console-prod

Verify the env var is set before proceeding:

heroku config:get FREESCOUT_OPERATIONS_MAILBOX_ID --app raxx-console-staging
heroku config:get FREESCOUT_OPERATIONS_MAILBOX_ID --app raxx-console-prod

Both must return a non-empty numeric value.

2. FREESCOUT_API_KEY provisioned

The console app needs API access to FreeScout to create tickets.

heroku config:get FREESCOUT_API_KEY --app raxx-console-staging
heroku config:get FREESCOUT_API_KEY --app raxx-console-prod

If missing, retrieve from Infisical and set (silenced):

FS_KEY=$(infisical secrets get FREESCOUT_API_KEY --path /MooseQuest/freescout --env prod --plain)
heroku config:set FREESCOUT_API_KEY="$FS_KEY" --app raxx-console-staging >/dev/null 2>&1
heroku config:set FREESCOUT_API_KEY="$FS_KEY" --app raxx-console-prod >/dev/null 2>&1

3. FreeScout instance healthy

Confirm FreeScout is reachable:

curl -sI https://tickets.raxx.app/ | head -3

Expected: HTTP/2 200. If FreeScout is down, do not proceed — tickets will fail to create and the failure may be mistaken for a flag misconfiguration. See docs/ops/runbooks/freescout.md for FreeScout incident response.

Rollout sequence

The two flags are independent. They can be flipped in either order or at the same time. The staging-first sequence below applies to each flag individually.

Step 1 — Flip on staging

Navigate to https://console.raxx.app/console/flags with the staging env active (purple banner).

For the Investigate button: 1. Locate FLAG_CONSOLE_INVESTIGATE_FROM_STATUS in the flags table. 2. Toggle to on. 3. Confirm the source column changes to db.

For the alerts auto-ticket: 1. Locate FLAG_CONSOLE_ALERTS_AUTO_TICKET in the flags table. 2. Toggle to on. 3. Confirm the source column changes to db.

Step 2 — Verify on staging

Investigate button (Entry point A): 1. Navigate to /console/status on staging (purple banner). 2. Find any surface showing DEGRADED (or manually trigger a test degradation if all surfaces are healthy). 3. Hover the degraded tile and click Investigate. 4. Confirm: the console shows a FreeScout ticket URL; the ticket appears in FreeScout operations mailbox with tag auto:status. 5. Click Investigate again within 60 minutes. Confirm: no duplicate ticket — the same URL is returned.

Alerts auto-ticket (Entry point B): 1. Wait for or trigger a P2+ alert on staging (check Heroku logs for alert_aggregator emit entries). 2. Confirm a ticket appears in the FreeScout operations mailbox with tag auto:alert. 3. Confirm the Slack DM also fired (the two are additive). 4. Check the audit log: SELECT * FROM audit_log WHERE action = 'console.alerts.ticket_filed' ORDER BY created_at DESC LIMIT 5;

Step 3 — Soak on staging (~24h)

Leave both flags on staging for approximately 24 hours. Monitor for: - Ticket flood: repeated console.alerts.ticket_filed entries for the same alert_key in the audit log (see runbook failure mode "Ticket flood"). - Tickets not creating: FLAG_CONSOLE_ALERTS_AUTO_TICKET on but no tickets despite P2+ alerts in logs. - FreeScout API errors in Heroku logs: heroku logs --app raxx-console-staging | grep -i freescout

If any of these appear during staging soak, do not promote to prod. See docs/ops/runbooks/auto-ticketing-runbook.md.

Step 4 — Promote to prod

After staging soak passes, use the mark-promote → promote flow (see docs/ops/feature-flags-runbook.md section 3).

Alternatively, flip directly on prod via the console with the prod env active (red banner): 1. Confirm env banner is red. 2. Toggle FLAG_CONSOLE_INVESTIGATE_FROM_STATUS to on. 3. Toggle FLAG_CONSOLE_ALERTS_AUTO_TICKET to on.

Step 5 — Verify on prod

Repeat the verification steps from Step 2 on the prod env. The degraded tile test may require waiting for a naturally occurring degradation or using a known-degraded surface.

Confirm audit log entries are appearing on prod:

SELECT created_at, action, payload
FROM audit_log
WHERE action IN ('console.status.ticket_filed', 'console.alerts.ticket_filed')
  AND created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC;

Rollback

Rollback is a single flag flip. No data migration is required.

Via console UI (preferred): 1. Navigate to /console/flags. 2. Toggle the flag off. Change takes effect within 30 seconds (cache TTL).

Via Heroku CLI (break-glass, if console is unavailable):

heroku config:set FLAG_CONSOLE_INVESTIGATE_FROM_STATUS=0 --app raxx-console-prod >/dev/null 2>&1
heroku config:set FLAG_CONSOLE_ALERTS_AUTO_TICKET=0 --app raxx-console-prod >/dev/null 2>&1

After a break-glass flip, write a reconciling DB row via the console UI when it recovers, so the audit log reflects the true state. See docs/ops/feature-flags-runbook.md section 6.

Effect of rollback: - Entry point A (Investigate button): the button disappears from the status grid. No effect on tickets already filed. - Entry point B (alerts auto-ticket): the alerts pipeline stops creating tickets. Slack DMs continue to fire at all severities regardless of flag state. - Entry point C (customer ticket view): unaffected by either flag.

Tickets that were created before rollback remain open in FreeScout. The operator must close or resolve them manually if they are no longer relevant.

Flag state reference

Update this table after each rollout step.