FreeScout runbook

System: FreeScout helpdesk — tickets.raxx.app Owner: operator Last incident: 2026-06-12 (#3280 GAP-NEW — customer portal paths behind CF Access gate) Last reviewed: 2026-07-16 (bridge Lambda BFM/UA fix — Failure mode J; E2E pipeline verified) Module IaC: terraform/freescout/modules.tf (PR # — replaces bash install for normal ops)

Infrastructure

CF Access applications

CF Access IdPs in use:

Design intent (updated 2026-06-19):

• /login — CF Access Google SSO gate; auto_redirect_to_identity=true (single IdP — no login-method picker). Policy allows @raxx.app email domain + kris@moosequest.net explicitly. FreeScout's own / -> /login redirect means any authenticated-only path FreeScout redirects to /login is also effectively gated before reaching the CF challenge.
• /new-conversation, /portal, /contact — decision=bypass (include=everyone). Beta testers and future customers reach the ticket submission form without a Google challenge.
• /api/* — Lambda service token (non_identity). Machine-to-machine only. Separate CF Access app; changing the human gate IdP has no effect on this path.
• All other paths with no matching CF Access app — fall through to the FreeScout origin; FreeScout's own auth middleware redirects unauthenticated requests to /login, which then hits the CF gate.
• Human login app and service-token app are separate CF Access applications with separate policy sets. Any IdP change to one does NOT affect the other.

Adding a new operator:

1. Create a FreeScout user under Manage -> Users with the operator's @raxx.app email.
2. No CF Access policy change needed — the @raxx.app email domain is already in the allow policy.
3. The operator signs in at tickets.raxx.app/login -> Google SSO -> FreeScout email match -> session issued.

IaC backfill: These apps are outside Terraform state until #3343 Terraform step lands. The comment block in terraform/freescout/dns.tf is the durable record of all app UUIDs.

Verification assertions (copy-paste):

# /login MUST redirect to CF Access (302) -> then to accounts.google.com (Google SSO)
curl -sI -A "Mozilla/5.0 (compatible; raxx-sre-probe/1.0)" --max-redirs 0 https://tickets.raxx.app/login \
  | grep -E "^HTTP|^[Ll]ocation:"
# Expected: HTTP/2 302 + location: https://moosequest.cloudflareaccess.com/...
# If 200: the Google SSO gate is missing — re-check app fa5939cd via CF API (see Failure mode D below)
# Note: if you follow the redirect chain, the next hop will be accounts.google.com (Google OAuth), NOT an OTP form

# /new-conversation must NOT redirect to CF Access
curl -sI -A "Mozilla/5.0 (compatible; raxx-sre-probe/1.0)" https://tickets.raxx.app/new-conversation \
  | grep -E "^HTTP|^[Ll]ocation:"
# Expected: HTTP/2 200 (FreeScout form) or HTTP/2 404 (contact-form module not installed)
# If 302 to cloudflareaccess.com: bypass app 753f1573 is missing — re-create it

# /portal must NOT redirect to CF Access
curl -sI -A "Mozilla/5.0 (compatible; raxx-sre-probe/1.0)" https://tickets.raxx.app/portal \
  | grep -E "^HTTP|^[Ll]ocation:"
# Expected: 200 or 404 — NOT 302 to cloudflareaccess.com

History: - 2026-05-01 (#716): Only CF Access app (scoped to /admin, non-existent route) deleted. /login unprotected. - 2026-06-05: qa-agent detected gap via #3280. Incident #3283 filed. - 2026-06-06: Root-domain gate (tickets.raxx.app) restored by sre-agent. RCA docs/ops/incidents/2026-06-06-freescout-login-unprotected.md. - 2026-06-12 (#3280 GAP-NEW): Root-domain app narrowed to /login path only; three customer-portal bypass apps created so beta testers can submit tickets without OTP challenge. - 2026-06-19 (#3343): Human login app fa5939cd migrated from OTP IdP to Google Workspace IdP (18c69c18). Policy updated: email_domain=raxx.app + email=kris@moosequest.net. auto_redirect_to_identity enabled. Service-token app ca6fd315 untouched; confirmed 200 via service token post-change.

tickets-agent-rw service token (#2329)

Provisioned: 2026-05-18 UTC
Token name: tickets-agent-rw
Token ID: 8e723042-182b-4822-a079-fad4c1ada810
CF Access app: FreeScout API — Lambda service token (ca6fd315-31c0-4fb5-91f4-ebfea8d49a98)
Policy ID: cab8010a-d63b-4519-b67a-03b9afef6409
Policy name: tickets-agent-rw-policy
Policy decision: non_identity (required for service tokens — allow would 302 due to missing identity context)
Expires: 2126-04-24 (effectively forever)

Vault paths:

Usage pattern (Python):

import os, urllib.request

client_id = vault.get_secret("CF_ACCESS_SVC_TICKETS_CLIENT_ID")
client_secret = vault.get_secret("CF_ACCESS_SVC_TICKETS_CLIENT_SECRET")
fs_api_key = vault.get_secret("FREESCOUT_API_KEY", path="/MooseQuest/freescout/")

req = urllib.request.Request(
    "https://tickets.raxx.app/api/conversations",
    headers={
        "CF-Access-Client-Id": client_id,
        "CF-Access-Client-Secret": client_secret,
        "X-FreeScout-API-Key": fs_api_key,
        "User-Agent": "raxx-agent/1.0",
    }
)

Critical notes:

• The policy decision MUST be non_identity. Service tokens carry no IdP identity; an allow decision causes a 302 redirect to the CF Access login page (false-positive 200 if the client follows redirects without checking final URL).
• This policy is additive — the existing Lambda bridge policy (b1bf50c4-ab02-470e-ba7d-ccc06693d227) is unmodified.
• HTTP 401 from the FreeScout /api/* endpoint is application-layer auth (missing X-FreeScout-API-Key), NOT a CF Access failure.
• HTTP 302 to *.cloudflareaccess.com is a CF Access failure.

Verification (copy-paste):

CLIENT_ID=$(infisical secrets get CF_ACCESS_SVC_TICKETS_CLIENT_ID \
  --path /MooseQuest/cloudflare --env prod --plain)
CLIENT_SECRET=$(infisical secrets get CF_ACCESS_SVC_TICKETS_CLIENT_SECRET \
  --path /MooseQuest/cloudflare --env prod --plain)
FS_KEY=$(infisical secrets get FREESCOUT_API_KEY \
  --path /MooseQuest/freescout --env prod --plain)

curl -sI \
  -H "CF-Access-Client-Id: ${CLIENT_ID}" \
  -H "CF-Access-Client-Secret: ${CLIENT_SECRET}" \
  -H "X-FreeScout-API-Key: ${FS_KEY}" \
  "https://tickets.raxx.app/api/conversations?page=1&pageSize=1"
# Expect: HTTP 200 with JSON body
# If HTTP 302 to cloudflareaccess.com: CF Access policy missing or wrong decision
# If HTTP 401 {"message":"Not Authorized"}: FreeScout API key missing/wrong

Rotation:

If the service token needs to be rotated (compromise, annual audit): 1. Mint a new token via CF API (see docs/ops/runbooks/cf-access-service-token-provisioning.md). 2. Write the new client_id and client_secret to vault, replacing the values above. 3. Update the CF Access policy include array with the new token_id. 4. Verify with the snippet above. 5. Delete the old token via DELETE /client/v4/accounts/{ACCOUNT_ID}/access/service_tokens/{OLD_TOKEN_ID}. 6. Update the Token ID in this runbook.

Blueprint choice rationale

lamp_8_bitnami was the original blueprint. As of 2026, it ships PHP 8.5.3. FreeScout's illuminate/support v5.5.40 declares ^7.1.3 (PHP 7.x only), so Composer refused to install on PHP 8.5.

lamp_ls_1_0 is the AWS-native standard LAMP blueprint (non-Bitnami). It ships PHP 8.2, which is the minimum PHP version FreeScout officially supports ("PHP 8.2 and newer are OK" per FreeScout installation guide). The bootstrap installs with --ignore-platform-req=php to bypass the Composer platform check for the PHP version only — not all platform requirements. Extension requirements (mbstring, xml, gd, etc.) are still enforced.

FreeScout does NOT support Laravel 9+ and has no PHP 8.5-compatible release as of April 2026.

Paths

How to tell it's broken

• https://tickets.raxx.app/ returns a non-FreeScout page (default Apache, 404, 502, 503)
• https://tickets.raxx.app/ returns 200 but no FreeScout login form
• Cloudflare Access gate does not appear (CF Access bypassed or CF not proxying)
• cloud-init status returns error on the instance
• /var/log/freescout-bootstrap.log contains FATAL or is missing

How to diagnose (in order)

1. curl -sI https://tickets.raxx.app/ — check status code and headers. FreeScout returns 200 with Content-Type: text/html; charset=UTF-8.
2. SSH to instance: ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null admin@54.146.13.200
3. Check bootstrap log: sudo cat /var/log/freescout-bootstrap.log
4. Check cloud-init status: cloud-init status
5. Check cloud-init output: sudo tail -50 /var/log/cloud-init-output.log
6. Check services: systemctl status apache2; systemctl status mariadb
7. Check if FreeScout is cloned: ls /var/www/html/freescout/
8. Check Apache vhost: ls /etc/apache2/sites-enabled/
9. Check Apache error log: sudo tail -50 /var/log/apache2/freescout-error.log

Known failure modes

Failure mode A: cloud-init error — bootstrap script failed mid-run

Symptom: - /var/log/freescout-bootstrap.log contains FATAL or ends before "completed at" - cloud-init status returns error - FreeScout directory does not exist or is incomplete

Cause: Bootstrap script exited before completing. Common sub-causes: 1. apt lock held too long (if loop exhausted before lock cleared) 2. Composer checksum mismatch (network issue fetching installer.sig) 3. MariaDB not accepting connections within 60s window 4. Composer install failed (network, disk, or dependency resolution) 5. Artisan migrate failed (DB connection refused or wrong credentials)

Diagnose:

ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu@54.146.13.200
sudo cat /var/log/freescout-bootstrap.log

Look for the last successful step, then the error. Each step is prefixed [freescout-bootstrap].

Manual recovery — run as root on the instance:

If the bootstrap failed partway, source the variables from user-data and re-run the failed step forward:

# Source non-sensitive variables (passwords are already on disk)
eval "$(sudo grep -E '^(DB_NAME|DB_USER|DB_PASSWORD|ADMIN_EMAIL|ADMIN_PASSWORD|APP_KEY_SEED|SMTP_HOST|SMTP_PORT|SMTP_USERNAME|SMTP_PASSWORD|DOMAIN|APP_NAME|MAILBOX_EMAIL)=' \
  /var/lib/cloud/instance/user-data.txt | head -20)"

# If MariaDB setup incomplete — connect as root (root uses unix socket before password is set)
sudo mysql -u root <<SQL
CREATE DATABASE IF NOT EXISTS \`freescout\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER IF NOT EXISTS 'freescout'@'localhost' IDENTIFIED BY '$DB_PASSWORD';
GRANT ALL PRIVILEGES ON \`freescout\`.* TO 'freescout'@'localhost';
FLUSH PRIVILEGES;
SQL

# If FreeScout not cloned:
# IMPORTANT: use freescout-help-desk/freescout (with the dash), NOT freescout-helpdesk/freescout
# The dash-less name is a stale mirror that does not receive security patches.
sudo git clone --depth 1 --branch master \
  https://github.com/freescout-help-desk/freescout.git \
  /var/www/html/freescout

# If Composer install not done:
# --ignore-platform-req=php: bypasses illuminate/support ^7.1.3 platform check only.
# FreeScout is PHP 8.2 compatible per official docs. Extension requirements still enforced.
cd /var/www/html/freescout
sudo -u www-data /usr/local/bin/composer install \
  --no-dev --optimize-autoloader --no-interaction \
  --ignore-platform-req=php

# If .env not written:
APP_KEY="base64:$(echo -n "${APP_KEY_SEED}$(date -u +%s)" | sha256sum | xxd -r -p | base64 | tr -d '\n')"
sudo tee /var/www/html/freescout/.env <<ENV
APP_NAME="$APP_NAME"
APP_ENV=production
APP_KEY=$APP_KEY
APP_DEBUG=false
APP_URL=https://$DOMAIN
DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=$DB_NAME
DB_USERNAME=$DB_USER
DB_PASSWORD=$DB_PASSWORD
MAIL_DRIVER=smtp
MAIL_HOST=$SMTP_HOST
MAIL_PORT=$SMTP_PORT
MAIL_USERNAME=$SMTP_USERNAME
MAIL_PASSWORD=$SMTP_PASSWORD
MAIL_ENCRYPTION=tls
MAIL_FROM_ADDRESS=$MAILBOX_EMAIL
MAIL_FROM_NAME="$APP_NAME"
FREESCOUT_ADMIN_EMAIL=$ADMIN_EMAIL
FREESCOUT_ADMIN_PASSWORD=$ADMIN_PASSWORD
TRUSTED_PROXIES=*
ENV
sudo chmod 640 /var/www/html/freescout/.env
sudo chown -R www-data:www-data /var/www/html/freescout
sudo chmod -R 775 /var/www/html/freescout/storage /var/www/html/freescout/bootstrap/cache

# If migrations not run:
cd /var/www/html/freescout
sudo -u www-data /usr/bin/php artisan key:generate --force
# NOTE: "freescout:migrate" does not exist; use "migrate --force"
sudo -u www-data /usr/bin/php artisan migrate --force
# NOTE: freescout:create-user flags in current version are --firstName, --lastName
# (NOT --name, --first_name)
sudo -u www-data /usr/bin/php artisan freescout:create-user \
  --firstName="Kristerpher" --email="$ADMIN_EMAIL" --password="$ADMIN_PASSWORD" --role=admin 2>/dev/null || \
  echo "create-user failed — seed manually after first login"
sudo -u www-data /usr/bin/php artisan storage:link 2>/dev/null || true
sudo -u www-data /usr/bin/php artisan config:cache
# NOTE: route:cache fails if any route has a Closure (polycast plugin causes this)
# Skip it — FreeScout works without route cache
sudo -u www-data /usr/bin/php artisan route:cache 2>/dev/null || echo "route:cache skipped (expected)"

# If Apache vhost not configured:
# NOTE: lamp_ls_1_0 enables SSL + snakeoil cert by default. Cloudflare connects
# to origin on port 443. BOTH vhosts (80 + 443) are required.
sudo a2enmod rewrite headers remoteip ssl
sudo a2dissite 000-default.conf default-ssl.conf 2>/dev/null || true

sudo tee /etc/apache2/sites-available/freescout.conf <<VHOST
<VirtualHost *:80>
  ServerName $DOMAIN
  DocumentRoot "/var/www/html/freescout/public"
  RewriteEngine On
  RewriteCond %{HTTP:X-Forwarded-Proto} !https
  RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
  <Directory "/var/www/html/freescout/public">
    Options -Indexes +FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>
  RemoteIPHeader CF-Connecting-IP
  ErrorLog "/var/log/apache2/freescout-error.log"
  CustomLog "/var/log/apache2/freescout-access.log" combined
</VirtualHost>
VHOST

sudo tee /etc/apache2/sites-available/freescout-ssl.conf <<SSLVHOST
<VirtualHost *:443>
  ServerName $DOMAIN
  DocumentRoot "/var/www/html/freescout/public"
  SSLEngine on
  SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
  SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
  <Directory "/var/www/html/freescout/public">
    Options -Indexes +FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>
  RemoteIPHeader CF-Connecting-IP
  ErrorLog "/var/log/apache2/freescout-error.log"
  CustomLog "/var/log/apache2/freescout-access.log" combined
</VirtualHost>
SSLVHOST

sudo a2ensite freescout.conf freescout-ssl.conf
sudo systemctl restart apache2

Verification: curl -sI https://tickets.raxx.app/ should return 200 with PHP headers and FreeScout login HTML.

Failure mode B: Apache vhost not applied — serving default or 404

Symptom: curl returns 200 but body is the default Apache page, or 404 for all paths

Cause: Apache vhost not written or enabled, or Apache not restarted

Fix:

# Check enabled sites
ls /etc/apache2/sites-enabled/
# If freescout.conf not present:
sudo a2ensite freescout.conf
sudo systemctl restart apache2
# Verify rewrite module is enabled:
apache2ctl -M | grep rewrite
# If missing: sudo a2enmod rewrite && sudo systemctl restart apache2

Verification: curl -sI https://tickets.raxx.app/ returns Laravel/FreeScout headers

Failure mode C: MariaDB not accepting connections

Symptom: Laravel .env connects but migrations fail with "Connection refused" or "Access denied"

Cause: MariaDB stopped or user not created

Fix:

# Check MariaDB status
systemctl status mariadb
# If stopped:
sudo systemctl start mariadb
# Check DB user:
sudo mysql -u root -p"$(sudo cat /root/.mariadb_root_pass)" \
  -e "SELECT User,Host FROM mysql.user WHERE User='freescout';"
# If missing, re-run DB creation commands from Failure mode A above

Failure mode D: CF Access gate on /login not appearing (operator reaches FreeScout without Google SSO)

Symptom: Anonymous GET https://tickets.raxx.app/login returns HTTP 200 (FreeScout form) instead of HTTP 302 to moosequest.cloudflareaccess.com. Operator can reach FreeScout login without a Google SSO challenge.

Note: As of 2026-06-19 (#3343), this app uses Google Workspace SSO (IdP 18c69c18), not OTP. The fix below reflects the post-migration state.

Cause: CF Access application fa5939cd (path tickets.raxx.app/login) has been deleted, or its domain was widened / changed, or its policy was removed.

Diagnose:

# Requires CF_TOKEN from vault (CLOUDFLARE_ACCESS_MGMT_TOKEN at /MooseQuest/cloudflare)
CF_TOKEN=$(infisical secrets get CLOUDFLARE_ACCESS_MGMT_TOKEN --path /MooseQuest/cloudflare/ --plain)
curl -sS -H "Authorization: Bearer $CF_TOKEN" \
  -A "raxx-sre-probe/1.0" \
  "https://api.cloudflare.com/client/v4/accounts/22b5c35090724fbf05db6d4f501ac821/access/apps/fa5939cd-ab76-40da-9fae-25bd40278c54" \
  | python3 -c "
import sys,json; r=json.load(sys.stdin).get('result',{})
print('domain:', r.get('domain','NOT FOUND'))
print('name:', r.get('name','?'))
print('allowed_idps:', r.get('allowed_idps'))
print('auto_redirect:', r.get('auto_redirect_to_identity'))
"
# Expected: domain: tickets.raxx.app/login, allowed_idps: ['18c69c18-14db-4f12-94f6-6c69d27cb04b']
# If 404 or domain is tickets.raxx.app (no /login path): re-create the app (see Fix below)

Fix: If the app is missing, re-create it via CF API with the Google IdP. Do NOT re-create as a root-domain app — that would gate /new-conversation and block beta testers. Do NOT use the OTP IdP (1035b261) — it was retired 2026-06-19.

curl -sS -X POST \
  -H "Authorization: Bearer $CF_TOKEN" \
  -H "Content-Type: application/json" \
  -A "raxx-sre-probe/1.0" \
  "https://api.cloudflare.com/client/v4/accounts/22b5c35090724fbf05db6d4f501ac821/access/apps" \
  -d '{
    "name": "FreeScout login — tickets.raxx.app/login (Google SSO)",
    "domain": "tickets.raxx.app/login",
    "type": "self_hosted",
    "session_duration": "8h",
    "app_launcher_visible": false,
    "allowed_idps": ["18c69c18-14db-4f12-94f6-6c69d27cb04b"],
    "auto_redirect_to_identity": true
  }'
# Capture the new app UUID from the response. Then add the Google SSO policy:
# POST /access/apps/<new-uuid>/policies with:
# {
#   "name": "Operator access — Google SSO (@raxx.app + kris@moosequest.net)",
#   "decision": "allow", "precedence": 1,
#   "include": [
#     {"email_domain": {"domain": "raxx.app"}},
#     {"email": {"email": "kris@moosequest.net"}}
#   ]
# }
# Update terraform/freescout/dns.tf and this runbook with the new UUID.

Verification:

curl -sI -A "Mozilla/5.0 (compatible; raxx-sre-probe/1.0)" --max-redirs 0 https://tickets.raxx.app/login \
  | grep -E "^HTTP|^[Ll]ocation:"
# Expected: HTTP/2 302 + location: https://moosequest.cloudflareaccess.com/...
# Following that redirect should auto-redirect to accounts.google.com (Google OAuth)

Failure mode E: Composer fails with PHP version error

Symptom: Bootstrap log shows illuminate/support ... php ^7.1.3 -> your php version (X.X.X) does not satisfy that requirement

Cause: PHP version on the instance is outside Laravel 5.5's platform requirement. On lamp_ls_1_0 (PHP 8.2), this should not occur because the bootstrap uses --ignore-platform-req=php. If it does occur, the flag may have been lost in a template edit.

Fix:

cd /var/www/html/freescout
sudo -u www-data /usr/local/bin/composer install \
  --no-dev --optimize-autoloader --no-interaction \
  --ignore-platform-req=php

Note: Do NOT use --ignore-platform-reqs (plural, no =php). That flag ignores all platform requirements including PHP extensions. Only the PHP version check should be bypassed.

Failure mode F: Queue worker silently drops all outbound mail — broken crontab (fixed in #1085)

Symptom: - FreeScout Settings > Email test sends show no delivery errors in the UI, but email never arrives - sudo tail -20 /var/log/freescout-queue.log is empty or missing entirely - sudo mysql -u root freescout -e "SELECT COUNT(*) FROM jobs;" returns a growing backlog - crontab -u www-data -l shows queue:work --stop-when-empty or only default queue

Root cause (four bugs, all introduced from day 0): 1. /var/log/freescout-queue.log was not created before crontab install — www-data cannot write to a root-owned path; every queue:work invocation exits with Permission denied silently. 2. --stop-when-empty does not exist in this artisan version — the worker process exited immediately on every invocation with an "unknown option" error. 3. --queue=default only — FreeScout dispatches mail jobs to the emails queue; the default queue was never consumed. 4. No schedule:run entry — Laravel scheduler (maintenance tasks, notification cleanup) never fired.

Diagnosis:

# Check crontab
sudo crontab -u www-data -l
# Check log file ownership
ls -la /var/log/freescout-queue.log 2>/dev/null || echo "log file missing"
# Check jobs backlog
sudo mysql -u root freescout -e "SELECT COUNT(*) FROM jobs;"

Fix (applies to instances bootstrapped before PR #1085 was merged):

# 1. Create log with correct ownership
sudo touch /var/log/freescout-queue.log
sudo chown www-data:www-data /var/log/freescout-queue.log
sudo chmod 640 /var/log/freescout-queue.log

# 2. Replace broken crontab
sudo crontab -u www-data - <<'CRON'
* * * * * /usr/bin/php /var/www/html/freescout/artisan queue:work --queue=emails,default --once --timeout=30 >> /var/log/freescout-queue.log 2>&1
* * * * * /usr/bin/php /var/www/html/freescout/artisan schedule:run >> /var/log/freescout-queue.log 2>&1
CRON

# 3. Verify
sudo crontab -u www-data -l

Verification (wait ~2 minutes after applying fix):

sudo tail -20 /var/log/freescout-queue.log
# Should show queue:work and schedule:run lines, not Permission denied
sudo mysql -u root freescout -e "SELECT COUNT(*) FROM jobs;"
# Should be 0 or decreasing under normal operation

Re-run the FreeScout Settings > Email test. Email should arrive within 2 minutes.

Note: New Lightsail instances provisioned from Terraform after PR #1085 is merged receive the corrected crontab automatically and do not need this manual fix.

Emergency stop

sudo systemctl stop apache2
# MariaDB: leave running unless disk is full

Escalation

Wake the operator when: - FreeScout database has data loss or corruption - SSL certificate is expired and CF fails to renew - Lightsail instance is unresponsive (SSH down, Lightsail console shows stopped) - MariaDB data partition is full (40 GB SSD)

Admin account security

TOTP two-factor authentication (#718)

FreeScout has built-in TOTP support. The admin account (kris@moosequest.net) must have TOTP enabled as a defense-in-depth layer behind Cloudflare Access. This is an operator-only action — it cannot be automated.

This is an operator action. Kristerpher must complete this personally.

Enrollment steps:

1. Log in to https://tickets.raxx.app (passes through CF Access first).
2. In the top-right corner, click your avatar / name, then Your Profile.
3. In the left sidebar, click Security (or look for Two-Factor Authentication directly on the profile page).
4. Under Two-Factor Authentication, click Enable.
5. A QR code is displayed. Scan it with an authenticator app (use the same app as other Raxx TOTP codes — Google Authenticator, Authy, 1Password, etc.).
6. Enter the 6-digit confirmation code from the app to confirm enrollment.
7. FreeScout will display one-time recovery codes. Copy all recovery codes before closing this page. They are shown only once.
8. Store recovery codes in Infisical at path /MooseQuest/freescout/FREESCOUT_TOTP_RECOVERY_CODES (as a multi-line secret value, one code per line).

Verification:

Log out of FreeScout, then log back in through CF Access. After CF Access passes, FreeScout should show a TOTP prompt before granting access to the inbox.

Recovery procedure (if authenticator app is lost):

1. Retrieve recovery codes from Infisical at /MooseQuest/freescout/FREESCOUT_TOTP_RECOVERY_CODES.
2. On the FreeScout login screen, after entering your password, click Lost your device? or Use recovery code.
3. Enter one recovery code (each code is single-use).
4. Once logged in, go to Profile > Security > Two-Factor Authentication and re-enroll with a new authenticator app.
5. Store the new recovery codes in Infisical, replacing the previous set.

If TOTP is not available on the deployed version:

Check FreeScout version: https://tickets.raxx.app/dashboard (visible in footer or Settings > General). FreeScout's built-in 2FA was introduced in version 1.8.x. If unavailable, document the gap in a comment on #718 and rely solely on CF Access WebAuthn from #695 until a FreeScout update can be applied.

Important: Do not close issue #718 until TOTP is active AND recovery codes are stored in Infisical. Documentation alone is not sufficient for this card.

Vault path reference for admin credentials

Google OAuth SSO (#961)

FreeScout operators sign in with their @raxx.app Google Workspace account via the Tagras oauthlogin module (installed 2026-05-12 UTC). Local FreeScout passwords are retired after successful configuration.

Full setup and operations procedure: docs/ops/runbooks/freescout-google-oauth.md

Quick-reference vault paths:

Operator action required — see the Google OAuth runbook for the Cloud Console OAuth app creation steps and FreeScout module configuration. Neither step can be automated (Cloud Console requires Workspace admin login; module config is UI-only).

Branding and company settings (#720, #1895)

Current state (2026-06-19, #1895 complete): - .env APP_NAME="Raxx Support" — set - .env MAIL_FROM_NAME="Raxx Support" — set - DB options.company_name = "Raxx Support" — set (this is what FreeScout actually uses for display; config/app.php hardcodes "FreeScout" and is NOT used for the company name) - DB options.email_branding = "" — cleared (was 1; removing this suppresses the "powered by FreeScout" footer in outbound emails) - Mailbox name verified via API: "Raxx Support" (mailbox ID 1, support@raxx.app)

FreeScout's login page (tickets.raxx.app/login) is the operator-facing gateway. It is protected by Cloudflare Access and is not customer-reachable under normal operating conditions. Nevertheless, a minimal Raxx wordmark should be set so the login page does not display the default FreeScout logo.

Access decision: tickets.raxx.app/login is operator-only (behind CF Access — kris@moosequest.net allowlist only). Customers interact exclusively with support.raxx.app (portal, #651). Full branding is not launch-blocking but is a pre-launch should-do.

How to update company name and logo (operator action, ~10 min)

This is an operator action. Kristerpher must complete this in the FreeScout admin UI.

1. Log in to https://tickets.raxx.app.
2. Navigate to Settings (gear icon, top navigation bar).
3. Click General in the left sidebar.
4. In the Company Name field, enter: Raxx Support
5. In the Company Logo field: upload docs/design/brand/logos/raxx-wordmark-c.svg (the canonical Confidence Engine wordmark). If the field requires a raster image, export the SVG at 2x (300 px wide, white background if needed) and upload the PNG.
6. In the Favicon field (if present): upload console/app/static/favicon-32.png (32x32).
7. Click Save.

Verification:

Navigate to https://tickets.raxx.app/login (in an incognito window after passing CF Access). The FreeScout default logo should be replaced by the Raxx wordmark. The page title should read "Raxx Support" rather than "FreeScout".

Canonical brand assets:

Note on logo hosting for email: Do NOT use a repo-local path as the logo URL in email templates. Cloudflare Access protects tickets.raxx.app — any image URL that points to a CF-Access-gated resource will fail to render in external email clients. The branded email templates in freescout/templates/ inline the logo as a base64 data URI to avoid this. See docs/ops/runbooks/freescout-templates.md for the full email template deployment procedure (#712).

Outbound email template branding (#712)

Raxx-branded HTML + plaintext email templates for all outbound FreeScout emails live at:

freescout/templates/
  autoreply.html         — auto-reply sent when a customer creates a new ticket
  autoreply.txt          — plaintext fallback for auto-reply
  operator-reply.html    — sent when an operator replies to a customer
  operator-reply.txt     — plaintext fallback for operator reply
  notification.html      — ticket status change and system notifications to customers
  notification.txt       — plaintext fallback for notifications

Full deployment procedure, variable reference, durability notes, and revert steps: docs/ops/runbooks/freescout-templates.md

Template file locations on the Lightsail instance (verify on your FreeScout version):

Confirmed paths for FreeScout 1.8.218 on raxx-tickets (2026-06-19): The operator-reply template is reply_fancy.blade.php (HTML) and reply_fancy_text.blade.php (plaintext). Older docs and some FreeScout guides reference reply.blade.php — that path does NOT exist in 1.8.218. Use reply_fancy*.blade.php exclusively.

Operator-reply template override — persistence across FreeScout updates (#657)

Repo source files (canonical):

Instance target paths:

Durability: Core file edits at /var/www/html/freescout/resources/views/ are overwritten by git pull on the instance. After any FreeScout core update, re-apply the branded templates using:

# From the instance (SSH as admin):
sudo cp /tmp/reply_fancy.blade.php \
  /var/www/html/freescout/resources/views/emails/customer/reply_fancy.blade.php
sudo cp /tmp/reply_fancy_text.blade.php \
  /var/www/html/freescout/resources/views/emails/customer/reply_fancy_text.blade.php
sudo chown www-data:www-data \
  /var/www/html/freescout/resources/views/emails/customer/reply_fancy.blade.php \
  /var/www/html/freescout/resources/views/emails/customer/reply_fancy_text.blade.php
cd /var/www/html/freescout
sudo -u www-data /usr/bin/php artisan view:clear
sudo -u www-data /usr/bin/php artisan config:cache

Alternatively, use the Customization module's template-override path if the module is active and its override directory is confirmed. See docs/ops/runbooks/freescout-templates.md §"Durability" for the override-path evaluation procedure and the Customization module status.

Template variables (Blade syntax) in reply_fancy.blade.php:

CTA deep-link pattern (must resolve to the portal, not tickets.raxx.app):

https://support.raxx.app/tickets/{{ $conversation->id }}

After any FreeScout core update (git pull on the instance), re-apply the templates. See the templates runbook for the post-update checklist.

Paid module install and management

FreeScout's 10 paid modules (API & Webhooks, Reports, OAuth, Custom Fields, Tags, Workflows, Customization, Saved Replies, Slack, Custom Folders) are managed declaratively via terraform/freescout/modules.tf.

Normal path — terraform apply

# From repo root (requires infisical CLI + AWS credentials in environment):
make freescout-install
# Review the plan output, then:
cd terraform/freescout && terraform apply tfplan

Expected timing: 5–10 minutes for all 10 modules.

Terraform takes a Lightsail snapshot before and after the install batch. After apply completes, verify all 10 modules show status Active under https://tickets.raxx.app/modules.

Re-installing a single module (e.g., after license key rotation)

1. Update the vault secret at /MooseQuest/freescout/<NAME>_LICENSE_KEY.
2. Re-export just that TF_VAR_license_* variable.
3. Run terraform plan — only that module's null_resource shows as "must replace".
4. Run terraform apply.

Forcing a full reinstall of all modules

Bump module_version_pin in terraform/freescout/terraform.tfvars:

module_version_pin = "YYYY-MM-DD"  # new date

Next apply re-fires all 10 install+activate commands.

Break-glass: bash script fallback

If Terraform is unavailable (e.g., running directly on the instance without a local workstation), use scripts/ops/install-freescout-modules.sh. This script reads all license keys from Infisical at runtime and runs the same artisan commands. It is the operational reference that modules.tf was derived from. Do NOT delete it.

ssh -i ~/.ssh/lightsail_us_east_1.pem -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null admin@54.146.13.200
curl -fsSL https://raw.githubusercontent.com/raxx-app/TradeMasterAPI/main/scripts/ops/install-freescout-modules.sh \
  -o /tmp/install.sh
chmod +x /tmp/install.sh
sudo \
  INFISICAL_HOST="https://vault.raxx.app" \
  INFISICAL_PROJECT_ID="29b77751-f761-4afa-b3fa-2c842988f95c" \
  INFISICAL_CLIENT_ID="<from break-glass>" \
  INFISICAL_CLIENT_SECRET="<from break-glass>" \
  /tmp/install.sh

After using the break-glass script, run terraform plan to confirm state matches. If Terraform shows drift, either import the state or accept that the next apply will re-run install+activate (idempotent at the FreeScout layer).

Failure mode G: E2E smoke fails at Step 2 — /new-conversation returns 404

Symptom: tickets-e2e-smoke.yml Step 2 FAIL: HTTP 404 from https://tickets.raxx.app/new-conversation

Cause: FreeScout 1.8.x has no built-in public /new-conversation route at the root domain. All conversation routes require the auth middleware (operator login). The E2E smoke was written expecting the customer portal surface to be live — either the support.raxx.app proxy (epic #651) or a FreeScout customer-portal module (not installed). Until epic #651 ships, Steps 2-3 of the smoke will fail.

Step 1 (CF Access detection) and Step 6 (invalid-path 404 check) PASS correctly.

Status: Known pre-launch blocker. The smoke correctly identifies that the customer-facing form submission path is not yet configured. This is not a regression — it is a capability gap.

Resolution path: Ship epic #651 (support.raxx.app proxy + FreeScout API bridge), then re-run gh workflow run tickets-e2e-smoke.yml.

Temporary posture: The smoke runs daily on schedule and will automatically start passing once the /new-conversation form endpoint exists. No action needed until #651 is ready.

Failure mode J: Bridge Lambda gets HTTP 403 from FreeScout — Cloudflare Bot Fight Mode blocks Python-urllib default User-Agent

Symptom: - E2E synthetic test publishes to SNS (HTTP 200 from APIGW) but no conversation appears in FreeScout - SQS FIFO queue is empty (messages were consumed by bridge Lambda) - DLQ is empty (Lambda did not put messages there) - CloudWatch logs for raxx-email-inbound-bridge show: {"event_type": "freescout_client_error", "http_status": 403, "outcome": "poison_deleted", ...} - Direct probe to /api/mailboxes with explicit User-Agent returns HTTP 200 (auth works) - Direct probe to /api/conversations with Python-urllib default UA (Python-urllib/3.x) returns HTTP 403 with CF challenge HTML

Root cause: Cloudflare Bot Fight Mode (BFM) runs before the CF Access application layer. BFM matches the Python-urllib/3.x default User-Agent as an automated bot and returns HTTP 403 (CF error 1010 challenge page) before the CF Access service token is evaluated. The bridge Lambda did not set an explicit User-Agent header, so every POST to FreeScout was blocked by BFM regardless of valid CF Access and FreeScout API key headers.

The 4xx (403) outcome causes the Lambda to classify the SQS record as a poison message and delete it. No DLQ entry is written. Emails are silently dropped.

Detected: 2026-07-16 — sre-agent E2E test after prior agent's 500-fix

Fix (already applied 2026-07-16): Added "User-Agent": "raxx-email-inbound-bridge/1.0" to _call_freescout in lambdas/email-inbound-freescout-bridge/handler.py and redeployed:

cd /tmp
cp /path/to/repo/lambdas/email-inbound-freescout-bridge/handler.py ./
zip raxx-email-inbound-bridge.zip handler.py
aws lambda update-function-code \
  --function-name raxx-email-inbound-bridge \
  --zip-file fileb://raxx-email-inbound-bridge.zip \
  --region us-east-1

Verification:

# After deploying the fix, re-run synthetic E2E test (see §End-to-end test).
# CloudWatch should show:
# {"event_type": "freescout_response_content_type", "content_type": "application/json", "http_status": 201}
# {"event_type": "conversation_created", ..., "http_status": 201, "outcome": "success"}

Architecture note: A WAF skip rule keyed on CF-Access-Client-Id header is the proper defense-in-depth fix. The skip rule would bypass BFM for any request that carries a valid service token header, preventing this class of issue from recurring if the UA is ever lost. Tracked in GitHub type:reliability issue. Until that issue ships, the User-Agent header is the primary safeguard.

Post-incident actions for any new failure mode

1. Update this runbook with the new failure mode
2. Add a terraform/README.md lessons-learned entry
3. File a type:reliability GitHub issue

Post-apply bootstrap smoke test (#526)

Added: 2026-05-12 UTC RCA origin: docs/ops/incidents/2026-04-29-freescout-cloud-init-fatal.md action item #3

Why this exists

terraform apply returns success as soon as the Lightsail instance resource is created. It does NOT wait for cloud-init to finish. The 2026-04-29 incident saw 6 consecutive applies succeed from Terraform's perspective while cloud-init failed with FATAL every time. This smoke test catches that gap.

What it checks (in order)

1. SSH connectivity to the instance (30s timeout per attempt; 60s total)
2. cloud-init status polls until status is no longer running (20-minute max)
3. cloud-init final status is NOT error or degraded
4. /var/log/freescout-bootstrap.log exists and contains NO FATAL
5. /var/log/freescout-bootstrap.log contains the completion marker: [freescout-bootstrap] completed at
6. apache2 is active (systemctl)
7. mariadb is active (systemctl)

How to run it

Via CI (preferred path — triggered after terraform apply via the GH Actions workflow):

Navigate to Actions → "FreeScout — Terraform apply + bootstrap smoke test" and dispatch with module_only=false. The smoke test runs automatically as a separate job after apply succeeds.

Standalone invocation (workstation, after a manual apply):

# SSH key from SSM (matches CI path):
KEY_FILE=$(mktemp /tmp/lightsail_key_XXXXXX.pem)
aws ssm get-parameter \
  --name /raxx/freescout/ssh_key \
  --with-decryption \
  --region us-east-1 \
  --query "Parameter.Value" \
  --output text > "$KEY_FILE"
chmod 600 "$KEY_FILE"

export SSH_KEY_PATH="$KEY_FILE"
export INSTANCE_IP="54.146.13.200"   # raxx-tickets-ip
bash scripts/freescout/smoke-test-bootstrap.sh

Environment variables accepted by the script:

What the script records

The script writes a before+after state log to ${SMOKE_LOG_DIR}/freescout-bootstrap-smoke-<RUN_ID>.log.

Before state (captured immediately after SSH connects): - cloud-init status output - Last 20 lines of /var/log/freescout-bootstrap.log (if present) - Instance uptime

After state (captured after all assertions pass or fail): - Final cloud-init status output - Last 10 lines of the bootstrap log - /var/www/html/freescout/public directory presence - /var/www/html/freescout/.env file presence - Apache enabled sites list

In CI, the log is uploaded as a GitHub Actions artifact (freescout-bootstrap-smoke-log, retained 7 days).

Interpreting failures

Exit codes

Skipping the smoke test (module-only applies)

When running terraform apply for module installs only (no Lightsail instance change), the smoke test is not meaningful. Pass module_only=true to the GH Actions workflow or skip the script entirely when no new instance was created:

# Check if the instance resource changed in this apply before running smoke test:
# terraform show -json tfplan | jq '.resource_changes[] | select(.address == "aws_lightsail_instance.freescout") | .change.actions'
# If output is ["no-op"], skip the smoke test.

Files

RaxxTheme module

Module location (repo): tickets/freescout-modules/RaxxTheme/ Module location (instance): /var/www/html/freescout/Modules/RaxxTheme/ Issue: #1894

Tier A theming module — injects CE palette CSS and suppresses vendor attribution.

Deploy

# From repo root
bash tickets/freescout-modules/RaxxTheme/deploy.sh [/path/to/lightsail_key.pem]

Default key path is /tmp/lightsail_us_east_1.pem.

Pre-requisite: place raxx-logo-light.png (340x102px) in tickets/freescout-modules/RaxxTheme/Resources/assets/img/ before running. See Resources/assets/img/README.txt for export instructions.

Known deploy friction (2026-05-30): The Modules/ directory is owned by www-data. rsync from the workstation's admin user will fail with Permission denied unless the target directory is temporarily chowned to admin first:

# Run this before deploy.sh if target dir already exists owned by www-data:
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
  admin@54.146.13.200 \
  "sudo chown -R admin:admin /var/www/html/freescout/Modules/RaxxTheme/ 2>/dev/null || \
   sudo mkdir -p /var/www/html/freescout/Modules/RaxxTheme/ && \
   sudo chown -R admin:admin /var/www/html/freescout/Modules/RaxxTheme/"
# Then run deploy.sh normally — it re-chowns to www-data on completion.

ServiceProvider hook (PR #3117 fix): The ServiceProvider uses \Eventy::addFilter('stylesheets', ...) — NOT \FreeScout::registerStylesheet(). The latter does not exist in FreeScout 1.8.x. If you see Class "FreeScout" not found during cache:clear / view:clear, the worktree is missing the PR #3117 fix.

CSS scoping policy

CE dark-theme overrides in theme.css apply to exactly two contexts:

All admin pages (Settings, Mailbox config, Conversations, Users, Reports) run vanilla FreeScout / AdminLTE light theme — no CE overrides. This boundary is enforced by PHPUnit regression tests added in PR #3120. If you see a "global body" or unscoped CE color rule in theme.css, the tests will fail and the PR is blocked.

raxxtheme-post-deploy-checklist

After deploy.sh completes:

1. Open https://tickets.raxx.app/login — verify: - Page background is #0B0F14 (near-black) - "Sign in" button is moss green (#5B8C5A) - No vendor name in browser tab title (should read "Raxx Support — Sign in")
2. Open https://tickets.raxx.app/settings/general — verify: - Background is vanilla light (white or #ecf0f5 AdminLTE default) - All form labels (Company Name, Timezone, etc.) are dark, readable text - No CE dark surface visible anywhere on the Settings page
3. Admin > Customization module (if installed): - Footer: Raxx Support — support.raxx.app - APP_NAME: Raxx Support - Primary color: #5B8C5A
4. Admin > Settings > General > Logo: upload raxx-wordmark-dark-512x128.png
5. Replace public/favicon.ico with the Raxx mark ICO (48/32/16 multi-size)
6. Check php artisan module:list shows RaxxTheme as active
7. Check https://tickets.raxx.app/modules/raxxtheme/css/theme.css returns 200

Re-apply after FreeScout core update

FreeScout's Modules/ directory may be reset by git pull on the instance root. After any git pull or core update on the instance, re-run deploy.sh to restore the module:

bash tickets/freescout-modules/RaxxTheme/deploy.sh

Run module tests (no running FreeScout required)

cd tickets/freescout-modules/RaxxTheme
composer install --dev
vendor/bin/phpunit

Tests validate file structure, CSS palette tokens, vendor-attribution suppression, and config defaults without requiring a live instance.

New-ticket acknowledgment email (#713)

When FLAG_FREESCOUT_NEW_TICKET_ACK is on, Raptor sends an acknowledgment email to the customer via Postmark every time FreeScout fires a conversation.created webhook.

Mechanism: Postmark transactional send — NOT FreeScout's native auto-reply. FreeScout's native auto-reply is UI-only and has no API configuration path.

Handler: POST /api/webhooks/freescout/support (existing endpoint) Service: backend_v2/api/services/support_ack_notification.py Flag: freescout_new_ticket_ack (default off; promotion queued by migration 0226)

Ack email spec

Ack copy (plain text, Variant A — "1 business day")

Hi [first name],

Got it — your message is in the queue. We'll get back to you within 1 business day.

Your ticket: [#<id>] <subject>

If you're seeing something broken on your end, check the status page first — it's often faster:

  https://status.raxx.app

You can view and reply to this ticket anytime at:

  https://support.raxx.app/tickets/<id>

Or reply to this email — your reply will be added to the thread.

— Raxx Support

Prerequisites before enabling

1. POSTMARK_SERVER_TOKEN must be set in Heroku config (vault: /Raxx/Backend/).
2. The freescout_support webhook must be registered in FreeScout with event conversation.created (see §"Outbound webhooks" below — webhook ID 5 to be added).
3. FREESCOUT_SUPPORT_WEBHOOK_SECRET must be set (vault: /raxx/prod/).

Enable / disable

# Enable
heroku config:set FLAG_FREESCOUT_NEW_TICKET_ACK=1 --app raxx-api-prod >/dev/null 2>&1

# Disable
heroku config:set FLAG_FREESCOUT_NEW_TICKET_ACK=0 --app raxx-api-prod >/dev/null 2>&1

# Override response-time copy (optional)
heroku config:set SUPPORT_ACK_SLA_COPY="4 business hours" --app raxx-api-prod >/dev/null 2>&1

Anti-loop behaviour

• The ack is skipped when customer.email is absent from the webhook payload (system-generated or internally-created tickets).
• FreeScout's own queue worker will not generate a conversation.created event for auto-replies (which are threaded to an existing conversation), so the ack does not self-trigger.
• Phase 2+ smart suppression (X-Auto-Submitted header detection, duplicate-conversation guard) is tracked in issue #713 Phase 2+.

Verify ack is working

1. Send a test email to support@raxx.app from an external address.
2. Wait up to 2 minutes.
3. Confirm: a reply arrives from no-reply@raxx.app with subject Re: <your subject> — Got it.
4. Check Postmark activity log: tag support-ticket-ack, status Delivered.
5. Check Raptor logs: support_ack_notification: sending conversation_id=...

Failure mode H: ack email not arriving

Symptom: Test email to support@raxx.app triggers no reply from no-reply@raxx.app.

Diagnose in order:

1. Confirm flag is on: heroku config:get FLAG_FREESCOUT_NEW_TICKET_ACK --app raxx-api-prod
2. Confirm webhook is registered for conversation.created (see §"Outbound webhooks"): bash curl -sk -H "X-FreeScout-API-Key: ${API_KEY}" -H 'Host: tickets.raxx.app' \ -H 'User-Agent: raxx-sre-probe/1.0' \ 'https://127.0.0.1/api/webhooks' | python3 -m json.tool
3. Check Raptor logs for freescout_support_webhook and support_ack_notification log lines.
4. Check Postmark activity (tag support-ticket-ack) for delivery or bounce events.
5. Confirm POSTMARK_SERVER_TOKEN is set: bash heroku run --app raxx-api-prod python -c \ "from api.services.postmark_client import test_postmark_token; test_postmark_token()"
6. If webhook is not firing: verify FreeScout webhook registration includes conversation.created event and the target URL is reachable from the instance.

Fix: Register or update the FreeScout webhook to include conversation.created (see §"Register a new webhook" below).

Custom fields — customer_raxx_id (#656)

Created: 2026-07-18 (sre-agent, #656)

What it is

customer_raxx_id is a single-line text custom field (FreeScout TYPE_SINGLE_LINE = 2) that stores the Raxx account ID (usr_*) for the customer associated with a ticket. Operators see it in the conversation sidebar. It is NOT rendered in the customer portal (FreeScout's CustomFields module has no customer-portal rendering path).

Field registry

The field is associated with all four active mailboxes so that Raptor can gate/join ticket visibility by Raxx identity across all inbound queues.

Population path

When a new ticket arrives in FreeScout, customer_raxx_id is populated as follows:

1. Lambda bridge path (primary): The raxx-email-inbound-bridge Lambda creates conversations via PUT /api/conversations/{id}/custom_fields. It resolves the sender's email to a raxx_user_id using the freescout_crm_ticket_cache (see #4258). If a match is found, it passes {"id": <field_id_for_mailbox>, "value": "usr_..."} in the PUT body at conversation creation time.

2. CRM cache backfill (fallback): If the Lambda cannot resolve the sender email at creation time (user not yet in cache), the #4258 webhook fires when the cache is populated and backfills the field via the same PUT /api/conversations/{id}/custom_fields endpoint.

3. Manual: Operators can set the field in the FreeScout UI sidebar if neither automated path resolves it.

Population dependency: The automated population requires #4258 (freescout_crm_ticket_cache webhook). The field existence and operator-visibility are complete as of 2026-07-18; population wiring is tracked in #4258.

API endpoint for setting the field (from within the instance, bypassing CF Access):

# From the instance
API_KEY=$(sudo -u www-data /usr/bin/php -r "
define('LARAVEL_START', microtime(true));
require '/var/www/html/freescout/vendor/autoload.php';
\$app = require_once '/var/www/html/freescout/bootstrap/app.php';
\$kernel = \$app->make(Illuminate\Contracts\Console\Kernel::class);
\$kernel->bootstrap();
echo \ApiWebhooks::getApiKey();
" 2>/dev/null)

# Set customer_raxx_id on conversation <CONV_ID> (mailbox 1 -> field id 2)
curl -sk -X PUT \
  -H "X-FreeScout-API-Key: ${API_KEY}" \
  -H 'Host: tickets.raxx.app' \
  -H 'Content-Type: application/json' \
  -H 'User-Agent: raxx-sre-agent/1.0' \
  -d '[{"id": 2, "value": "usr_XXXX"}]' \
  "https://127.0.0.1/api/conversations/<CONV_ID>/custom_fields"

Use field id 2 for mailbox 1 (support@), 3 for mailbox 2 (ops@), 4 for mailbox 3 (account-management@), 5 for mailbox 5 (privacy@).

How to verify the field is in place

# Query DB (authoritative)
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
  admin@54.146.13.200 \
  "sudo mysql freescout -e 'SELECT id,mailbox_id,name,type,required,show_in_list FROM custom_fields WHERE name=\"customer_raxx_id\" ORDER BY mailbox_id;'"
# Expected: 4 rows, one per mailbox (1, 2, 3, 5), type=2

# Verify via application ORM
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
  admin@54.146.13.200 \
  "sudo -u www-data /usr/bin/php -r \"
define('LARAVEL_START', microtime(true));
require '/var/www/html/freescout/vendor/autoload.php';
\\\$app = require_once '/var/www/html/freescout/bootstrap/app.php';
\\\$kernel = \\\$app->make(Illuminate\\\Contracts\\\Console\\\Kernel::class);
\\\$kernel->bootstrap();
\\\$fields = \\\Modules\\\CustomFields\\\Entities\\\CustomField::getMailboxCustomFields(1, false);
foreach (\\\$fields as \\\$f) {
    echo 'id=' . \\\$f->id . ' name=' . \\\$f->name . ' type=' . \\\$f->type . PHP_EOL;
}
\" 2>&1"
# Expected: includes 'id=2 name=customer_raxx_id type=2'

Slug note

FreeScout's CustomFields module derives the API slug from the field name at query time (not stored in the DB). The slug for customer_raxx_id is customer_raxx_id (already snake_case, no transformation applied). The conversation_custom_field join table uses custom_field_id (the integer DB id), not slugs.

Outbound webhooks (#1893)

FreeScout fires outbound webhooks via the ApiWebhooks module. Webhooks are stored in the webhooks table.

Current webhooks (as of 2026-07-21)

Note on webhook delivery errors (updated 2026-07-21 — supersedes the 2026-06-30 CF-BFM note above): BFM is now off on raxx.app (project_bfm_off_pre_launch), so deliveries reach Raptor. Both webhook 3 and webhook 5 now hit 401 {"error":"invalid signature"} from the Raptor receiver on every real FreeScout-originated delivery — confirmed via webhook_logs during a live E2E test (conversation #53, 2026-07-21 ~20:15 UTC, see Failure mode K below). This is NOT a CF/BFM issue and NOT a missing/misconfigured secret — it is a receiver-side signature-scheme bug, see Failure mode K.

Failure mode K: Raptor webhook receivers can never verify a real FreeScout signature (SHA1/base64 vs SHA256/hex)

Discovered: 2026-07-21, sre-agent, during FREESCOUT_SUPPORT_WEBHOOK_SECRET provisioning + FREESCOUT_AUDIT_WEBHOOK_SECRET FreeScout-side rotation follow-up.

Symptom: webhook_logs on raxx-tickets shows every real delivery to webhook 3 (audit) and webhook 5 (support) failing with 401 Unauthorized {"error":"invalid signature"} from Raptor, regardless of what value FREESCOUT_AUDIT_WEBHOOK_SECRET / FREESCOUT_SUPPORT_WEBHOOK_SECRET is set to in vault + Heroku. A synthetic request signed the way the Raptor code expects (HMAC-SHA256, hex digest, using the vault secret as the HMAC key) DOES verify — proving the endpoints and secrets are wired correctly; only real FreeScout traffic fails.

Root cause — three independent mismatches between FreeScout's ApiWebhooks module (v1.0.101, Modules/ApiWebhooks/Entities/Webhook.php) and the Raptor receivers (freescout_audit_webhook.py, freescout_support_webhook.py):

1. Signing algorithm/encoding. FreeScout signs with base64_encode(hash_hmac('sha1', $payload, self::getSecretKey(), true)) — HMAC-SHA1, base64-encoded. Raptor verifies with hmac.new(secret, payload, hashlib.sha256).hexdigest() — HMAC-SHA256, hex-encoded. These can never match, for any secret value.
2. Signing key is not configurable. Webhook::getSecretKey() returns md5(config('app.key').'webhook_key') — derived solely from FreeScout's own APP_KEY. There is no admin-UI field, DB column, or module setting to set an arbitrary shared secret (confirmed: the webhooks table has no secret column; Modules/ApiWebhooks/Resources/views/settings.blade.php only displays \Webhook::getSecretKey() as a read-only label). FREESCOUT_AUDIT_WEBHOOK_SECRET / FREESCOUT_SUPPORT_WEBHOOK_SECRET are Raptor-side-only values with no FreeScout-side counterpart to "rotate to match" — rotating them only ever affects Raptor.
3. Event type + name. FreeScout sends the event name in the X-FreeScout-Event HTTP header (see Webhook::run()), not in the JSON body. Both Raptor handlers read the event name from a body field (payload.get("type") / payload.get("event")) instead, and check for conversation.* event names (e.g. conversation.replied, conversation.created) that FreeScout never sends — FreeScout's actual event vocabulary is convo.* (convo.created, convo.status, convo.customer.reply.created, convo.agent.reply.created, etc., see Webhook::$events in Webhook.php). Even if signature verification passed, every real delivery would currently be dispatched to the "unhandled event — no-op" branch.

Impact: The audit cache (freescout_crm_ticket_cache) has never been populated by a real webhook delivery, and the operator-reply-email notification path (support_operator_reply_email flag, #657) will never fire from a real FreeScout webhook, despite both flags being ON in prod and staging. Prior to 2026-07-21 this was masked by Cloudflare Bot Fight Mode returning 403 before the signature check ever ran, so the signature/event mismatch was never observed against live traffic.

This is a receiver-side code bug, not a secret/config/vault issue — out of SRE scope to fix. Recommended fix (feature-developer): reimplement _verify_signature / verify_freescout_hmac as base64_encode(hmac.new(secret_key, payload, hashlib.sha1).digest()), read the event name from the X-FreeScout-Event header, and map convo.* → the handler's internal event names (or switch the handlers to match FreeScout's native convo.* names directly). The "secret" in this scheme is md5(FREESCOUT_APP_KEY + 'webhook_key') computed from the FreeScout instance's APP_KEY (see /MooseQuest/freescout/FREESCOUT_APP_KEY_SEED — note this is the seed, not literally FreeScout's Laravel APP_KEY; the actual APP_KEY lives in FreeScout's .env on the instance and would need to be read from there or re-derived at signing-key-computation time).

Verification of the fix: After the code change ships, repeat the live E2E test in this runbook (create a test conversation → post an operator reply via API → check webhook_logs on the instance for status_code/error on the corresponding row). Expect no error and finished=1 with a 2xx from Raptor.

Webhook secret

FreeScout ApiWebhooks module uses a derived signing key for all webhooks:

// Webhook.php
public static function getSecretKey() {
    return md5(config('app.key').'webhook_key');
}

This is NOT the WEBHOOK_SECRET_AGENT_REPLIED secret stored in Infisical. The Infisical secret is for use by the Raptor/Queue receiver to verify incoming webhook payloads. The FreeScout-side signing is automatic (no config needed on FreeScout).

Vault path: /Raxx/FreeScout/WEBHOOK_SECRET_AGENT_REPLIED — generated 2026-06-19. Pending vault write (blocked in agent session — operator action required — see below).

2026-07-21 update: This same derived-secret / no-config-on-FreeScout-side constraint applies identically to FREESCOUT_AUDIT_WEBHOOK_SECRET (webhook 3) and FREESCOUT_SUPPORT_WEBHOOK_SECRET (webhook 5). See Failure mode K above — this is the reason both webhooks 401 on every real delivery regardless of vault secret rotation.

Register a new webhook (copy-paste, from localhost)

# SSH to instance
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null admin@54.146.13.200

# Get API key
API_KEY=$(sudo -u www-data /usr/bin/php -r "
require '/var/www/html/freescout/vendor/autoload.php';
\$app = require_once '/var/www/html/freescout/bootstrap/app.php';
\$kernel = \$app->make(Illuminate\Contracts\Console\Kernel::class);
\$kernel->bootstrap();
echo \ApiWebhooks::getApiKey();
" 2>/dev/null)

# Register webhook
curl -sk -X POST \
  -H "X-FreeScout-API-Key: ${API_KEY}" \
  -H 'Host: tickets.raxx.app' \
  -H 'Content-Type: application/json' \
  -H 'User-Agent: raxx-sre-agent/1.0' \
  -d '{"url":"https://your-endpoint","events":["convo.agent.reply.created"]}' \
  'https://127.0.0.1/api/webhooks'

Operator action required: write WEBHOOK_SECRET_AGENT_REPLIED to vault

The webhook secret was generated on the instance at /tmp/webhook_secret_agent_replied.txt (2026-06-19). Vault write was blocked in the sre-agent session. Complete by running:

# On the instance
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null admin@54.146.13.200 \
  "cat /tmp/webhook_secret_agent_replied.txt"
# Then write the value to Infisical at /Raxx/FreeScout/WEBHOOK_SECRET_AGENT_REPLIED (prod env)