FreeScout runbook

System: FreeScout helpdesk — tickets.raxx.app Owner: operator Last incident: 2026-06-12 (#3280 GAP-NEW — customer portal paths behind CF Access gate) Last reviewed: 2026-06-19 (#1892 #1893 #1895) Module IaC: terraform/freescout/modules.tf (PR # — replaces bash install for normal ops)

Infrastructure

CF Access applications

CF Access IdPs in use:

Design intent (updated 2026-06-19):

• /login — CF Access Google SSO gate; auto_redirect_to_identity=true (single IdP — no login-method picker). Policy allows @raxx.app email domain + kris@moosequest.net explicitly. FreeScout's own / -> /login redirect means any authenticated-only path FreeScout redirects to /login is also effectively gated before reaching the CF challenge.
• /new-conversation, /portal, /contact — decision=bypass (include=everyone). Beta testers and future customers reach the ticket submission form without a Google challenge.
• /api/* — Lambda service token (non_identity). Machine-to-machine only. Separate CF Access app; changing the human gate IdP has no effect on this path.
• All other paths with no matching CF Access app — fall through to the FreeScout origin; FreeScout's own auth middleware redirects unauthenticated requests to /login, which then hits the CF gate.
• Human login app and service-token app are separate CF Access applications with separate policy sets. Any IdP change to one does NOT affect the other.

Adding a new operator:

1. Create a FreeScout user under Manage -> Users with the operator's @raxx.app email.
2. No CF Access policy change needed — the @raxx.app email domain is already in the allow policy.
3. The operator signs in at tickets.raxx.app/login -> Google SSO -> FreeScout email match -> session issued.

IaC backfill: These apps are outside Terraform state until #3343 Terraform step lands. The comment block in terraform/freescout/dns.tf is the durable record of all app UUIDs.

Verification assertions (copy-paste):

# /login MUST redirect to CF Access (302) -> then to accounts.google.com (Google SSO)
curl -sI -A "Mozilla/5.0 (compatible; raxx-sre-probe/1.0)" --max-redirs 0 https://tickets.raxx.app/login \
  | grep -E "^HTTP|^[Ll]ocation:"
# Expected: HTTP/2 302 + location: https://moosequest.cloudflareaccess.com/...
# If 200: the Google SSO gate is missing — re-check app fa5939cd via CF API (see Failure mode D below)
# Note: if you follow the redirect chain, the next hop will be accounts.google.com (Google OAuth), NOT an OTP form

# /new-conversation must NOT redirect to CF Access
curl -sI -A "Mozilla/5.0 (compatible; raxx-sre-probe/1.0)" https://tickets.raxx.app/new-conversation \
  | grep -E "^HTTP|^[Ll]ocation:"
# Expected: HTTP/2 200 (FreeScout form) or HTTP/2 404 (contact-form module not installed)
# If 302 to cloudflareaccess.com: bypass app 753f1573 is missing — re-create it

# /portal must NOT redirect to CF Access
curl -sI -A "Mozilla/5.0 (compatible; raxx-sre-probe/1.0)" https://tickets.raxx.app/portal \
  | grep -E "^HTTP|^[Ll]ocation:"
# Expected: 200 or 404 — NOT 302 to cloudflareaccess.com

History: - 2026-05-01 (#716): Only CF Access app (scoped to /admin, non-existent route) deleted. /login unprotected. - 2026-06-05: qa-agent detected gap via #3280. Incident #3283 filed. - 2026-06-06: Root-domain gate (tickets.raxx.app) restored by sre-agent. RCA docs/ops/incidents/2026-06-06-freescout-login-unprotected.md. - 2026-06-12 (#3280 GAP-NEW): Root-domain app narrowed to /login path only; three customer-portal bypass apps created so beta testers can submit tickets without OTP challenge. - 2026-06-19 (#3343): Human login app fa5939cd migrated from OTP IdP to Google Workspace IdP (18c69c18). Policy updated: email_domain=raxx.app + email=kris@moosequest.net. auto_redirect_to_identity enabled. Service-token app ca6fd315 untouched; confirmed 200 via service token post-change.

tickets-agent-rw service token (#2329)

Provisioned: 2026-05-18 UTC
Token name: tickets-agent-rw
Token ID: 8e723042-182b-4822-a079-fad4c1ada810
CF Access app: FreeScout API — Lambda service token (ca6fd315-31c0-4fb5-91f4-ebfea8d49a98)
Policy ID: cab8010a-d63b-4519-b67a-03b9afef6409
Policy name: tickets-agent-rw-policy
Policy decision: non_identity (required for service tokens — allow would 302 due to missing identity context)
Expires: 2126-04-24 (effectively forever)

Vault paths:

Usage pattern (Python):

import os, urllib.request

client_id = vault.get_secret("CF_ACCESS_SVC_TICKETS_CLIENT_ID")
client_secret = vault.get_secret("CF_ACCESS_SVC_TICKETS_CLIENT_SECRET")
fs_api_key = vault.get_secret("FREESCOUT_API_KEY", path="/MooseQuest/freescout/")

req = urllib.request.Request(
    "https://tickets.raxx.app/api/conversations",
    headers={
        "CF-Access-Client-Id": client_id,
        "CF-Access-Client-Secret": client_secret,
        "X-FreeScout-API-Key": fs_api_key,
        "User-Agent": "raxx-agent/1.0",
    }
)

Critical notes:

• The policy decision MUST be non_identity. Service tokens carry no IdP identity; an allow decision causes a 302 redirect to the CF Access login page (false-positive 200 if the client follows redirects without checking final URL).
• This policy is additive — the existing Lambda bridge policy (b1bf50c4-ab02-470e-ba7d-ccc06693d227) is unmodified.
• HTTP 401 from the FreeScout /api/* endpoint is application-layer auth (missing X-FreeScout-API-Key), NOT a CF Access failure.
• HTTP 302 to *.cloudflareaccess.com is a CF Access failure.

Verification (copy-paste):

CLIENT_ID=$(infisical secrets get CF_ACCESS_SVC_TICKETS_CLIENT_ID \
  --path /MooseQuest/cloudflare --env prod --plain)
CLIENT_SECRET=$(infisical secrets get CF_ACCESS_SVC_TICKETS_CLIENT_SECRET \
  --path /MooseQuest/cloudflare --env prod --plain)
FS_KEY=$(infisical secrets get FREESCOUT_API_KEY \
  --path /MooseQuest/freescout --env prod --plain)

curl -sI \
  -H "CF-Access-Client-Id: ${CLIENT_ID}" \
  -H "CF-Access-Client-Secret: ${CLIENT_SECRET}" \
  -H "X-FreeScout-API-Key: ${FS_KEY}" \
  "https://tickets.raxx.app/api/conversations?page=1&pageSize=1"
# Expect: HTTP 200 with JSON body
# If HTTP 302 to cloudflareaccess.com: CF Access policy missing or wrong decision
# If HTTP 401 {"message":"Not Authorized"}: FreeScout API key missing/wrong

Rotation:

If the service token needs to be rotated (compromise, annual audit): 1. Mint a new token via CF API (see docs/ops/runbooks/cf-access-service-token-provisioning.md). 2. Write the new client_id and client_secret to vault, replacing the values above. 3. Update the CF Access policy include array with the new token_id. 4. Verify with the snippet above. 5. Delete the old token via DELETE /client/v4/accounts/{ACCOUNT_ID}/access/service_tokens/{OLD_TOKEN_ID}. 6. Update the Token ID in this runbook.

Blueprint choice rationale

lamp_8_bitnami was the original blueprint. As of 2026, it ships PHP 8.5.3. FreeScout's illuminate/support v5.5.40 declares ^7.1.3 (PHP 7.x only), so Composer refused to install on PHP 8.5.

lamp_ls_1_0 is the AWS-native standard LAMP blueprint (non-Bitnami). It ships PHP 8.2, which is the minimum PHP version FreeScout officially supports ("PHP 8.2 and newer are OK" per FreeScout installation guide). The bootstrap installs with --ignore-platform-req=php to bypass the Composer platform check for the PHP version only — not all platform requirements. Extension requirements (mbstring, xml, gd, etc.) are still enforced.

FreeScout does NOT support Laravel 9+ and has no PHP 8.5-compatible release as of April 2026.

Paths

How to tell it's broken

• https://tickets.raxx.app/ returns a non-FreeScout page (default Apache, 404, 502, 503)
• https://tickets.raxx.app/ returns 200 but no FreeScout login form
• Cloudflare Access gate does not appear (CF Access bypassed or CF not proxying)
• cloud-init status returns error on the instance
• /var/log/freescout-bootstrap.log contains FATAL or is missing

How to diagnose (in order)

1. curl -sI https://tickets.raxx.app/ — check status code and headers. FreeScout returns 200 with Content-Type: text/html; charset=UTF-8.
2. SSH to instance: ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null admin@54.146.13.200
3. Check bootstrap log: sudo cat /var/log/freescout-bootstrap.log
4. Check cloud-init status: cloud-init status
5. Check cloud-init output: sudo tail -50 /var/log/cloud-init-output.log
6. Check services: systemctl status apache2; systemctl status mariadb
7. Check if FreeScout is cloned: ls /var/www/html/freescout/
8. Check Apache vhost: ls /etc/apache2/sites-enabled/
9. Check Apache error log: sudo tail -50 /var/log/apache2/freescout-error.log

Known failure modes

Failure mode A: cloud-init error — bootstrap script failed mid-run

Symptom: - /var/log/freescout-bootstrap.log contains FATAL or ends before "completed at" - cloud-init status returns error - FreeScout directory does not exist or is incomplete

Cause: Bootstrap script exited before completing. Common sub-causes: 1. apt lock held too long (if loop exhausted before lock cleared) 2. Composer checksum mismatch (network issue fetching installer.sig) 3. MariaDB not accepting connections within 60s window 4. Composer install failed (network, disk, or dependency resolution) 5. Artisan migrate failed (DB connection refused or wrong credentials)

Diagnose:

ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu@54.146.13.200
sudo cat /var/log/freescout-bootstrap.log

Look for the last successful step, then the error. Each step is prefixed [freescout-bootstrap].

Manual recovery — run as root on the instance:

If the bootstrap failed partway, source the variables from user-data and re-run the failed step forward:

# Source non-sensitive variables (passwords are already on disk)
eval "$(sudo grep -E '^(DB_NAME|DB_USER|DB_PASSWORD|ADMIN_EMAIL|ADMIN_PASSWORD|APP_KEY_SEED|SMTP_HOST|SMTP_PORT|SMTP_USERNAME|SMTP_PASSWORD|DOMAIN|APP_NAME|MAILBOX_EMAIL)=' \
  /var/lib/cloud/instance/user-data.txt | head -20)"

# If MariaDB setup incomplete — connect as root (root uses unix socket before password is set)
sudo mysql -u root <<SQL
CREATE DATABASE IF NOT EXISTS \`freescout\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER IF NOT EXISTS 'freescout'@'localhost' IDENTIFIED BY '$DB_PASSWORD';
GRANT ALL PRIVILEGES ON \`freescout\`.* TO 'freescout'@'localhost';
FLUSH PRIVILEGES;
SQL

# If FreeScout not cloned:
# IMPORTANT: use freescout-help-desk/freescout (with the dash), NOT freescout-helpdesk/freescout
# The dash-less name is a stale mirror that does not receive security patches.
sudo git clone --depth 1 --branch master \
  https://github.com/freescout-help-desk/freescout.git \
  /var/www/html/freescout

# If Composer install not done:
# --ignore-platform-req=php: bypasses illuminate/support ^7.1.3 platform check only.
# FreeScout is PHP 8.2 compatible per official docs. Extension requirements still enforced.
cd /var/www/html/freescout
sudo -u www-data /usr/local/bin/composer install \
  --no-dev --optimize-autoloader --no-interaction \
  --ignore-platform-req=php

# If .env not written:
APP_KEY="base64:$(echo -n "${APP_KEY_SEED}$(date -u +%s)" | sha256sum | xxd -r -p | base64 | tr -d '\n')"
sudo tee /var/www/html/freescout/.env <<ENV
APP_NAME="$APP_NAME"
APP_ENV=production
APP_KEY=$APP_KEY
APP_DEBUG=false
APP_URL=https://$DOMAIN
DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=$DB_NAME
DB_USERNAME=$DB_USER
DB_PASSWORD=$DB_PASSWORD
MAIL_DRIVER=smtp
MAIL_HOST=$SMTP_HOST
MAIL_PORT=$SMTP_PORT
MAIL_USERNAME=$SMTP_USERNAME
MAIL_PASSWORD=$SMTP_PASSWORD
MAIL_ENCRYPTION=tls
MAIL_FROM_ADDRESS=$MAILBOX_EMAIL
MAIL_FROM_NAME="$APP_NAME"
FREESCOUT_ADMIN_EMAIL=$ADMIN_EMAIL
FREESCOUT_ADMIN_PASSWORD=$ADMIN_PASSWORD
TRUSTED_PROXIES=*
ENV
sudo chmod 640 /var/www/html/freescout/.env
sudo chown -R www-data:www-data /var/www/html/freescout
sudo chmod -R 775 /var/www/html/freescout/storage /var/www/html/freescout/bootstrap/cache

# If migrations not run:
cd /var/www/html/freescout
sudo -u www-data /usr/bin/php artisan key:generate --force
# NOTE: "freescout:migrate" does not exist; use "migrate --force"
sudo -u www-data /usr/bin/php artisan migrate --force
# NOTE: freescout:create-user flags in current version are --firstName, --lastName
# (NOT --name, --first_name)
sudo -u www-data /usr/bin/php artisan freescout:create-user \
  --firstName="Kristerpher" --email="$ADMIN_EMAIL" --password="$ADMIN_PASSWORD" --role=admin 2>/dev/null || \
  echo "create-user failed — seed manually after first login"
sudo -u www-data /usr/bin/php artisan storage:link 2>/dev/null || true
sudo -u www-data /usr/bin/php artisan config:cache
# NOTE: route:cache fails if any route has a Closure (polycast plugin causes this)
# Skip it — FreeScout works without route cache
sudo -u www-data /usr/bin/php artisan route:cache 2>/dev/null || echo "route:cache skipped (expected)"

# If Apache vhost not configured:
# NOTE: lamp_ls_1_0 enables SSL + snakeoil cert by default. Cloudflare connects
# to origin on port 443. BOTH vhosts (80 + 443) are required.
sudo a2enmod rewrite headers remoteip ssl
sudo a2dissite 000-default.conf default-ssl.conf 2>/dev/null || true

sudo tee /etc/apache2/sites-available/freescout.conf <<VHOST
<VirtualHost *:80>
  ServerName $DOMAIN
  DocumentRoot "/var/www/html/freescout/public"
  RewriteEngine On
  RewriteCond %{HTTP:X-Forwarded-Proto} !https
  RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
  <Directory "/var/www/html/freescout/public">
    Options -Indexes +FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>
  RemoteIPHeader CF-Connecting-IP
  ErrorLog "/var/log/apache2/freescout-error.log"
  CustomLog "/var/log/apache2/freescout-access.log" combined
</VirtualHost>
VHOST

sudo tee /etc/apache2/sites-available/freescout-ssl.conf <<SSLVHOST
<VirtualHost *:443>
  ServerName $DOMAIN
  DocumentRoot "/var/www/html/freescout/public"
  SSLEngine on
  SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
  SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
  <Directory "/var/www/html/freescout/public">
    Options -Indexes +FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>
  RemoteIPHeader CF-Connecting-IP
  ErrorLog "/var/log/apache2/freescout-error.log"
  CustomLog "/var/log/apache2/freescout-access.log" combined
</VirtualHost>
SSLVHOST

sudo a2ensite freescout.conf freescout-ssl.conf
sudo systemctl restart apache2

Verification: curl -sI https://tickets.raxx.app/ should return 200 with PHP headers and FreeScout login HTML.

Failure mode B: Apache vhost not applied — serving default or 404

Symptom: curl returns 200 but body is the default Apache page, or 404 for all paths

Cause: Apache vhost not written or enabled, or Apache not restarted

Fix:

# Check enabled sites
ls /etc/apache2/sites-enabled/
# If freescout.conf not present:
sudo a2ensite freescout.conf
sudo systemctl restart apache2
# Verify rewrite module is enabled:
apache2ctl -M | grep rewrite
# If missing: sudo a2enmod rewrite && sudo systemctl restart apache2

Verification: curl -sI https://tickets.raxx.app/ returns Laravel/FreeScout headers

Failure mode C: MariaDB not accepting connections

Symptom: Laravel .env connects but migrations fail with "Connection refused" or "Access denied"

Cause: MariaDB stopped or user not created

Fix:

# Check MariaDB status
systemctl status mariadb
# If stopped:
sudo systemctl start mariadb
# Check DB user:
sudo mysql -u root -p"$(sudo cat /root/.mariadb_root_pass)" \
  -e "SELECT User,Host FROM mysql.user WHERE User='freescout';"
# If missing, re-run DB creation commands from Failure mode A above

Failure mode D: CF Access gate on /login not appearing (operator reaches FreeScout without Google SSO)

Symptom: Anonymous GET https://tickets.raxx.app/login returns HTTP 200 (FreeScout form) instead of HTTP 302 to moosequest.cloudflareaccess.com. Operator can reach FreeScout login without a Google SSO challenge.

Note: As of 2026-06-19 (#3343), this app uses Google Workspace SSO (IdP 18c69c18), not OTP. The fix below reflects the post-migration state.

Cause: CF Access application fa5939cd (path tickets.raxx.app/login) has been deleted, or its domain was widened / changed, or its policy was removed.

Diagnose:

# Requires CF_TOKEN from vault (CLOUDFLARE_ACCESS_MGMT_TOKEN at /MooseQuest/cloudflare)
CF_TOKEN=$(infisical secrets get CLOUDFLARE_ACCESS_MGMT_TOKEN --path /MooseQuest/cloudflare/ --plain)
curl -sS -H "Authorization: Bearer $CF_TOKEN" \
  -A "raxx-sre-probe/1.0" \
  "https://api.cloudflare.com/client/v4/accounts/22b5c35090724fbf05db6d4f501ac821/access/apps/fa5939cd-ab76-40da-9fae-25bd40278c54" \
  | python3 -c "
import sys,json; r=json.load(sys.stdin).get('result',{})
print('domain:', r.get('domain','NOT FOUND'))
print('name:', r.get('name','?'))
print('allowed_idps:', r.get('allowed_idps'))
print('auto_redirect:', r.get('auto_redirect_to_identity'))
"
# Expected: domain: tickets.raxx.app/login, allowed_idps: ['18c69c18-14db-4f12-94f6-6c69d27cb04b']
# If 404 or domain is tickets.raxx.app (no /login path): re-create the app (see Fix below)

Fix: If the app is missing, re-create it via CF API with the Google IdP. Do NOT re-create as a root-domain app — that would gate /new-conversation and block beta testers. Do NOT use the OTP IdP (1035b261) — it was retired 2026-06-19.

curl -sS -X POST \
  -H "Authorization: Bearer $CF_TOKEN" \
  -H "Content-Type: application/json" \
  -A "raxx-sre-probe/1.0" \
  "https://api.cloudflare.com/client/v4/accounts/22b5c35090724fbf05db6d4f501ac821/access/apps" \
  -d '{
    "name": "FreeScout login — tickets.raxx.app/login (Google SSO)",
    "domain": "tickets.raxx.app/login",
    "type": "self_hosted",
    "session_duration": "8h",
    "app_launcher_visible": false,
    "allowed_idps": ["18c69c18-14db-4f12-94f6-6c69d27cb04b"],
    "auto_redirect_to_identity": true
  }'
# Capture the new app UUID from the response. Then add the Google SSO policy:
# POST /access/apps/<new-uuid>/policies with:
# {
#   "name": "Operator access — Google SSO (@raxx.app + kris@moosequest.net)",
#   "decision": "allow", "precedence": 1,
#   "include": [
#     {"email_domain": {"domain": "raxx.app"}},
#     {"email": {"email": "kris@moosequest.net"}}
#   ]
# }
# Update terraform/freescout/dns.tf and this runbook with the new UUID.

Verification:

curl -sI -A "Mozilla/5.0 (compatible; raxx-sre-probe/1.0)" --max-redirs 0 https://tickets.raxx.app/login \
  | grep -E "^HTTP|^[Ll]ocation:"
# Expected: HTTP/2 302 + location: https://moosequest.cloudflareaccess.com/...
# Following that redirect should auto-redirect to accounts.google.com (Google OAuth)

Failure mode E: Composer fails with PHP version error

Symptom: Bootstrap log shows illuminate/support ... php ^7.1.3 -> your php version (X.X.X) does not satisfy that requirement

Cause: PHP version on the instance is outside Laravel 5.5's platform requirement. On lamp_ls_1_0 (PHP 8.2), this should not occur because the bootstrap uses --ignore-platform-req=php. If it does occur, the flag may have been lost in a template edit.

Fix:

cd /var/www/html/freescout
sudo -u www-data /usr/local/bin/composer install \
  --no-dev --optimize-autoloader --no-interaction \
  --ignore-platform-req=php

Note: Do NOT use --ignore-platform-reqs (plural, no =php). That flag ignores all platform requirements including PHP extensions. Only the PHP version check should be bypassed.

Failure mode F: Queue worker silently drops all outbound mail — broken crontab (fixed in #1085)

Symptom: - FreeScout Settings > Email test sends show no delivery errors in the UI, but email never arrives - sudo tail -20 /var/log/freescout-queue.log is empty or missing entirely - sudo mysql -u root freescout -e "SELECT COUNT(*) FROM jobs;" returns a growing backlog - crontab -u www-data -l shows queue:work --stop-when-empty or only default queue

Root cause (four bugs, all introduced from day 0): 1. /var/log/freescout-queue.log was not created before crontab install — www-data cannot write to a root-owned path; every queue:work invocation exits with Permission denied silently. 2. --stop-when-empty does not exist in this artisan version — the worker process exited immediately on every invocation with an "unknown option" error. 3. --queue=default only — FreeScout dispatches mail jobs to the emails queue; the default queue was never consumed. 4. No schedule:run entry — Laravel scheduler (maintenance tasks, notification cleanup) never fired.

Diagnosis:

# Check crontab
sudo crontab -u www-data -l
# Check log file ownership
ls -la /var/log/freescout-queue.log 2>/dev/null || echo "log file missing"
# Check jobs backlog
sudo mysql -u root freescout -e "SELECT COUNT(*) FROM jobs;"

Fix (applies to instances bootstrapped before PR #1085 was merged):

# 1. Create log with correct ownership
sudo touch /var/log/freescout-queue.log
sudo chown www-data:www-data /var/log/freescout-queue.log
sudo chmod 640 /var/log/freescout-queue.log

# 2. Replace broken crontab
sudo crontab -u www-data - <<'CRON'
* * * * * /usr/bin/php /var/www/html/freescout/artisan queue:work --queue=emails,default --once --timeout=30 >> /var/log/freescout-queue.log 2>&1
* * * * * /usr/bin/php /var/www/html/freescout/artisan schedule:run >> /var/log/freescout-queue.log 2>&1
CRON

# 3. Verify
sudo crontab -u www-data -l

Verification (wait ~2 minutes after applying fix):

sudo tail -20 /var/log/freescout-queue.log
# Should show queue:work and schedule:run lines, not Permission denied
sudo mysql -u root freescout -e "SELECT COUNT(*) FROM jobs;"
# Should be 0 or decreasing under normal operation

Re-run the FreeScout Settings > Email test. Email should arrive within 2 minutes.

Note: New Lightsail instances provisioned from Terraform after PR #1085 is merged receive the corrected crontab automatically and do not need this manual fix.

Emergency stop

sudo systemctl stop apache2
# MariaDB: leave running unless disk is full

Escalation

Wake the operator when: - FreeScout database has data loss or corruption - SSL certificate is expired and CF fails to renew - Lightsail instance is unresponsive (SSH down, Lightsail console shows stopped) - MariaDB data partition is full (40 GB SSD)

Admin account security

TOTP two-factor authentication (#718)

FreeScout has built-in TOTP support. The admin account (kris@moosequest.net) must have TOTP enabled as a defense-in-depth layer behind Cloudflare Access. This is an operator-only action — it cannot be automated.

This is an operator action. Kristerpher must complete this personally.

Enrollment steps:

1. Log in to https://tickets.raxx.app (passes through CF Access first).
2. In the top-right corner, click your avatar / name, then Your Profile.
3. In the left sidebar, click Security (or look for Two-Factor Authentication directly on the profile page).
4. Under Two-Factor Authentication, click Enable.
5. A QR code is displayed. Scan it with an authenticator app (use the same app as other Raxx TOTP codes — Google Authenticator, Authy, 1Password, etc.).
6. Enter the 6-digit confirmation code from the app to confirm enrollment.
7. FreeScout will display one-time recovery codes. Copy all recovery codes before closing this page. They are shown only once.
8. Store recovery codes in Infisical at path /MooseQuest/freescout/FREESCOUT_TOTP_RECOVERY_CODES (as a multi-line secret value, one code per line).

Verification:

Log out of FreeScout, then log back in through CF Access. After CF Access passes, FreeScout should show a TOTP prompt before granting access to the inbox.

Recovery procedure (if authenticator app is lost):

1. Retrieve recovery codes from Infisical at /MooseQuest/freescout/FREESCOUT_TOTP_RECOVERY_CODES.
2. On the FreeScout login screen, after entering your password, click Lost your device? or Use recovery code.
3. Enter one recovery code (each code is single-use).
4. Once logged in, go to Profile > Security > Two-Factor Authentication and re-enroll with a new authenticator app.
5. Store the new recovery codes in Infisical, replacing the previous set.

If TOTP is not available on the deployed version:

Check FreeScout version: https://tickets.raxx.app/dashboard (visible in footer or Settings > General). FreeScout's built-in 2FA was introduced in version 1.8.x. If unavailable, document the gap in a comment on #718 and rely solely on CF Access WebAuthn from #695 until a FreeScout update can be applied.

Important: Do not close issue #718 until TOTP is active AND recovery codes are stored in Infisical. Documentation alone is not sufficient for this card.

Vault path reference for admin credentials

Google OAuth SSO (#961)

FreeScout operators sign in with their @raxx.app Google Workspace account via the Tagras oauthlogin module (installed 2026-05-12 UTC). Local FreeScout passwords are retired after successful configuration.

Full setup and operations procedure: docs/ops/runbooks/freescout-google-oauth.md

Quick-reference vault paths:

Operator action required — see the Google OAuth runbook for the Cloud Console OAuth app creation steps and FreeScout module configuration. Neither step can be automated (Cloud Console requires Workspace admin login; module config is UI-only).

Branding and company settings (#720, #1895)

Current state (2026-06-19, #1895 complete): - .env APP_NAME="Raxx Support" — set - .env MAIL_FROM_NAME="Raxx Support" — set - DB options.company_name = "Raxx Support" — set (this is what FreeScout actually uses for display; config/app.php hardcodes "FreeScout" and is NOT used for the company name) - DB options.email_branding = "" — cleared (was 1; removing this suppresses the "powered by FreeScout" footer in outbound emails) - Mailbox name verified via API: "Raxx Support" (mailbox ID 1, support@raxx.app)

FreeScout's login page (tickets.raxx.app/login) is the operator-facing gateway. It is protected by Cloudflare Access and is not customer-reachable under normal operating conditions. Nevertheless, a minimal Raxx wordmark should be set so the login page does not display the default FreeScout logo.

Access decision: tickets.raxx.app/login is operator-only (behind CF Access — kris@moosequest.net allowlist only). Customers interact exclusively with support.raxx.app (portal, #651). Full branding is not launch-blocking but is a pre-launch should-do.

How to update company name and logo (operator action, ~10 min)

This is an operator action. Kristerpher must complete this in the FreeScout admin UI.

1. Log in to https://tickets.raxx.app.
2. Navigate to Settings (gear icon, top navigation bar).
3. Click General in the left sidebar.
4. In the Company Name field, enter: Raxx Support
5. In the Company Logo field: upload docs/design/brand/logos/raxx-wordmark-c.svg (the canonical Confidence Engine wordmark). If the field requires a raster image, export the SVG at 2x (300 px wide, white background if needed) and upload the PNG.
6. In the Favicon field (if present): upload console/app/static/favicon-32.png (32x32).
7. Click Save.

Verification:

Navigate to https://tickets.raxx.app/login (in an incognito window after passing CF Access). The FreeScout default logo should be replaced by the Raxx wordmark. The page title should read "Raxx Support" rather than "FreeScout".

Canonical brand assets:

Note on logo hosting for email: Do NOT use a repo-local path as the logo URL in email templates. Cloudflare Access protects tickets.raxx.app — any image URL that points to a CF-Access-gated resource will fail to render in external email clients. The branded email templates in freescout/templates/ inline the logo as a base64 data URI to avoid this. See docs/ops/runbooks/freescout-templates.md for the full email template deployment procedure (#712).

Outbound email template branding (#712)

Raxx-branded HTML + plaintext email templates for all outbound FreeScout emails live at:

freescout/templates/
  autoreply.html         — auto-reply sent when a customer creates a new ticket
  autoreply.txt          — plaintext fallback for auto-reply
  operator-reply.html    — sent when an operator replies to a customer
  operator-reply.txt     — plaintext fallback for operator reply
  notification.html      — ticket status change and system notifications to customers
  notification.txt       — plaintext fallback for notifications

Full deployment procedure, variable reference, durability notes, and revert steps: docs/ops/runbooks/freescout-templates.md

Template file locations on the Lightsail instance (verify on your FreeScout version):

After any FreeScout core update (git pull on the instance), re-apply the templates. See the templates runbook for the post-update checklist.

Paid module install and management

FreeScout's 10 paid modules (API & Webhooks, Reports, OAuth, Custom Fields, Tags, Workflows, Customization, Saved Replies, Slack, Custom Folders) are managed declaratively via terraform/freescout/modules.tf.

Normal path — terraform apply

# From repo root (requires infisical CLI + AWS credentials in environment):
make freescout-install
# Review the plan output, then:
cd terraform/freescout && terraform apply tfplan

Expected timing: 5–10 minutes for all 10 modules.

Terraform takes a Lightsail snapshot before and after the install batch. After apply completes, verify all 10 modules show status Active under https://tickets.raxx.app/modules.

Re-installing a single module (e.g., after license key rotation)

1. Update the vault secret at /MooseQuest/freescout/<NAME>_LICENSE_KEY.
2. Re-export just that TF_VAR_license_* variable.
3. Run terraform plan — only that module's null_resource shows as "must replace".
4. Run terraform apply.

Forcing a full reinstall of all modules

Bump module_version_pin in terraform/freescout/terraform.tfvars:

module_version_pin = "YYYY-MM-DD"  # new date

Next apply re-fires all 10 install+activate commands.

Break-glass: bash script fallback

If Terraform is unavailable (e.g., running directly on the instance without a local workstation), use scripts/ops/install-freescout-modules.sh. This script reads all license keys from Infisical at runtime and runs the same artisan commands. It is the operational reference that modules.tf was derived from. Do NOT delete it.

ssh -i ~/.ssh/lightsail_us_east_1.pem -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null admin@54.146.13.200
curl -fsSL https://raw.githubusercontent.com/raxx-app/TradeMasterAPI/main/scripts/ops/install-freescout-modules.sh \
  -o /tmp/install.sh
chmod +x /tmp/install.sh
sudo \
  INFISICAL_HOST="https://vault.raxx.app" \
  INFISICAL_PROJECT_ID="29b77751-f761-4afa-b3fa-2c842988f95c" \
  INFISICAL_CLIENT_ID="<from break-glass>" \
  INFISICAL_CLIENT_SECRET="<from break-glass>" \
  /tmp/install.sh

After using the break-glass script, run terraform plan to confirm state matches. If Terraform shows drift, either import the state or accept that the next apply will re-run install+activate (idempotent at the FreeScout layer).

Failure mode G: E2E smoke fails at Step 2 — /new-conversation returns 404

Symptom: tickets-e2e-smoke.yml Step 2 FAIL: HTTP 404 from https://tickets.raxx.app/new-conversation

Cause: FreeScout 1.8.x has no built-in public /new-conversation route at the root domain. All conversation routes require the auth middleware (operator login). The E2E smoke was written expecting the customer portal surface to be live — either the support.raxx.app proxy (epic #651) or a FreeScout customer-portal module (not installed). Until epic #651 ships, Steps 2-3 of the smoke will fail.

Step 1 (CF Access detection) and Step 6 (invalid-path 404 check) PASS correctly.

Status: Known pre-launch blocker. The smoke correctly identifies that the customer-facing form submission path is not yet configured. This is not a regression — it is a capability gap.

Resolution path: Ship epic #651 (support.raxx.app proxy + FreeScout API bridge), then re-run gh workflow run tickets-e2e-smoke.yml.

Temporary posture: The smoke runs daily on schedule and will automatically start passing once the /new-conversation form endpoint exists. No action needed until #651 is ready.

Post-incident actions for any new failure mode

1. Update this runbook with the new failure mode
2. Add a terraform/README.md lessons-learned entry
3. File a type:reliability GitHub issue

Post-apply bootstrap smoke test (#526)

Added: 2026-05-12 UTC RCA origin: docs/ops/incidents/2026-04-29-freescout-cloud-init-fatal.md action item #3

Why this exists

terraform apply returns success as soon as the Lightsail instance resource is created. It does NOT wait for cloud-init to finish. The 2026-04-29 incident saw 6 consecutive applies succeed from Terraform's perspective while cloud-init failed with FATAL every time. This smoke test catches that gap.

What it checks (in order)

1. SSH connectivity to the instance (30s timeout per attempt; 60s total)
2. cloud-init status polls until status is no longer running (20-minute max)
3. cloud-init final status is NOT error or degraded
4. /var/log/freescout-bootstrap.log exists and contains NO FATAL
5. /var/log/freescout-bootstrap.log contains the completion marker: [freescout-bootstrap] completed at
6. apache2 is active (systemctl)
7. mariadb is active (systemctl)

How to run it

Via CI (preferred path — triggered after terraform apply via the GH Actions workflow):

Navigate to Actions → "FreeScout — Terraform apply + bootstrap smoke test" and dispatch with module_only=false. The smoke test runs automatically as a separate job after apply succeeds.

Standalone invocation (workstation, after a manual apply):

# SSH key from SSM (matches CI path):
KEY_FILE=$(mktemp /tmp/lightsail_key_XXXXXX.pem)
aws ssm get-parameter \
  --name /raxx/freescout/ssh_key \
  --with-decryption \
  --region us-east-1 \
  --query "Parameter.Value" \
  --output text > "$KEY_FILE"
chmod 600 "$KEY_FILE"

export SSH_KEY_PATH="$KEY_FILE"
export INSTANCE_IP="54.146.13.200"   # raxx-tickets-ip
bash scripts/freescout/smoke-test-bootstrap.sh

Environment variables accepted by the script:

What the script records

The script writes a before+after state log to ${SMOKE_LOG_DIR}/freescout-bootstrap-smoke-<RUN_ID>.log.

Before state (captured immediately after SSH connects): - cloud-init status output - Last 20 lines of /var/log/freescout-bootstrap.log (if present) - Instance uptime

After state (captured after all assertions pass or fail): - Final cloud-init status output - Last 10 lines of the bootstrap log - /var/www/html/freescout/public directory presence - /var/www/html/freescout/.env file presence - Apache enabled sites list

In CI, the log is uploaded as a GitHub Actions artifact (freescout-bootstrap-smoke-log, retained 7 days).

Interpreting failures

Exit codes

Skipping the smoke test (module-only applies)

When running terraform apply for module installs only (no Lightsail instance change), the smoke test is not meaningful. Pass module_only=true to the GH Actions workflow or skip the script entirely when no new instance was created:

# Check if the instance resource changed in this apply before running smoke test:
# terraform show -json tfplan | jq '.resource_changes[] | select(.address == "aws_lightsail_instance.freescout") | .change.actions'
# If output is ["no-op"], skip the smoke test.

Files

RaxxTheme module

Module location (repo): tickets/freescout-modules/RaxxTheme/ Module location (instance): /var/www/html/freescout/Modules/RaxxTheme/ Issue: #1894

Tier A theming module — injects CE palette CSS and suppresses vendor attribution.

Deploy

# From repo root
bash tickets/freescout-modules/RaxxTheme/deploy.sh [/path/to/lightsail_key.pem]

Default key path is /tmp/lightsail_us_east_1.pem.

Pre-requisite: place raxx-logo-light.png (340x102px) in tickets/freescout-modules/RaxxTheme/Resources/assets/img/ before running. See Resources/assets/img/README.txt for export instructions.

Known deploy friction (2026-05-30): The Modules/ directory is owned by www-data. rsync from the workstation's admin user will fail with Permission denied unless the target directory is temporarily chowned to admin first:

# Run this before deploy.sh if target dir already exists owned by www-data:
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
  admin@54.146.13.200 \
  "sudo chown -R admin:admin /var/www/html/freescout/Modules/RaxxTheme/ 2>/dev/null || \
   sudo mkdir -p /var/www/html/freescout/Modules/RaxxTheme/ && \
   sudo chown -R admin:admin /var/www/html/freescout/Modules/RaxxTheme/"
# Then run deploy.sh normally — it re-chowns to www-data on completion.

ServiceProvider hook (PR #3117 fix): The ServiceProvider uses \Eventy::addFilter('stylesheets', ...) — NOT \FreeScout::registerStylesheet(). The latter does not exist in FreeScout 1.8.x. If you see Class "FreeScout" not found during cache:clear / view:clear, the worktree is missing the PR #3117 fix.

CSS scoping policy

CE dark-theme overrides in theme.css apply to exactly two contexts:

All admin pages (Settings, Mailbox config, Conversations, Users, Reports) run vanilla FreeScout / AdminLTE light theme — no CE overrides. This boundary is enforced by PHPUnit regression tests added in PR #3120. If you see a "global body" or unscoped CE color rule in theme.css, the tests will fail and the PR is blocked.

raxxtheme-post-deploy-checklist

After deploy.sh completes:

1. Open https://tickets.raxx.app/login — verify: - Page background is #0B0F14 (near-black) - "Sign in" button is moss green (#5B8C5A) - No vendor name in browser tab title (should read "Raxx Support — Sign in")
2. Open https://tickets.raxx.app/settings/general — verify: - Background is vanilla light (white or #ecf0f5 AdminLTE default) - All form labels (Company Name, Timezone, etc.) are dark, readable text - No CE dark surface visible anywhere on the Settings page
3. Admin > Customization module (if installed): - Footer: Raxx Support — support.raxx.app - APP_NAME: Raxx Support - Primary color: #5B8C5A
4. Admin > Settings > General > Logo: upload raxx-wordmark-dark-512x128.png
5. Replace public/favicon.ico with the Raxx mark ICO (48/32/16 multi-size)
6. Check php artisan module:list shows RaxxTheme as active
7. Check https://tickets.raxx.app/modules/raxxtheme/css/theme.css returns 200

Re-apply after FreeScout core update

FreeScout's Modules/ directory may be reset by git pull on the instance root. After any git pull or core update on the instance, re-run deploy.sh to restore the module:

bash tickets/freescout-modules/RaxxTheme/deploy.sh

Run module tests (no running FreeScout required)

cd tickets/freescout-modules/RaxxTheme
composer install --dev
vendor/bin/phpunit

Tests validate file structure, CSS palette tokens, vendor-attribution suppression, and config defaults without requiring a live instance.

Outbound webhooks (#1893)

FreeScout fires outbound webhooks via the ApiWebhooks module. Webhooks are stored in the webhooks table.

Current webhooks (as of 2026-06-19)

Note on webhook delivery errors: Both webhooks currently hit 403 from the Raptor receiver (CF BFM / WAF skip rule gap — tracked separately). This is a pre-existing issue and does not indicate a webhook registration problem.

Webhook secret

FreeScout ApiWebhooks module uses a derived signing key for all webhooks:

// Webhook.php
public static function getSecretKey() {
    return md5(config('app.key').'webhook_key');
}

This is NOT the WEBHOOK_SECRET_AGENT_REPLIED secret stored in Infisical. The Infisical secret is for use by the Raptor/Queue receiver to verify incoming webhook payloads. The FreeScout-side signing is automatic (no config needed on FreeScout).

Vault path: /Raxx/FreeScout/WEBHOOK_SECRET_AGENT_REPLIED — generated 2026-06-19. Pending vault write (blocked in agent session — operator action required — see below).

Register a new webhook (copy-paste, from localhost)

# SSH to instance
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null admin@54.146.13.200

# Get API key
API_KEY=$(sudo -u www-data /usr/bin/php -r "
require '/var/www/html/freescout/vendor/autoload.php';
\$app = require_once '/var/www/html/freescout/bootstrap/app.php';
\$kernel = \$app->make(Illuminate\Contracts\Console\Kernel::class);
\$kernel->bootstrap();
echo \ApiWebhooks::getApiKey();
" 2>/dev/null)

# Register webhook
curl -sk -X POST \
  -H "X-FreeScout-API-Key: ${API_KEY}" \
  -H 'Host: tickets.raxx.app' \
  -H 'Content-Type: application/json' \
  -H 'User-Agent: raxx-sre-agent/1.0' \
  -d '{"url":"https://your-endpoint","events":["convo.agent.reply.created"]}' \
  'https://127.0.0.1/api/webhooks'

Operator action required: write WEBHOOK_SECRET_AGENT_REPLIED to vault

The webhook secret was generated on the instance at /tmp/webhook_secret_agent_replied.txt (2026-06-19). Vault write was blocked in the sre-agent session. Complete by running:

# On the instance
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null admin@54.146.13.200 \
  "cat /tmp/webhook_secret_agent_replied.txt"
# Then write the value to Infisical at /Raxx/FreeScout/WEBHOOK_SECRET_AGENT_REPLIED (prod env)