FreeScout runbook

System: FreeScout helpdesk — tickets.raxx.app Owner: operator Last incident: 2026-04-29 (see RCA docs/incidents/2026-04-29-freescout-cloud-init-fatal.md) Last reviewed: 2026-05-02 Module IaC: terraform/freescout/modules.tf (PR # — replaces bash install for normal ops)

Infrastructure

Blueprint choice rationale

lamp_8_bitnami was the original blueprint. As of 2026, it ships PHP 8.5.3. FreeScout's illuminate/support v5.5.40 declares ^7.1.3 (PHP 7.x only), so Composer refused to install on PHP 8.5.

lamp_ls_1_0 is the AWS-native standard LAMP blueprint (non-Bitnami). It ships PHP 8.2, which is the minimum PHP version FreeScout officially supports ("PHP 8.2 and newer are OK" per FreeScout installation guide). The bootstrap installs with --ignore-platform-req=php to bypass the Composer platform check for the PHP version only — not all platform requirements. Extension requirements (mbstring, xml, gd, etc.) are still enforced.

FreeScout does NOT support Laravel 9+ and has no PHP 8.5-compatible release as of April 2026.

Paths

How to tell it's broken

• https://tickets.raxx.app/ returns a non-FreeScout page (default Apache, 404, 502, 503)
• https://tickets.raxx.app/ returns 200 but no FreeScout login form
• Cloudflare Access gate does not appear (CF Access bypassed or CF not proxying)
• cloud-init status returns error on the instance
• /var/log/freescout-bootstrap.log contains FATAL or is missing

How to diagnose (in order)

1. curl -sI https://tickets.raxx.app/ — check status code and headers. FreeScout returns 200 with Content-Type: text/html; charset=UTF-8.
2. SSH to instance: ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null admin@54.146.13.200
3. Check bootstrap log: sudo cat /var/log/freescout-bootstrap.log
4. Check cloud-init status: cloud-init status
5. Check cloud-init output: sudo tail -50 /var/log/cloud-init-output.log
6. Check services: systemctl status apache2; systemctl status mariadb
7. Check if FreeScout is cloned: ls /var/www/html/freescout/
8. Check Apache vhost: ls /etc/apache2/sites-enabled/
9. Check Apache error log: sudo tail -50 /var/log/apache2/freescout-error.log

Known failure modes

Failure mode A: cloud-init error — bootstrap script failed mid-run

Symptom: - /var/log/freescout-bootstrap.log contains FATAL or ends before "completed at" - cloud-init status returns error - FreeScout directory does not exist or is incomplete

Cause: Bootstrap script exited before completing. Common sub-causes: 1. apt lock held too long (if loop exhausted before lock cleared) 2. Composer checksum mismatch (network issue fetching installer.sig) 3. MariaDB not accepting connections within 60s window 4. Composer install failed (network, disk, or dependency resolution) 5. Artisan migrate failed (DB connection refused or wrong credentials)

Diagnose:

ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu@54.146.13.200
sudo cat /var/log/freescout-bootstrap.log

Look for the last successful step, then the error. Each step is prefixed [freescout-bootstrap].

Manual recovery — run as root on the instance:

If the bootstrap failed partway, source the variables from user-data and re-run the failed step forward:

# Source non-sensitive variables (passwords are already on disk)
eval "$(sudo grep -E '^(DB_NAME|DB_USER|DB_PASSWORD|ADMIN_EMAIL|ADMIN_PASSWORD|APP_KEY_SEED|SMTP_HOST|SMTP_PORT|SMTP_USERNAME|SMTP_PASSWORD|DOMAIN|APP_NAME|MAILBOX_EMAIL)=' \
  /var/lib/cloud/instance/user-data.txt | head -20)"

# If MariaDB setup incomplete — connect as root (root uses unix socket before password is set)
sudo mysql -u root <<SQL
CREATE DATABASE IF NOT EXISTS \`freescout\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER IF NOT EXISTS 'freescout'@'localhost' IDENTIFIED BY '$DB_PASSWORD';
GRANT ALL PRIVILEGES ON \`freescout\`.* TO 'freescout'@'localhost';
FLUSH PRIVILEGES;
SQL

# If FreeScout not cloned:
# IMPORTANT: use freescout-help-desk/freescout (with the dash), NOT freescout-helpdesk/freescout
# The dash-less name is a stale mirror that does not receive security patches.
sudo git clone --depth 1 --branch master \
  https://github.com/freescout-help-desk/freescout.git \
  /var/www/html/freescout

# If Composer install not done:
# --ignore-platform-req=php: bypasses illuminate/support ^7.1.3 platform check only.
# FreeScout is PHP 8.2 compatible per official docs. Extension requirements still enforced.
cd /var/www/html/freescout
sudo -u www-data /usr/local/bin/composer install \
  --no-dev --optimize-autoloader --no-interaction \
  --ignore-platform-req=php

# If .env not written:
APP_KEY="base64:$(echo -n "${APP_KEY_SEED}$(date -u +%s)" | sha256sum | xxd -r -p | base64 | tr -d '\n')"
sudo tee /var/www/html/freescout/.env <<ENV
APP_NAME="$APP_NAME"
APP_ENV=production
APP_KEY=$APP_KEY
APP_DEBUG=false
APP_URL=https://$DOMAIN
DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=$DB_NAME
DB_USERNAME=$DB_USER
DB_PASSWORD=$DB_PASSWORD
MAIL_DRIVER=smtp
MAIL_HOST=$SMTP_HOST
MAIL_PORT=$SMTP_PORT
MAIL_USERNAME=$SMTP_USERNAME
MAIL_PASSWORD=$SMTP_PASSWORD
MAIL_ENCRYPTION=tls
MAIL_FROM_ADDRESS=$MAILBOX_EMAIL
MAIL_FROM_NAME="$APP_NAME"
FREESCOUT_ADMIN_EMAIL=$ADMIN_EMAIL
FREESCOUT_ADMIN_PASSWORD=$ADMIN_PASSWORD
TRUSTED_PROXIES=*
ENV
sudo chmod 640 /var/www/html/freescout/.env
sudo chown -R www-data:www-data /var/www/html/freescout
sudo chmod -R 775 /var/www/html/freescout/storage /var/www/html/freescout/bootstrap/cache

# If migrations not run:
cd /var/www/html/freescout
sudo -u www-data /usr/bin/php artisan key:generate --force
# NOTE: "freescout:migrate" does not exist; use "migrate --force"
sudo -u www-data /usr/bin/php artisan migrate --force
# NOTE: freescout:create-user flags in current version are --firstName, --lastName
# (NOT --name, --first_name)
sudo -u www-data /usr/bin/php artisan freescout:create-user \
  --firstName="Kristerpher" --email="$ADMIN_EMAIL" --password="$ADMIN_PASSWORD" --role=admin 2>/dev/null || \
  echo "create-user failed — seed manually after first login"
sudo -u www-data /usr/bin/php artisan storage:link 2>/dev/null || true
sudo -u www-data /usr/bin/php artisan config:cache
# NOTE: route:cache fails if any route has a Closure (polycast plugin causes this)
# Skip it — FreeScout works without route cache
sudo -u www-data /usr/bin/php artisan route:cache 2>/dev/null || echo "route:cache skipped (expected)"

# If Apache vhost not configured:
# NOTE: lamp_ls_1_0 enables SSL + snakeoil cert by default. Cloudflare connects
# to origin on port 443. BOTH vhosts (80 + 443) are required.
sudo a2enmod rewrite headers remoteip ssl
sudo a2dissite 000-default.conf default-ssl.conf 2>/dev/null || true

sudo tee /etc/apache2/sites-available/freescout.conf <<VHOST
<VirtualHost *:80>
  ServerName $DOMAIN
  DocumentRoot "/var/www/html/freescout/public"
  RewriteEngine On
  RewriteCond %{HTTP:X-Forwarded-Proto} !https
  RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
  <Directory "/var/www/html/freescout/public">
    Options -Indexes +FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>
  RemoteIPHeader CF-Connecting-IP
  ErrorLog "/var/log/apache2/freescout-error.log"
  CustomLog "/var/log/apache2/freescout-access.log" combined
</VirtualHost>
VHOST

sudo tee /etc/apache2/sites-available/freescout-ssl.conf <<SSLVHOST
<VirtualHost *:443>
  ServerName $DOMAIN
  DocumentRoot "/var/www/html/freescout/public"
  SSLEngine on
  SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
  SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
  <Directory "/var/www/html/freescout/public">
    Options -Indexes +FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>
  RemoteIPHeader CF-Connecting-IP
  ErrorLog "/var/log/apache2/freescout-error.log"
  CustomLog "/var/log/apache2/freescout-access.log" combined
</VirtualHost>
SSLVHOST

sudo a2ensite freescout.conf freescout-ssl.conf
sudo systemctl restart apache2

Verification: curl -sI https://tickets.raxx.app/ should return 200 with PHP headers and FreeScout login HTML.

Failure mode B: Apache vhost not applied — serving default or 404

Symptom: curl returns 200 but body is the default Apache page, or 404 for all paths

Cause: Apache vhost not written or enabled, or Apache not restarted

Fix:

# Check enabled sites
ls /etc/apache2/sites-enabled/
# If freescout.conf not present:
sudo a2ensite freescout.conf
sudo systemctl restart apache2
# Verify rewrite module is enabled:
apache2ctl -M | grep rewrite
# If missing: sudo a2enmod rewrite && sudo systemctl restart apache2

Verification: curl -sI https://tickets.raxx.app/ returns Laravel/FreeScout headers

Failure mode C: MariaDB not accepting connections

Symptom: Laravel .env connects but migrations fail with "Connection refused" or "Access denied"

Cause: MariaDB stopped or user not created

Fix:

# Check MariaDB status
systemctl status mariadb
# If stopped:
sudo systemctl start mariadb
# Check DB user:
sudo mysql -u root -p"$(sudo cat /root/.mariadb_root_pass)" \
  -e "SELECT User,Host FROM mysql.user WHERE User='freescout';"
# If missing, re-run DB creation commands from Failure mode A above

Failure mode D: Cloudflare Access gate not appearing

Symptom: https://tickets.raxx.app/ loads FreeScout directly without CF Access login prompt

Cause: CF Access application or policy missing/misconfigured in Cloudflare

Diagnose:

cd terraform/freescout && terraform state list | grep cloudflare
# If cf_access resources are absent, re-apply CF Access portion:
terraform apply -target=cloudflare_access_application.freescout \
  -target=cloudflare_access_policy.freescout

Failure mode E: Composer fails with PHP version error

Symptom: Bootstrap log shows illuminate/support ... php ^7.1.3 -> your php version (X.X.X) does not satisfy that requirement

Cause: PHP version on the instance is outside Laravel 5.5's platform requirement. On lamp_ls_1_0 (PHP 8.2), this should not occur because the bootstrap uses --ignore-platform-req=php. If it does occur, the flag may have been lost in a template edit.

Fix:

cd /var/www/html/freescout
sudo -u www-data /usr/local/bin/composer install \
  --no-dev --optimize-autoloader --no-interaction \
  --ignore-platform-req=php

Note: Do NOT use --ignore-platform-reqs (plural, no =php). That flag ignores all platform requirements including PHP extensions. Only the PHP version check should be bypassed.

Failure mode F: Queue worker silently drops all outbound mail — broken crontab (fixed in #1085)

Symptom: - FreeScout Settings > Email test sends show no delivery errors in the UI, but email never arrives - sudo tail -20 /var/log/freescout-queue.log is empty or missing entirely - sudo mysql -u root freescout -e "SELECT COUNT(*) FROM jobs;" returns a growing backlog - crontab -u www-data -l shows queue:work --stop-when-empty or only default queue

Root cause (four bugs, all introduced from day 0): 1. /var/log/freescout-queue.log was not created before crontab install — www-data cannot write to a root-owned path; every queue:work invocation exits with Permission denied silently. 2. --stop-when-empty does not exist in this artisan version — the worker process exited immediately on every invocation with an "unknown option" error. 3. --queue=default only — FreeScout dispatches mail jobs to the emails queue; the default queue was never consumed. 4. No schedule:run entry — Laravel scheduler (maintenance tasks, notification cleanup) never fired.

Diagnosis:

# Check crontab
sudo crontab -u www-data -l
# Check log file ownership
ls -la /var/log/freescout-queue.log 2>/dev/null || echo "log file missing"
# Check jobs backlog
sudo mysql -u root freescout -e "SELECT COUNT(*) FROM jobs;"

Fix (applies to instances bootstrapped before PR #1085 was merged):

# 1. Create log with correct ownership
sudo touch /var/log/freescout-queue.log
sudo chown www-data:www-data /var/log/freescout-queue.log
sudo chmod 640 /var/log/freescout-queue.log

# 2. Replace broken crontab
sudo crontab -u www-data - <<'CRON'
* * * * * /usr/bin/php /var/www/html/freescout/artisan queue:work --queue=emails,default --once --timeout=30 >> /var/log/freescout-queue.log 2>&1
* * * * * /usr/bin/php /var/www/html/freescout/artisan schedule:run >> /var/log/freescout-queue.log 2>&1
CRON

# 3. Verify
sudo crontab -u www-data -l

Verification (wait ~2 minutes after applying fix):

sudo tail -20 /var/log/freescout-queue.log
# Should show queue:work and schedule:run lines, not Permission denied
sudo mysql -u root freescout -e "SELECT COUNT(*) FROM jobs;"
# Should be 0 or decreasing under normal operation

Re-run the FreeScout Settings > Email test. Email should arrive within 2 minutes.

Note: New Lightsail instances provisioned from Terraform after PR #1085 is merged receive the corrected crontab automatically and do not need this manual fix.

Emergency stop

sudo systemctl stop apache2
# MariaDB: leave running unless disk is full

Escalation

Wake the operator when: - FreeScout database has data loss or corruption - SSL certificate is expired and CF fails to renew - Lightsail instance is unresponsive (SSH down, Lightsail console shows stopped) - MariaDB data partition is full (40 GB SSD)

Admin account security

TOTP two-factor authentication (#718)

FreeScout has built-in TOTP support. The admin account (kris@moosequest.net) must have TOTP enabled as a defense-in-depth layer behind Cloudflare Access. This is an operator-only action — it cannot be automated.

This is an operator action. Kristerpher must complete this personally.

Enrollment steps:

1. Log in to https://tickets.raxx.app (passes through CF Access first).
2. In the top-right corner, click your avatar / name, then Your Profile.
3. In the left sidebar, click Security (or look for Two-Factor Authentication directly on the profile page).
4. Under Two-Factor Authentication, click Enable.
5. A QR code is displayed. Scan it with an authenticator app (use the same app as other Raxx TOTP codes — Google Authenticator, Authy, 1Password, etc.).
6. Enter the 6-digit confirmation code from the app to confirm enrollment.
7. FreeScout will display one-time recovery codes. Copy all recovery codes before closing this page. They are shown only once.
8. Store recovery codes in Infisical at path /MooseQuest/freescout/FREESCOUT_TOTP_RECOVERY_CODES (as a multi-line secret value, one code per line).

Verification:

Log out of FreeScout, then log back in through CF Access. After CF Access passes, FreeScout should show a TOTP prompt before granting access to the inbox.

Recovery procedure (if authenticator app is lost):

1. Retrieve recovery codes from Infisical at /MooseQuest/freescout/FREESCOUT_TOTP_RECOVERY_CODES.
2. On the FreeScout login screen, after entering your password, click Lost your device? or Use recovery code.
3. Enter one recovery code (each code is single-use).
4. Once logged in, go to Profile > Security > Two-Factor Authentication and re-enroll with a new authenticator app.
5. Store the new recovery codes in Infisical, replacing the previous set.

If TOTP is not available on the deployed version:

Check FreeScout version: https://tickets.raxx.app/dashboard (visible in footer or Settings > General). FreeScout's built-in 2FA was introduced in version 1.8.x. If unavailable, document the gap in a comment on #718 and rely solely on CF Access WebAuthn from #695 until a FreeScout update can be applied.

Important: Do not close issue #718 until TOTP is active AND recovery codes are stored in Infisical. Documentation alone is not sufficient for this card.

Vault path reference for admin credentials

Branding and company settings (#720)

FreeScout's login page (tickets.raxx.app/login) is the operator-facing gateway. It is protected by Cloudflare Access and is not customer-reachable under normal operating conditions. Nevertheless, a minimal Raxx wordmark should be set so the login page does not display the default FreeScout logo.

Access decision: tickets.raxx.app/login is operator-only (behind CF Access — kris@moosequest.net allowlist only). Customers interact exclusively with support.raxx.app (portal, #651). Full branding is not launch-blocking but is a pre-launch should-do.

How to update company name and logo (operator action, ~10 min)

This is an operator action. Kristerpher must complete this in the FreeScout admin UI.

1. Log in to https://tickets.raxx.app.
2. Navigate to Settings (gear icon, top navigation bar).
3. Click General in the left sidebar.
4. In the Company Name field, enter: Raxx Support
5. In the Company Logo field: upload docs/design/brand/logos/raxx-wordmark-c.svg (the canonical Confidence Engine wordmark). If the field requires a raster image, export the SVG at 2x (300 px wide, white background if needed) and upload the PNG.
6. In the Favicon field (if present): upload console/app/static/favicon-32.png (32x32).
7. Click Save.

Verification:

Navigate to https://tickets.raxx.app/login (in an incognito window after passing CF Access). The FreeScout default logo should be replaced by the Raxx wordmark. The page title should read "Raxx Support" rather than "FreeScout".

Canonical brand assets:

Note on logo hosting for email: Do NOT use a repo-local path as the logo URL in email templates. Cloudflare Access protects tickets.raxx.app — any image URL that points to a CF-Access-gated resource will fail to render in external email clients. The branded email templates in freescout/templates/ inline the logo as a base64 data URI to avoid this. See docs/ops/runbooks/freescout-templates.md for the full email template deployment procedure (#712).

Outbound email template branding (#712)

Raxx-branded HTML + plaintext email templates for all outbound FreeScout emails live at:

freescout/templates/
  autoreply.html         — auto-reply sent when a customer creates a new ticket
  autoreply.txt          — plaintext fallback for auto-reply
  operator-reply.html    — sent when an operator replies to a customer
  operator-reply.txt     — plaintext fallback for operator reply
  notification.html      — ticket status change and system notifications to customers
  notification.txt       — plaintext fallback for notifications

Full deployment procedure, variable reference, durability notes, and revert steps: docs/ops/runbooks/freescout-templates.md

Template file locations on the Lightsail instance (verify on your FreeScout version):

After any FreeScout core update (git pull on the instance), re-apply the templates. See the templates runbook for the post-update checklist.

Paid module install and management

FreeScout's 10 paid modules (API & Webhooks, Reports, OAuth, Custom Fields, Tags, Workflows, Customization, Saved Replies, Slack, Custom Folders) are managed declaratively via terraform/freescout/modules.tf.

Normal path — terraform apply

# From repo root (requires infisical CLI + AWS credentials in environment):
make freescout-install
# Review the plan output, then:
cd terraform/freescout && terraform apply tfplan

Expected timing: 5–10 minutes for all 10 modules.

Terraform takes a Lightsail snapshot before and after the install batch. After apply completes, verify all 10 modules show status Active under https://tickets.raxx.app/modules.

Re-installing a single module (e.g., after license key rotation)

1. Update the vault secret at /MooseQuest/freescout/<NAME>_LICENSE_KEY.
2. Re-export just that TF_VAR_license_* variable.
3. Run terraform plan — only that module's null_resource shows as "must replace".
4. Run terraform apply.

Forcing a full reinstall of all modules

Bump module_version_pin in terraform/freescout/terraform.tfvars:

module_version_pin = "YYYY-MM-DD"  # new date

Next apply re-fires all 10 install+activate commands.

Break-glass: bash script fallback

If Terraform is unavailable (e.g., running directly on the instance without a local workstation), use scripts/ops/install-freescout-modules.sh. This script reads all license keys from Infisical at runtime and runs the same artisan commands. It is the operational reference that modules.tf was derived from. Do NOT delete it.

ssh -i ~/.ssh/lightsail_us_east_1.pem -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null admin@54.146.13.200
curl -fsSL https://raw.githubusercontent.com/raxx-app/TradeMasterAPI/main/scripts/ops/install-freescout-modules.sh \
  -o /tmp/install.sh
chmod +x /tmp/install.sh
sudo \
  INFISICAL_HOST="https://vault.raxx.app" \
  INFISICAL_PROJECT_ID="29b77751-f761-4afa-b3fa-2c842988f95c" \
  INFISICAL_CLIENT_ID="<from break-glass>" \
  INFISICAL_CLIENT_SECRET="<from break-glass>" \
  /tmp/install.sh

After using the break-glass script, run terraform plan to confirm state matches. If Terraform shows drift, either import the state or accept that the next apply will re-run install+activate (idempotent at the FreeScout layer).

Post-incident actions for any new failure mode

1. Update this runbook with the new failure mode
2. Add a terraform/README.md lessons-learned entry
3. File a type:reliability GitHub issue