FreeScout TLS / certificate renewal runbook

System: FreeScout helpdesk — tickets.raxx.app Owner: operator Last incident: 2026-06-19 (see RCA docs/incidents/2026-06-19-freescout-526-ssl-strict.md) Last reviewed: 2026-06-19

Architecture overview

tickets.raxx.app uses a two-layer TLS model. There is NO certbot on the Lightsail instance. This is correct by design.

Installed on origin: - Cert: /etc/ssl/certs/cf-origin.pem (Cloudflare Origin Certificate, 15-year validity) - Key: /etc/ssl/private/cf-origin.key (RSA private key, generated 2026-06-19, root:root 600) - CF cert ID: 54595718253663594481859538935220508661437382197 (stored in vault: CF_ORIGIN_CERT_TICKETS_RAXX_APP_ID at /MooseQuest/cloudflare) - Vault expiry companion: CF_ORIGIN_CERT_TICKETS_RAXX_APP_EXPIRES = 2041-06-15T19:36:00Z

Why a CF Origin Certificate (not snakeoil): The raxx.app zone SSL mode is full_strict. Full Strict validates the origin certificate chain — self-signed (snakeoil) certs are rejected with HTTP 526. A Cloudflare Origin Certificate is issued by the Cloudflare Origin CA and is accepted by CF Full Strict. It is NOT trusted by browsers directly (users never see it — they see the LE edge cert).

Why Cloudflare handles the public cert: Cloudflare Universal SSL is provisioned and renewed automatically for all proxied (orange-cloud) DNS records. No operator action is required unless the record is un-proxied or the zone's Universal SSL feature is disabled.

How to tell TLS is broken

Routine verification procedure

Run this check monthly or after any Cloudflare / Lightsail change.

1. Public-facing cert (Cloudflare Universal SSL)

# From any machine with openssl
echo | openssl s_client \
  -connect tickets.raxx.app:443 \
  -servername tickets.raxx.app 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

Expected output:

subject=CN=raxx.app
issuer=C=US, O=Let's Encrypt, CN=E7
notBefore=<date>
notAfter=<date 90 days out>

If notAfter is within 30 days, see "Fix: Cloudflare Universal SSL stalled" below.

2. Confirm Cloudflare is proxying

curl -sI https://tickets.raxx.app/ | grep -i "cf-ray\|server"

Expected: server: cloudflare and a cf-ray: header. If absent, the DNS record has been set to DNS-only — re-enable the Cloudflare proxy (orange cloud) for the tickets A record.

3. Origin cert (CF Origin Cert — not user-visible but required for Full Strict)

ssh -i /tmp/lightsail_us_east_1.pem \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null \
  admin@54.146.13.200 \
  'sudo openssl s_client -connect localhost:443 -servername tickets.raxx.app \
    </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates'

Expected:

subject=O=CloudFlare, Inc., OU=CloudFlare Origin CA, CN=CloudFlare Origin Certificate
issuer=C=US, O=CloudFlare, Inc., OU=CloudFlare Origin SSL Certificate Authority, ...
notBefore=Jun 19 19:36:00 2026 GMT
notAfter=Jun 15 19:36:00 2041 GMT

If the output shows CN=ip-172-26-11-76 (snakeoil), the origin cert was reverted or the Apache vhost was reset — see Failure Mode E.

4. Confirm zone SSL mode is Full Strict

# Requires CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN from vault /MooseQuest/cloudflare
ZONE_ID="f12dbb5cac57d5591a5058874498a6d1"
curl -sS \
  -H "Authorization: Bearer ${CF_TOKEN}" \
  -H "User-Agent: raxx-ops/1.0" \
  "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/settings/ssl" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["result"]["value"])'

Expected: strict. If full or other value, zone security posture has drifted — investigate.

5. Confirm certbot is NOT installed (expected state)

ssh -i /tmp/lightsail_us_east_1.pem \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null \
  admin@54.146.13.200 \
  'which certbot 2>/dev/null || echo "certbot not installed — expected"'

Expected: certbot not installed — expected.

Known failure modes

Failure mode E: CF Origin Certificate missing or expired — Full Strict returns 526

Symptom: - curl -sI https://tickets.raxx.app/ returns 526 Invalid SSL Certificate - CF-Ray header present (CF is proxying — the 526 is from CF rejecting the origin cert) - Origin Apache is healthy (active, port 443 listening) - Origin is serving snakeoil or an expired cert

Cause: The origin is not presenting a chain-trusted cert. Full Strict requires a CF Origin Certificate (or valid LE cert). Common triggers: - CF Origin Certificate not installed (new instance, snakeoil still active) - CF zone SSL mode changed to strict without a valid origin cert pre-installed - Origin cert expired (CF Origin Certs last 15 years — unlikely until 2041) - Apache vhost reset or redeployed pointing at snakeoil

Diagnosis:

# Check what the origin is serving
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null admin@54.146.13.200 \
  'sudo openssl s_client -connect localhost:443 -servername tickets.raxx.app \
    </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates'
# If CN=ip-172-26-11-76: snakeoil — origin cert not installed
# If CN=CloudFlare Origin Certificate + notAfter is past: expired — renew

Fix — issue a new CF Origin Certificate:

Prerequisites: - SSH key: aws ssm get-parameter --name /raxx/freescout/ssh_key --with-decryption --region us-east-1 --query Parameter.Value --output text > /tmp/lightsail_us_east_1.pem && chmod 600 /tmp/lightsail_us_east_1.pem - A CF API token with Zone:SSL and Certificates:Write for raxx.app zone. Mint via CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN (vault /MooseQuest/cloudflare):

# Read automation token from vault
AUTOMATION_TOKEN=$(infisical secrets get CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN \
  --env=prod --path=/MooseQuest/cloudflare --plain --domain=https://vault.raxx.app \
  --projectId="${INFISICAL_PROJECT_ID}")

ZONE_ID="f12dbb5cac57d5591a5058874498a6d1"
EXPIRES_AT=$(date -u -v+1d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d "+1 day" +%Y-%m-%dT%H:%M:%SZ)

MINT_RESP=$(curl -sS \
  -H "Authorization: Bearer ${AUTOMATION_TOKEN}" \
  -H "Content-Type: application/json" \
  -H "User-Agent: raxx-ops/1.0" \
  -X POST "https://api.cloudflare.com/client/v4/user/tokens" \
  -d "{
    \"name\": \"CF_SSL_CERT_ORIGIN_CA_RAXX_APP_TMP\",
    \"policies\": [{
      \"effect\": \"allow\",
      \"resources\": {\"com.cloudflare.api.account.zone.${ZONE_ID}\": \"*\"},
      \"permission_groups\": [
        {\"id\": \"c03055bc037c4ea9afb9a9f104b7b721\"},
        {\"id\": \"c8fed203ed3043cba015a93ad1616f1f\"}
      ]
    }],
    \"not_before\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
    \"expires_on\": \"${EXPIRES_AT}\"
  }")

SSL_TOKEN_ID=$(echo "$MINT_RESP" | python3 -c 'import sys,json; print(json.load(sys.stdin)["result"]["id"])')
SSL_TOKEN_VALUE=$(echo "$MINT_RESP" | python3 -c 'import sys,json; print(json.load(sys.stdin)["result"]["value"])')

Generate a new key and CSR on the origin (if /etc/ssl/private/cf-origin.key exists and is not lost, skip key gen):

ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null admin@54.146.13.200 '
sudo openssl genrsa -out /etc/ssl/private/cf-origin.key 2048
sudo chmod 600 /etc/ssl/private/cf-origin.key
sudo openssl req -new \
  -key /etc/ssl/private/cf-origin.key \
  -out /tmp/cf-origin.csr \
  -subj "/C=US/ST=PA/L=Erie/O=MooseQuest LLC/CN=tickets.raxx.app"
sudo cat /tmp/cf-origin.csr
'

Issue the cert (capture CSR from above, paste into API call):

CSR="<paste CSR PEM from above>"
CERT_RESP=$(curl -sS \
  -H "Authorization: Bearer ${SSL_TOKEN_VALUE}" \
  -H "Content-Type: application/json" \
  -H "User-Agent: raxx-ops/1.0" \
  -X POST "https://api.cloudflare.com/client/v4/certificates" \
  -d "{
    \"csr\": $(python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))' <<< "$CSR"),
    \"hostnames\": [\"tickets.raxx.app\"],
    \"request_type\": \"origin-rsa\",
    \"requested_validity\": 5475
  }")

CERT_ID=$(echo "$CERT_RESP" | python3 -c 'import sys,json; print(json.load(sys.stdin)["result"]["id"])')
CERT_PEM=$(echo "$CERT_RESP" | python3 -c 'import sys,json; print(json.load(sys.stdin)["result"]["certificate"])')
CERT_EXPIRY=$(echo "$CERT_RESP" | python3 -c 'import sys,json; print(json.load(sys.stdin)["result"]["expires_on"])')
echo "Cert ID: $CERT_ID  Expires: $CERT_EXPIRY"

Install on origin:

ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null admin@54.146.13.200 \
  "sudo tee /etc/ssl/certs/cf-origin.pem > /dev/null" <<< "$CERT_PEM"

# If vhost still points at snakeoil, update it:
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null admin@54.146.13.200 '
sudo sed -i "s|SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem|SSLCertificateFile /etc/ssl/certs/cf-origin.pem|" \
  /etc/apache2/sites-available/freescout-ssl.conf
sudo sed -i "s|SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key|SSLCertificateKeyFile /etc/ssl/private/cf-origin.key|" \
  /etc/apache2/sites-available/freescout-ssl.conf
sudo apache2ctl configtest && sudo systemctl reload apache2
'

Post-install:

# Revoke the temporary SSL token
curl -sS -H "Authorization: Bearer ${AUTOMATION_TOKEN}" -H "User-Agent: raxx-ops/1.0" \
  -X DELETE "https://api.cloudflare.com/client/v4/user/tokens/${SSL_TOKEN_ID}"

# Write new cert metadata to vault
infisical secrets set \
  "CF_ORIGIN_CERT_TICKETS_RAXX_APP_ID=${CERT_ID}" \
  "CF_ORIGIN_CERT_TICKETS_RAXX_APP_EXPIRES=${CERT_EXPIRY}" \
  --env=prod --path=/MooseQuest/cloudflare \
  --domain=https://vault.raxx.app --projectId="${INFISICAL_PROJECT_ID}"

Verification:

# Must be 302 (login redirect) — NOT 526
curl -sS -o /dev/null -w "%{http_code}" https://tickets.raxx.app/
# Expected: 302

# CF-Ray present
curl -sI https://tickets.raxx.app/ | grep -i "cf-ray"

# Origin cert is CF Origin Cert (not snakeoil)
ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null admin@54.146.13.200 \
  'sudo openssl s_client -connect localhost:443 -servername tickets.raxx.app \
    </dev/null 2>/dev/null | openssl x509 -noout -issuer'
# Expected: issuer=...CloudFlare Origin SSL Certificate Authority...

Next renewal: 2041-06-15. Add a calendar reminder 90 days prior (2041-03-17). Toil reduction follow-up: add CF Origin Cert to terraform/freescout/ stack as a cloudflare_origin_ca_certificate resource so renewal is IaC-managed.

Failure mode A: Cloudflare Universal SSL stalled

Symptom: - openssl s_client from outside shows an expired cert or a CF edge error cert - Browser shows a security warning for tickets.raxx.app - notAfter on the public cert is in the past or within 7 days

Cause: Cloudflare Universal SSL auto-renewal failed. Rare. Common triggers: zone plan downgrade, CAA DNS record blocking Let's Encrypt, or CF temporarily unable to reach ACME servers.

Fix: 1. CF dashboard → raxx.app zone → SSL/TLS → Edge Certificates → Universal SSL: confirm "Active" 2. Check CAA records: dig CAA tickets.raxx.app — should return no records or records permitting letsencrypt.org 3. If stalled: disable then re-enable Universal SSL (triggers re-issuance) 4. Wait up to 24h for CF to provision the new cert

Verification:

echo | openssl s_client -connect tickets.raxx.app:443 \
  -servername tickets.raxx.app 2>/dev/null \
  | openssl x509 -noout -dates

Failure mode B: DNS record switched to DNS-only (grey cloud)

Symptom: - curl -sI https://tickets.raxx.app/ shows no CF-Ray header - Browser connects directly to the Lightsail IP - CF Access gate does not appear

Cause: Someone changed the DNS record for tickets.raxx.app from proxied to DNS-only.

Fix:

# Via CF dashboard: DNS → records → tickets A record → toggle to proxied
# Or via Terraform:
cd terraform/freescout
terraform apply -target=cloudflare_record.tickets

Verification: curl -sI https://tickets.raxx.app/ | grep cf-ray returns a value.

Failure mode C: Apache not listening on port 443

Symptom: Cloudflare returns 525 SSL Handshake Failed

Cause: Apache SSL module disabled, SSL vhost not enabled, or Apache stopped.

Fix:

ssh -i /tmp/lightsail_us_east_1.pem -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null admin@54.146.13.200 '
sudo systemctl status apache2
sudo apache2ctl -M | grep ssl
# If ssl module missing:
sudo a2enmod ssl && sudo systemctl restart apache2
# If freescout-ssl.conf not enabled:
ls /etc/apache2/sites-enabled/
sudo a2ensite freescout-ssl.conf && sudo systemctl restart apache2
'

Failure mode D: Certbot installed unexpectedly and modified Apache vhosts

Context: Certbot should NOT be on this instance. If found, it was installed manually.

Impact: Certbot HTTP-01 renewal requires port 80 from Let's Encrypt IPs. Lightsail firewall restricts port 80 to Cloudflare IPs only. Auto-renewal WILL fail silently.

Fix options (escalate to operator): 1. Remove certbot; restore CF Origin Certificate to Apache vhost (preferred) 2. Keep certbot; switch to DNS-01 challenge with CF API token — significant IaC scope

Emergency stop

If TLS is broken and cannot be resolved quickly, temporarily set the zone to full (not strict) to restore access while the origin cert issue is diagnosed:

# TEMPORARY ONLY — do NOT leave at 'full' permanently
# Restores access but removes origin cert chain validation
CF_TOKEN="<CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN from vault>"
curl -sS -X PATCH \
  -H "Authorization: Bearer ${CF_TOKEN}" \
  -H "Content-Type: application/json" \
  -H "User-Agent: raxx-ops/1.0" \
  "https://api.cloudflare.com/client/v4/zones/f12dbb5cac57d5591a5058874498a6d1/settings/ssl" \
  -d '{"value":"full"}'
# Expected: {"success":true,"result":{"value":"full",...}}

After using the emergency stop, immediately install a CF Origin Certificate (Failure Mode E above) and restore to full_strict.

Monitoring

• Synthetic probe (action item from this incident): curl https://tickets.raxx.app/ alerting on 526/525 — see docs/ops/runbooks/synth-probes.md.
• Zone SSL mode drift check (action item): daily GET /zones/{id}/settings/ssl poll confirming strict.
• Cert expiry: CF Origin Cert expires 2041-06-15. Next check: 2041-03-17.

Escalation

Wake the operator when: - Public cert is expired and CF Universal SSL re-issuance has been tried and failed after 24h - Origin Apache is unresponsive to port 443 and Apache restart does not restore it - Zone SSL mode needs to remain downgraded to full for more than 1 hour - Certbot found installed and has caused a renewal loop or broken Apache config

References

• RCA: docs/incidents/2026-06-19-freescout-526-ssl-strict.md
• Related runbook: docs/ops/runbooks/freescout.md
• Terraform module: terraform/freescout/
• CF Origin Certificates: https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/
• CF API — issue cert: POST https://api.cloudflare.com/client/v4/certificates
• Vault cert metadata: /MooseQuest/cloudflare/CF_ORIGIN_CERT_TICKETS_RAXX_APP_ID
• Issue #715 (cert monitoring — open)