Raxx · internal docs

internal · gated

Inbound Email Pipeline — Google Workspace → Postmark → FreeScout

System: Customer/SLA inbound email for @raxx.app support addresses → FreeScout at tickets.raxx.app Owner: operator (Kristerpher) + SRE Status: LIVE — first real inbound email → ticket confirmed 2026-07-17 Supersedes the inbound sections of: docs/architecture/email-routing.md, docs/ops/runbooks/freescout-e2e-test.md (the alias-domain + generic-Routing design described there is obsolete; see "How it actually works" below).

The one-line summary: a customer emails support@raxx.app → a Google Workspace Default routing rule accepts it and rewrites the envelope recipient to Postmark's inbound address → Postmark (Pro plan) processes it and POSTs to our API Gateway → a Lambda chain lands it as a FreeScout ticket. All four support addresses share one Postmark inbound address; the bridge sorts them into mailboxes by the To: header.

Architecture (the working path)

Customer email to support@raxx.app
  └─► Google MX (raxx.app — secondary domain in the moosequest.net Workspace)
        └─► Gmail DEFAULT ROUTING rule (NOT the generic "Routing" setting):
              • Envelope recipient = support@raxx.app  (single recipient match)
              • Change envelope recipient → Replace recipient →
                  bb43f79c8bfe23d564079fb431583c30@inbound.postmarkapp.com
              • Add X-Gm-Original-To header
              • Affects: Inbound + Internal-Receiving; account types incl. Unrecognized
        └─► Postmark inbound (server `raxx-production`, id 19052422, PRO plan)
              • one inbound address (hash) for the whole server
              • InboundHookUrl → API Gateway (HTTP Basic auth in the URL userinfo)
        └─► API Gateway  tzkznyft9c.execute-api.us-east-1.amazonaws.com  /v1/email/inbound
              └─► authorizer Lambda  raxx-email-inbound-authorizer  (3 checks — see gotcha #5)
              └─► webhook Lambda     raxx-email-inbound-webhook
        └─► SNS FIFO  raxx-inbound-email.fifo
        └─► SQS FIFO  inbound-email-freescout-bridge.fifo
        └─► bridge Lambda  raxx-email-inbound-bridge
              • reads To: from Postmark ToFull[0].Email
              • mailbox routing map (SSM /raxx/email/freescout_mailbox_routing_map):
                  support@raxx.app=1  ops@raxx.app=2  account-management@raxx.app=3  privacy@raxx.app=5
              • POST tickets.raxx.app/api/conversations  (CF Access non_identity + FreeScout API key)
        └─► FreeScout ticket in the mapped mailbox

One inbound address, many senders — how the four stay separated

Postmark provides one inbound address per server. All four support addresses Replace-recipient to that same hash. Differentiation happens downstream: the routing rule rewrites only the envelope, so Postmark still sees the original To: header; the bridge reads it and maps to the right FreeScout mailbox. This is the "generated / many addresses, one inbound endpoint" pattern from the Postmark guide (references below). Adding a new SLA address = one more Default routing rule pointing at the same hash + one entry in the routing map.

CC nuance: the bridge routes on ToFull[0].Email. If an address appears only in Cc (not To), it currently falls back to mailbox 1 (support). Fine for the common case; enhance the bridge to scan all recipients if precise Cc routing is needed.

Google side: Default routing vs Routing (this was the crux)

Google Workspace has two distinct settings under Apps → Google Workspace → Gmail:

Setting What it does Accepts unknown addresses?
Routing Re-handles mail Google has already accepted for a known recipient (a real user/group). Rewrites/redirects. No — a non-existent recipient bounces before this runs.
Default routing Governs delivery for addresses that don't map to a user/group — the catch-all / unrecognized-recipient layer. Accepts and reroutes. Yes — this is the one that stops the bounce.

Use Default routing. On a secondary domain, once you delete the group, the support address has no mailbox — so acceptance must come from Default routing, or Google returns 550 and bounces (this is exactly what happened when groups were deleted with only a generic Routing rule in place). See the Stack Overflow reference below.

Exact rule config (per the Postmark guide + hard-won experience): - Email messages to affect: Inbound and Internal - Receiving (the commonly-missed one — with DMARC alignment on the sender domain Google can class some mail as internal). - Envelope recipient → Single recipient → the address (scope it; do NOT make a domain-wide catch-all, or spam to random @raxx.app addresses becomes tickets). - Modify message → Add X-Gm-Original-To header (preserves the true recipient). - Change envelope recipient → Replace recipient → bb43f79c...@inbound.postmarkapp.com. - Account types: include Unrecognized (that's what does the work once the group is gone).

Do not also keep a generic Routing rule for the same address — one or the other.

Postmark side

The five root causes we peeled off (2026-07 restoration)

Inbound had never actually worked end-to-end; it failed at five independent layers, each hidden behind the previous:

  1. Google acceptance — after alias→secondary domain conversion, support addresses were no longer auto-accepted; groups intercepted mail (a group is a terminal delivery target that swallows the message, so routing rules never fire). Fix: delete the group; use a Default routing rule (accepts unrecognized + rewrites to Postmark).
  2. Postmark plan gate — the account was on Basic, which locks inbound streams; mail was accepted at SMTP then dropped. Fix: upgrade to Pro (+$1.50/mo).
  3. Authorizer Date-header bug — raxx-email-inbound-authorizer Check 3 (replay protection) assumed Postmark sends an RFC 2822 Date header. It does on outbound delivery events but not on inbound webhooks → every inbound POST Denied 403, and API Gateway's 300s authorizer cache then buried Postmark's auto-retries. Fix: skip Check 3 when Date is absent (mirrors the existing HMAC skip). PR #4255.
  4. Bridge BFM 403 — raxx-email-inbound-bridge called tickets.raxx.app/api with urllib's default UA; Cloudflare Bot Fight Mode (runs before CF Access) blocked it 403 and the Lambda dropped the message as poison. Fix: explicit User-Agent. PR #4255.
  5. FreeScout storage perms (earlier) — root-owned Laravel cache dirs 500'd the API; fixed by chown www-data + cache clear.

Durable follow-up: a Cloudflare WAF skip rule on tickets.raxx.app keyed on CF-Access-Client-Id would make the API robust to any non-AWS automation client (folds into the BFM re-enable work). The bridge's explicit UA is the current safeguard.

Outbound (ticket replies) — every mailbox needs Postmark SMTP

FreeScout sends an agent's reply using the mailbox's own outgoing config, and a mailbox defaults to out_method = 1 (PHP mail()). On this box PHP mail() hands off to local Exim, which cannot relay to remote domains (R=nonlocal: Mailing to remote domains not supported) — the reply freezes in the Exim queue and never delivers, silently. This bit ops@/account-management@/privacy@ on 2026-07-17 (only support@ had been wired).

Every mailbox that will send replies must be set to Postmark SMTP, matching mailbox 1:

out_method = 3 (SMTP)   out_server = smtp.postmarkapp.com   out_port = 587
out_encryption = 3 (STARTTLS)   out_username = out_password = <Postmark server token>

Set it in FreeScout admin (Mailbox → Connection Settings → Sending) or by copying mailbox 1's out_* columns (the out_password blob is Laravel-encrypted with APP_KEY and copies verbatim between rows). Clear caches as www-data, never as root (root-owned cache dirs 500 the whole site). Verify: SELECT id,email,out_method,out_server FROM mailboxes; → all sending mailboxes show out_method=3 + smtp.postmarkapp.com.

Outbound sender identity: raxx.app domain DKIM is verified in Postmark, so any @raxx.app From address delivers; no-reply@raxx.app has 189+ delivered as proof. Watch the Postmark outbound suppression list — a recipient that hard-bounced during testing (e.g. from May) gets suppressed and silently drops replies until removed.

Known-dark (2026-07-17): FreeScout's webhooks back to Raptor (api.raxx.app/api/internal/freescout-webhook/{audit,agent-replied}) are 403'd by Cloudflare Bot Fight Mode on every event — same class as the inbound-bridge BFM bug. The audit trail of ticket events is dark until the CF WAF skip rule (keyed on CF-Access-Client-Id) lands (#987 / [[feedback_cf_access_does_not_bypass_bot_fight_mode]]). Does not affect email in/out.

Add a new SLA-bearing address (e.g. abuse@, dpo@)

  1. Google: duplicate the support@ Default routing rule; change only the Envelope recipient to the new address. Replace-recipient → the same Postmark inbound hash. Delete any group for the address first.
  2. Routing map: add "<address>": <freescout_mailbox_id> to SSM /raxx/email/freescout_mailbox_routing_map.
  3. FreeScout: ensure the target mailbox exists.
  4. Send a test; confirm the ticket lands (see verification below).

Verify end-to-end

References