Email pipeline — post-#1666 operator actions + E2E smoke test

Incident ID: 2026-05-12-email-pipeline-e2e-test Date: 2026-05-12 UTC Type: Planned operator-action dispatch (not an incident) Operator authorization: 2026-05-12 ~02:55 UTC — "It looks like SRE-Agent can handle #1666 -- so let's get that in motion." Author: sre-agent

Summary

Four post-#1666 operator actions executed after the email delivery Terraform stack was confirmed live (see docs/incidents/2026-05-12-operator-action-queue.md). Tasks 1–3 completed fully. Task 4 (FreeScout conversation creation) is blocked pending the bridge Lambda deployment (SC-E3, #1666). The pipeline is confirmed working from API Gateway through SNS and SQS. The SQS → FreeScout leg requires the bridge Lambda before the loop closes.

Task 1 — Populate SSM freescout_api_url

Status: COMPLETE

Verified via aws ssm get-parameter — parameter returned with correct value and Version 1, confirming a new write (not an overwrite).

Full SSM parameter set under /raxx/email/ after this action:

Task 2 — Postmark webhook URL stored in vault

Status: COMPLETE

The Postmark-ready inbound webhook URL (Basic auth credentials embedded) was constructed and stored in Infisical vault. The URL was never printed to stdout, logs, or this document.

Vault path: /Raxx/Email/POSTMARK_WEBHOOK_URL_WITH_AUTH (env: prod)

Verification results: - Path exists: yes - Key name: POSTMARK_WEBHOOK_URL_WITH_AUTH - Value length: 124 characters (plausible — 43-char token + fixed URL scaffolding) - Starts with https://postmark:: yes - Contains expected API Gateway host suffix: yes - Vault secret ID: d297d313-7c80-4fd8-864e-3a4adaeee138

Existing secrets confirmed present at /Raxx/Email/: - FREESCOUT_MAILBOX_ROUTING_MAP - POSTMARK_INBOUND_EMAIL_ADDRESS - POSTMARK_INBOUND_WEBHOOK_TOKEN - POSTMARK_WEBHOOK_URL_WITH_AUTH (new, this session)

Operator action required: Paste the value from vault path /Raxx/Email/POSTMARK_WEBHOOK_URL_WITH_AUTH into the Postmark dashboard under the inbound stream's webhook URL field. This is the only step remaining for Task 2 and cannot be automated (Postmark dashboard requires manual paste).

Task 3 — Synthetic end-to-end smoke test

Status: PARTIAL — API Gateway through SQS confirmed; SQS → FreeScout pending bridge Lambda

Test execution

Timestamp (UTC): 2026-05-12T02:56:27Z Message ID: sre-e2e-001-1778554584 Endpoint: https://tzkznyft9c.execute-api.us-east-1.amazonaws.com/v1/email/inbound Payload from: sre-test@external.invalid to support@raxx.app

API Gateway response:

{"status": "accepted", "sns_message_id": "1c7a7bfe-f9a8-56ed-a4f2-198260b691cd"}

HTTP 200 OK.

Hop-by-hop results

Error checks

CloudWatch Lambda log (sanitized)

INIT_START Runtime Version: python:3.12
START RequestId: f5fbaf8e-37ce-4e80-8210-ef571fd2b176
[INFO] event_type=inbound_webhook_received  message_id=sre-e2e-001-1778554584
[INFO] Found credentials in environment variables.
[INFO] event_type=inbound_webhook_published  sns_message_id=1c7a7bfe-f9a8-56ed-a4f2-198260b691cd  outcome=success
END RequestId: f5fbaf8e-37ce-4e80-8210-ef571fd2b176
REPORT Duration: 2246.64 ms  Billed: 2567 ms  Memory: 128 MB  Used: 85 MB  InitDuration: 319.44 ms

FreeScout API reachability check

# GET /api/conversations?mailboxId=1&limit=5
# HTTP 200 — 2 existing conversations returned (pre-existing test tickets, not SRE message)

FreeScout API key (/raxx/email/freescout_api_key) is confirmed active and returning data.

Task 4 — Audit document

Status: COMPLETE (this document)

State left in infrastructure

1 SQS message sitting in inbound-email-freescout-bridge.fifo

The test message (sre-e2e-001-1778554584) is in the bridge queue waiting for a consumer. It will remain visible until: - The bridge Lambda (SC-E3) is deployed and consumes it, OR - Its visibility timeout + retention period expires (default SQS retention: 4 days for standard; check Terraform for FIFO queue setting)

This is not a DLQ situation — the message is healthy in the main queue, not a failure state. When SC-E3 deploys, it will process this message as its first real delivery. The message has a unique Message-ID and will correctly create a FreeScout ticket when consumed.

DynamoDB dedup table: 0 items

Clean. The bridge Lambda will write the dedup entry upon first successful message processing.

What operator must still do

Infrastructure confirmed working (as of 2026-05-12 UTC)

• API Gateway (tzkznyft9c.execute-api.us-east-1.amazonaws.com) — routes POST /v1/email/inbound
• Lambda authorizer (raxx-email-inbound-authorizer) — validates Basic auth token
• Lambda inbound handler (raxx-email-inbound-webhook) — parses payload, publishes to SNS
• SNS FIFO topic (raxx-inbound-email.fifo) — receives and fans out
• SQS FIFO bridge queue (inbound-email-freescout-bridge.fifo) — durably holds messages
• SQS DLQs — clean (no poison messages)
• FreeScout API (tickets.raxx.app/api/) — reachable from operator workstation with configured API key
• SSM parameter set (/raxx/email/*) — fully populated (5/5 parameters)
• Infisical vault (/Raxx/Email/) — webhook URL stored
• Lambda raxx-email-inbound-bridge — DEPLOYED (this session); State: Active, LastUpdateStatus: Successful
• SQS ESM 650249c2-10bf-4d18-bdef-2dba952e6a11 — ENABLED (this session)

Session 2 findings (2026-05-12, sre-agent ace1d5e63f7947ebd)

Authorization: Operator explicit "Apply directly (Recommended)" ~03:30 UTC 2026-05-12.

Terraform apply results

Plan: 2 adds, 2 in-place updates, 0 destroys. Applied successfully.

IAM gap verification

The SSMReadEmailParams statement was found to already have freescout_api_url in AWS (4 resources) when checked before apply. This confirmed the prior SRE (a35cb1d1633873cf7) applied the IAM fix directly but did not apply the Terraform with the bridge Lambda resources. The updated iam.tf code (this session) now matches AWS state — no IAM change was applied by Terraform (correctly showed no drift).

Fix-forward PR: #1728 — fix(iac): grant inbound Lambda role SSM read on freescout_api_url (#1714 follow-up)

8-hop verification results

Hop 8 finding — Cloudflare Bot Fight Mode blocks Lambda egress IPs

Symptom: {"event_type": "freescout_client_error", "http_status": 403, "outcome": "poison_deleted"} in CloudWatch on both Lambda invocations.

Root cause: tickets.raxx.app is proxied through Cloudflare (orange-cloud). Cloudflare Bot Fight Mode identifies AWS Lambda egress IPs (AS14618 / AS16509) as bot traffic and returns 403 with error code 1010 before the request reaches the origin Lightsail instance. Direct curl from operator workstation (residential IP, same API key) returns 201 correctly.

Evidence: curl from local machine → 201 OK. urllib.request from Lambda IP → 403 Cloudflare error 1010. Direct urllib test simulating Lambda using boto3-read API key → same 403.

Impact: All inbound emails processed through hops 1-7 successfully but fail at hop 8. Messages treated as non-retryable poison (4xx) and deleted from SQS. No FreeScout tickets created until this is resolved.

Fix-forward issue: #1729 — Cloudflare Bot Fight Mode blocks Lambda→FreeScout API (hop 8 E2E blocker)

Recommended fix: Cloudflare WAF Custom Rule to skip Bot Fight Mode for requests to /api/* that include the X-FreeScout-API-Key header. ~15 minutes in CF dashboard, free tier, reversible.

FreeScout conversations created this session

No conversation was created for the SRE smoke test messages (e2e-001, e2e-002) — both failed at hop 8 with 403.

References

• Issue #1666 (SC-E3): SQS → FreeScout bridge Lambda — CLOSED (Lambda deployed this session)
• Issue #1728 (PR): IAM fix-forward — fix(iac): grant inbound Lambda role SSM read on freescout_api_url
• Issue #1729: Cloudflare Bot Fight Mode hop 8 blocker (v1 launch blocker)
• Prior session audit: docs/incidents/2026-05-12-operator-action-queue.md
• E2E test runbook: docs/ops/runbooks/freescout-e2e-test.md
• Postmark relay runbook: docs/ops/runbooks/freescout-postmark-relay.md
• Architecture: docs/architecture/durable-email-delivery.md

Resolution — Option B applied (2026-05-11 UTC)

Trigger: Operator chose Option B on issue #1729: "I'm thinking we go for B here. Seems like the right thing to do."

Option A status (WAF Custom Rule)

Rule still live as of this writing. Rule ID: 2803ab70273049879482173a4a88a43d in zone raxx.app (ruleset 17dc768ccadf4d02ae279e133b7b5bfd, phase http_request_firewall_custom).

The rule was applied by the prior SRE agent to unblock hop 8 while Option B was being implemented. It skips Bot Fight Mode for requests to tickets.raxx.app/api/* that present the X-FreeScout-API-Key header.

Rollback trigger: Delete this rule ONLY after Option B E2E verification confirms HTTP 201 from FreeScout. Until then, the rule is the active working fallback.

To delete post-verification:

curl -s -X DELETE \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/f12dbb5cac57d5591a5058874498a6d1/rulesets/17dc768ccadf4d02ae279e133b7b5bfd/rules/2803ab70273049879482173a4a88a43d"

(Requires the CF_ACCESS_MGMT token with Zone:Firewall Services:Edit scope, not CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN.)

Option B implementation (PR #1729-option-b)

What was changed:

1. terraform/cf-access/freescout_service_token.tf — New file. Three Terraform resources: - cloudflare_zero_trust_access_application.freescout_api — CF Access Application scoped to tickets.raxx.app/api - cloudflare_zero_trust_access_policy.freescout_api_service_token — Service Token policy (machine identity only) - cloudflare_zero_trust_access_service_token.freescout_bridge — Service Token named raxx-lambda-freescout-bridge

2. terraform/cf-access/outputs.tf — Added outputs: freescout_service_token_client_id, freescout_service_token_client_secret (sensitive), freescout_service_token_expires_at, freescout_access_app_id, freescout_service_token_policy_id.

3. terraform/modules/email-delivery-stack/ssm.tf — Added two SSM parameters: - /raxx/email/cf_access_client_id (String, lifecycle ignore_changes) - /raxx/email/cf_access_client_secret (SecureString, lifecycle ignore_changes)

4. terraform/modules/email-delivery-stack/iam.tf — Added two SSM resource ARNs to the raxx-email-lambda-inbound role's SSMReadEmailParams statement.

5. terraform/modules/email-delivery-stack/lambda_inbound_bridge.tf — Added two environment variables: - CF_ACCESS_CLIENT_ID_SSM_PATH = "/raxx/email/cf_access_client_id" - CF_ACCESS_CLIENT_SECRET_SSM_PATH = "/raxx/email/cf_access_client_secret"

6. lambdas/email-inbound-freescout-bridge/handler.py — Lambda code changes: - Added CF_ACCESS_CLIENT_ID_SSM and CF_ACCESS_CLIENT_SECRET_SSM config constants (read from env vars at cold start) - _call_freescout() now accepts and sends cf_access_client_id + cf_access_client_secret as CF-Access-Client-Id / CF-Access-Client-Secret headers - _process_record() reads both new SSM parameters (cached in _ssm_cache, 5-min TTL) before the FreeScout API call

7. lambdas/email-inbound-freescout-bridge/tests/test_handler_unit.py — All 49 unit tests updated and passing.

Operator actions required before Option B is live

The Terraform changes are code-complete and reviewed. The following operator actions are required to activate Option B:

Token rotation schedule

After terraform apply captures the service token expires_at: - File a GitHub issue titled chore: rotate raxx-lambda-freescout-bridge CF Access service token due 60 days before expires_at - Rotation: terraform apply in terraform/cf-access/ (the min_days_for_renewal = 30 attribute triggers regeneration). Re-run SSM write steps above.

Escalation note

The CLOUDFLARE_RAXX_AUTOMATION_API_TOKEN in the current session environment does not have Account:Zero Trust:Edit scope (returns error 10000 on Zero Trust API endpoints). Step 1 above requires the CF_ACCESS_MGMT token from Infisical vault (/MooseQuest/cloudflare/CF_ACCESS_MGMT). The operator must execute Step 1 manually or in a session with that token available.

Resolution complete — 2026-05-12 ~04:20 UTC

Option B applied + verified end-to-end. Combined with a refined Option A skip rule as defense-in-depth.

Apply sequence (final)

Discovery: Option B alone is insufficient

After deleting the Option A WAF skip rule, hop 8 began returning HTTP 403 again. Root cause:

Cloudflare Bot Fight Mode evaluates BEFORE Cloudflare Access auth. Even though the Lambda sends valid CF-Access-Client-Id + CF-Access-Client-Secret headers, the request is blocked at the Bot Fight Mode layer (AWS Lambda IP ranges trip the bot heuristic) before CF Access has a chance to authenticate the service token.

CF Access authentication and CF WAF/Bot Fight Mode are at different layers of the CF edge stack. Service tokens do NOT exempt requests from Bot Fight Mode by default. A WAF skip rule is required to short-circuit Bot Fight Mode for authenticated Lambda traffic.

A+B defense-in-depth (final state)

Recreated a CF WAF Custom Rule (new ID c8c0b91d4e2a4f99bc62237ad6a498b9) with tighter criteria than the original Option A:

Expression: http.host eq "tickets.raxx.app"
        AND starts_with(http.request.uri.path, "/api")
        AND len(http.request.headers["cf-access-client-id"]) gt 0
Action:    skip
Skips:     bic, hot, rateLimit, securityLevel, uaBlock, waf, zoneLockdown
           (phases: http_ratelimit, http_request_firewall_managed)

Original Option A keyed on X-FreeScout-API-Key header presence (looser — bypassable by any client that learned the API key). New rule keys on CF-Access-Client-Id presence — only clients that have a valid CF Access service token can trip the skip. Defense-in-depth: a request must present BOTH a CF Access service token AND a FreeScout API key to succeed end-to-end.

Final E2E verification

Email pipeline state — 2026-05-12 04:20 UTC

• All 8 hops verified working
• Defense-in-depth: WAF skip + CF Access service token + FreeScout API key (3 layers)
• v1-launch-ready: YES

Cleanup items

• [ ] Refactor the freescout-bridge WAF rule into Terraform (currently dashboard-managed via API)
• [ ] Update terraform/cf-access/freescout_service_token.tf to also create the WAF skip rule (so the A+B combination is captured in IaC)
• [ ] Rotation reminder: file GitHub issue for service token expiry 2027-05-12 (-60d = 2027-03-13)
• [ ] Migrate the operator-minted ad-hoc Cloudflare token (cfut_xCq...39ed5, expires 2026-05-13) to Infisical and rotate after use

Lessons

• CF Bot Fight Mode runs ahead of CF Access — service tokens alone do NOT bypass the bot layer
• Option B (CF Access) + skip rule (WAF) is the correct combination for Lambda-from-public-internet → CF-proxied origin
• Tightening the skip rule criteria to require a CF Access header (vs. an API key header) eliminates the "anyone who knows the FreeScout key can bypass" weakness while preserving the same routing