Durable Email Delivery — SNS/SQS/SES with DLQ at Both Layers

Status: Accepted
Date: 2026-05-11 UTC
Operator-locked: yes — architecture spec locked by operator 2026-05-11 UTC
Refs: #1657, docs/architecture/adr/0072-durable-email-sns-sqs-ses.md

V1 Implementation: Hybrid (Postmark + SNS/SQS/Lambda)

Updated 2026-05-11 UTC — See docs/architecture/adr/0074-email-delivery-hybrid-postmark-v1.md.

What builds in v1: The SNS/SQS/DLQ/Lambda durability topology below ships exactly as designed. The delivery leaf node is Postmark (not SES) for v1.

Topology delta from full-SES design:

Card delta: - SC-E1 (#1664): CLOSED — Postmark already verified + sandbox-out complete - SC-E2 (#1665): SAME (minor IAM adjustment: no ses:SendEmail, add Postmark token SSM path) - SC-E3 (#1666): MODIFIED — Postmark JSON input instead of raw MIME - SC-E4 (#1667): MODIFIED — Postmark API call instead of ses:SendEmail - SC-E5 (#1668): MODIFIED (scope reduced) — Raptor publishes to SNS; Lambda calls Postmark - SC-E6 (#1669): CLOSED — FreeScout stays on Postmark SMTP - SC-E7 (#1670): SAME - SC-E8 (#1671): SAME - SC-E9 (#1672): CLOSED — Postmark stays in v1 - SC-E10: NEW — API Gateway endpoint for Postmark webhook → SNS publish

Critical path impact: SC-E1 deadline (2026-05-16 UTC) is eliminated. No SES sandbox-out needed before launch. New critical path: SC-E2 → SC-E10 + SC-E3 before 2026-05-23 UTC.

The SES migration path remains open post-v1. Switching from hybrid to full SES requires only a Lambda consumer swap — all upstream topology is delivery-agent-agnostic.

Context

Raxx's current email path runs through Postmark: outbound transactional mail fires directly from Raptor via postmark_client.py; inbound support mail arrives via Postmark's inbound webhook, hits POST /webhooks/postmark/inbound in Raptor, and is forwarded to FreeScout's API. This is a synchronous, single-hop architecture with no durability layer and no replay capability.

The current path has four structural weaknesses: 1. If Postmark is degraded or rejects the call, the mail is gone — no retry, no DLQ. 2. If FreeScout's API is down when the Postmark webhook fires, the ticket is lost. 3. Raptor's inbound dedup table (SQLite, 1 000-row cap) is load-bearing for idempotency. 4. There is no mechanism to detect that the entire email path has silently failed end-to-end.

Operator direction (2026-05-11 UTC): design durable email delivery using SNS FIFO + SQS FIFO + DLQs at both layers + Lambda + SES. Start with failure modes, not features.

Invariants

Section 1 — Failure Mode Matrix

This section precedes all architecture. Every row names a failure, its customer impact, how we detect it (with specific metric and threshold), and the recovery path.

Design principle (operator verbatim): "We shouldn't discover that it has 'also' failed." If a failure mode has no detection, that is the bug.

Note on v1 hybrid changes: Rows F2, F8, F11, F18, and F19 are updated below for the hybrid architecture. Original SES-specific text is preserved with a [Full-SES target] note for reference.

Note on F15 (AWS region SPOF): Cross-region failover for SNS/SQS is out of scope for v1. In the hybrid architecture, Postmark handles outbound delivery independently of AWS — an AWS region outage stalls inbound queuing but does not prevent messages already in transit via Postmark from delivering. This is an improvement over full SES. Multi-region email durability is a post-v1 improvement path.

Section 2 — Component Layout

SNS FIFO Topics

SQS FIFO Queues

Lambda Functions

V1 hybrid (ADR-0074): inbound Lambda receives Postmark JSON (not SES raw MIME); outbound Lambda calls Postmark API (not ses:SendEmail).

Full-SES target (post-v1): outbound Lambda role adds ses:SendEmail, ses:SendRawEmail; removes Postmark token SSM path. Inbound Lambda role unchanged.

API Gateway (v1 hybrid addition)

CloudWatch Alarms

All alarms created in terraform/email-delivery/alarms.tf. SNS notification target: ops@raxx.app (pre-launch: digest mode per feedback_pre_launch_digest_notifications.md; post-launch: per-event).

IAM Roles (least-privilege summary)

raxx-email-sns-publisher: sns:Publish on raxx-email-inbound.fifo and raxx-email-outbound.fifo only.
raxx-email-lambda-inbound: sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes, sqs:ChangeMessageVisibility on inbound queue; ssm:GetParameter on /raxx/email/*; logs:CreateLogGroup, logs:CreateLogEvent.
raxx-email-lambda-outbound (v1 hybrid): same SQS permissions on outbound queue; ssm:GetParameter on /raxx/email/*; logs:CreateLogGroup, logs:CreateLogEvent. No ses:* permissions in v1.

No role has * actions. No role has cross-account permissions at v1.

Section 3 — Topic and Queue Topology

Decision: two separate SNS topics (one inbound, one outbound). See ADR-0072 for full rationale. Summary:

• Failure scoping: an outbound delivery failure does not block inbound ticket creation. Separate topics isolate blast radius.
• Dead-letter isolation: SNS DLQ for inbound failures is separate from outbound failures; operator can investigate one without the other.
• IAM isolation: Raptor's publisher role can be granted sns:Publish on outbound only; API Gateway integration publishes to inbound topic only.

One SQS queue per consumer. At v1, two consumers: FreeScout bridge (inbound) and Postmark sender (outbound). A third consumer for attachment AV scanning can subscribe to the inbound topic independently when that card lands. This pattern allows adding consumers without modifying existing queues.

Postmark inbound webhook
  └─► API Gateway POST /webhooks/postmark/inbound
        └─► (signature validation)
              └─► SNS FIFO: raxx-email-inbound.fifo
                    ├─► SNS DLQ: raxx-email-inbound-sns-dlq.fifo
                    └─► SQS FIFO: raxx-email-inbound-bridge.fifo
                          ├─► SQS DLQ: raxx-email-inbound-bridge-dlq.fifo
                          └─► Lambda: raxx-email-inbound-bridge
                                └─► FreeScout API POST /api/conversations

Raptor / Console / FreeScout (publisher)
  └─► SNS FIFO: raxx-email-outbound.fifo
        ├─► SNS DLQ: raxx-email-outbound-sns-dlq.fifo
        └─► SQS FIFO: raxx-email-outbound-send.fifo
              ├─► SQS DLQ: raxx-email-outbound-send-dlq.fifo
              └─► Lambda: raxx-email-outbound-sender
                    └─► Postmark API POST /email

Full-SES target topology (post-v1): Replace "Postmark inbound webhook → API Gateway" with "SES receipt rule". Replace "Postmark API POST /email" with "ses:SendEmail". All SNS/SQS/DLQ/Lambda layers are unchanged.

Section 4 — Multi-Mailbox Routing Logic

Day 1 mailboxes: support@raxx.app, ops@raxx.app, account-management@raxx.app.

The inbound bridge Lambda dispatches by destination To address to the correct FreeScout mailbox ID.

Routing map storage: SSM Parameter Store at /raxx/email/mailbox_routing_map as a JSON string:

{
  "support@raxx.app":            101,
  "ops@raxx.app":                102,
  "account-management@raxx.app": 103
}

Reasons for SSM over Lambda env vars: - Rotatable without Lambda redeploy (update SSM value; Lambda reads at invocation time or via cached refresh on cold start). - Consistent with feedback_aws_workloads_use_ssm_not_vault.md. - Auditable: SSM parameter updates generate CloudTrail events.

Lambda routing logic (pseudocode — implementation in sub-card):

to_address = normalize(postmark_event["To"])
routing_map = get_cached_ssm_param("/raxx/email/mailbox_routing_map")
mailbox_id = routing_map.get(to_address) or routing_map.get("support@raxx.app")
freescout_client.create_conversation(mailbox_id=mailbox_id, ...)

If no match, fall back to support@raxx.app (mailbox 101) and log routing_fallback event for audit.

Section 5 — Idempotency Contract

FIFO SQS with content-based deduplication provides at-most-once delivery within the 5-minute deduplication window. Outside that window, redelivery is possible.

Inbound bridge Lambda: - Dedup key: MessageID field from the Postmark inbound webhook payload (equivalent to RFC 2822 Message-ID). - Check: query a DynamoDB table raxx-email-inbound-dedup (partition key: message_id) before creating a FreeScout conversation. - On hit: log "event_type": "idempotency_skip" and delete from SQS (return success to avoid re-delivery). - On miss: create conversation, then write message_id to dedup table with TTL = 90 days. - Why DynamoDB (not Postgres / Raptor DB): Lambda is a separate compute boundary; avoiding a Raptor DB dependency prevents tight coupling and DB connection exhaustion under Lambda burst.

Outbound sender Lambda: - Dedup key: correlation_id provided by the caller in the SNS message attribute. - Check: same DynamoDB table with separate prefix outbound#<correlation_id>. - Callers (Raptor, Console) generate correlation_id as a UUID4 on first publish. Retry of the same logical send reuses the same correlation_id. - SQS FIFO MessageDeduplicationId = correlation_id (5-min window hardening).

Section 6 — Visibility Timeout Sizing

Lambda maximum execution timeout: 15 minutes (900 s).

Recommended SQS visibility timeout: 20 minutes (1200 s).

Math: Lambda max execution (900 s) + safety margin (300 s) = 1200 s = 20 min.

The safety margin accounts for Lambda cold-start latency (typically < 5 s but provisioned concurrency eliminates this), Postmark/FreeScout API call latency, and DynamoDB dedup read/write.

Implications: - If a Lambda invocation crashes hard (OOM, unhandled exception), the message becomes visible again after 20 minutes. - With maxReceiveCount = 3 on the SQS queue, a poison message will attempt 3 times (60 min total delay) before moving to the DLQ. - 60-minute delay is acceptable for v1 support volume; reduces noise from transient FreeScout/Postmark hiccups.

Section 7 — Reserved Concurrency

Outbound sender Lambda: - Postmark API published rate limit: 1 000 messages/second — significantly higher headroom than SES new-account 14/s. - At v1 outbound volume (10 000/mo ≈ 333/day ≈ 14/hour), burst is negligible. - Reserved concurrency: 5. Conservative guard against code bugs or requeue storms. - If Postmark rate limits surface, reduce to 1 (single-threaded processing).

Inbound bridge Lambda: - FreeScout API rate limit: unknown. Conservative assumption: 10 req/s. - Inbound v1 volume: ~100/mo. Spikes unlikely. - Reserved concurrency: 3. Prevents runaway invocations from hitting FS API. - If FreeScout API rate limits surface in production, reduce to 1 (single-threaded processing).

Note: Reserved concurrency also acts as a kill-switch. Setting reserved concurrency to 0 halts all processing (messages remain durably in SQS). This satisfies the platform invariant for kill-switches on critical execution paths.

Section 8 — Migration Plan

V1 Hybrid Plan (ADR-0074 — active)

Phase 0: None needed. Postmark is already production-ready, domain-verified, DKIM configured, sandbox-out complete. No AWS Support case required before v1 launch.

Phase 1 — Inbound durability before v1 launch (target: before 2026-05-23 UTC) 1. SC-E2: Terraform stack (SNS/SQS/DLQ/DynamoDB/alarms/IAM/API Gateway). 2. SC-E10: API Gateway endpoint POST /webhooks/postmark/inbound → SNS publish. 3. SC-E3 (modified): inbound Lambda (Postmark JSON → FreeScout API). 4. SC-E8: DLQ runbook. 5. SC-E7: Synthetic probe Lambda + alarm. 6. Cutover: update Postmark inbound server webhook URL to API Gateway endpoint. Feature-flag postmark_inbound_to_freescout OFF in Raptor.

Phase 2 — Outbound queue (post-v1 acceptable) 1. SC-E4 (modified): outbound Lambda (Postmark API call). 2. SC-E5 (modified): Raptor postmark_client.py → sns_publisher.py (feature-flagged). 3. 7-day soak period.

FreeScout continues using Postmark SMTP for outbound replies throughout v1. Unchanged.

Phase 3 (optional, post-v1): Full SES migration

If deliverability, cost, or vendor concerns justify it after launch:

1. Verify raxx.app domain in SES: add DKIM CNAME records to Cloudflare DNS.
2. Open AWS Support case: "Production Access Request" for SES.
3. Swap outbound Lambda consumer: Postmark API → ses:SendEmail. No publisher code changes.
4. FreeScout SMTP cutover: change from Postmark SMTP to SES SMTP endpoint.
5. Postmark retirement (cancel subscription, archive logs, remove credentials from Heroku apps, clean up dead code in Raptor).

This is an optional, undated future card — not a v1 commitment. The Lambda consumer is the only component that changes; all upstream topology (SNS/SQS/DLQ/publishers) is delivery-agent-agnostic.

Original SES Plan (ADR-0072 target state — reference only, superseded for v1 by ADR-0074)

Phase 0 — SES Domain Verification + Sandbox-Out (Operator action, size:s, ~1 day) - Verify raxx.app domain in SES: add DKIM CNAME records to DNS (Cloudflare, zone raxx.app). - Add SPF record if not present. - Open AWS Support case: "Production Access Request" for SES. - Do NOT proceed to Phase 1 for outbound until sandbox-out is confirmed.

Phase 1 — Inbound Durability (size:l) - Terraform: provision SNS inbound topic + SQS inbound queue + DLQs + alarms. - SES: configure receipt rule set for raxx.app → S3 action → SNS publish action to raxx-email-inbound.fifo. - Deploy raxx-email-inbound-bridge Lambda (SES raw MIME from S3 input). - Cut DNS MX record from Postmark to SES inbound.

Phase 2 — Outbound Queue Validation (size:m) - Terraform: provision SNS outbound topic + SQS outbound queue + DLQs. - Deploy raxx-email-outbound-sender Lambda (SES SendEmail). - Migrate postmark_client.py → sns_publisher.py in Raptor (feature-flagged). - Monitor for a 7-day soak period.

Phase 3 — FreeScout SMTP Cutover (size:s) - Change FreeScout SMTP settings from Postmark to SES SMTP endpoint. - SES SMTP credentials stored in SSM at /raxx/email/ses_smtp_password.

Phase 4 — Postmark Retirement (size:s, ~30 days after Phase 3 soak) - Cancel Postmark subscription. - Archive Postmark delivery logs (export to S3 for GDPR retention period). - Remove POSTMARK_SERVER_TOKEN from all Heroku apps. - Remove postmark_client.py, postmark_inbound.py, postmark_collector.py in a cleanup PR.

Section 9 — Replay SOP for DLQ

When a CloudWatch alarm fires for SQS DLQ depth > 0, follow this procedure.

Step 1 — Identify the affected DLQ

Alarm name format: raxx-email-<direction>-<type>-dlq-depth. Determine which DLQ.

Step 2 — Examine DLQ messages

# Receive up to 10 messages (does not delete them)
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/<account-id>/raxx-email-inbound-bridge-dlq.fifo \
  --max-number-of-messages 10 \
  --attribute-names All \
  --message-attribute-names All \
  --visibility-timeout 300 \
  --region us-east-1

Examine the Body field (JSON-encoded SNS notification wrapping the original email event). Look for: - ApproximateReceiveCount: if > 3, it hit max-receive-count. - SentTimestamp: when the message first arrived.

Step 3 — Correlate with Lambda logs

# Find Lambda invocation errors around the message's SentTimestamp
aws logs filter-log-events \
  --log-group-name /aws/lambda/raxx-email-inbound-bridge \
  --start-time <epoch-ms-from-SentTimestamp> \
  --filter-pattern '"ERROR"' \
  --region us-east-1

Check for: FreeScout API 5xx, connection timeout, malformed payload, IAM permission error.

Step 4 — Determine disposition

Step 5 — Redrive (AWS Console preferred for audit trail)

AWS Console method: 1. SQS Console → queue → "Dead-letter queue" tab → "Start DLQ redrive". 2. Set destination to the source queue. 3. Confirm. AWS generates a CloudTrail event.

CLI method (for scripting):

aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:<account-id>:raxx-email-inbound-bridge-dlq.fifo \
  --destination-arn arn:aws:sqs:us-east-1:<account-id>:raxx-email-inbound-bridge.fifo \
  --region us-east-1

Step 6 — Post-redrive audit

After redrive, verify: 1. DLQ depth returns to 0 (CloudWatch metric ApproximateNumberOfMessagesVisible). 2. Lambda successfully processes the message (check CloudWatch Logs for successful invocations). 3. FreeScout conversation created (or Postmark delivery confirmed). 4. Write an ops note in the FreeScout ops@ ticket if the failure affected a customer-visible email path. Note: timestamp (UTC), root cause, resolution, message count affected.

Section 10 — Synthetic Probe

Design

• Frequency: every 5 minutes (CloudWatch Events / EventBridge rule).
• Sender: a verified external email address controlled by the operator (e.g., kris@moosequest.net).
• Recipient: support@raxx.app.
• Subject format: PROBE-<UUID4> (probe ID embedded in subject, not body, for easy FS query).
• Expected outcome: FreeScout conversation appears within 60 seconds of send; conversation subject contains the probe ID.
• Probe runtime: Lambda (raxx-email-probe) queries FreeScout API GET /api/conversations?subject=PROBE-<UUID4> after 90 seconds.
• Alert condition: if no conversation found with matching probe ID within 5 minutes of send, CloudWatch alarm fires → ops@raxx.app alert.

Why external sender

Probe must originate outside AWS. If API Gateway, SNS, SQS, Lambda, or FreeScout API all fail together (e.g., F15 — region outage), a probe sent via AWS Lambda would also be affected and never trigger. Using the operator's personal email as sender ensures the signal originates from a completely independent path.

Probe Lambda IAM

raxx-email-probe-role: ssm:GetParameter on /raxx/email/freescout_api_key only. No SNS, no SQS permissions — read-only health check.

Failure response

Probe alarm fires → ops@ alert with probe ID → operator checks: 1. Did the email arrive at Postmark inbound? (Postmark dashboard: inbound activity log) 2. Did API Gateway receive the webhook? (API Gateway access logs) 3. Did SNS publish? (SNS console: delivery status) 4. Did SQS receive? (SQS console: messages received metric) 5. Did Lambda invoke? (CloudWatch Logs) 6. Did FreeScout receive the API call? (FS admin → activity log)

This six-layer walkthrough isolates the failing component within 10 minutes.

Section 11 — Cost Analysis

At v1 volume: 10 000 outbound/mo + 100 inbound/mo.

V1 Hybrid (Postmark delivery leaf)

Full SES (post-v1 target)

Cost delta at v1: $13.97/mo (hybrid) vs $2.48/mo (full SES). Difference: ~$11.50/mo.

When does the SES migration pay for itself? The migration is approximately 3–4 engineering-days of work. At a conservative $500/day cost, break-even at v1 volume is ~1 000 months. The migration is a cost decision at scale, not at v1. At 100k/mo outbound volume, the difference is ~$112/mo — meaningful enough to revisit.

Section 12 — ADRs

• ADR-0072: docs/architecture/adr/0072-durable-email-sns-sqs-ses.md — durability topology (Accepted, target state)
• ADR-0074: docs/architecture/adr/0074-email-delivery-hybrid-postmark-v1.md — v1 implementation scope (Accepted, amends ADR-0072 for v1)

Section 13 — Card Breakdown for PM

V1 Hybrid Card Slate

Estimated total engineering (v1 hybrid): ~8.5 days across feature-developer + sre-agent. SC-E2 (Terraform) is the critical path. SC-E2 → SC-E10 → SC-E3 must land before 2026-05-23 UTC.

Migrations

No Raptor database schema migrations are introduced by this design. The dedup table moves from Raptor SQLite (postmark_inbound_dedup) to a standalone DynamoDB table in the email-delivery stack. This DynamoDB table is provisioned by the email-delivery Terraform stack (SC-E2) and populated by the Lambda (SC-E3).

The postmark_inbound_dedup SQLite table in Raptor remains for a 30-day observation window after Phase 1 cutover, then is dropped in a schema migration in a follow-up cleanup card.

GDPR and Security Checklist

• PII collected: email address (sender, recipient), email subject, body. All PII transits SNS/SQS/Lambda ephemerally. The DynamoDB dedup table stores only hashed Message-ID headers (no email addresses, no content).
• Retention: Lambda logs retained 90 days in CloudWatch (then deleted). DynamoDB dedup records TTL = 90 days. SQS messages: max retention 4 days (default). DLQ messages: retain until operator drains (alarm fires on day 0 of DLQ depth > 0 — this is not a silent accumulation). Postmark retains inbound messages for 45 days (useful for replay/audit).
• DSR (deletion on data subject request): Email content is never stored in the queue layer. FreeScout is the system of record for conversation content — DSR erasure runs against FreeScout per existing runbook. Lambda logs contain to_hash (SHA-256 truncated) not raw addresses.
• Audit logging: all Lambda invocations produce structured JSON log events to CloudWatch Logs with event_type, message_id_hash, mailbox_id, outcome, timestamp_utc. All DLQ redrives produce CloudTrail events. SNS publish calls produce CloudTrail events.
• Credential replay: no credentials are stored or logged. Postmark server token is read from SSM at invocation time; never written to Lambda environment variables in plaintext in Terraform (use aws_ssm_parameter data source + Lambda environment via Terraform, marked sensitive = true). FreeScout API key: same pattern.
• Breach notification: if a DLQ message contains PII (email body from Postmark webhook payload), the message is visible to anyone with sqs:ReceiveMessage on the DLQ. IAM policy restricts DLQ access to raxx-ops-break-glass role. If unauthorized access detected: notify operator within 72 hours per GDPR Art. 33 requirement. Breach notification automation tracked in docs/architecture/adr/0003-gdpr-by-default.md.
• Secrets location: all secrets (FreeScout API key, Postmark server token, probe FreeScout API key, Postmark inbound webhook token) in SSM Parameter Store (/raxx/email/*), SecureString type, KMS-encrypted. Rotatable without Lambda redeploy (Lambda reads SSM at invocation time with a 5-minute in-memory cache). Never in Lambda environment variables in Terraform state.
• Kill-switch: setting Lambda reserved concurrency to 0 halts email processing without message loss (SQS retains messages). This is the kill-switch for both inbound and outbound paths.

Open Questions

These must be resolved before sub-cards SC-E3 and SC-E10 can be claimed:

1. FreeScout mailbox IDs for support@, ops@, account-management@: operator must confirm the numeric mailbox IDs from FreeScout admin so the SSM routing map can be populated.
2. DynamoDB provisioning mode: on-demand (pay-per-request) vs provisioned (fixed capacity). Recommendation is on-demand at v1 volume, but operator should confirm cost tolerance.
3. Probe sender address: confirm which external email address (e.g., kris@moosequest.net) will be the synthetic probe sender. It must be a real inbox the operator controls to validate end-to-end delivery — and it must be outside AWS.
4. Postmark inbound webhook token: confirm the token value to populate SSM /raxx/email/postmark_inbound_webhook_token. This is the static token Postmark sends in the webhook request header for validation.
5. API Gateway auth method: API Gateway → SNS direct integration (no Lambda hop) requires careful IAM role configuration. Confirm whether sre-agent should use a direct integration (mapping templates) or a thin Lambda shim for the signature validation step. Direct integration is more efficient; Lambda shim is more debuggable.

Note: Open questions 2 (SES receipt rule set region) and 5 (S3 attachment handling) from the original list are eliminated by the hybrid architecture — Postmark handles MIME parsing and attachment metadata inline in the webhook payload; no S3 involvement for inbound.