AWS-Native Migration (May 2026)
This page records the architectural shift performed in May 2026 to replace external dependencies (Redis Cloud, SMTP relay, IMAP polling, direct frontend↔background-tasks calls) with AWS-native services. It's intended for engineers who need to understand why the runtime looks the way it does — what was replaced, what's running where, and how to flip behaviour at runtime.
Why we migrated
Three pressures stacked up:
- External dependency cost — Redis Cloud was a paid SPOF outside our AWS footprint. Killing it removes ~$30–100/month and one outage vector.
- Polling latency — IMAP polling (30s for inbound mail/CV) and DB polling (30s for agent docs) were wasted compute and laggy UX. SES inbound + S3 EventBridge cuts that to under 5s.
- Two public ingresses, one ALB cost — the frontend used to talk to both
api.aiqlick.com(NestJS backend) andai.aiqlick.com(background-tasks GraphQL). Consolidating onto NestJS-as-sole-gateway halves the auth surface, simplifies CORS, and lets us put background-tasks on a private subnet.
Bonus: the gateway consolidation enforces the architectural invariant that all business logic lives in NestJS — background-tasks is now a pure compute service.
Architectural invariant — frontend ↔ backend isolation
After this migration the rule is:
| Allowed | Forbidden |
|---|---|
| Frontend → NestJS (HTTPS, WSS, SSE) | Frontend → background-tasks (any protocol) |
| Frontend → Stripe Checkout (redirect only) | Frontend → AWS direct (S3/Bedrock/Transcribe) |
| NestJS → background-tasks (private VPC, internal-service JWT) | Frontend importing background-tasks GraphQL client libs |
| NestJS → SES / SQS / SNS / EventBridge / S3 | Background-tasks publishing public endpoints |
| Background-tasks → RDS / Bedrock / S3 | — |
Every new feature must respect this. Adding any frontend → background-tasks call is a regression — fix the gateway instead.
Service replacement matrix
| Before | After | Phase |
|---|---|---|
Bull mailQueue (Redis) + Nodemailer SMTP | SQS aiqlick-mail-* + SES SendEmail | 1 |
Bull notifications (Redis) | SQS aiqlick-notifications-delivery-* | 4 |
| Redis pub/sub (notifications + messaging) | In-process graphql-subscriptions PubSub | 4 |
| IMAP polling (jobs / CVs from one.com) | SES inbound → S3 → SQS → BG consumer | 2 |
| Agent docs DB poll (30s) | S3 EventBridge → SQS → BG consumer | 5.1 |
| Redis pub/sub for cost-config invalidation | EventBridge custom bus | 4.4 |
| Frontend → background-tasks (4 env vars, 8+ WS clients) | Frontend → NestJS gateway (one Apollo client) | 3 |
Public ai-dev.aiqlick.com ALB hop | Backend → BG via private VPC IP, plain HTTP | 3.6 |
| Per-deploy env-var feature toggles | AWS AppConfig (60s runtime polls) | 3.0 |
X-Request-Id log grep | AWS X-Ray distributed tracing | 0.3 |
| Manual smoke tests | CloudWatch Synthetics canaries | 0.5 |
Architecture after migration
Phase ledger
The work happened in 6 phases over a few days. Phase 0 was prep, Phases 1–5 are the substantive migration, Phase 6 is deferred until we go multi-pod.
| Phase | Scope | Status |
|---|---|---|
| 0 — Observability prep | AWS Budgets, AWS Config, Trusted Advisor, Parameter Store, X-Ray, Synthetics canaries | ✅ live (dev + prod) |
| 1 — Mail (SQS + transport switch) | MailService.queueMail() → SQS → SqsMailConsumer → SES or SMTP via MAIL_TRANSPORT | ✅ live (dev + prod). Outbound mail is fully cut over to MAIL_TRANSPORT=ses after SES production access was granted. |
| 2 — SES inbound | inbound.aiqlick.com → SES rule → S3 → EventBridge → SQS → BG consumer | ✅ live (dev + prod). IMAP cutover complete on prod 2026-05-02. |
| 3 — NestJS as sole gateway | subscribeAi / queryAi / mutateAi proxies for all 10 AI ops + voice WS | ✅ live (dev + prod), all AppConfig flags ON |
| 4 — Redis purge | SQS for notifications, in-process pubsub, Bull deleted | ✅ Redis Cloud subscription cancellation pending billing-side step |
| 5 — Event-driven agent docs + cross-service bus | EventBridge aiqlick-events-* with two rules | ✅ live (dev + prod) |
| 6 — EventBridge Scheduler for cron + multi-pod pubsub | Replaces in-process @Cron and in-process graphql-subscriptions PubSub | ✅ done as part of Phase 7 |
| 7 — EC2 → ECS Fargate migration | Backend + BG move to multi-task Fargate behind public + internal ALBs, deployment circuit breaker, autoscaling. See Phase 7 runbook. | ✅ live |
What's running where (dev)
| Service | Resource |
|---|---|
| AppConfig application | e6qub9k (env kkx4z4c, profile zatlet7) |
| EventBridge bus | aiqlick-events-dev |
| SES sender domain | aiqlick.com (Easy DKIM, all 3 CNAMEs) |
| SES inbound subdomain | inbound-dev.aiqlick.com (verified; MX at one.com pending) |
| SQS queues | aiqlick-mail-dev, aiqlick-notifications-delivery-dev, aiqlick-notifications-fanout-dev, aiqlick-inbound-jobs-dev, aiqlick-inbound-cvs-dev, aiqlick-agent-docs-dev, aiqlick-bg-events-dev (each with a -dlq peer) |
| S3 inbound bucket | aiqlick-inbound-emails-dev |
| X-Ray daemon | local on each EC2 (AWS_XRAY_DAEMON_ADDRESS=127.0.0.1:2000) |
| Synthetics canaries | aiqlick-api-health-dev, aiqlick-ai-gateway-health-dev, aiqlick-frontend-signin-dev (15-min cadence; alarms wired) |
| Synthetics S3 artifacts | aiqlick-canaries-dev |
| Synthetics IAM role | aiqlick-canary-role-dev |
| BG dev SG (post-cutover) | sg-080a50d29673d01d3 (only port 22 + port 8000 from backend SG) |
Runtime configuration (dev secret)
The dev backend + BG share aiqlick-backend/development in Secrets Manager. The migration added these keys (besides the queue URLs):
MAIL_BACKEND=sqs
NOTIFICATIONS_BACKEND=sqs
NOTIFICATIONS_DELIVERY_QUEUE_URL=https://sqs.eu-north-1.amazonaws.com/842697652860/aiqlick-notifications-delivery-dev
NOTIFICATIONS_FANOUT_QUEUE_URL=https://sqs.eu-north-1.amazonaws.com/842697652860/aiqlick-notifications-fanout-dev
SES_FROM_ADDRESS=no-reply@aiqlick.com
INBOUND_EMAIL_BACKEND=sqs
INBOUND_CV_BACKEND=sqs
INBOUND_EMAIL_QUEUE_URL=https://sqs.eu-north-1.amazonaws.com/842697652860/aiqlick-inbound-jobs-dev
INBOUND_CV_QUEUE_URL=https://sqs.eu-north-1.amazonaws.com/842697652860/aiqlick-inbound-cvs-dev
INBOUND_EMAIL_S3_BUCKET=aiqlick-inbound-emails-dev
INBOUND_EMAIL_MAILBOX_ADDRESS=jobs@inbound-dev.aiqlick.com
INBOUND_CV_MAILBOX_ADDRESS=cv@inbound-dev.aiqlick.com
BILLING_PUBSUB_BACKEND=eventbridge
BG_EVENTS_QUEUE_URL=https://sqs.eu-north-1.amazonaws.com/842697652860/aiqlick-bg-events-dev
AGENT_DOCS_QUEUE_URL=https://sqs.eu-north-1.amazonaws.com/842697652860/aiqlick-agent-docs-dev
AWS_EVENT_BUS_NAME=aiqlick-events-dev
AWS_APPCONFIG_APPLICATION_ID=e6qub9k
AWS_APPCONFIG_ENVIRONMENT_ID=kkx4z4c
AWS_APPCONFIG_PROFILE_ID=zatlet7
INTERNAL_SERVICE_SECRET=<generated HS256 secret>
AI_BACKEND_GRAPHQL_URL=http://172.31.17.4:8000/graphql # private VPC IP after Phase 3.6
AI_BACKEND_GRAPHQL_WS_URL=ws://172.31.17.4:8000/graphql
AI_BACKEND_VOICE_WS_URL=ws://172.31.17.4:8000/voice/ws
XRAY_ENABLED=true
The 11 dead REDIS_* keys (and the deactivated Redis Cloud password) were removed from the secret as part of the cleanup.
Promoting to production
Each phase shipped dev → main as its own PR. The cutover landed on 2026-05-02 in a single staged rollout — see Inbound email — prod cutover & runbook for the full ordered checklist (queues → S3 → EventBridge → SES rules → IAM → secret keys → container restart → slug backfill → sentinel test → IMAP cutover).
What's live on prod (post-cutover)
| Resource | Prod name |
|---|---|
| EventBridge bus | aiqlick-events-prod |
| SES sender domain | aiqlick.com (Easy DKIM verified, all 3 CNAMEs) |
| SES inbound subdomain | inbound.aiqlick.com (verified; MX → inbound-smtp.eu-north-1.amazonaws.com) |
| SES receipt rules | jobs-to-s3-prod, cvs-to-s3-prod (in the active rule set aiqlick-receipt-rules alongside the dev rules) |
| SQS queues | aiqlick-mail-prod, aiqlick-notifications-delivery-prod, aiqlick-notifications-fanout-prod, aiqlick-inbound-jobs-prod, aiqlick-inbound-cvs-prod, aiqlick-agent-docs-prod, aiqlick-bg-events-prod (each with a -dlq peer) |
| S3 inbound bucket | aiqlick-inbound-emails-prod (AES256, EventBridge enabled, 30-day expiry) |
| IAM (post-Phase-7) | The legacy custom policies (aiqlick-backend-mail-policy, -events-policy, -inbound-policy, aiqlick-ai-inbound-policy, -events-policy, -bg-events-policy) and their EC2 roles (aiqlick-backend-ec2-role, aiqlick-ai-ec2-role) were retired with the Phase 7 cleanup of 2026-05-14. All those grants now live in the inline BackendTaskPolicy (on aiqlick-ecs-backend-task-role-{dev,prod}) and BgTaskPolicy (on aiqlick-ecs-bg-task-role-{dev,prod}). |
Outbound transport on prod
MAIL_TRANSPORT=ses on prod — outbound flows natively through Amazon SES. SES production access was successfully granted (case 177763885000822). The old SMTP Nodemailer configuration is maintained as a fallback, but the primary transport is now SES. Inbound SES continues to power the cv@inbound.aiqlick.com / jobs@inbound.aiqlick.com flow.
IMAP cutover (2026-05-02)
After ~30min of parallel-run soak the legacy IMAP poller was turned off on prod by setting INBOUND_EMAIL_ENABLED=false and INBOUND_CV_ENABLED=false, then stop+remove+running the BG container. Logs confirm:
Inbound email worker disabled (INBOUND_EMAIL_ENABLED=false)
Inbound CV worker disabled (INBOUND_CV_ENABLED=false)
SQS inbound email consumer started (Phase 2)
SQS inbound CV consumer started (Phase 2)
The same secret update stripped 27 unused keys: 7 REDIS_*, 16 EMAIL_IMAP_* / CV_IMAP_*, and 4 INBOUND_*_RETRY_*. The IMAP mailboxes themselves on imap.one.com (jobs@aiqlick.com, cv@aiqlick.com) still exist and can be retired at one.com whenever convenient — they're decoupled from the platform.
Known gotchas
AppConfig payload shape
The hosted-config upload format wraps flag values under .values, but the appconfigdata.GetLatestConfiguration runtime API returns a flat map of just the resolved values: { "flag-name": { "enabled": true, ...attrs } }. Our parser used to expect the wrapped shape — every flag came back false, and the gateway looked dark for half a day before the /health/ai-gateway endpoint surfaced it. The current parser accepts both shapes.
X-Ray "context missing" log noise
X-Ray's HTTP/SDK patchers log an error every time an outbound AWS call happens outside an HTTP request scope (i.e., from a background SQS poll loop). It's noisy but harmless — the calls still complete. Set AWS_XRAY_CONTEXT_MISSING=IGNORE_ERROR in the secret to silence it.
AuthService event emits
requestPasswordReset / resetPassword / verifyEmail historically did NOT emit the user.password.reset.requested / .completed / user.email.verified events that UserCompanyNotificationListener subscribes to. The email path worked (because MailService.queueMail() was called directly), but in-app SSE notifications for these events were silently dropped. Fixed in commit 7fb62c0. If you add a new auth flow, make sure to emit the matching event so the listener fires.
SES production access and transport cutover
SES production-access case 177763885000822 was successfully granted (status GRANTED per aws sesv2 get-account). Outbound mail now flows natively through Amazon SES on both dev and prod by setting MAIL_TRANSPORT=ses in Secrets Manager. The one.com SMTP configuration remains in the codebase as a fallback transport. See Mail & Email — Transport selection for the full table.
SES inbound is independent and is unaffected by the sandbox; it continues to handle cv+/jobs+ mail (Phase 2 went live without SES production access for outbound).
Prisma 7 Docker build needs a placeholder DATABASE_URL
A subtle deploy bug bit during the prod cutover: Prisma 7 reads datasource.url from prisma.config.ts at config-load time, and env('DATABASE_URL') in that file throws PrismaConfigEnvError during prisma generate because the env var only gets injected at docker run time. Fixed in aiqlick-backend@fb36c1b5 by passing a placeholder URL during the build (prisma generate doesn't connect, just needs the var to be defined). See Prisma 7 migration.
Related
- NestJS AI Gateway — the new gateway architecture
- Mail & Email — current SQS+SES mail path
- Real-time delivery (SSE) — current notification flow
- AWS Infrastructure — full account inventory