EC2 → ECS Fargate Migration (Phase 7)
This is the architectural shift that took aiqlick-backend and
background-tasks off single-EC2-per-env Docker hosts and onto ECS Fargate
with multi-task autoscaling. It also closed two pieces of debt that
aws-native-migration.md explicitly
deferred: cron orchestration (Phase 6.A) and multi-pod-safe pub/sub fan-out
(Phase 6.B).
Why we migrated
- Single-EC2 = single point of failure. Today each env runs one container on one t3.micro/small. A hardware fault, a botched deploy, or an OOMKill takes the env down for the duration of the next deploy pipeline.
- No horizontal scale. The earlier migration capped at one task per
service because cron decorators fire on every pod and the
graphql-subscriptionsPubSub is in-process. Both block multi-pod. - Caddy + Let's Encrypt is the wrong layer for an AWS shop. TLS lives on the box, certificate renewal is a per-instance cron, ports 22/80/443 are open to the world. ALB + ACM moves all of that into managed AWS.
- SSH-driven deploys leak instance IPs into GitHub Actions secrets.
Every deploy SSHes via
EC2_HOST/EC2_DEV_HOST/AI_EC2_HOST/AI_EC2_DEV_HOST. The new pipeline uses OIDC + ECS APIs only.
Post-cleanup state (2026-05-14)
The host-decommission tail of Phase E was executed on 2026-05-14T17:33 UTC. What ran (in order, all reversible up until step 3):
- Released 4 EIPs back to the account pool —
eipalloc-080c42040a66984b6,eipalloc-0fa3f4eb782a10a9e,eipalloc-04b786b8c101900ac,eipalloc-0205ee345a2187390(the13.62.166.68,13.62.32.225,13.63.47.99,51.21.229.163IPs). - Terminated 4 EC2 instances —
aiqlick-backend(i-0b9f31f3236c25b7f),aiqlick-ai(i-053ddc2f75899c06a),aiqlick-backend-dev(i-0b7d632f4de8a5a9a),aiqlick-ai-dev(i-0912ca82a42fd6ba4). - Deleted 2 security groups —
aiqlick-backend-sg,aiqlick-ai-dev-private-sg. Also revoked the cross-reference from the default VPC SG's:5432ingress that pointed ataiqlick-backend-sg. - Deleted IAM —
aiqlick-{backend,ai}-ec2-role,aiqlick-{backend,ai}-ec2-profile, and the 9 custom policies (aiqlick-backend-{s3-secrets,events,appconfig,inbound,mail}-policy,aiqlick-background-tasks-policy,aiqlick-ai-{bg-events,events,inbound}-policy). - GHA self-hosted runner terminated —
i-0b13fda708cf21e54no longer exists. All workflows run on GitHub-hostedubuntu-latestwith OIDC. See self-hosted-runner for historical context.
Verified after cleanup: all four ECS services
(backend-{dev,prod}, bg-tasks-{dev,prod}) at desired count,
https://api.aiqlick.com/health and
https://api-dev.aiqlick.com/health both 200 with DB latency 2ms.
Estimated savings: ~$48/mo EC2 + ~$15/mo EIPs (released, no longer
billed when not associated to a running instance) = ~$63/mo recurring.
The 4 freed EIPs also unblock NAT EIP allocation
(case bed0e56e1dd345288afcdd9df55e3d23CR4fmolZ is no longer
strictly required).
DNS cleanup (done 2026-05-14)
The ai.aiqlick.com and ai-dev.aiqlick.com records were deleted
from the one.com control panel by the operator on the same day. Both
hostnames now resolve to one.com's parking IP 46.30.213.40 (the
default response for any unconfigured subdomain on a one.com-hosted
zone). The pointer at AWS-released EIPs is gone; subdomain-takeover
risk is closed. No further DNS work is required.
Architectural invariants this migration enforces
| Allowed | Forbidden |
|---|---|
| Frontend → public ALB → backend tasks | Frontend → background-tasks (still — Phase 3.6 invariant) |
| Backend → internal ALB → BG tasks (private VPC) | Backend → BG via hardcoded private IP |
| Tasks → AWS services via VPC endpoints | Tasks bypassing the task IAM role with hand-crafted credentials |
EventBridge Scheduler → backend /internal/cron/:name | @Cron(...) firing in-process on multi-task services |
pubSubFanout.publish() for any cross-task event | direct new PubSub().publish() in feature modules |
Architecture (post-cutover)
Service replacement matrix
| Before | After |
|---|---|
4× EC2 (aiqlick-backend{,-dev}, aiqlick-ai{,-dev}) running Docker | 4× ECS services (backend-{dev,prod}, bg-tasks-{dev,prod}) on Fargate |
| Caddy on each EC2 + Let's Encrypt | aiqlick-public-alb + ACM cert on api/api-dev.aiqlick.com |
Hardcoded BG private IP http://172.31.17.4:8000 | aiqlick-internal-alb host-header routing + Cloud Map bg-{env}.aiqlick.local |
fetch-secrets.sh → .env → docker run --env-file | Task definition BUNDLE_SECRET_ARN env var → app-side loadSecretBundle() at startup |
SSH deploy via EC2_HOST GitHub secrets | OIDC + aws-actions/amazon-ecs-deploy-task-definition@v2 |
:previous tag manual rollback in BG workflow | ECS deployment circuit breaker (auto-rollback) |
@Cron decorators firing in-process on every pod | EventBridge Scheduler → POST /internal/cron/:name + CronExecution idempotency table |
notification-system/services/notification-pubsub.service.ts (in-process PubSub) | Same surface, internally delegates to PubSubFanoutService (in-process when 1 task, SNS+SQS when ≥2) |
messaging/services/messaging-pubsub.service.ts (in-process PubSub) | Same as above |
Direct EC2 logs via SSH + docker logs | CloudWatch Logs /ecs/aiqlick-{service}-{env} (30-day retention) |
Phase ledger
The work was split into 6 phases (A–F). All landed in a single PR — none of them are useful alone, and the deployment circuit breaker / Cloud Map / SNS plumbing has to come up together.
| Phase | Scope | Status |
|---|---|---|
| A — Foundation | 3 private subnets in 3 AZs, 9 VPC endpoints + S3 gateway, aiqlick-public-alb, aiqlick-internal-alb, Cloud Map namespace aiqlick.local, ECS cluster aiqlick, 4 security groups | Live (NAT GW + ACM cert deferred — see "Deferred items" below) |
| B — Task defs + IAM | 4 task defs (backend-{dev,prod}, bg-tasks-{dev,prod}) + 1 prisma-migrate one-shot, 5 IAM roles, 4 ECS services with deployment circuit breaker, sticky-session backend target groups, target-tracking auto scaling | All 4 services Live and at desired count (dev 1+1, prod 2+2 — 6 healthy Fargate tasks) |
| C — Phase 6 work | Cron migration code (12 jobs → registry + signed /internal/cron/:name + CronExecution idempotency), PubSub fan-out code (SNS topic per env + per-task ephemeral SQS subscription with cleanup at shutdown), SQS consumer idempotency audit | Code merged. CRON_MODE=external and PUBSUB_FANOUT_BACKEND=sns_sqs not yet flipped on — flip when ready (see "Post-cutover unlocks") |
| D — CI/CD rewrite | Both aws-deploy.yml workflows replaced — OIDC + ECR build/push + render task def + deploy. Backend gets a one-shot Fargate task running prisma db push between build and deploy when prisma/schema.prisma changed. | Live — first dev+prod CI runs green on 2026-05-09 |
| E — Cutover | Per-env DNS flip from EC2 IP → ALB CNAME, 24h drain, 7-day soak, EC2 termination + EIP release, aiqlick-backend-sg 80/443 ingress revoke. See Cutover runbook. | Complete. api.* DNS cut over 2026-05-09 22:55 UTC. EC2 termination + EIP release + IAM/SG cleanup ran 2026-05-14 17:33 UTC. ai.* DNS records deleted from one.com same day — see "Post-cleanup state" + "DNS cleanup" below. |
| F — Documentation | This page + the new ECS reference. Per-doc updates to aws-overview, network, ec2, deployment/{backend,ai-service}, aws-native-migration. CLAUDE.md updates in both repos and at workspace root. | Live |
Deploy log (what actually shipped 2026-05-09)
PRs
aiqlick-backend#186— backend code + infra/ecs/ + newaws-deploy.ymlbackground-tasks#90— BG code + infra/ecs/ + newaws-deploy.ymldocumentation#62— this page + ECS reference + per-doc updates
All three merged within seconds of each other; first dev CI run cleared at 2026-05-09 17:43 UTC, first prod CI run cleared at ~2026-05-09 18:14 UTC.
Live AWS resources (post-deploy snapshot)
| Resource | ID / Name |
|---|---|
| ECS cluster | aiqlick (eu-north-1, Container Insights enabled) |
| Backend services | backend-dev (1/1), backend-prod (2/2) |
| BG services | bg-tasks-dev (1/1), bg-tasks-prod (2/2) |
| Public ALB | aiqlick-public-alb (DNS aiqlick-public-alb-2032682590.eu-north-1.elb.amazonaws.com, currently HTTP:80 only — see "Deferred items") |
| Internal ALB | aiqlick-internal-alb (DNS internal-aiqlick-internal-alb-217681152.eu-north-1.elb.amazonaws.com) |
| Cloud Map namespace | aiqlick.local (ns-xg62geye6d43rnop) |
| SNS topics | aiqlick-pubsub-dev, aiqlick-pubsub-prod |
| Backend execution roles | aiqlick-ecs-execution-role-{dev,prod} |
| Backend task roles | aiqlick-ecs-backend-task-role-{dev,prod} |
| BG task roles | aiqlick-ecs-bg-task-role-{dev,prod} |
| Scheduler role | aiqlick-scheduler-role (idle until Phase C cron flips on) |
| GH Actions ECS deploy policy | aiqlick-ecs-deploy-policy inline on github-actions-role (grants iam:PassRole on the new task roles + ECS RegisterTaskDefinition / UpdateService / RunTask) |
Bugs fixed mid-deploy (canonical scripts updated, carried forward to prod)
| # | Bug | Fix |
|---|---|---|
| 1 | awslogs-create-group: false rejected by ECS — must be true or omitted | Removed from all 5 task defs |
| 2 | Cloud Map service registries don't accept containerPort (only SRV records do) | Removed from all 4 service scripts |
| 3 | log_* helpers wrote to stdout, contaminating command-substitution captures (PRIV_A=$(create_or_find_subnet ...) got "Created subnet ... id" instead of just id) | Helpers redirect to stderr |
| 4 | Backend SG had no path to BG SG :8000 directly — only via internal-alb-sg. Cloud Map name resolves to BG task ENI but couldn't connect | Added rule: aiqlick-ecs-bg-sg allows :8000 ← aiqlick-ecs-backend-sg |
| 5 | Backend stripe.service.ts reads STRIPE_SECRET_KEY at module-import time, before async bundle loader could populate process.env | New src/bootstrap-env.ts synchronously parses ECS-injected _BUNDLE_JSON BEFORE any other import, plus loads .env via dotenv for local dev |
| 6 | Dockerfile's ENV DATABASE_URL= empty-string reset (left over from Prisma 7 placeholder dance) was treated as "set" by the loader, blocking the bundle from filling it in | bootstrap-env now treats '' as not-set |
| 7 | github-actions-role lacked iam:PassRole on the new ECS task roles AND ecs:RegisterTaskDefinition / UpdateService / RunTask permissions | New inline policy aiqlick-ecs-deploy-policy attached |
| 8 | Prisma migrate one-shot ECS task ran node -e "execSync('npx prisma db push')" without bundle hydration, so prisma.config.ts's env('DATABASE_URL') threw PrismaConfigEnvError | Task def now injects _BUNDLE_JSON via secrets; the inline node -e parses + populates env before spawning prisma |
| 9 | 02-{backend,bg-tasks}-prod.sh bootstrapped with :latest image which doesn't exist before the first prod CI run | Both scripts now accept IMAGE_TAG=... env override (default latest); first prod provisioning used IMAGE_TAG=dev-latest to bootstrap, then prod CI rolled forward |
| 10 | EIP quota was at 5 (account already has 10 in use via legacy raises). New NAT EIP allocation hit AddressLimitExceeded | New SKIP_NAT=true flag on 01-vpc-foundation.sh defers NAT until cutover when 4 EC2 EIPs auto-release. Quota raise to 20 also requested (Service Quotas case bed0e56e1dd345288afcdd9df55e3d23CR4fmolZ, PENDING) |
| 11 | ACM cert needs DNS validation TXT records on one.com — operator-only, blocks HTTPS:443 listener on public ALB | New USE_HTTP_ONLY=true flag on 07-public-alb.sh for HTTP-only smoke testing. Re-run without the flag once cert is ISSUED |
Verified end-to-end against the new infra
Production tasks were proven to serve correctly via the public ALB (Host: api.aiqlick.com override, no DNS change yet):
/healthreturns 200 with DB latency 1ms{ __typename }GraphQL query returns valid Apollo responsepublicPlansreturns 4 real plans from prod RDS/health/ai-gatewayreportsok=true, signerReady=true, httpClientReady=true, urls=bg-prod.aiqlick.local, 10 AI ops registered- ALB target health:
tg-backend-prod2/2,tg-bg-prod2/2
Dev was further exercised with real Bedrock workloads:
subscribeAi extractCv— 5 streamed events in 1.5s (pipeline reached BG, Bedrock OCR was called, returned the expected "couldn't OCR text-as-image" failure for our test.txtpayload)subscribeAi parseJob— 10 streamed events in 10.6s, ending withJobParsingSuccesscarrying structured output (title, salary range, benefits) via Claude Opus 4.7- Live event delivery:
supportTicketEventarrived ~150ms aftersendTicketMessagemutation - SSE
/notifications/streamopened cleanly, returnedevent: open+ sessionId - All 11 GraphQL subscriptions reachable through the new ALB
Deferred items (not blocking the cutover, intentionally deferred)
These are not "missing" — they're explicit choices to defer to specific moments in the cutover sequence.
| Item | Why deferred | When to land it |
|---|---|---|
| NAT gateway | EIP quota at 5 (account already over). 4 EIPs auto-release when the legacy EC2 instances are terminated post-cutover; allocate the NAT EIP from that pool. AWS Service Quotas request bed0e56e1dd345288afcdd9df55e3d23CR4fmolZ (PENDING) is also in flight — whichever lands first | After EC2 termination (cutover step 6) OR quota raise approval (whichever is first) |
| HTTP:80 → HTTPS:443 redirect on public ALB | The :80 listener forwards plain HTTP to the same backend target groups as :443 (host-header rules mirrored). Plain-HTTP requests are served, not redirected. | Replace the :80 host-header forward actions with redirect actions targeting HTTPS://#{host}:443/#{path}?#{query} (HTTP 301). Default action stays as the existing 404 fixed-response for unknown hosts. |
| EventBridge Scheduler entries | Code path is wired (InternalCronModule + signed POST /internal/cron/:name) but CRON_MODE=external is unset, so all 12 in-process @Cron handlers still fire normally | Once an HMAC-signing path for Scheduler is built (Lambda or static rotation), provision schedules via bash 10-eventbridge-scheduler.sh, set CRON_MODE=external in the prod task def, redeploy |
PUBSUB_FANOUT_BACKEND=sns_sqs | In-process pubsub still works for desiredCount=1/2 since sticky sessions pin clients to a single task; multi-pod fan-out is only required for desiredCount>2 or when sticky sessions can't guarantee affinity | When you want to scale beyond 2 prod tasks AND see SSE notifications stay live across the spread |
Permanent product decisions (not pending)
These were considered and explicitly chosen — not on any roadmap to change:
- DNS stays on one.com. No Route 53 migration. ACM TXT records added manually.
- Outbound mail runs on AWS SES.
MAIL_TRANSPORT=sesis active in both dev and prod. SES production access was successfully granted (Case177763885000822).
Current cutover state
As of 2026-05-09 22:55 UTC+3 — DNS cutover complete for both environments.
- ACM cert (
644abb95-5a71-4d53-8118-e42630ad8096, eu-north-1) issued at 22:55:58 with bothapi.aiqlick.comandapi-dev.aiqlick.comvalidated. aiqlick-public-albnow serves both hostnames on:443(forward totg-backend-prod/tg-backend-devby host header) and on:80(plain-HTTP forward; HTTP→HTTPS redirect is a deferred item).api.aiqlick.comandapi-dev.aiqlick.comat one.com are now ALIAS/CNAME records pointing ataiqlick-public-alb-2032682590.eu-north-1.elb.amazonaws.com. Bothhttps://api.aiqlick.com/healthandhttps://api-dev.aiqlick.com/healthreturn 200 from AWS-owned IPs (eu-north-1).- The legacy backend EC2 IPs (
13.62.166.68,13.62.32.225) no longer receive real user traffic and were terminated as part of the post-cleanup execution on 2026-05-14.
While DNS was being re-pointed in the one.com control panel, the records briefly fell back to one.com's parking infrastructure (46.30.213.40), which served an Apache 500 ("contact support@one.com"). If you see that error again on a *.aiqlick.com hostname, the DNS record at one.com is missing entirely — see Network & DNS → Troubleshooting.
What's running where
Dev
| Resource | Value |
|---|---|
| ECS cluster | aiqlick (single, both envs) |
| Backend service | backend-dev (1 desired, max 4) |
| BG service | bg-tasks-dev (1 desired, max 4) |
| Public ALB | aiqlick-public-alb (shared, listener rule host=api-dev.aiqlick.com) |
| Internal ALB | aiqlick-internal-alb (shared, listener rule host=bg-dev.aiqlick.local) |
| Task definitions | aiqlick-backend-dev:N, aiqlick-bg-tasks-dev:N (revision N rolled by CI) |
| Execution role | aiqlick-ecs-execution-role-dev |
| Backend task role | aiqlick-ecs-backend-task-role-dev |
| BG task role | aiqlick-ecs-bg-task-role-dev |
| SNS pubsub topic | aiqlick-pubsub-dev |
| Log groups | /ecs/aiqlick-backend-dev, /ecs/aiqlick-bg-tasks-dev, /ecs/aiqlick-prisma-migrate |
| Cloud Map services | backend-dev.aiqlick.local, bg-dev.aiqlick.local |
Prod
Same shape with -prod / -prod everywhere, min=2 max=8, ALB listener
rule host=api.aiqlick.com.
EventBridge Scheduler entries (prod only)
| Schedule | Expression | Target |
|---|---|---|
credit-replenishment-prod | cron(0 2 * * ? *) | POST /internal/cron/credit-replenishment |
subscription-expiry-prod | cron(0 9 * * ? *) | POST /internal/cron/subscription-expiry |
interview-reminder-prod | rate(1 hour) | POST /internal/cron/interview-reminder |
meeting-reminder-prod | rate(1 hour) | POST /internal/cron/meeting-reminder |
daily-digest-prod | cron(0 9 * * ? *) | POST /internal/cron/daily-digest |
weekly-digest-prod | cron(0 9 ? * MON *) | POST /internal/cron/weekly-digest |
slot-offer-expiration-prod | rate(1 hour) | POST /internal/cron/slot-offer-expiration |
enterprise-invoice-generation-prod | cron(0 3 1 * ? *) | POST /internal/cron/enterprise-invoice-generation |
enterprise-invoice-overdue-prod | cron(0 10 * * ? *) | POST /internal/cron/enterprise-invoice-overdue |
ticket-auto-close-prod | rate(1 hour) | POST /internal/cron/ticket-auto-close |
recommendation-expiry-prod | cron(0 2 * * ? *) | POST /internal/cron/recommendation-expiry |
recommendation-reminder-prod | cron(0 10 * * ? *) | POST /internal/cron/recommendation-reminder |
Dev cron runs in-process (CRON_MODE=in-process in the dev task def).
This is intentional — running scheduled jobs against the dev database
spams real users with reminder emails / churns credits / etc. To enable
dev cron during a feature test, change the env var on the dev task and
roll the service.
Runtime configuration
ECS migration adds these keys to the Secrets Manager bundles
(aiqlick-backend/development and aiqlick-backend/production):
CRON_MODE in-process | external
INTERNAL_CRON_SECRET <hex 32 bytes> — HMAC for /internal/cron POSTs
INTERNAL_CRON_ALLOW_UNSIGNED true | false (false in prod, true while signing Lambda is being wired)
PUBSUB_FANOUT_BACKEND in-process | sns_sqs
PUBSUB_SNS_TOPIC_ARN arn:aws:sns:eu-north-1:842697652860:aiqlick-pubsub-{env}
AI_BACKEND_GRAPHQL_URL http://bg-{env}.aiqlick.local:8000/graphql (was: hardcoded private IP)
AI_BACKEND_GRAPHQL_WS_URL ws://bg-{env}.aiqlick.local:8000/graphql
AI_BACKEND_VOICE_WS_URL ws://bg-{env}.aiqlick.local:8000/voice/ws
The task definition environment block sets these directly (so they're
visible in ECS console without secret reads). The Secrets Manager bundle
hosts the rest of the existing keys (DATABASE_URL, JWT_SECRET, Stripe, SES,
etc.) and loadSecretBundle() in src/main.ts populates process.env from
that bundle at boot.
Cutover
Per-env steps live in aiqlick-backend/infrastructure/ecs/shared/CUTOVER.md.
Order: dev → soak 7 days → prod. Each cutover takes ~25h end-to-end:
T-25h to lower DNS TTL, T-24h to deploy ECS services with the same image
SHA as EC2, T-1h smoke test via temp api-{env}-ecs.aiqlick.com alias,
T-0 DNS flip in the one.com control panel, T+1h monitoring window, T+24h
drain old EC2 container, T+7d terminate the EC2 + revoke 80/443 ingress
on aiqlick-backend-sg.
Rollback works for the first 24h: revert the DNS A record in one.com, the
old container on the EC2 (still running, ALB just isn't pointing at it)
takes traffic back. Beyond 24h the EC2 container is docker stop'd, so
rollback requires docker start on the host.
Known gotchas
CRON_MODE=external short-circuits in-process @Cron
Every existing @Cron(...)-decorated method checks
if (process.env.CRON_MODE === 'external') return; at the top of its body.
When the env var is external the method exits immediately — EventBridge
Scheduler is the sole driver. Forgetting this on a single task def turns
into "cron jobs fire twice" because both the local scheduler and Scheduler
target the same handler. Always set CRON_MODE=external together with
provisioning Scheduler entries.
Two cron jobs were missing from the original Phase 6 plan
recommendation-expiry and recommendation-reminder (in
recommendations/jobs/)
weren't in the AWS-native migration's Phase 6 inventory — they shipped as
part of the recommendations feature in late April. They are wired into the
EventBridge Scheduler entries here. If a new cron job is added later,
the migration touch-points are:
- Add the file in
src/<feature>/jobs/. - Add the
if (process.env.CRON_MODE === 'external') return;guard. - Add the entry to
src/internal-cron/cron-jobs.registry.ts. - Add the EventBridge Scheduler entry to
aiqlick-backend/infrastructure/ecs/shared/10-eventbridge-scheduler.sh.
The canary at /health/cron (TODO) cross-checks the registry length
against CRON_JOB_NAMES.length to catch drift between the two lists.
Cloud Map TTL must be 60s
The internal ALB target group binds tasks by IP. ECS deregisters a task
from the Cloud Map service on stop and registers the replacement on
start, but the per-record DNS TTL still applies. We set
DnsRecords[].TTL=60 (the lowest legal value) — anything higher means
backend → BG calls during a rolling deploy hit dead ENIs for a minute or
more. The internal ALB itself has stable DNS so the primary call path
is unaffected; the Cloud Map name is a fallback used by direct
discovery callers (e.g. ECS Exec sessions).
NAT gateway is the only egress for cross-region calls
The 9 interface VPC endpoints cover everything ECS tasks need in
eu-north-1. Rekognition / Transcribe / Polly are in eu-west-1 and don't
have eu-north-1 endpoints — those calls go via the NAT gateway in AZ-A.
This adds ~$32/month and ~20-30ms cross-region latency, both already
present on EC2 (Rekognition latency was the same shape there). If the
NAT goes down, BG's face-extraction / transcription pipelines stop —
status-check on aiqlick-nat-eu-north-1a should be in any pager rotation.
ACM DNS validation against one.com is manual (permanent)
DNS stays on one.com as a permanent architectural decision — no Route
53 migration is planned. aws acm request-certificate issues validation
CNAMEs that must be added to the one.com control panel by hand. Budget 1
day for cert issuance the first time you run
infrastructure/ecs/shared/05-acm-cert.sh — it will print the records
and poll until issuance completes. Subsequent renewals are automatic
once the validation CNAMEs are in place (ACM keeps them across
renewal cycles, so this is a one-time operator step per cert).
prisma db push runs as a one-shot ECS task between build and deploy
The CI workflow runs the new image with the
aiqlick-prisma-migrate task family BEFORE
amazon-ecs-deploy-task-definition rolls the long-running service. If
db push fails (Prisma incompatibility, RDS connectivity, etc.), the
deploy aborts and the existing service stays on the old image. There
is NO retry — fix the schema and re-run the workflow.
Cost delta
Approximate monthly figures, eu-north-1 list price, both envs combined:
| Component | Before | After | Δ |
|---|---|---|---|
| 4× EC2 (t3.micro/small + EBS) | ~$48 | $0 | −$48 |
| 4× Fargate (avg 0.75 vCPU / 1.5 GB) | $0 | ~$80 | +$80 |
| Public ALB | $0 | ~$22 | +$22 |
| Internal ALB | $0 | ~$22 | +$22 |
| NAT gateway (1× single-AZ) | $0 | ~$32 | +$32 |
| 9× Interface VPC endpoints | $0 | ~$50 | +$50 |
| Net | ~$48 | ~$206 | +$158 |
The +$158/month buys: HA across 3 AZs (multi-task scheduling), zero-touch ALB-managed TLS, deployment circuit breaker auto-rollback, removal of SSH-based deploys, and unblocking horizontal scale. Eliminating Caddy + Let's Encrypt + manual EC2 maintenance pays for the rest.
Operational notes
How to check if a service is healthy
aws ecs describe-services --profile Administrator-842697652860 --region eu-north-1 \
--cluster aiqlick \
--services backend-prod backend-dev bg-tasks-prod bg-tasks-dev \
--query 'services[*].{name:serviceName,desired:desiredCount,running:runningCount,pending:pendingCount,rollout:deployments[0].rolloutState}' \
--output table
rollout should be COMPLETED and running == desired.
Open a shell inside a running task
TASK=$(aws ecs list-tasks --profile Administrator-842697652860 --region eu-north-1 \
--cluster aiqlick --service-name backend-dev --query 'taskArns[0]' --output text)
aws ecs execute-command --profile Administrator-842697652860 --region eu-north-1 \
--cluster aiqlick --task "$TASK" --container aiqlick-backend \
--interactive --command '/bin/sh'
Replaces ssh ec2-user@$EC2_HOST from the EC2 era. The task role grants
the ssmmessages:* actions needed for ECS Exec.
Tail logs in real time
aws logs tail --profile Administrator-842697652860 --region eu-north-1 \
--follow --format short /ecs/aiqlick-backend-dev
Force a redeploy (no code change)
aws ecs update-service --profile Administrator-842697652860 --region eu-north-1 \
--cluster aiqlick --service backend-dev --force-new-deployment
aws ecs wait services-stable --profile Administrator-842697652860 --region eu-north-1 \
--cluster aiqlick --services backend-dev
The deployment circuit breaker auto-rolls back if the new task definition doesn't pass health checks within 10 minutes.
Where the source of truth lives
| Concern | File |
|---|---|
| Provisioning scripts | aiqlick-backend/infrastructure/ecs/shared/ |
| Backend task defs | aiqlick-backend/infrastructure/ecs/task-definition-{dev,prod}.json |
| Backend IAM | aiqlick-backend/infrastructure/ecs/iam/ |
| Backend service registration | aiqlick-backend/infrastructure/ecs/services/ |
| BG task defs | background-tasks/infrastructure/ecs/task-definition-{dev,prod}.json |
| BG IAM | background-tasks/infrastructure/ecs/iam/ |
| BG service registration | background-tasks/infrastructure/ecs/services/ |
| Cutover runbook | aiqlick-backend/infrastructure/ecs/shared/CUTOVER.md |
| SQS idempotency audit | aiqlick-backend/infrastructure/ecs/sqs-idempotency-audit.md |
| Internal cron module | aiqlick-backend/src/internal-cron/ |
| PubSub fanout module | aiqlick-backend/src/common/pubsub-fanout/ |
Related
- ECS reference — canonical inventory of cluster, services, ALBs
- AWS Infrastructure — full account inventory
- AWS-Native Migration — Phases 0–5 (this is Phase 7)
- Network & DNS — updated topology after Phase 7