Skip to main content

SSM Operations

AWS Systems Manager (SSM) is the only path into running workloads — there are no inbound SSH ports and no per-host SSH keys to rotate. Two flavours apply, depending on whether you need to reach an ECS Fargate task or a standalone EC2 instance:

WorkloadPathNotes
backend-{dev,prod} taskaws ecs execute-commandDaily-driver. ECS Exec is enabled on every aiqlick service.
bg-tasks-{dev,prod} taskaws ecs execute-commandSame as above.
aiqlick-jitsi EC2aws ssm start-session / send-commandOnly EC2 left in the account.

The legacy aiqlick-{ai,backend}{,-dev} EC2s were terminated in the Phase 7 cleanup of 2026-05-14. For application debugging on backend or background-tasks, use ECS Exec — there is no fallback path anymore.

All commands below assume --profile Administrator-842697652860 --region eu-north-1.

ECS Exec (primary path for backend + BG)​

Open a shell in a running ECS task. Replaces ssh ec2-user@$EC2_HOST from the EC2 era. Each task has its own ENI, so you're sshing into a specific replica — autoscaling and rolling deploys mean the task ID changes; always re-resolve before connecting.

One-liner: open a shell in the current backend-dev task​

TASK=$(aws ecs list-tasks \
  --profile Administrator-842697652860 --region eu-north-1 \
  --cluster aiqlick --service-name backend-dev \
  --query 'taskArns[0]' --output text)

aws ecs execute-command \
  --profile Administrator-842697652860 --region eu-north-1 \
  --cluster aiqlick --task "$TASK" --container aiqlick-backend \
  --interactive --command '/bin/sh'

Same shape for the other three services — substitute service + container name:

ServiceContainer name
backend-dev, backend-prodaiqlick-backend
bg-tasks-dev, bg-tasks-prodbackground-tasks

Container env vars (post-bundle resolution)​

ECS injects secrets via a single bundle JSON, not as discrete env vars. Inside the container, process.env.DATABASE_URL is set by bootstrap-env.ts after parsing _BUNDLE_JSON. Reading the env via docker exec-style probes from outside doesn't work the same way as on EC2 — see ECS Fargate for the bundle resolution model.

To get a specific value out of _BUNDLE_JSON from a session:

echo "$_BUNDLE_JSON" | python3 -c 'import json,sys,os; \
  print(json.loads(sys.stdin.read()).get("DATABASE_URL", ""))'

One-shot diagnostic without an interactive shell​

aws ecs execute-command \
  --profile Administrator-842697652860 --region eu-north-1 \
  --cluster aiqlick --task "$TASK" --container aiqlick-backend \
  --interactive --command "sh -c 'cat /etc/os-release; node --version'"

Tail logs (CloudWatch, not docker logs)​

ECS Fargate writes to CloudWatch via the awslogs driver — there's no local docker logs to read.

aws logs tail \
  --profile Administrator-842697652860 --region eu-north-1 \
  --follow --format short \
  /ecs/aiqlick-backend-prod

Available log groups: /ecs/aiqlick-{backend,bg-tasks}-{dev,prod}, /ecs/aiqlick-prisma-migrate.

Force a redeploy without code change​

aws ecs update-service \
  --profile Administrator-842697652860 --region eu-north-1 \
  --cluster aiqlick --service backend-prod --force-new-deployment

The deployment circuit breaker auto-rolls back on health-check failure.

EC2 SSM (Jitsi + zombies)​

The classic SSM commands still apply for the EC2 instances. There's exactly one production case: Jitsi. The other four EC2s are still reachable via SSM but you almost never want to use them.

Instance ID reference​

InstanceInstance IDPurpose
aiqlick-jitsii-0620d2e23695f5bfcJitsi Meet — only EC2 in the account

The four legacy backend / AI EC2s (aiqlick-{backend,ai}{,-dev}) and the GHA self-hosted runner (i-0b13fda708cf21e54) were terminated in the Phase 7 cleanup of 2026-05-14. All workflows now run on ubuntu-latest.

Interactive session​

aws ssm start-session \
  --profile Administrator-842697652860 --region eu-north-1 \
  --target i-0620d2e23695f5bfc        # Jitsi

Requires the Session Manager plugin: brew install --cask session-manager-plugin.

Run a command without an interactive session​

aws ssm send-command \
  --profile Administrator-842697652860 --region eu-north-1 \
  --instance-ids i-0620d2e23695f5bfc \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker ps --format \"table {{.Names}}\t{{.Status}}\""]' \
  --output json

Then retrieve the output via the returned CommandId:

aws ssm get-command-invocation \
  --profile Administrator-842697652860 --region eu-north-1 \
  --command-id "<CommandId>" \
  --instance-id i-0620d2e23695f5bfc \
  --query 'StandardOutputContent' --output text

Common Jitsi operations​

# Container status
aws ssm send-command --profile Administrator-842697652860 --region eu-north-1 \
  --instance-ids i-0620d2e23695f5bfc --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker ps --filter name=jitsi"]'

# Tail JVB logs
aws ssm send-command --profile Administrator-842697652860 --region eu-north-1 \
  --instance-ids i-0620d2e23695f5bfc --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker logs --tail 100 jitsi-jvb-1"]'

# Restart Jicofo
aws ssm send-command --profile Administrator-842697652860 --region eu-north-1 \
  --instance-ids i-0620d2e23695f5bfc --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker restart jitsi-jicofo-1"]'

SSM in CI/CD​

Only the Jitsi deploy uses SSM in CI/CD today (see CI/CD pipelines). The backend, background-tasks, frontend, and documentation deploys all use ECS APIs / Amplify / S3 sync — not SSM.

# Jitsi deploy (one-line dispatch from GitHub Actions)
aws ssm send-command \
  --instance-ids i-0620d2e23695f5bfc \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["cd /opt/jitsi/jitsi-deploy && git pull && docker compose pull && docker compose up -d --force-recreate"]' \
  --region eu-north-1

Troubleshooting​

"TargetNotConnected" on ECS Exec​

Either the task isn't running ECS Exec (verify enableExecuteCommand: true in the task description) or the SSM agent inside the container hasn't started yet — give a freshly-deployed task ~30 seconds and retry.

"InvalidInstanceID.NotFound" on SSM send-command​

Instance was terminated. Most likely you're targeting one of the legacy aiqlick-{backend,ai}{,-dev} IDs or the old GHA runner (i-0b13fda708cf21e54) — all gone since the Phase 7 cleanup. Only aiqlick-jitsi (i-0620d2e23695f5bfc) remains.

Output truncated​

send-command truncates output at 24,000 characters. For large outputs (verbose logs), use --output-s3-bucket-name to write to S3, or use Session Manager (no truncation):

aws ssm send-command --profile Administrator-842697652860 --region eu-north-1 \
  --instance-ids i-0620d2e23695f5bfc --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker logs --tail 500 jitsi-jvb-1"]' \
  --output-s3-bucket-name aiqlick-deployments \
  --output-s3-key-prefix ssm-output/