Monitoring and Operations

This page covers health checking, observability, deployment verification, and operational tooling for the AIQLick platform.

Health Check Endpoints

Each service exposes health endpoints used by deployment pipelines and monitoring systems.

Service	Endpoint	Type	Description
Backend	/health	Liveness	Basic process health
Backend	/healthz	Liveness	ALB health check target
AI Service	/health	Liveness	Basic process health
AI Service	/health/ready	Readiness	Component-level status
Jitsi	https://book.aiqlick.com/	HTTP	Web interface reachability

Backend Health

The backend exposes two liveness endpoints (/health and /healthz) that return a 200 status when the NestJS process is running and able to serve requests.

During deployment, the CI/CD pipeline additionally verifies GraphQL availability by running an introspection query against /graphql.

Background Tasks Readiness

The /health/ready endpoint on the background tasks service returns detailed component status:

{
  "status": "healthy",
  "components": {
    "database": "healthy",
    "match_score_generator": "healthy",
    "email_worker": "healthy",
    "cv_worker": "healthy",
    "document_worker": "healthy"
  }
}

Each component reports independently. A degraded component does not necessarily mean the entire service is unhealthy -- other operations may continue to function.

Match Score Generator Metrics

The background tasks service exposes operational metrics for the match score generator at:

GET /api/v1/background-tasks/match-score-generator/metrics

Response fields:

Metric	Description
total_processed	Total number of match scores calculated since last restart
processing_rate	Current throughput (matches per second)
error_rate	Percentage of failed match calculations
circuit_breaker_state	Current state: CLOSED, OPEN, or HALF-OPEN

Circuit Breaker Pattern

The match score generator implements a circuit breaker to prevent cascading failures when downstream services (Bedrock LLM) are degraded.

CLOSED ──(5 consecutive failures)──> OPEN ──(60s cooldown)──> HALF-OPEN
  ^                                                              |
  |                                                              |
  └────────────────(success)─────────────────────────────────────┘
                                     |
                              (failure) -> OPEN

State	Behavior
CLOSED	Normal operation. Requests pass through to the LLM. Failures are counted.
OPEN	All requests are immediately rejected without calling the LLM. Entered after 5 consecutive failures.
HALF-OPEN	After 60 seconds in OPEN state, a single test request is allowed through. Success returns to CLOSED; failure returns to OPEN.

Deployment Verification

Every CI/CD pipeline includes post-deployment health checks to verify successful rollouts.

Health Check Configuration

Parameter	Value
Maximum attempts	20
Interval between attempts	3 seconds
Total timeout	60 seconds

Backend Deployment Checks

1. GET /health returns 200.
2. GraphQL introspection query at /graphql succeeds.
3. If prisma db push was run, schema migration is verified.

Background Tasks Deployment Checks

1. GET /health returns 200.
2. GET /health/ready returns healthy status for all components.

Automatic Rollback (Background Tasks)

The background tasks deployment pipeline maintains a :previous Docker image tag in ECR. If health checks fail after deploying a new image:

1. The new container is stopped.
2. The :previous image is pulled and started.
3. Health checks run again to confirm the rollback succeeded.

This rollback mechanism is unique to the background tasks service due to the complexity of its dependencies (Bedrock, Rekognition, Transcribe, database workers).

Logging

Structured Logging

The background tasks service uses structured logging with contextual fields. Log entries include:

• Timestamp
• Log level
• Module/component name
• Request or operation context (correlation IDs, user IDs, company IDs)
• Duration for timed operations

Admin Live Log Streaming

Super administrators can access real-time log streaming through the frontend admin panel. This is powered by the logStream GraphQL subscription:

subscription {
  logStream {
    timestamp
    level
    message
    context
  }
}

The subscription is filtered to super admin users only. It connects via WebSocket through the backend's GraphQL subscription endpoint, which uses Redis pub/sub for message delivery.

The frontend admin panel provides two log viewing interfaces:

• LiveTail: Real-time streaming of log entries as they occur.
• LogExplorer: Historical log search and filtering.

Useful AWS CLI Commands

Check EC2 Instance Status

aws ec2 describe-instance-status \
  --profile Administrator-842697652860 \
  --region eu-north-1 \
  --instance-ids i-0b9f31f3236c25b7f i-053ddc2f75899c06a \
  --query 'InstanceStatuses[].{ID:InstanceId,State:InstanceState.Name,Status:InstanceStatus.Status}' \
  --output table

View Recent ECR Images

aws ecr describe-images \
  --profile Administrator-842697652860 \
  --region eu-north-1 \
  --repository-name aiqlick-backend \
  --query 'imageDetails | sort_by(@, &imagePushedAt) | [-3:]' \
  --output table

Check RDS Instance Status

aws rds describe-db-instances \
  --profile Administrator-842697652860 \
  --region eu-north-1 \
  --query 'DBInstances[].{ID:DBInstanceIdentifier,Status:DBInstanceStatus,Class:DBInstanceClass}' \
  --output table

Run Remote Command via SSM

# Check Docker containers on production backend
aws ssm send-command \
  --profile Administrator-842697652860 \
  --region eu-north-1 \
  --instance-ids i-0b9f31f3236c25b7f \
  --document-name "AWS-RunShellScript" \
  --parameters commands=["docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"] \
  --output json

Check S3 Bucket Size

aws s3 ls s3://aiqlick-uploads/ \
  --profile Administrator-842697652860 \
  --summarize --human-readable

Service URLs

Production

Service	URL
Frontend	https://www.aiqlick.com
Backend API	https://api.aiqlick.com
Background Tasks	https://ai.aiqlick.com
Jitsi Meet	https://book.aiqlick.com
File Storage CDN	https://storage.aiqlick.com

Development

Service	URL
Frontend	https://dev.aiqlick.com
Backend API	https://api-dev.aiqlick.com
Background Tasks	https://ai-dev.aiqlick.com
File Storage CDN	https://storage-dev.aiqlick.com