Voice Service

The voice service enables spoken conversations with AI agents. Users speak into the microphone, audio is streamed to background-tasks over WebSocket, transcribed via AWS Transcribe, and the transcript is sent as a chat message through the existing agentChat pipeline. AI responses can be synthesized back to speech via Amazon Polly.

info

This is a UI convenience layer — not an intelligent agent. The voice service handles only STT (speech-to-text) and TTS (text-to-speech). All intelligence, RAG, tools, and conversation management is handled by the existing agentChat subscription.

Architecture

Key points:

• STT uses AWS Transcribe Streaming (same provider as meeting transcription) — NOT Bedrock
• TTS uses Amazon Polly Neural — NOT Bedrock
• Both services run in eu-west-1 (Ireland) — Polly neural voices and Transcribe are not available in eu-north-1
• The WebSocket connection is a simple audio transport layer; the agentChat subscription handles all AI logic

WebSocket Protocol

Endpoint

wss://{AI_SERVICE_URL}/voice/ws/{conversation_id}

Environment	URL
Production	wss://ai.aiqlick.com/voice/ws/{conversation_id}
Development	wss://ai-dev.aiqlick.com/voice/ws/{conversation_id}

The frontend connects via the NestJS voice WS proxy at ${NEXT_PUBLIC_API_URL}/voice/ws/<conversationId>?token=<userJwt> — the gateway terminates the user JWT, mints an internal-service JWT, and opens an upstream WS to background-tasks. There is no direct frontend → background-tasks voice connection.

Client → Server Messages

Start session:

{"type": "start", "language": "en-US", "voice_id": "tiffany"}

Audio data: Binary PCM chunks (16kHz, 16-bit, mono). No JSON wrapping — raw bytes sent directly.

Text-to-speech request:

{"type": "synthesize", "text": "Hello world", "voice_id": "tiffany", "language": "en-US"}

End session: Triggers final transcription of remaining audio buffer and returns a non-partial transcript.

{"type": "end"}

Server → Client Messages

Session started:

{"type": "session_started", "session_id": "uuid", "language": "en-US", "voice_id": "tiffany"}

Transcript (committed segment):

{"type": "transcript", "text": "What are my top candidates", "is_partial": false, "confidence": 0.95}

Each transcript event is a committed, non-overlapping audio segment — is_partial is always false. The handler emits one segment whenever the audio buffer crosses the process threshold, and one final segment when the client sends {"type": "end"}. Concatenating the text of every emitted segment in order yields the full utterance with no duplicate words.

Behaviour change (SUP-00153, 2026-04-24)

Previously the partial path retained a 0.5s audio overlap between chunks and emitted is_partial: true for the in-progress text. Because consecutive partials transcribed overlapping audio, joining them at end-of-session produced doubled words at every chunk boundary (e.g. "…this is a test test with several…"). The current implementation clears the buffer after each transcribe call so every emitted segment is self-contained, and flips is_partial to false since there is no longer an interim/final relationship to manage. The trade-off is no growing-word "live caption" — captions arrive in ~1s committed bursts.

Audio response (TTS):

{"type": "audio", "data": "base64-encoded-pcm", "duration_ms": 1234, "voice_id": "tiffany", "is_final": true}

Session complete:

{"type": "complete", "transcript": "full transcript text", "session_id": "uuid"}

Error:

{"type": "error", "code": "TRANSCRIPTION_FAILED", "message": "Speech transcription failed."}

Audio Specification

Property	Value
Sample rate	16,000 Hz (16kHz)
Bit depth	16-bit signed integer
Channels	1 (mono)
Format	Linear PCM (little-endian)
Output sample rate	16,000 Hz (same as input)

Buffer Management

The WebSocket handler buffers incoming audio to optimize transcription calls:

Threshold	Size	Purpose
Minimum buffer	16,000 bytes (~0.5s)	Minimum audio before processing
Process threshold	32,000 bytes (~1s)	Triggers partial transcription
Maximum buffer	960,000 bytes (~30s)	Memory protection limit

After each partial transcribe the buffer is fully cleared so the next segment is self-contained — no overlap, no duplicated words at chunk boundaries (SUP-00153). When the buffer exceeds the maximum size before the threshold fires, it is truncated to keep the last 30 seconds as a memory-safety guard.

Supported Languages

10 languages supported for both STT and TTS:

Language	Code
English (US)	en-US
English (UK)	en-GB
English (Indian)	en-IN
English (Australian)	en-AU
Spanish (Spain)	es-ES
German	de-DE
French	fr-FR
Italian	it-IT
Portuguese (Brazilian)	pt-BR
Hindi	hi-IN

warning

Swedish is not supported. This is a limitation of the underlying AWS services (Transcribe language support and Polly neural voice availability).

Available Voices

Two voice personas mapped to language-specific Amazon Polly neural voices:

Voice ID	Gender	Default
tiffany	Female	Yes
matthew	Male	No

Unlike polyglot models, Polly neural voices are language-specific. Each (voice_id, language) combination maps to a different Polly VoiceId:

Tiffany (Female) Voice Mapping

Language	Polly VoiceId
en-US	Joanna
en-GB	Amy
en-IN	Kajal
en-AU	Olivia
es-ES	Lucia
de-DE	Vicki
fr-FR	Lea
it-IT	Bianca
pt-BR	Camila
hi-IN	Kajal

Matthew (Male) Voice Mapping

Language	Polly VoiceId
en-US	Matthew
en-GB	Brian
en-IN	Kajal*
en-AU	Olivia*
es-ES	Sergio
de-DE	Daniel
fr-FR	Remi
it-IT	Adriano
pt-BR	Thiago
hi-IN	Kajal*

*No male neural voice available for this language — falls back to the best available voice.

Session Management

Each voice conversation creates a VoiceSession with the following lifecycle:

Property	Value
Session timeout	600 seconds (10 minutes)
Cleanup interval	Every 60 seconds
Stale session handling	Auto-ended after timeout

Sessions track total audio bytes, partial transcripts, language, voice, and active status.

Billing Integration

Voice operations are metered through the credit system as VOICE_CHAT operations. Both entry points — the WebSocket (streaming STT+TTS) and the synthesizeSpeech GraphQL mutation (standalone TTS) — run through BillingGate.

WebSocket path (/voice/ws/{conversation_id})

1. Pre-check — BillingGate.pre_check() validates subscription and credit balance when the WebSocket connects. If billing fails, the connection is accepted then closed with code 1008 and a BILLING_ERROR message.
2. STT consumption — Credits consumed per transcription call (both partial and final). Logged with description "Voice STT (partial/final): N bytes".
3. TTS consumption — Credits consumed per synthesis call. Logged with description "Voice TTS: N chars -> N bytes".
4. Logging — All operations recorded in AIOperationLog with model_id="nova-sonic" and operation_type="VOICE_CHAT".

GraphQL mutation path (synthesizeSpeech)

As of 2026-04-20 the standalone synthesizeSpeech mutation also runs through the full BillingGate lifecycle (it previously bypassed billing entirely):

1. Input — SynthesizeSpeechInput accepts optional company_id and user_id; unauthenticated / system calls silently skip billing (matches cv_extraction convention).
2. Pre-check — BillingGate.pre_check(company_id, "VOICE_CHAT", model_id="amazon.nova-2-sonic-v1:0", user_id=...). Enforces subscription status (incl. the UNPAID enterprise block), plan limits, and credit balance. Failure → log_rejected + SynthesizeSpeechResponse(success=false).
3. Synthesis — runs Amazon Polly (or Bedrock Nova Sonic when configured). On exception → log_failure + SynthesizeSpeechResponse(success=false).
4. Consumption — BillingGate.check_and_consume(..., input_tokens=len(text), output_tokens=0, duration_ms=...). Credit cost is driven by the CreditCostConfig row for amazon.nova-2-sonic-v1:0 (flat-cost pricing: $0.0012 × (1 + markup_percent/100)). Enterprise POSTPAID companies are logged with metadata.billing_mode = "POSTPAID" and NOT charged.

note

model_id differs between the two paths:

• WebSocket: "nova-sonic" (historical short form)
• GraphQL mutation: "amazon.nova-2-sonic-v1:0" (full Bedrock model ID)

Both resolve to the same CreditCostConfig row via the model-default fallback (_resolve_rates_with_source in cost_calculator.py — exact {modelId}:VOICE_CHAT match takes precedence, then {modelId}: model default, then the hardcoded NOVA_SONIC_PRICING entry in aws_pricing.py).

info

For enterprise (POSTPAID) subscriptions: pre_check returns balance=infinity without consuming credits; check_and_consume writes the log entry with billing_mode=POSTPAID but does NOT deduct. UNPAID POSTPAID is blocked via BillingGate._check_enterprise_blocked before synthesis runs.

Cost Estimates

Service	Pricing	Example
AWS Transcribe	$0.024/min	1 hour of transcription: ~$1.44
Amazon Polly Neural	$16.00/1M chars	10,000 chars (~2,000 words): ~$0.16

Error Codes

Code	Description	Connection
VOICE_DISABLED	Service is disabled via configuration	Closed (1008)
SERVICE_UNAVAILABLE	Health check failed	Closed (1013)
BILLING_ERROR	Insufficient credits or inactive subscription	Closed (1008)
SYNTHESIS_FAILED	Text-to-speech synthesis failed	Stays open
TRANSCRIPTION_FAILED	Speech-to-text transcription failed	Stays open
INVALID_JSON	Malformed JSON control message	Stays open
INTERNAL_ERROR	Unexpected server error	Stays open

Configuration

Environment Variable	Default	Description
NOVA_SONIC_ENABLED	true	Enable/disable voice service
NOVA_SONIC_MODEL_ID	amazon.nova-2-sonic-v1:0	Model ID (kept for config compatibility)
NOVA_SONIC_REGION	eu-north-1	Region for Bedrock client (health check only)
NOVA_SONIC_DEFAULT_LANGUAGE	en-US	Default language
NOVA_SONIC_DEFAULT_VOICE	tiffany	Default voice
NOVA_SONIC_SAMPLE_RATE	16000	Audio sample rate (Hz)
NOVA_SONIC_SESSION_TIMEOUT	600	Session timeout in seconds

note

The NOVA_SONIC_* env var names are retained for backward compatibility. STT actually uses AWS Transcribe (eu-west-1) and TTS uses Amazon Polly (eu-west-1). The Bedrock client (configured by NOVA_SONIC_REGION) is only used for the health check endpoint.

IAM Permissions

The aiqlick-ecs-bg-task-role-{dev,prod} ECS task roles carry the following permissions for voice services via the inline BgTaskPolicy:

{
  "Sid": "PollyTextToSpeech",
  "Effect": "Allow",
  "Action": ["polly:SynthesizeSpeech", "polly:DescribeVoices"],
  "Resource": "*"
},
{
  "Sid": "TranscribeSpeechToText",
  "Effect": "Allow",
  "Action": [
    "transcribe:StartTranscriptionJob",
    "transcribe:GetTranscriptionJob",
    "transcribe:DeleteTranscriptionJob",
    "transcribe:ListTranscriptionJobs",
    "transcribe:StartStreamTranscription"
  ],
  "Resource": "*"
}

The canonical source lives at background-tasks/infrastructure/ecs/iam/.

Implementation Files

File	Purpose
background-tasks/app/services/agent/voice_service.py	Core service: STT via AWS Transcribe, TTS via Polly, session management
background-tasks/app/api/voice_ws.py	WebSocket handler: audio buffering, billing, error reporting
background-tasks/app/graphql/mutations/voice.py	GraphQL synthesizeSpeech / startVoiceSession / voiceHealthCheck mutations — standalone TTS with full BillingGate integration
background-tasks/app/graphql/types/voice.py	SynthesizeSpeechInput (optional company_id / user_id), response types, voice enums
aiqlick-frontend/lib/hooks/useVoiceChat.ts	React hook: microphone capture, WebSocket transport, audio playback
aiqlick-frontend/components/reusable/chatbot/ChatModal.tsx	UI integration: voice mode toggle, push-to-talk, transcript handling

Related

• Voice Chat (Frontend) — Frontend hook and ChatModal integration
• AI Models — Model inventory and pricing
• Bedrock & AI Services — AWS Bedrock configuration
• AI Agents — Agent system that voice integrates with