Prerequisites

Clone & Configure

git clone https://github.com/Pzharyuk/live-translator-node.git && cd live-translator-node
cp .env.example .env

Edit .env and set your API key:

ELEVENLABS_API_KEY=sk-your-key-here
ADMIN_PASSWORD=your-secure-password

Start Infrastructure

# Start Redis + LibreTranslate
docker compose -f docker-compose.local.yml up -d

# Wait for LibreTranslate to download language models (~500 MB)
docker logs -f $(docker ps -qf "name=libretranslate") 2>&1 | grep -i "running"

Start Backend

cd backend
npm install
npm run dev  # nodemon watches for changes

Start Frontend

cd frontend
npm install
npm run dev  # Vite hot-reload on localhost:5173

System Overview

Data Flow

Key Architecture Decisions

Connection Lifecycle

1. Client sends start_session with source type (mic or youtube) and optional voiceId
2. Backend opens a WebSocket to wss://api.elevenlabs.io/v1/speech-to-text/realtime
3. For YouTube: spawns yt-dlp | ffmpeg child processes to extract PCM audio
4. For Microphone: awaits audio_chunk events from the frontend

Audio Streaming

Audio chunks are sent to Scribe as JSON messages:

{
  "message_type": "input_audio_chunk",
  "audio_base_64": "UklGR..."  // PCM 16-bit LE, 16kHz, mono
}

Scribe Responses

Commit Merge Buffer

After receiving a committed_transcript, the backend waits 2.5 seconds (COMMIT_MERGE_MS) to collect additional commits before translating. This prevents fragmented translations from aggressive VAD.

Stability Timeout

If VAD stalls (no new commits), a 3.5 second fallback timer (STABILITY_TIMEOUT_MS) fires to translate whatever new text has accumulated, preventing indefinite silence.

Text Validation

Before translation, text is validated against EN/RU/UK character regex patterns. This filters out hallucinated text from the STT model (common with silence or background noise).

Provider Chain

The system supports three translation providers with automatic fallback:

Fallback Logic

1. Try primary provider (admin-selected)
2. If primary fails → try configured fallback
3. If fallback fails → try LibreTranslate (last resort)
4. If all fail → emit error event

Language Detection

The app uses a two-layer auto-detection approach:

Layer 1: Script-based Pre-detection

Before calling any translation API, the backend checks Unicode character scripts:

• Cyrillic characters (Unicode 0x0400–0x04FF) → if >50% of matched letters are Cyrillic, detected as Russian
• Latin characters → detected as English
• This avoids low-confidence results from LibreTranslate's /detect endpoint on short text

Layer 2: STT Language Code

When the auto_language_detect flag is enabled, ElevenLabs Scribe returns a language_code with each transcript commit. The backend uses this to correctly route EN/RU/UK without relying solely on script detection.

Language Gating

Detected languages are checked against the admin-approved pool. If a detected language isn't in the allowed set, the translation is rejected to prevent hallucinated language outputs.

TTS Pipeline

After translation, the text is sent to ElevenLabs TTS:

const stream = await client.textToSpeech.stream(voiceId, {
  text: translatedText,
  model_id: "eleven_multilingual_v2",
  output_format: "mp3_44100_128",
  voice_settings: {
    stability: 0.5,
    similarity_boost: 0.75,
    style: 0.0,
    speed: 1.0,
    use_speaker_boost: true
  }
});

Audio Delivery

TTS audio is streamed to a Buffer, then emitted as a base64-encoded MP3 via the tts_audio Socket.io event.

Frontend Playback Queue

The frontend maintains an audio queue to prevent overlapping playback:

1. Received tts_audio events are queued
2. Each segment plays to completion before the next starts
3. A configurable pause (600ms default) is inserted between segments
4. The pause duration is controlled by tts_segment_pause_ms (adjustable in admin)

Microphone Input

1. User selects "Mic" tab and chooses a TTS voice
2. Browser captures audio via Web Audio API's ScriptProcessor
3. PCM 16-bit LE at 16kHz sample rate sent to backend via Socket.io
4. Backend pipes audio to ElevenLabs Scribe v2 Realtime WebSocket
5. Language auto-detected (EN/RU/UK), text translated and synthesized
6. TTS audio returned and played back with inter-segment pauses

YouTube Input

1. User pastes a YouTube URL (live stream or video)
2. Backend spawns yt-dlp | ffmpeg child processes
3. Audio extracted as PCM stream (16kHz, 16-bit LE, mono)
4. Piped to Scribe v2, same pipeline as microphone
5. Stream ends when YouTube content ends or user stops

User Interface

The user view features a dark cavern theme with:

• Waveform visualizer — Canvas-based bar chart with orange gradient and cyan tips
• Transcript display — White translated text scrolls upward with fade masks
• Partial transcript — Shown in italic orange while STT is processing
• Source tabs — Toggle between Mic and YouTube (controlled by feature flags)

How It Works

The backend uses yt-dlp and ffmpeg as child processes to extract audio from YouTube URLs:

yt-dlp (best audio) → ffmpeg (PCM 16kHz 16-bit LE mono) → Scribe v2

Supported Sources

• Live streams — Translates in real-time as the stream progresses
• Regular videos — Processes the full audio track
• Any URL supported by yt-dlp (YouTube, etc.)

Requirements

Both yt-dlp and ffmpeg must be installed and available in the system PATH. On macOS:

brew install yt-dlp ffmpeg

Overview

The Biblical Transcript Simulator is an admin-only feature that generates biblical text passages using Google's Gemini API (gemini-2.5-flash), then routes them through the full translation pipeline. This provides a hands-free way to test STT → Translation → TTS without a live audio source.

Language Styles

Flow

1. Admin selects language (EN/RU/UK)
2. Backend calls Gemini 2.5 Flash with streaming
3. Gemini generates 6-8 biblical passages, 3-5 sentences each
4. Stream is buffered until 140+ characters AND complete sentences
5. Chunks emitted with 1800ms smooth pacing between them
6. Each chunk flows through the standard pipeline:
   • Emitted as transcript (isFinal: true)
   • Auto-translated via configured provider
   • TTS synthesized and audio returned
7. Frontend plays audio with standard inter-segment pause

Overview

Voice Training uses ElevenLabs' Instant Voice Cloning (IVC) API to create custom voices from audio samples. Once cloned, the voice appears in the voice selector immediately.

From Microphone

1. Open the Voice Training section in the admin panel
2. Click Generate Text to get an AI-generated reading passage (via Gemini) — gives the speaker natural, phonetically diverse text to read aloud
3. Record multiple audio clips using your browser microphone while reading the generated text
4. Provide a name for the voice
5. Clips are uploaded to ElevenLabs IVC API
6. Cloned voice is available for TTS immediately
7. Click Preview Voice to hear the cloned voice speak a sample sentence via TTS

From YouTube

1. Paste a YouTube URL in the Voice Training section
2. Backend extracts N × 30-second clips via yt-dlp + ffmpeg
3. Clips are uploaded to ElevenLabs IVC API
4. Resulting voice is stored in your ElevenLabs account

Concepts

Admin Controls

• Change the active language pair via the admin panel
• Changes broadcast to all connected clients in real-time
• Manage the available languages pool for viewer selection

Viewer Selection

When the user_language_selector feature flag is enabled, viewers can override the admin-set language pair by selecting their own preferred languages from the available pool.

Overview

Two people can video call each other through the app, each speaking their own language. The app transcribes, translates, and synthesizes speech in real-time so each participant hears the other in their language.

How It Works

1. Create a room — Person A selects their language, picks a TTS voice, and clicks "Create Room". A 6-character room code is generated.
2. Share the code — Person A shares the room code with Person B (copy button provided).
3. Join the room — Person B enters the code, selects their language and TTS voice, and clicks "Join".
4. WebRTC connection — The app establishes a peer-to-peer video connection via WebRTC (signaled through Socket.io). Video flows directly between browsers.
5. Audio translation — Each participant's microphone audio is simultaneously:
   • Sent to the peer via WebRTC (but muted on their end)
   • Captured as PCM chunks and sent to the backend via Socket.io for STT
6. Translation pipeline — Each participant has their own independent Scribe STT session. Transcribed text is translated to the other participant's language, then synthesized via ElevenLabs TTS and sent back to the peer.
7. Playback — The peer hears the TTS translation instead of the raw audio. Translated transcript is displayed below the video.

Architecture

Person A (Browser)         Server              Person B (Browser)
├─ getUserMedia            ├─ Socket.io         ├─ getUserMedia
├─ WebRTC P2P ═══video═══►│  (signaling)  ◄═══ ├─ WebRTC P2P
│                          │                    │
├─ PCM chunks ──Socket.io─►├─ ScribeA(STT)      │
│                          │  ↓ translate       │
│                          │  ↓ TTS ───────────►├─ Plays TTS
│                          │                    │
│  Plays TTS ◄─────────────├─ ScribeB(STT) ◄───├─ PCM chunks
│  (remote video muted)    │  ↓ translate       │  (remote video muted)
└──────────────────────────┴────────────────────┘

Socket Events

Room Lifecycle

• Rooms are stored in Redis with key video_room:{code} and a 4-hour TTL
• Maximum 2 participants per room
• When one participant disconnects, the other is notified and the call ends
• Scribe sessions are automatically cleaned up on disconnect

The Mac Audio Agent has moved to its own public repository:

github.com/Pzharyuk/live-translator-agent

It is a lightweight Node.js daemon that runs as a macOS LaunchAgent and streams microphone audio to the live-translator backend via Socket.io — eliminating the need to open a browser for the Remote Audio Source role.

Pre-shared key authentication

Any socket that emits register_audio_source must present the server's pre-shared key in the Socket.IO handshake (auth.agentPsk). This stops random clients from connecting to the backend and impersonating an agent.

• Server: set AGENT_PSK (env var) — surfaces as auth.agent_psk in application.yaml. An empty value disables enforcement and logs a warning on every registration.
• Mac daemon: add agentPsk to ~/.config/live-translator-agent/config.json (or set the AGENT_PSK env var — env wins).
• Browser /audio-source: paste the key into the new Agent Pre-Shared Key field; it is stored in localStorage on that device only and travels in the handshake (never in event payloads).
• Mismatch behaviour: server logs register_audio_source REJECTED ... invalid or missing PSK, emits agent_auth_error to the client, then disconnects.

File Structure

The APP_ENV environment variable (local or prod) determines which overlay file is loaded on top of the base config.

Full Configuration Reference

server:
  port: 3001
  cors_origin: "http://localhost:5173"

elevenlabs:
  api_key: "${ELEVENLABS_API_KEY}"
  default_voice_id: "kxj9qk6u5PfI0ITgJwO0"
  tts_model: "eleven_multilingual_v2"
  tts_settings:
    stability: 0.5
    similarity_boost: 0.75
    style: 0.0
    speed: 1.0
    use_speaker_boost: true
  stt_model: "scribe_v2"

anthropic:
  api_key: "${ANTHROPIC_API_KEY}"

deepl:
  api_key: "${DEEPL_API_KEY}"

libretranslate:
  url: "http://libretranslate:5000"
  api_key: ""

redis:
  host: "redis"
  port: 6379
  password: ""

feature_flags:
  youtube_input: true
  mic_input: true
  auto_language_detect: true
  user_language_selector: false
  audio_device_selector: true
  video_translation: false
  video_voice_cloning: false
  broadcast: false

audio:
  sample_rate: 16000
  channels: 1
  chunk_duration_ms: 250

translation:
  source_lang: "auto"
  target_lang_en: "en"
  target_lang_ru: "ru"
  provider: "libretranslate"
  fallback: "libretranslate"

Environment Variable Interpolation

YAML values using ${VAR_NAME} syntax are automatically replaced with the corresponding environment variable at startup.

API Keys Management

Body: { elevenlabs?: string; anthropic?: string; deepl?: string; libretranslate?: string; google?: string; youtube?: string }

Voice Management

Body: { voiceIds: string[] }

TTS & STT Settings

Body: Partial<{ stability: number; similarity_boost: number; style: number; speed: number; use_speaker_boost: boolean }>

Body: Partial<SttTimingSettings> (commit_merge_ms, stability_timeout_ms, vad_threshold, etc.)

Body: { text: string; voiceId?: string; format?: 'mp3' | 'pcm' }

Languages & Translation

Body: { languages: [string, string] }

Body: { languages: string[] }

Body: { provider: 'google' | 'deepl' | 'claude' | 'libretranslate' }

Body: { model: string }

Audio Device

Body: { deviceId?: string; label?: string }

Video Call Settings

Body: Partial<{ stability_ms: number; commit_merge_ms: number; translation_provider: string }>

Feature Flags

Body: { value: boolean }

Broadcast Schedule

Body: { events: ScheduleEvent[] }

Voice Training & Cloning

Body: { name: string; clips: string[]; mimeType?: string }

Body: { name: string; youtubeUrl: string; clipCount?: number; startOffset?: number }

Content Generation

Body: { apiKey?: string; language?: string; sentences?: number }

YouTube Integration

Body: { channelId: string }

Monitoring & Diagnostics

Body: { words: string[] }

Video Room Moderation

Session History

User Management

Body: { isAdmin?: boolean; roleId?: string | null; roleIds?: string[] }

Body: { password: string }

Roles & Permissions

Body: { name: string; permissions: Permission[] }

Body: { name: string; permissions: Permission[] }

Legacy / Deprecated

SDK

Uses the official @elevenlabs/elevenlabs-js SDK (v2). The client is lazy-loaded on first use.

Speech-to-Text (Scribe v2 Realtime)

Connects via native WebSocket to wss://api.elevenlabs.io/v1/speech-to-text/realtime. Handles:

• VAD-based commit buffering with configurable merge window
• Stability timeout fallback for stalled VAD
• Text validation (EN/RU/UK character regex filtering)
• Partial and final transcript emission

Text-to-Speech

Uses client.textToSpeech.stream() with the eleven_multilingual_v2 model. Audio is collected into a Buffer and emitted as base64 MP3.

Voice Management

• client.voices.getAll() — fetches all voices from account
• Admin can filter which voices are available to viewers
• Voice cloning via IVC API (from recordings or YouTube)

Key File

backend/src/services/elevenlabs.service.ts

Provider Details

Google Translate

Google Cloud Translation API v2. Fast (~200ms), deterministic, and reliable. Requires GOOGLE_TRANSLATE_API_KEY with the Cloud Translation API enabled in Google Cloud Console. Ensure the API key has no HTTP referrer restrictions (server-side requests have no referrer).

File: backend/src/services/google-translate.service.ts

LibreTranslate

Self-hosted in Docker. No API key required by default. Provides language detection and translation via REST API.

File: backend/src/services/libretranslate.service.ts

DeepL

Premium translation API. Auto-detects free vs. paid endpoint based on the API key format.

File: backend/src/services/deepl.service.ts

Claude (Anthropic)

AI-powered translation using claude-haiku-4-5 for speed. Includes language detection and auto-flip logic.

File: backend/src/services/claude-translate.service.ts

Routing

Provider routing is handled by backend/src/services/translation.provider.ts:

1. Try admin-selected primary provider
2. On failure, try configured fallback provider
3. LibreTranslate is always the last-resort fallback

Connection

Uses ioredis with automatic retry strategy. Falls back to in-memory/YAML defaults if Redis is unavailable.

Key Patterns

Key File

backend/src/services/redis.service.ts

Local Development

Use docker-compose.local.yml for Redis and LibreTranslate only (backend/frontend run natively):

docker compose -f docker-compose.local.yml up -d

Production

Use docker-compose.yml for all services:

docker compose up -d --build

Services

Configuration

ELEVENLABS_API_KEY=sk-your-production-key
ADMIN_PASSWORD=strong-secure-password
FRONTEND_URL=https://translate.example.com
APP_ENV=prod
REDIS_PASSWORD=redis-secret

Deploy

docker compose up -d --build

Reverse Proxy

When running behind Nginx or another reverse proxy:

• Set LISTEN_PORT in .env (e.g., 8080)
• Proxy pass to localhost:8080
• Important: Ensure WebSocket upgrades are forwarded for the /socket.io/ path

server {
    listen 443 ssl;
    server_name translate.example.com;

    location / {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }
}

Monitoring

# Check all services
docker compose ps

# View backend logs
docker compose logs -f backend

# Health check
curl http://localhost:3001/api/health