Live Translator - Documentation

⚙

Installation

Set up the project locally with Docker, Redis, and LibreTranslate in minutes.

▦

Architecture

Understand the STT → Translation → TTS pipeline and real-time Socket.io communication.

▶

Live Translation

Stream from YouTube or microphone with automatic EN/RU/UK language detection and voice output.

📜

Biblical Simulator

Test the full pipeline with AI-generated biblical passages in King James, Church Slavonic, or Ukrainian style.

🎤

Voice Training

Clone custom voices from microphone recordings or YouTube videos using ElevenLabs IVC.

💡

Looking for examples?

Check the Quickstart guide for a complete walkthrough, or jump to the API Reference for endpoint documentation.

Prerequisites

●

Node.js 20+ Runtime for backend and build tools

●

Docker + Docker Compose For Redis and LibreTranslate services

●

yt-dlp + ffmpeg Required for YouTube audio extraction

●

ElevenLabs API Key For speech-to-text and text-to-speech

Clone & Configure

Terminal

git clone https://github.com/Pzharyuk/live-translator-node.git && cd live-translator-node
cp .env.example .env

Edit .env and set your API key:

.env

ELEVENLABS_API_KEY=sk-your-key-here
ADMIN_PASSWORD=your-secure-password

Start Infrastructure

Terminal

# Start Redis + LibreTranslate
docker compose -f docker-compose.local.yml up -d

# Wait for LibreTranslate to download language models (~500 MB)
docker logs -f $(docker ps -qf "name=libretranslate") 2>&1 | grep -i "running"

Start Backend

Terminal

cd backend
npm install
npm run dev  # nodemon watches for changes

Start Frontend

Terminal

cd frontend
npm install
npm run dev  # Vite hot-reload on localhost:5173

✓

You're all set!

Open http://localhost:5173 — log in with user / changeme and you will be redirected to /translate. Admin panel: http://localhost:5173/admin (admin password: admin123).

1

Start the services

Follow the Installation guide to get Docker services, backend, and frontend running.

2

Open the Admin Panel

Navigate to http://localhost:5173/admin and enter the admin password.

3

Select a Voice

Choose a TTS voice from the dropdown. The voice list is fetched from your ElevenLabs account.

4

Test with Text

Use the free-text area in the admin panel to type a phrase. Click translate to hear the TTS output instantly.

5

Go Live

Open the user view at http://localhost:5173/translate. Select "Mic" as input, pick a voice, and click Start. Speak into your microphone and watch real-time translation appear with audio playback.

💡

Try the Biblical Simulator

For a hands-free demo, enable the biblical_simulator feature flag in admin, enter an Anthropic API key, select a language, and click "Generate". The system will produce biblical passages through the full STT → Translation → TTS pipeline.

System Overview

Frontend

React 19 + Vite

Socket.io Client

Web Audio API

↔

Backend

Express + Socket.io

TypeScript

Service Layer

ElevenLabs

Scribe v2 (STT)

TTS Streaming

Voice Cloning

Translation

LibreTranslate (self-hosted)

DeepL (premium API)

Claude / Anthropic (AI)

Redis

Feature Flags

Settings Store

Anthropic

Biblical Simulator

Claude Translation

DeepL

Free & Pro tiers

Auto endpoint detection

Data Flow

1 Audio Input (Mic / YouTube / Simulator)

↓

2 PCM 16-bit LE @ 16kHz via Socket.io chunks

↓

3 ElevenLabs Scribe v2 WebSocket STT

↓

4 Commit Merge Buffer 2.5s VAD aggregation

↓

5 Translation Provider LibreTranslate / DeepL / Claude

↓

6 ElevenLabs TTS Voice synthesis streaming

↓

7 Audio Playback Queued with 600ms pause

Key Architecture Decisions

Two-layer Language Detection

LibreTranslate's /detect endpoint returns 0-confidence for short Cyrillic phrases. The app uses script-based pre-detection (Unicode 0x0400–0x04FF = Cyrillic) combined with ElevenLabs Scribe's language_code output for reliable EN/RU/UK auto-detection.

VAD Commit Merging

Voice Activity Detection can fire aggressively on speaker breathing. Commits are buffered for 2.5 seconds before translation to merge fragments into meaningful phrases.

Feature Flag Merging

YAML config defaults are merged with Redis runtime overrides. Redis values take priority, falling back to YAML if Redis is unavailable.

API Key Hierarchy

Keys resolve in order: Runtime Cache → Redis → Config File → Empty. This allows hot-swapping keys without restarts.

Connection Lifecycle

Client sends start_session with source type (mic or youtube) and optional voiceId
Backend opens a WebSocket to wss://api.elevenlabs.io/v1/speech-to-text/realtime
For YouTube: spawns yt-dlp | ffmpeg child processes to extract PCM audio
For Microphone: awaits audio_chunk events from the frontend

Audio Streaming

Audio chunks are sent to Scribe as JSON messages:

WebSocket Message

{
  "message_type": "input_audio_chunk",
  "audio_base_64": "UklGR..."  // PCM 16-bit LE, 16kHz, mono
}

Scribe Responses

Response Type	Meaning	Action
`partial_transcript`	Live partial text (speculative)	Emitted as non-final `transcript` event
`committed_transcript`	VAD fired — complete phrase	Buffered for commit merge window

Commit Merge Buffer

After receiving a committed_transcript, the backend waits 2.5 seconds (COMMIT_MERGE_MS) to collect additional commits before translating. This prevents fragmented translations from aggressive VAD.

Stability Timeout

If VAD stalls (no new commits), a 3.5 second fallback timer (STABILITY_TIMEOUT_MS) fires to translate whatever new text has accumulated, preventing indefinite silence.

Text Validation

Before translation, text is validated against EN/RU/UK character regex patterns. This filters out hallucinated text from the STT model (common with silence or background noise).

Provider Chain

The system supports three translation providers with automatic fallback:

Default

LibreTranslate

Self-hosted, no API key required. Runs in Docker alongside the app. Best for privacy and cost.

Premium

DeepL

High-quality translations. Supports both free and paid API tiers. Auto-detects endpoint.

AI

Claude

Anthropic's Claude for context-aware translations. Uses claude-haiku-4-5 for speed.

Fallback Logic

Provider Resolution

1. Try primary provider (admin-selected)
2. If primary fails → try configured fallback
3. If fallback fails → try LibreTranslate (last resort)
4. If all fail → emit error event

Language Detection

The app uses a two-layer auto-detection approach:

Layer 1: Script-based Pre-detection

Before calling any translation API, the backend checks Unicode character scripts:

Cyrillic characters (Unicode 0x0400–0x04FF) → if >50% of matched letters are Cyrillic, detected as Russian
Latin characters → detected as English
This avoids low-confidence results from LibreTranslate's /detect endpoint on short text

Layer 2: STT Language Code

When the auto_language_detect flag is enabled, ElevenLabs Scribe returns a language_code with each transcript commit. The backend uses this to correctly route EN/RU/UK without relying solely on script detection.

Note: For LibreTranslate, both Russian and Ukrainian Cyrillic text is passed with source ru since LibreTranslate handles Ukrainian text acceptably via the Russian model. DeepL and Claude providers distinguish Ukrainian natively and handle uk as a proper source language.

Language Gating

Detected languages are checked against the admin-approved pool. If a detected language isn't in the allowed set, the translation is rejected to prevent hallucinated language outputs.

TTS Pipeline

After translation, the text is sent to ElevenLabs TTS:

TypeScript

const stream = await client.textToSpeech.stream(voiceId, {
  text: translatedText,
  model_id: "eleven_multilingual_v2",
  output_format: "mp3_44100_128",
  voice_settings: {
    stability: 0.5,
    similarity_boost: 0.75,
    style: 0.0,
    speed: 1.0,
    use_speaker_boost: true
  }
});

Audio Delivery

TTS audio is streamed to a Buffer, then emitted as a base64-encoded MP3 via the tts_audio Socket.io event.

Frontend Playback Queue

The frontend maintains an audio queue to prevent overlapping playback:

Received tts_audio events are queued
Each segment plays to completion before the next starts
A configurable pause (600ms default) is inserted between segments
The pause duration is controlled by tts_segment_pause_ms (adjustable in admin)

Microphone Input

User selects "Mic" tab and chooses a TTS voice
Browser captures audio via Web Audio API's ScriptProcessor
PCM 16-bit LE at 16kHz sample rate sent to backend via Socket.io
Backend pipes audio to ElevenLabs Scribe v2 Realtime WebSocket
Language auto-detected (EN/RU/UK), text translated and synthesized
TTS audio returned and played back with inter-segment pauses

YouTube Input

User pastes a YouTube URL (live stream or video)
Backend spawns yt-dlp | ffmpeg child processes
Audio extracted as PCM stream (16kHz, 16-bit LE, mono)
Piped to Scribe v2, same pipeline as microphone
Stream ends when YouTube content ends or user stops

User Interface

The user view features a dark cavern theme with:

Waveform visualizer — Canvas-based bar chart with orange gradient and cyan tips
Transcript display — White translated text scrolls upward with fade masks
Partial transcript — Shown in italic orange while STT is processing
Source tabs — Toggle between Mic and YouTube (controlled by feature flags)

How It Works

The backend uses yt-dlp and ffmpeg as child processes to extract audio from YouTube URLs:

Pipeline

yt-dlp (best audio) → ffmpeg (PCM 16kHz 16-bit LE mono) → Scribe v2

Supported Sources

Live streams — Translates in real-time as the stream progresses
Regular videos — Processes the full audio track
Any URL supported by yt-dlp (YouTube, etc.)

Requirements

Both yt-dlp and ffmpeg must be installed and available in the system PATH. On macOS:

Terminal

brew install yt-dlp ffmpeg

⚠

Feature Flag Required

YouTube input is controlled by the youtube_input feature flag. Enable it in the admin panel to show the YouTube tab in the user view.

Overview

The Biblical Transcript Simulator is an admin-only feature that generates biblical text passages using Anthropic's Claude API, then routes them through the full translation pipeline. This provides a hands-free way to test STT → Translation → TTS without a live audio source.

Language Styles

Language	Style	Example
`en`	King James English	"In the beginning was the Word..."
`ru`	Church Slavonic Russian	"В начале было Слово..."
`uk`	Traditional Ukrainian	"На початку було Слово..."

Flow

Admin provides Anthropic API key and selects language
Backend calls Claude with streaming (uses claude-sonnet-4-6)
Claude generates 6-8 biblical passages, 3-5 sentences each
Stream is buffered until 140+ characters AND complete sentences
Chunks emitted with 1800ms smooth pacing between them
Each chunk flows through the standard pipeline:
- Emitted as transcript (isFinal: true)
- Auto-translated via configured provider
- TTS synthesized and audio returned
Frontend plays audio with standard inter-segment pause

💡

Feature Flag

Enable biblical_simulator in the admin feature flags panel. The Anthropic API key is provided at runtime in the UI — it's never stored in config files.

Overview

Voice Training uses ElevenLabs' Instant Voice Cloning (IVC) API to create custom voices from audio samples. Once cloned, the voice appears in the voice selector immediately.

From Microphone

Open the Voice Training section in the admin panel
Record multiple audio clips using your browser microphone
Provide a name for the voice
Clips are uploaded to ElevenLabs IVC API
Cloned voice is available for TTS immediately

From YouTube

Paste a YouTube URL in the Voice Training section
Backend extracts N × 30-second clips via yt-dlp + ffmpeg
Clips are uploaded to ElevenLabs IVC API
Resulting voice is stored in your ElevenLabs account

⚠

ElevenLabs Account

Cloned voices are stored in your ElevenLabs account, not locally. Ensure your plan supports voice cloning.

Concepts

Concept	Description
Active Language Pair	The current pair used for translation (e.g., EN ↔ RU, EN ↔ UK, or RU ↔ UK). Set by admin.
Available Languages	The pool of languages viewers can select from (if `user_language_selector` is enabled).

Admin Controls

Change the active language pair via the admin panel
Changes broadcast to all connected clients in real-time
Manage the available languages pool for viewer selection

Viewer Selection

When the user_language_selector feature flag is enabled, viewers can override the admin-set language pair by selecting their own preferred languages from the available pool.

Overview

Two people can video call each other through the app, each speaking their own language. The app transcribes, translates, and synthesizes speech in real-time so each participant hears the other in their language.

Feature flag: Video call is gated behind the video_translation flag. Enable it in the admin panel or set video_translation: true in your YAML config.

How It Works

Create a room — Person A selects their language, picks a TTS voice, and clicks "Create Room". A 6-character room code is generated.
Share the code — Person A shares the room code with Person B (copy button provided).
Join the room — Person B enters the code, selects their language and TTS voice, and clicks "Join".
WebRTC connection — The app establishes a peer-to-peer video connection via WebRTC (signaled through Socket.io). Video flows directly between browsers.
Audio translation — Each participant's microphone audio is simultaneously:
- Sent to the peer via WebRTC (but muted on their end)
- Captured as PCM chunks and sent to the backend via Socket.io for STT
Translation pipeline — Each participant has their own independent Scribe STT session. Transcribed text is translated to the other participant's language, then synthesized via ElevenLabs TTS and sent back to the peer.
Playback — The peer hears the TTS translation instead of the raw audio. Translated transcript is displayed below the video.

Architecture

Person A (Browser)         Server              Person B (Browser)
├─ getUserMedia            ├─ Socket.io         ├─ getUserMedia
├─ WebRTC P2P ═══video═══►│  (signaling)  ◄═══ ├─ WebRTC P2P
│                          │                    │
├─ PCM chunks ──Socket.io─►├─ ScribeA(STT)      │
│                          │  ↓ translate       │
│                          │  ↓ TTS ───────────►├─ Plays TTS
│                          │                    │
│  Plays TTS ◄─────────────├─ ScribeB(STT) ◄───├─ PCM chunks
│  (remote video muted)    │  ↓ translate       │  (remote video muted)
└──────────────────────────┴────────────────────┘

Socket Events

Event	Direction	Purpose
`video_create_room`	C→S	Create a new room with language + voice
`video_room_created`	S→C	Returns the 6-char room code
`video_join_room`	C→S	Join an existing room
`video_room_joined`	S→C	Sent to both participants, triggers WebRTC
`video_signal_offer/answer/ice`	C↔S	WebRTC signaling relay
`video_audio_chunk`	C→S	PCM audio for STT processing
`video_transcript`	S→C	Transcript sent to the speaker
`video_translation`	S→C	Translation sent to the listener
`video_tts_audio`	S→C	TTS audio sent to the listener
`video_leave_room`	C→S	Leave the room
`video_room_closed`	S→C	Notify peer when other leaves

Room Lifecycle

Rooms are stored in Redis with key video_room:{code} and a 4-hour TTL
Maximum 2 participants per room
When one participant disconnects, the other is notified and the call ends
Scribe sessions are automatically cleaned up on disconnect

Feature Flags

Feature flags control which routes, UI elements, and capabilities are enabled. Defaults are defined in application.yaml and can be overridden at runtime via Redis. The admin API merges both sources, with Redis taking precedence.

All Feature Flags

Flag	Default	Description
`youtube_input`	true	Allow audio input from YouTube URLs.
`mic_input`	true	Allow audio input from microphone.
`auto_language_detect`	true	Automatically detect source language during speech recognition.
`user_language_selector`	false	Allow users to manually select language pairs (else admin-configured only).
`audio_device_selector`	true	Allow users to select audio input device.
`video_translation`	false	Enable `/video` route for video call translation.
`video_voice_cloning`	false	Enable Clone Voice button in video lobby (premium feature).
`broadcast`	false	Enable `/broadcast` route — public receiver page for live translation broadcasts.
`translate`	false	Enable `/translate` route — private translator page for personal sessions.

Storage & Precedence

Feature flags are stored in two places. When a flag is requested, Redis values take precedence over YAML defaults. This allows administrators to toggle features at runtime without restarting the server. If a flag is not set in Redis, the YAML default is used. New flags added to application.yaml are immediately available without a restart.

Admin API

Retrieve, set, and manage feature flags via the admin panel:

GET /admin/flags
Returns all feature flags merged from YAML defaults + Redis overrides.

Response:
{
  "flags": {
    "youtube_input": true,
    "mic_input": true,
    "auto_language_detect": true,
    "user_language_selector": false,
    "audio_device_selector": true,
    "video_translation": false,
    "video_voice_cloning": false,
    "broadcast": false,
    "translate": false
  }
}

GET /admin/flags/:flag
Get a single flag value.

POST /admin/flags/:flag
Set a flag value (persists to Redis).

Request Body:
{
  "value": true
}

Response:
{
  "flag": "translate",
  "value": true
}

Socket.IO Events

When a flag is updated via the admin API, all connected clients receive a real-time broadcast over Socket.IO:

feature_flags → Emitted to all connected sockets
Payload:
{
  "youtube_input": true,
  "mic_input": true,
  "auto_language_detect": true,
  "user_language_selector": false,
  "audio_device_selector": true,
  "video_translation": false,
  "video_voice_cloning": false,
  "broadcast": false,
  "translate": false
}

Sent on initial connection and whenever any flag is changed.

Usage Examples

Enable the broadcast route:

POST /admin/flags/broadcast
{
  "value": true
}

Check if a feature is available (frontend):

// After receiving feature_flags event
if (flags.video_translation) {
  // Show /video link in nav
}

File Structure

File	Purpose
`config/application.yaml`	Base defaults for all environments
`config/application-local.yaml`	Local development overrides (localhost URLs)
`config/application-prod.yaml`	Production overrides (Docker service names)

The APP_ENV environment variable (local or prod) determines which overlay file is loaded on top of the base config.

Full Configuration Reference

config/application.yaml

server:
  port: 3001
  cors_origin: "http://localhost:5173"

elevenlabs:
  api_key: "${ELEVENLABS_API_KEY}"
  default_voice_id: "kxj9qk6u5PfI0ITgJwO0"
  tts_model: "eleven_multilingual_v2"
  tts_settings:
    stability: 0.5
    similarity_boost: 0.75
    style: 0.0
    speed: 1.0
    use_speaker_boost: true
  stt_model: "scribe_v2"

anthropic:
  api_key: "${ANTHROPIC_API_KEY}"

deepl:
  api_key: "${DEEPL_API_KEY}"

libretranslate:
  url: "http://libretranslate:5000"
  api_key: ""

redis:
  host: "redis"
  port: 6379
  password: ""

feature_flags:
  youtube_input: true
  mic_input: true
  auto_language_detect: true
  user_language_selector: false
  audio_device_selector: true
  video_translation: false
  video_voice_cloning: false
  broadcast: false

audio:
  sample_rate: 16000
  channels: 1
  chunk_duration_ms: 250

translation:
  source_lang: "auto"
  target_lang_en: "en"
  target_lang_ru: "ru"
  provider: "libretranslate"
  fallback: "libretranslate"

Environment Variable Interpolation

YAML values using ${VAR_NAME} syntax are automatically replaced with the corresponding environment variable at startup.

Variable	Required	Default	Description
`ELEVENLABS_API_KEY`	Yes	—	API key for ElevenLabs speech synthesis and recognition services.
`ELEVENLABS_VOICE_ID`	No	`kxj9qk6u5PfI0ITgJwO0`	Default voice ID for text-to-speech output when not specified by user.
`ANTHROPIC_API_KEY`	No	—	API key for Anthropic Claude models used for sermon generation and translation.
`GOOGLE_TRANSLATE_API_KEY`	No	—	Google Cloud Translation API key; required if provider is set to Google.
`DEEPL_API_KEY`	No	—	DeepL API key for translation; required if provider is set to DeepL.
`TRANSLATION_PROVIDER`	No	`google`	Primary translation provider: `google`, `deepl`, `claude`, or `libretranslate`.
`APP_ENV`	No	`local`	Application environment: `local` for development or `prod` for Docker.
`FRONTEND_URL`	No	`http://localhost`	Frontend base URL used for CORS origin configuration in production.
`LISTEN_PORT`	No	`80`	Port the frontend listens on.
`REDIS_PASSWORD`	No	—	Password for Redis authentication; leave empty if no authentication is required.
`LIBRETRANSLATE_API_KEY`	No	—	Optional API key for LibreTranslate if the instance requires authentication.
`ADMIN_PASSWORD`	No	`admin123`	Legacy socket authentication password for admin access; must be changed in production.
`APP_ADMIN_USERNAME`	No	`admin`	Admin user seeded into the database on first boot.
`APP_ADMIN_PASSWORD`	No	`admin123`	Initial admin password; user must change on first login if `force_password_change=true`.
`APP_USERNAME`	No	`user`	Standard user login username; must be changed in production.
`APP_PASSWORD`	No	`changeme`	Standard user login password; must be changed in production.
`JWT_SECRET`	No	—	Secret key for JWT session cookie signing; generate with `openssl rand -hex 32` in production.
`COOKIE_SECURE`	No	`true`	Set to `true` when serving over HTTPS; required for Cloudflare Tunnel.
`DB_PASSWORD`	No	—	PostgreSQL database password for the translator database user.

TTS Settings

API Endpoints

GET /admin/tts-settings
Response: { "settings": { ... } }

POST /admin/tts-settings
Content-Type: application/json
Body: { "stability": 0.5, "similarity_boost": 0.75, ... }
Response: { "settings": { ... } }

Configuration Table

Setting	Range	Default	Description
`stability`	0.0 – 1.0	0.5	Voice consistency; higher = less variation in emotional tone.
`similarity_boost`	0.0 – 1.0	0.75	Accent similarity to the selected voice; higher = closer match.
`style`	0.0 – 1.0	0.0	Exaggeration of voice style; 0.0 = neutral, higher = more expressive.
`speed`	0.5 – 2.0	1.0	Playback speed multiplier; 1.0 = normal, <1.0 = slower, >1.0 = faster.
`use_speaker_boost`	boolean	true	Enable speaker boost for clearer, louder audio output.

STT Timing Settings

Controls speech–to–text audio buffering and translation dispatch thresholds.

Setting	Range	Default	Description
`commit_merge_ms`	0 – 10000	2500	Buffer VAD commits for this duration before translating (ms); merges sentence fragments.
`stability_timeout_ms`	0 – 10000	2000	Wait for stable partial text before translating if unchanged for N ms.
`tts_segment_pause_ms`	0 – 2000	250	Pause between audio segment playback (ms); frontend uses this for pacing.
`max_accumulation_ms`	0 – 30000	10000	Force–dispatch accumulated words during continuous speech after N ms.
`vad_threshold`	0.0 – 1.0	0.5	Voice Activity Detection sensitivity; higher = stricter noise filtering.
`vad_silence_threshold_secs`	0.5 – 5.0	1.5	Seconds of silence before VAD commits transcript (seconds).
`min_speech_duration_ms`	50 – 1000	100	Ignore speech shorter than this threshold (ms).
`min_silence_duration_ms`	50 – 1000	100	Minimum silence gap within continuous speech (ms).
`flush_on_sentence_boundary`	boolean	true	Flush commit buffer at sentence boundaries (. ? ! ;) instead of all at once.

Video Call Settings

Separate STT/TTS tuning for low–latency video call translation pipeline.

Setting	Range	Default	Description
`stability_ms`	100 – 2000	500	Wait for stable partial text before translating in video calls (ms).
`commit_merge_ms`	0 – 500	50	Merge VAD commits in video calls with minimal latency (ms).
`translation_provider`	libretranslate \| claude \| deepl \| google	claude	Translation provider for video call sessions.

ElevenLabs Configuration

Core TTS engine settings from application.yaml.

Setting	Range / Options	Default	Description
`api_key`	string (env: ELEVENLABS_API_KEY)	—	ElevenLabs API credential; can be updated via Admin UI.
`default_voice_id`	string	kxj9qk6u5PfI0ITgJwO0	Voice ID used when none is explicitly selected.
`tts_model`	eleven_multilingual_v2 \| eleven_flash_v2_5	eleven_multilingual_v2	TTS model; flash variant used for low–latency video calls.
`stt_model`	scribe_v2_realtime \| scribe_v2	scribe_v2_realtime	STT model; realtime supports VAD–based auto–commits.

Related Endpoints

GET /admin/stt-timing → Retrieve STT timing settings.
POST /admin/stt-timing → Update STT timing (partial updates supported).
GET /admin/video-settings → Retrieve video call STT/TTS configuration.
POST /admin/video-settings → Update video call settings.
GET /admin/voices → Scan and list all available ElevenLabs voices.
GET /admin/available-voices → Get admin–filtered voice whitelist.
POST /admin/available-voices → Set allowed voice IDs for viewers.

Notes

TTS settings persist to Redis and are loaded on application startup.
STT timing affects both live translation and broadcast pipelines.
Video call settings use a separate provider chain for optimized latency.
All settings are runtime–configurable via the Admin UI; no restart required.
ElevenLabs API key can be rotated via POST /admin/api-keys without downtime.

STT Timing Settings

Configure speech-to-text (STT) recognition timing, voice activity detection (VAD), and transcript buffering behavior. These settings control how long to wait before triggering translation and how aggressively to filter noise.

Settings Reference

Setting	Default	Description
`commit_merge_ms`	2500	Buffer VAD commits for this many milliseconds before translating (merges sentence fragments).
`stability_timeout_ms`	2000	Fire translation when partial text is unchanged for this duration — fallback when VAD doesn't commit.
`tts_segment_pause_ms`	250	Pause between consecutive TTS audio segments (frontend playback timing).
`max_accumulation_ms`	10000	Force-dispatch accumulated words during continuous speech if no translation fires within this time.
`vad_threshold`	0.5	Voice activity detection sensitivity (0–1, higher → stricter noise filter).
`vad_silence_threshold_secs`	1.5	Seconds of silence required before VAD commits a transcript segment.
`min_speech_duration_ms`	100	Ignore speech utterances shorter than this duration.
`min_silence_duration_ms`	100	Minimum silence gap in milliseconds between speech segments.
`flush_on_sentence_boundary`	`true`	Split buffered commits at sentence boundaries (. ? ! ;) instead of flushing all at once.

Tuning Guide

Faster response → Lower commit_merge_ms, stability_timeout_ms, and max_accumulation_ms; increase vad_threshold to be less strict.
Better accuracy → Increase vad_silence_threshold_secs and min_speech_duration_ms to avoid noise; decrease vad_threshold to be stricter.
Sermon/long speech → Lower max_accumulation_ms (e.g., 5000) to dispatch words even during continuous speech; enable flush_on_sentence_boundary.
Noisy environment → Increase vad_threshold (e.g., 0.7–0.9) and min_speech_duration_ms (e.g., 200–300).
Sentence buffering → Enable flush_on_sentence_boundary to split buffered text at . ? ! ; and dispatch complete sentences independently.

API Endpoints

GET /admin/stt-timing

Retrieve all current STT timing settings.

curl -X GET http://localhost:3001/admin/stt-timing \
  -H "Authorization: Bearer YOUR_JWT_TOKEN"

// Response:
{
  "settings": {
    "commit_merge_ms": 2500,
    "stability_timeout_ms": 2000,
    "tts_segment_pause_ms": 250,
    "max_accumulation_ms": 10000,
    "vad_threshold": 0.5,
    "vad_silence_threshold_secs": 1.5,
    "min_speech_duration_ms": 100,
    "min_silence_duration_ms": 100,
    "flush_on_sentence_boundary": true
  }
}

POST /admin/stt-timing

Update one or more STT timing settings. Partial updates are supported — omitted fields retain their current values.

curl -X POST http://localhost:3001/admin/stt-timing \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -d '{
    "commit_merge_ms": 1500,
    "stability_timeout_ms": 1000,
    "max_accumulation_ms": 5000,
    "vad_threshold": 0.7,
    "flush_on_sentence_boundary": true
  }'

// Response:
{
  "settings": {
    "commit_merge_ms": 1500,
    "stability_timeout_ms": 1000,
    "tts_segment_pause_ms": 250,
    "max_accumulation_ms": 5000,
    "vad_threshold": 0.7,
    "vad_silence_threshold_secs": 1.5,
    "min_speech_duration_ms": 100,
    "min_silence_duration_ms": 100,
    "flush_on_sentence_boundary": true
  }
}

Notes

Settings are persisted to Redis and survive server restarts.
Changes apply immediately to new STT sessions; active sessions use settings at their start time.
Socket.IO clients receive stt_timing events when settings are updated, allowing real-time UI synchronization.
The tts_segment_pause_ms setting is sent to the frontend for audio playback timing; the other VAD/buffer settings are server-side only.

Authentication: All endpoints require JWT cookie-based admin authentication. Users must have is_admin=true or possess at least one role with relevant permissions. Permission-gated endpoints are marked below.

API Keys Management

GET /admin/api-keys

Retrieve all API key statuses (shows whether keys are configured, not their values).

POST /admin/api-keys

Set or update API keys for elevenlabs, anthropic, deepl, libretranslate, & google.

Body: {
  "elevenlabs": "sk-...",
  "anthropic": "sk-ant-...",
  "deepl": "...",
  "libretranslate": "...",
  "google": "..."
}

GET /admin/anthropic-key

Retrieve the current Anthropic API key (for client-side sermon generation).

Voice Management

GET /admin/voices

Scan & list all available ElevenLabs voices, with detection of new voices not yet in the allowed pool.

GET /admin/available-voices

Get the current list of admin-allowed voice IDs that viewers can select from.

POST /admin/available-voices

Set the pool of allowed voice IDs; broadcasts to all connected clients in real-time.

Body: {
  "voiceIds": ["voice1", "voice2", "..."]
}

Feature Flags

GET /admin/flags

Get all feature flags merged from YAML config & Redis overrides.

GET /admin/flags/:flag

Get the current value of a single feature flag.

POST /admin/flags/:flag

Set a feature flag value and broadcast the updated flags to all connected clients via Socket.IO.

Body: {
  "value": true
}

TTS & STT Settings

GET /admin/tts-settings

Retrieve current TTS settings (stability, similarity boost, style, speed, speaker boost).

POST /admin/tts-settings

Update TTS settings (all fields optional).

Body: {
  "stability": 0.5,
  "similarity_boost": 0.75,
  "style": 0.0,
  "speed": 1.0,
  "use_speaker_boost": true
}

GET /admin/stt-timing

Get speech-to-text timing parameters (VAD thresholds, commit merge delays, accumulation timeout).

POST /admin/stt-timing

Update STT timing settings to control recognition responsiveness & translation dispatch.

Body: {
  "commit_merge_ms": 2500,
  "stability_timeout_ms": 2000,
  "tts_segment_pause_ms": 250,
  "max_accumulation_ms": 10000,
  "vad_threshold": 0.5,
  "vad_silence_threshold_secs": 1.5,
  "min_speech_duration_ms": 100,
  "min_silence_duration_ms": 100,
  "flush_on_sentence_boundary": true
}

GET /admin/video-settings

Get video call STT/TTS settings (separate from main live-translation settings).

POST /admin/video-settings

Update video call settings (stability, commit merge, translation provider).

Body: {
  "stability_ms": 500,
  "commit_merge_ms": 50,
  "translation_provider": "claude"
}

Languages

GET /admin/languages

Get the currently active language pair (e.g., ["en", "ru"]).

POST /admin/languages

Set the active language pair and broadcast to all viewers in real-time.

Body: {
  "languages": ["en", "ru"]
}

GET /admin/available-languages

Get the pool of languages available for viewers to select from.

POST /admin/available-languages

Set the pool of available languages and broadcast both the pool & active pair to all clients.

Body: {
  "languages": ["en", "ru", "uk", "es"]
}

Translation Provider

GET /admin/translation-provider

Get the active translation provider & list of available providers (google, deepl, claude, libretranslate).

POST /admin/translation-provider

Set the active translation provider.

Body: {
  "provider": "google"
}

GET /admin/claude-model

Get the currently selected Claude model & list of available models for translation.

POST /admin/claude-model

Set the Claude model to use for translation.

Body: {
  "model": "claude-opus-4-1-20250805"
}

Audio Device

GET /admin/audio-device

Get the admin-selected audio input device (overrides viewer's local choice).

POST /admin/audio-device

Set the admin-forced audio device & broadcast to all viewers.

Body: {
  "deviceId": "default",
  "label": "Built-in Microphone"
}

Voice Training & Cloning

POST /admin/voice-training/from-recording

Clone a voice from browser microphone recordings (base64-encoded audio blobs).

Body: {
  "name": "My Custom Voice",
  "clips": ["base64_audio_1", "base64_audio_2"],
  "mimeType": "audio/webm"
}

POST /admin/voice-training/from-youtube

Clone a voice from a YouTube video URL using yt-dlp & ffmpeg to extract audio clips.

Body: {
  "name": "My Custom Voice",
  "youtubeUrl": "https://www.youtube.com/watch?v=...",
  "clipCount": 3,
  "startOffset": 0
}

Content Generation

POST /admin/generate-sermon

Generate a random biblical sermon snippet via Anthropic Claude in the specified language.

Body: {
  "apiKey": "sk-ant-...",
  "language": "en"
}

Monitoring & Logs

GET /admin/hallucinations

Get hallucination detection statistics & log entries.

DELETE /admin/hallucinations

Clear the hallucination log.

GET /admin/translation-log

Retrieve all translation events with timing metrics (translation & TTS latency).

DELETE /admin/translation-log

Clear the translation log.

GET /admin/queue-depth

Get the current depth of the broadcast TTS job queue for monitoring latency.

Session History

GET /admin/sessions

List all broadcast & private translation sessions from PostgreSQL with metadata.

GET /admin/sessions/:id

Get detailed transcript & timing data for a specific session.

GET /admin/sessions/:id/export

Export session transcripts in JSON, CSV, or plain text format. Use query param ?format=csv|txt|json.

User Management (requires `user_management` permission)

GET /admin/users

List all users (password hashes & avatar data stripped).

POST /admin/users/:id/role

Update a user's admin status and/or assign roles.

Body: {
  "isAdmin": true,
  "roleIds": ["role1", "role2"]
}

POST /admin/users/:id/reset-password

Force-reset a user's password (minimum 6 characters).

Body: {
  "password": "newpassword123"
}

DELETE /admin/users/:id

Delete a user account (cannot delete yourself).

Roles & Permissions (requires `user_management` permission)

GET /admin/permissions

List all available permissions that can be assigned to roles.

GET /admin/roles

List all defined roles & their assigned permissions.

POST /admin/roles

Create a new role with a set of permissions.

Body: {
  "name": "Moderator",
  "permissions": ["user_management", "content_moderation"]
}

PUT /admin/roles/:id

Update an existing role's name & permissions.

Body: {
  "name": "Senior Moderator",
  "permissions": ["user_management", "content_moderation", "analytics"]
}

DELETE /admin/roles/:id

Delete a role (users assigned to it are unaffected; they keep other roles).

Socket.IO Events

Server → Client Events

Event	Payload	Description
`feature_flags`	`{ [flag: string]: boolean }`	Merged feature flags from config defaults & Redis overrides.
`languages`	`{ languages: string[] }`	Current active language pair (2-element array).
`available_languages`	`{ languages: string[] }`	Pool of languages viewers can select from.
`stt_timing`	`{ tts_segment_pause_ms: number }`	STT timing settings, specifically the TTS segment pause duration.
`broadcast_status`	`{ active: boolean; source?: 'mic' \| 'youtube' \| 'biblical' }`	Current broadcast on/off status & source type (global to all clients).
`broadcast_viewer_count`	`{ count: number }`	Number of viewers currently joined to the broadcast room.
`admin_audio_device`	`{ deviceId: string; label: string }`	Admin-selected audio input device override for forced device selection.
`transcript`	`{ text: string; isFinal: boolean; skipped?: boolean }`	Speech-to-text output — partial or final transcript from Scribe STT.
`translation`	`{ original: string; translated: string; detectedLanguage?: string }`	Translated text output from the translation pipeline.
`tts_audio`	`{ audio: string }`	Base64-encoded MP3 audio chunk from text-to-speech synthesis.
`tts_clear_queue`	`{}`	Signal to clear any pending TTS queue on the client.
`audio_level`	`{ data: number[] }`	Downsampled waveform data (64 samples) for real-time audio level visualization.
`stream_ended`	`{}`	YouTube stream or biblical simulation has finished.
`session_started`	`{ source: 'mic' \| 'youtube' }`	User’s private session has started with the specified audio source.
`session_stopped`	`{}`	User’s private session has stopped.
`error`	`{ message: string }`	Error message from speech recognition, TTS, or stream processing.
`admin_translate_result`	`{ original: string; translated: string; detectedLanguage?: string; audio: string }`	Result of admin’s instant translate & TTS test (private to admin socket).

Client → Server Events

Event	Payload	Description
`join_broadcast`	`{}`	Viewer joins the broadcast room to receive translated output & waveform.
`leave_broadcast`	`{}`	Viewer leaves the broadcast room.
`set_languages`	`{ languages: string[] }`	Viewer selects active language pair (validated against available pool).
`start_session`	`{ source: 'mic' \| 'youtube'; voiceId?: string; youtubeUrl?: string }`	Start a private translation session from mic or YouTube URL.
`stop_session`	`{}`	Stop the user’s private translation session.
`admin_start_broadcast`	`{ voiceId?: string; source: 'mic' \| 'youtube'; youtubeUrl?: string }`	Admin starts a broadcast session that streams to all connected clients.
`admin_stop_broadcast`	`{}`	Admin stops the active broadcast session.
`audio_chunk`	`{ audio: string }`	Base64-encoded PCM audio chunk from microphone (routes to broadcast or private session).
`admin_translate_test`	`{ text: string; voiceId?: string; sourceLang?: string; targetLang?: string }`	Admin performs instant translation & TTS synthesis for testing.
`start_biblical_sim`	`{ anthropicApiKey: string; language: BiblicalLanguage; voiceId?: string }`	Start a biblical text simulation broadcast using Anthropic API.
`stop_biblical_sim`	`{}`	Stop the active biblical text simulation.

SDK

Uses the official @elevenlabs/elevenlabs-js SDK (v2). The client is lazy-loaded on first use.

Speech-to-Text (Scribe v2 Realtime)

Connects via native WebSocket to wss://api.elevenlabs.io/v1/speech-to-text/realtime. Handles:

VAD-based commit buffering with configurable merge window
Stability timeout fallback for stalled VAD
Text validation (EN/RU/UK character regex filtering)
Partial and final transcript emission

Text-to-Speech

Uses client.textToSpeech.stream() with the eleven_multilingual_v2 model. Audio is collected into a Buffer and emitted as base64 MP3.

Voice Management

client.voices.getAll() — fetches all voices from account
Admin can filter which voices are available to viewers
Voice cloning via IVC API (from recordings or YouTube)

Key File

backend/src/services/elevenlabs.service.ts

Provider Details

LibreTranslate

Self-hosted in Docker. No API key required by default. Provides language detection and translation via REST API.

File: backend/src/services/libretranslate.service.ts

DeepL

Premium translation API. Auto-detects free vs. paid endpoint based on the API key format.

File: backend/src/services/deepl.service.ts

Claude (Anthropic)

AI-powered translation using claude-haiku-4-5 for speed. Includes language detection and auto-flip logic.

File: backend/src/services/claude-translate.service.ts

Routing

Provider routing is handled by backend/src/services/translation.provider.ts:

Try admin-selected primary provider
On failure, try configured fallback provider
LibreTranslate is always the last-resort fallback

Connection

Uses ioredis with automatic retry strategy. Falls back to in-memory/YAML defaults if Redis is unavailable.

Key Patterns

Pattern	Example	Purpose
`flag:<name>`	`flag:youtube_input`	Feature flag boolean values
`setting:<name>`	`setting:tts_settings`	JSON settings objects

Key File

backend/src/services/redis.service.ts

Local Development

Use docker-compose.local.yml for Redis and LibreTranslate only (backend/frontend run natively):

Terminal

docker compose -f docker-compose.local.yml up -d

Production

Use docker-compose.yml for all services:

Terminal

docker compose up -d --build

Services

Service	Image	Port	Notes
frontend	Nginx (custom build)	80 (exposed)	Serves React build, proxies API/WS to backend
backend	Node.js (custom build)	3001 (internal)	Express + Socket.io server
redis	redis:7-alpine	6379 (internal)	Feature flags and settings store
libretranslate	libretranslate/libretranslate	5000 (internal)	Self-hosted translation engine

Configuration

.env (production)

ELEVENLABS_API_KEY=sk-your-production-key
ADMIN_PASSWORD=strong-secure-password
FRONTEND_URL=https://translate.example.com
APP_ENV=prod
REDIS_PASSWORD=redis-secret

Deploy

Terminal

docker compose up -d --build

Reverse Proxy

When running behind Nginx or another reverse proxy:

Set LISTEN_PORT in .env (e.g., 8080)
Proxy pass to localhost:8080
Important: Ensure WebSocket upgrades are forwarded for the /socket.io/ path

Nginx Config (example)

server {
    listen 443 ssl;
    server_name translate.example.com;

    location / {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }
}

Monitoring

Terminal

# Check all services
docker compose ps

# View backend logs
docker compose logs -f backend

# Health check
curl http://localhost:3001/api/health

Shipped

v0.1 – v0.2 — Core Translation Engine

Real-time STT via ElevenLabs Scribe v2 Realtime
Multi-provider translation (LibreTranslate, DeepL, Claude)
TTS voice synthesis with ElevenLabs
Microphone and YouTube live input
Admin panel with feature flags, voice management, TTS tuning
Biblical Transcript Simulator for pipeline testing
Instant Voice Cloning from recordings and YouTube

Shipped

v0.3 — Audio Mixer & Device Selection

Browser-side audio device scanning with support for professional mixing consoles, virtual audio devices, and audio interfaces.

Browser-side device enumeration with permission flow
Virtual device detection (Loopback, BlackHole, VB-Audio, Voicemeeter, OBS)
Categorized device picker (Microphones vs Mixers / Virtual Devices)
Admin device override broadcast to all viewers via Socket.io
Real-time feature flag broadcasting

Shipped

v0.7 — Broadcast Service

The /translate route is now a true broadcast service. Admins start one global translation session from the admin panel and all connected viewers receive the live output simultaneously.

Single global broadcast session (one-to-many)
Admin "Broadcast Control" panel — Start/Stop with source + voice selection
Microphone and YouTube source both supported for broadcast
All translation output (transcript, translated text, TTS audio) io.emit’d to every viewer
Viewer shows Waiting for broadcast to start… status when off air
"On Air" / "Off Air" status pill visible to viewers in real-time
Broadcast ownership tracked by admin socket ID; auto-stops on admin disconnect
Biblical Transcript Simulator also broadcasts to all viewers

Shipped

v0.8 — Navigation, Broadcast FF & Transcript UX

Global persistent bottom navigation, feature-flag-gated route visibility, and a refined transcript reading experience.

Persistent bottom navigation bar on all pages (/translate, /broadcast, /video, /admin)
FF-gated nav links — Broadcast and Video Call entries only appear when their flags are enabled
No extra socket connection — nav reads flags from the page’s existing useSocket call via props
Nav renders a frosted dark background gradient so it never overlaps content
/broadcast route is now public (no login required); gated inside the page by the broadcast feature flag
broadcast feature flag added to YAML, backend config, and frontend FeatureFlags interface
Transcript panel: newest translation is always at the top; older lines scroll down and fade out at the bottom
Each new transcript entry animates in from above (transcriptIn keyframe)
Removed duplicate “Video Call” button from /translate and /broadcast header bars

Up Next

v0.4 — Direct Audio Interface Feed

Accept audio directly from professional mixing consoles and audio interfaces — bypass browser mic capture entirely for broadcast-quality input.

Direct audio interface input (ASIO / Core Audio / ALSA)
Multi-channel mixer feed support
Low-latency audio routing (sub-100ms)
Hardware device auto-discovery and selection
Professional broadcast integration (NDI, Dante)

Shipped

v0.5 — Video Call Translation

WebRTC peer-to-peer video calls with real-time bidirectional translation. Two people speak different languages and hear each other translated via TTS.

Built-in WebRTC video call with room codes
Full-duplex translation (each person hears the other translated)
Per-participant STT pipeline with independent Scribe sessions
Video grid UI with local PiP and remote full-screen
Mic/video mute controls, hang up, auto-cleanup on disconnect
Feature-flagged behind video_translation

Shipped

v0.6 — Auth, Mobile & Voice Cloning in /video

User-facing login page (/) with JWT cookie sessions (30-day sticky, HttpOnly)
All app routes protected — redirect to login if unauthenticated
Live translator moved to /translate
Mobile-responsive UI across Translator, Admin, and Video Call views
FaceTime-style full-screen in-call layout on mobile with safe-area insets
“Clone Voice” button in /video lobby, gated by video_voice_cloning feature flag
Voice cloning modal with mic recording or YouTube URL, admin-password gated

Planned

Future

Additional language pairs beyond EN/RU/UK
Speaker diarization (multi-speaker detection)
Translation memory and glossary support
Webhooks and API for third-party integrations
Multi-tenant deployment with user accounts

Introduction

Ship live translationswith confidence

Installation

Architecture

Live Translation

Biblical Simulator

Voice Training

Installation

Prerequisites

Clone & Configure

Start Infrastructure

Start Backend

Start Frontend

Quickstart

Start the services

Open the Admin Panel

Select a Voice

Test with Text

Go Live

Architecture

System Overview

Data Flow

Key Architecture Decisions

Two-layer Language Detection

VAD Commit Merging

Feature Flag Merging

API Key Hierarchy

Speech-to-Text Flow

Connection Lifecycle

Audio Streaming

Scribe Responses

Commit Merge Buffer

Stability Timeout

Text Validation

Translation Pipeline

Provider Chain

LibreTranslate

DeepL

Claude

Fallback Logic

Language Detection

Layer 1: Script-based Pre-detection

Layer 2: STT Language Code

Language Gating

TTS & Playback

TTS Pipeline

Audio Delivery

Frontend Playback Queue

Live Translation

Microphone Input

YouTube Input

User Interface

YouTube Input

How It Works

Supported Sources

Requirements

Biblical Transcript Simulator

Overview

Language Styles

Flow

Voice Training

Overview

From Microphone

From YouTube

Language Management

Concepts

Admin Controls

Viewer Selection

Video Call Translation

Overview

How It Works

Architecture

Socket Events

Room Lifecycle

Feature Flags

Feature Flags

All Feature Flags

Storage & Precedence

Admin API

Socket.IO Events

Ship live translations
with confidence

User Management (requires `user_management` permission)

Roles & Permissions (requires `user_management` permission)