Velora AI Studio's Voice Pipeline: How We Handle ElevenLabs, Audio Sync & Lip-Matching at Scale

Why Voice Is the Hardest Part

It's counterintuitive. Video generation — the complex multi-billion-parameter diffusion processes — feels like it should be the hard engineering problem. But the component that caused us the most production incidents, the most user complaints, and the most on-call pages was voice. Not because generating voice is hard, but because synchronizing voice with everything else — the video clips, subtitle timing, background music ducking, and lip-sync on AI avatars — is extraordinarily finicky at scale.

This post is a detailed breakdown of how Velora AI Studio's voice pipeline works: from the initial TTS call all the way to the final composite render with frame-perfect subtitle timing.

Stage 1: Script Segmentation

Before a single TTS call is made, our pipeline segments the generated script into scene-aligned narration chunks. Each scene in the video gets its own text segment, and each segment is tagged with estimated duration targets based on the target video length and scene complexity.

This segmentation matters because TTS models have optimal input lengths — too long and the prosody degrades (unnatural pacing, monotone delivery); too short and there's no room for natural intonation arcs. We target 2–4 sentence segments, roughly 15–35 seconds of audio per scene.

Stage 2: Multi-Provider TTS Synthesis

Like our video generation layer, the voice pipeline supports multiple TTS providers. ElevenLabs is our primary provider, handling the vast majority of generation volume due to the emotional range and naturalness of its voice models. Minimax provides our multilingual capacity, handling 28 languages with native-quality output for creators targeting non-English markets.

Each TTS call is made with the following parameters beyond the script text:

• →Voice ID: mapped from the creator's selected voice persona in their brand settings
• →Stability + Similarity Boost: tuned per voice model for optimal naturalness
• →Output Format: PCM 44.1kHz (not MP3) to preserve maximum fidelity for downstream processing
• →Timestamps: word-level alignment data requested alongside audio — this is the key to subtitle accuracy

Stage 3: Waveform Analysis & Timing Normalization

Raw TTS output rarely fits neatly into a target scene duration. A scene might be allocated 18 seconds, but the synthesized audio runs 21.4 seconds. Rather than truncating or stretching (both degrade quality dramatically), our waveform processor applies subtle time-scale modification (TSM) using the phase vocoder algorithm. For deviations under 15%, the result is perceptually transparent — listeners cannot detect the speed adjustment.

For deviations over 15%, the pipeline re-requests the TTS with adjusted SSML speed tags, nudging the speech rate up or down at the synthesis level. This produces much more natural results than post-processing stretching.

Stage 4: Subtitle Alignment & AI Avatar Lip-Sync

The word-level timestamp data from Stage 2 is processed by our subtitle engine. Each word gets a precise start and end timestamp, which drives our subtitle animations (karaoke-style highlighting, word-pop effects, and fade transitions) at the exact millisecond level. Subtitle drift — a common complaint with AI video tools that estimate subtitle timing from script length alone — is eliminated because our timing is grounded in the actual waveform, not approximation.

For AI avatar videos, the same timestamp data feeds our lip-sync layer. We use phoneme-level alignment (derived from forced alignment on the PCM audio) to drive the avatar's mouth shape animations. The result is lip-sync accurate to within ±12ms — comfortably below the 80ms threshold of human perceptual detection. Creators notice this immediately: Velora AI Studio avatars don't have the "slightly off" mouth movement that plagues other platforms.

Stage 5: Music Ducking & Final Audio Mix

The final stage combines the narration audio with background music from our AI Music Studio. Simple volume balancing produces poor results because music doesn't duck evenly — you want aggressive ducking during speech and natural volume during pauses. Our audio mixer performs dynamic loudness-based ducking using the narration waveform's RMS envelope as the control signal for the music attenuation curve. The result is a professional-quality audio mix where narration is always intelligible and background music feels supportive, not competing.