AI Dubbing and Video Translation in 2026: faster-whisper (24k Stars) + Coqui TTS Pipeline - Dub Any Video in Your Voice

A free local pipeline: faster-whisper (24,747 stars) transcribes with word timestamps, an LLM translates, and Coqui TTS (45,857) or RVC (37,104) re-synthesizes speech - full video dubbing without cloud fees.

💡 What You Will Learn

A free local pipeline: faster-whisper (24,747 stars) transcribes with word timestamps, an LLM translates, and Coqui TTS (45,857) or RVC (37,104) re-synthesizes speech - full video dubbing without clou

## The short answer Professional dubbing costs $50-200 per video minute. This open-source pipeline does it locally: **faster-whisper** (24,747 stars, MIT) transcribes with word-level timestamps, an LLM translates the text, and **Coqui TTS** (45,857 stars, MPL-2.0) generates the new language audio. ## The 4-step pipeline **Step 1 - Transcribe with word timestamps** ```python from faster_whisper import WhisperModel model = WhisperModel("medium", device="cuda", compute_type="float16") segments, _ = model.transcribe("video.mp4", word_timestamps=True) for seg in segments: print(seg.start, seg.end, seg.text) ``` **Step 2 - Translate** - send the transcript to any LLM: "Translate to French, keep it natural for dubbing, short sentences." **Step 3 - Synthesize with Coqui TTS** ```python from TTS.api import TTS tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to("cuda") tts.tts_to_file(text="Bonjour, bienvenue dans notre video...", speaker_wav="original_voice.wav", language="fr", file_path="dub_fr.wav") ``` **Step 4 - Mux** - use ffmpeg to replace the audio track: `ffmpeg -i video.mp4 -i dub_fr.wav -c:v copy -map 0:v -map 1:a out.mp4` ## Real numbers - faster-whisper is ~4x faster than the original Whisper (106,660 stars) implementation - real-time transcription of a 10-min video in ~2 min on a consumer GPU. - XTTS v2 clones a voice from 6+ seconds of reference audio. - Full pipeline cost: $0 software + electricity; cloud dubbing services charge $5-30 per minute. ## FAQ **Q: Can it keep the original speaker's voice?** A: XTTS v2 can clone timbre from a reference clip, but emotion and prosody transfer is limited - expect natural but not identical delivery. **Q: Does it handle lip sync?** A: Not automatically. For lip-synced output, pair with video tools like Wav2Lip after generating audio. **Q: What about subtitles only?** A: Skip step 3-4 - export WhisperX (23,436 stars) word timestamps as .srt for perfect subtitles.