AI Dubbing and Video Translation in 2026: faster-whisper (24k Stars) + Coqui TTS Pipeline - Dub Any Video in Your Voice

A free local pipeline: faster-whisper (24,747 stars) transcribes with word timestamps, an LLM translates, and Coqui TTS (45,857) or RVC (37,104) re-synthesizes speech - full video dubbing without cloud fees.

💡 What You Will Learn

A free local pipeline: faster-whisper (24,747 stars) transcribes with word timestamps, an LLM translates, and Coqui TTS (45,857) or RVC (37,104) re-synthesizes speech - full video dubbing without clou

📜 Table of Contents

• The short answer
• The 4-step pipeline
• Real numbers
• FAQ

The short answer

Professional dubbing costs $50-200 per video minute. This open-source pipeline does it locally: faster-whisper (24,747 stars, MIT) transcribes with word-level timestamps, an LLM translates the text, and Coqui TTS (45,857 stars, MPL-2.0) generates the new language audio.

The 4-step pipeline

Step 1 - Transcribe with word timestamps

from faster_whisper import WhisperModel
model = WhisperModel("medium", device="cuda", compute_type="float16")
segments, _ = model.transcribe("video.mp4", word_timestamps=True)
for seg in segments:
    print(seg.start, seg.end, seg.text)

Step 2 - Translate - send the transcript to any LLM: "Translate to French, keep it natural for dubbing, short sentences."

Step 3 - Synthesize with Coqui TTS

from TTS.api import TTS
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to("cuda")
tts.tts_to_file(text="Bonjour, bienvenue dans notre video...",
                speaker_wav="original_voice.wav",
                language="fr", file_path="dub_fr.wav")

Step 4 - Mux - use ffmpeg to replace the audio track: ffmpeg -i video.mp4 -i dub_fr.wav -c:v copy -map 0:v -map 1:a out.mp4

Real numbers

• faster-whisper is ~4x faster than the original Whisper (106,660 stars) implementation - real-time transcription of a 10-min video in ~2 min on a consumer GPU.
• XTTS v2 clones a voice from 6+ seconds of reference audio.
• Full pipeline cost: $0 software + electricity; cloud dubbing services charge $5-30 per minute.

FAQ

Q: Can it keep the original speaker's voice? A: XTTS v2 can clone timbre from a reference clip, but emotion and prosody transfer is limited - expect natural but not identical delivery.

Q: Does it handle lip sync? A: Not automatically. For lip-synced output, pair with video tools like Wav2Lip after generating audio.

Q: What about subtitles only? A: Skip step 3-4 - export WhisperX (23,436 stars) word timestamps as .srt for perfect subtitles.

❓ FAQ

Can it keep the original speaker's voice?

XTTS v2 can clone timbre from a reference clip, but emotion and prosody transfer is limited - expect natural but not identical delivery.

Does it handle lip sync?

Not automatically. For lip-synced output, pair with video tools like Wav2Lip after generating audio.

What about subtitles only?

Skip step 3-4 - export WhisperX (23,436 stars) word timestamps as .srt for perfect subtitles.