AI Subtitle Generator: Open-Source Automatic Subtitles with WhisperX
Generating subtitles is easy; getting timings and line breaks right is not. A full command chain: ffmpeg extracts audio, WhisperX gives word-level timestamps, lines break at 42 characters, then mux or burn in.
📜 Table of Contents
AI Subtitle Generator: Automatic SRT with WhisperX
Plenty of viewers watch muted, so subtitles decide completion rates. The hard part is not transcription accuracy — modern models are good — it is timing and line breaks. WhisperX adds word-level alignment so every word has start and end times; rules handle the breaks.
1. The chain
| Step | Tool | Output |
|---|---|---|
| extract audio | ffmpeg | 16kHz mono wav |
| transcribe + align | WhisperX | word-level JSON |
| diarization (optional) | WhisperX + pyannote | speaker labels |
| line breaking | Python | SRT |
| mux or burn | ffmpeg | soft track or hardcoded |
Everything runs locally; nothing is uploaded.
2. Install
python -m venv venv && source venv/bin/activate # Windows: venv\Scripts\activate
pip install whisperx
Install ffmpeg (winget install Gyan.FFmpeg on Windows). Confirm PyTorch is the CUDA build, otherwise it silently falls back to CPU. The first run downloads large-v3, several GB.
3. Run it
ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le audio.wav
whisperx audio.wav --model large-v3 --language zh --output_format srt --output_dir out
whisperx audio.wav --model large-v3 --language zh --diarize --output_format srt
4. Parameters
| Flag | Suggested | Note |
|---|---|---|
--model |
large-v3 |
medium when time is short |
--language |
always specify | detection costs time and can misfire |
--batch_size |
8–16 | lower it first on OOM |
--compute_type |
float16, int8 on CPU |
|
--vad_filter |
on | removes hallucinated lines over silence |
--diarize |
interviews, meetings | needs a Hugging Face token |
VAD matters: over silence and music, models invent text. Filtering voice activity removes most of it.
5. Line breaking decides readability
Roughly 42 characters per line, at most two lines, 1–6 seconds on screen, break at punctuation. For Chinese, 16–20 characters per line. Split by accumulating word-level timestamps until the limit, which keeps each line's timing exact without realignment.
6. Mux or burn
ffmpeg -i input.mp4 -i out/audio.srt -c copy -c:s mov_text output.mp4
ffmpeg -i input.mp4 -vf "subtitles=out/audio.srt:force_style='FontName=Source Han Sans,FontSize=20'" -c:a copy output.mp4
Short-video platforms usually ignore soft subtitle tracks, so burn in for those and keep soft tracks for archives. Specify a CJK font or you get boxes.
7. Pre-publish checklist
Verify names, brands and jargon — the highest error source. Check homophone number errors. Spot-check timing at start, middle and end; mid-video drift means a long silence was mis-segmented. Confirm on-screen duration is readable. Watch for whole missing passages, usually VAD filtering speech as noise.
FAQ
WhisperX or FunASR for Chinese? FunASR is often better for pure Chinese with accents and proper nouns; Whisper handles mixed language better. Run both and compare.
Bilingual subtitles? Translate the transcript with an LLM while keeping the original timestamps. Never let the model re-segment, or the timing breaks.
Timing shifted by seconds? Audio/video start offsets; correct with ffmpeg -itsoffset or shift the subtitle uniformly.
How long for an hour of video? Depends on the GPU. On CUDA, large-v3 runs faster than real time; on CPU, slower. Time a one-minute sample first.
Commercial use? Check the license of each model involved, including the diarization model.
Related
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.