Skip to main content

Fixed an issue where unsorted segments were left in SRT/VTT

whisperX: Fixed to leave alignment failure segments in SRT/VTT​

Link: https://github.com/m-bain/whisperX/pull/1491

This is a pull request to fix the issue reported at https://github.com/m-bain/whisperX/issues/1490. In iterate_subtitles(), if words is empty and the segment contains text, the segment is treated as a single word with its own start and end timestamps and output to the queue. Four test cases were added.

Background​

SceneMakerAI generates SRT/VTT/JSON from Korean broadcast audio using the following writers: whisperx.load_model(...).transcribe() → whisperx.align() → whisperx/utils.py. When Whisper attaches a long sentence to a very short segment (hallucination, timestamp error), the number of wav2vec2 frames is fewer than the number of characters, causing alignment to fail and Failed to align segment (...): backtrack failed, resorting to original to be logged. Although the log message states that the original content is preserved, in reality, the .txt, .tsv, and .json files generated from the same result contain sentences, while the .srt and .vtt files do not.

Causes​

  • whisperx/alignment.py:221-227: Initializes all output segments to "words": []. The three cases where sorting fails (:236-239: character not found in the dictionary; :241-244: start time exceeds audio length; :294-297: backtrack failure) are included in the result exactly as they are. In other words, text, start, and end remain, while only words is an empty list.
  • whisperx/utils.py:308: If the first segment contains the words key, it follows the word-based path.
  • whisperx/utils.py:269-270: iterate_subtitles() iterates only through segment["words"] to create the subtitle queue. If the list is empty, no words are added, so no queue is created for that segment. This path does not examine segment["text"] at all.
  • The segment timestamp fallback (utils.py:313-321) added in #1347 only handles cases where a word exists but start / end are missing, so it never reaches an empty words.

Reproduction​

This can be reproduced using only the standard library, without a model (repro_writer.py). When the words of the middle segment among the three is set to an empty list, passing the resulting dict to WriteTXT, WriteTSV, WriteSRT, WriteVTT, the output consists of only 3 lines for TXT/TSV and 2 queues for SRT/VTT.

I also verified the actual align() path (repro_align.py). If I insert a stub wav2vec2 model that returns only 5 frames for a 26-character segment to trigger a backtrack failure, align() returns the words: [] segment, and subsequently, WriteSRT omits that sentence.

Proposed fix (local branch, uncommitted)​

In iterate_subtitles(), treat segments where words is empty and text is not empty as a single word containing the segment’s start /end. whisperx/utils.py Added 6 lines; added 4 test cases tests/test_subtitles_unaligned_segments.py.

Outstanding issues: When using --highlight_words, the entire sentence with the sorting failure is underlined. Removing start /end from placeholders results in output to the regular queue, but this reduces the temporal accuracy of the queue in segment merge mode. Therefore, I’m raising this as an issue first instead of a PR to get feedback from the maintainer.

Verification​

  • Reproduced on upstream main HEAD 771b4a1 (3.8.7rc1), Python 3.12, and torch 2.8.0+cpu.
  • Before the fix, 3 out of 4 new tests failed(The remaining one was intended to verify the existing behavior of skipping empty text segments, so it passed even before the fix).
  • After the fix, all 16 pytest tests/ tests passed.
  • Not tested with an actual wav2vec2 model and real audio (verified using a stub without downloading the model).
  • Duplicate check: I reviewed #75, #404, #230, #1089, #1478, #1315, #1347, #1456, #1458, and #1487 and found no duplicates.