Skip to main content

07_Voice Analysis Process and Results

· 7 min read
sbin
sbin
SceneMakerAI 팀

Introduction​

This article provides a detailed

explanation of the speech-to-text conversion process.

It demonstrates how the output is formatted into JSON.

The speech analysis process can be summarized most simply as follows

1

: Noise removal (Denoise) 1

; detection of human speech segments (VAD) 1

; speaker separation (Pyannotate) 1

; language detection (LID) 1

; and automatic speech recognition (ASR).

After identifying various libraries available for optimization, I finalized the workflow as follows.

# worker-prep_stt/main.py

run()
 ├ 0) _load_audio: Load audio and remove noise (denoise) 
 ├ 1) vad.detect utterance interval (Silero VAD)
 ├ 1b) speaker.diarize  Speaker identification (pyannote)
 ├ 2) _classify_languages  Language Identification (LID) by Segment
 ├ 3) _transcribe_batched  ◀── This is the actual speech-to-text (ASR) process
 ├ 4) _assemble_segments   Reassembling words into sentences + speaker mapping
 └ 5) Save JSON

1. Noise Removal (DeepFilterNet v3)​

  • Removes background music, sound effects, and crowd noise → Reduces ASR hallucination. The length remains unchanged
DEEPFILTER_MODEL = "deepfilternet3"  
ATTEN_LIM_DB = -30   #  -30 = upper limit for noise attenuation while preserving speech (dB). None = full power,
CHUNK_SEC = 30 # Length of long audio chunks (DF spectrogram VRAM limit)

## lib/audio/denoise.py

> def process(audio_np: np.ndarray, sr: int) - tuple[np.ndarray, int]:
    ...
    audio = torch.from_numpy(audio_np)
    if audio.ndim > 1:
        audio = audio.mean(dim=1)        # → mono
    audio = audio.unsqueeze(0)           # [1, T]
    if sr != df_sr:
        audio = F.resample(audio, sr, df_sr)

    total = audio.shape[-1]
    chunk = CHUNK_SEC * df_sr
    
    # === Set the chunk time to 30 seconds and proceed ===
    
    if total <= chunk:
        # Actual noise removal function <- Noise removal is performed in this section
        enhanced = enhance(model=_model, df_state=_df_state, audio=audio,
                           pad=True, atten_lim_db=ATTEN_LIM_DB)
    else:
        n = (total + chunk - 1) // chunk
        log.info(f"denoise chunked: {total / df_sr:.1f}s → {n} chunks of {CHUNK_SEC}s")
        parts = []
        for s in range(0, total, chunk):
            parts.append(enhance(
                model=_model, df_state=_df_state, audio=audio[..., s:s + chunk],
                pad=True, atten_lim_db=ATTEN_LIM_DB,
            ))
        enhanced = torch.cat(parts, dim=-1)

    return enhanced.squeeze(0).cpu().numpy().astype(np.float32), df_sr

  • Since DF (DeepFilterNet) requires a sampling rate (sr) of 48 k, we temporarily convert from 16 k to 48 k.

    • Downsampling from 16 k will be required for future calls

    • Used only during ASR (transcription). Must not be used in LID or VAD segments.

  • If the audio is long, the spectrogram may grow too large, potentially causing a GPU out-of-memory (OOM) error.

    • Set CNUNK_SEC = 30 (maximum length of a single chunk is 30 seconds)

    • chunk = 30 × 48000 (30 seconds converted to the number of samples)

    • Conclusion

      • Increasing this value boosts throughput per chunk but risks VRAM overload

      • Reducing this value lowers the throughput per chunk but ensures VRAM safety

  • Sample Input/Output

Input:  [·♪♪·word♪♪♪·♪·wordword·♪♪♪·]    ← Sequence of sounds mixed with noise (♪)  
Output:  [···word·····wordword······] ← A sequence of sounds with only the noise removed

2. Speech Segment Extraction (VAD)​

  • Extract only the parts where a person is speaking
 model="silero_vad"

 # lib/audio/vad.py
 
> def detect(audio_np: np.ndarray, sr: int = 16000) - list[tuple[float, float]]:
    """Extracting timestamps for speech segments.

    audio_np: float32 1D NumPy (16 kHz recommended)
    sr: Sample rate (Silero VAD officially supports only 16k and 8k)

    Returns: [(start_sec, end_sec), ...]  — Utterance intervals (sorted)
    """

    audio_t = torch.from_numpy(audio_np).float()
    ts_list = _get_speech_timestamps(
        audio_t, _model, sampling_rate=sr,
        min_speech_duration_ms=int(MIN_SPEECH_S * 1000),
        max_speech_duration_s=MAX_SPEECH_S,
        min_silence_duration_ms=int(MIN_SILENCE_S * 1000),
    )
    return [(t["start"] / sr, t["end"] / sr) for t in ts_list]

  • Identifies segments where a person is speaking and sets the sampling rate to 16 kHz
  • Sample Input/Output
Input:  [Silence····Word·······Silence··WordWord····Silence]  ← Sound array # np.ndarray (float32 1D)
Output:  [(2.1, 5.8), (40.3, 47.2), ...] ← Only the timestamps of the spoken segments # list[tuple[float, float]]

3. Speaker Segmentation​

  • The video features various people, and their voices are naturally different.
  • This step involves distinguishing between speakers. We implemented the open-source library pyannotate.
 PYANNOTE_DIARIZE =  "pyannote-diarization"  # Speaker segmentation (community-1, self-contained)
 
 # lib/audio/speaker.py

 def diarize(audio_np: np.ndarray, sr: int = 16000) -> list[tuple[float, float, str]]:
    """Extracting the speaker turn timeline.

    audio_np : float32 1D numpy (16k mono)
    Returns: [(start_s, end_s, speaker_label), ...] (in chronological order)
    """
    #...
    
    waveform = torch.from_numpy(audio_np).unsqueeze(0).to(_device)
    out = _pipeline({"waveform": waveform, "sample_rate": sr})
    # pyannote 4.x: DiarizeOutput.speaker_diarization is an annotation
    turns = [
        (turn.start, turn.end, speaker)
        for turn, _, speaker in out.speaker_diarization.itertracks(yield_label=True)
    ]
    log.info(f"diarization: {len(turns)} turns, {len(set(t[2] for t in turns))} speakers")
    return turns

Example Extraction Results

![image](/img/blog/07-음성분석-처리-과정-및-결과/img-00.png

)

Input/Output Format

Input: Full audio, 16 kHz floating-point array
  
Output: Speaker Identification
[
    (0.0,   5.8,  "SPEAKER_00"),   # 0–5.8 seconds: Speaker 0
    (5.8,   12.3, "SPEAKER_01"),   # 5.8–12.3 seconds: Speaker 1
    (12.3,  15.0, "SPEAKER_00"),   # Back to Speaker 0
    (40.3,  47.2, "SPEAKER_02"),   # Speaker 2
    ...
]

Whisper-large-v3 is the latest speech recognition model released by OpenAI,

and we adopted it because it delivers outstanding performance in tasks such as multilingual speech recognition, automatic caption generation, and audio-to-text conversion.####

  1. Language Identification (LID)
  • This is the process of identifying the language.
MODEL : "whisper-large-v3"
COMPUTE_TYPE: "float16" # Since this is a GPU server, COMPUTE_TYPE is set to float16

 # lib/audio/whisper.py

> def detect_language(chunk: np.ndarray) - tuple[str, float]:
    """raw chunk → (lang_code, prob). LID is derived from the raw audio (PoC Strategy 2)."""

    lang_code, prob, _all_probs = _model.detect_language(chunk)
    return lang_code, prob

  • LID process for a .wav audio file.

    • ko=1.00 → This means it is recognized as “ko” with the highest probability.

  • We compared two models (Whisper LID / VoxLingua107 LID) and selected Whisper following a PoC.

2026-05-28 18:35:21 [INFO] lid_bench: DeepFilterNet3 loaded (sr=48000, device=cuda:0)
2026-05-28 18:35:21 [INFO] lid_bench: Loading Silero VAD...
2026-05-28 18:35:21 [INFO] lid_bench: Loading Whisper LID: mobiuslabsgmbh/faster-whisper-large-v3-turbo
2026-05-28 18:35:22 [INFO] lid_bench: Loading VoxLingua107 LID (savedir=/stg/models/voxlingua107)
2026-05-28 18:35:22 [INFO] lid_bench: === models loaded ===
2026-05-28 18:35:22 [INFO] lid_bench: === bench start: /stg/vod/scenemaker/sound_full/docu.wav ===
2026-05-28 18:35:23 [INFO] lid_bench: denoise chunked: 3520.7s → 118 chunks of 30s
2026-05-28 18:35:33 [INFO] lid_bench: denoise saved: output/denoise/docu.wav
2026-05-28 18:35:43 [INFO] lid_bench: audio 3520.7s → VAD 383 speech segments
2026-05-28 18:35:44 [INFO] lid_bench: [  11.5~  12.7s] W-raw=ko=1.00  V-raw=ko=0.99  W-den=ko=1.00  V-den=ko=0.80
2026-05-28 18:35:44 [INFO] lid_bench: [  13.1~  15.3s] W-raw=ko=1.00  V-raw=ko=1.00  W-den=ko=1.00  V-den=ko=1.00
...
026-05-28 18:35:45 [INFO] lid_bench: [ 278.4~ 281.1s] W-raw=ko=1.00  V-raw=ko=1.00  W-den=ko=1.00  V-den=ko=1.00
2026-05-28 18:35:45 [INFO] lid_bench: [ 282.6~ 284.3s] W-raw=en=0.30  V-raw=it=0.79  W-den=en=0.46  V-den=pt=0.32

5. Automatic Speech Recognition (ASR)​

 MODEL : "whisper-large-v3"
 COMPUTE_TYPE : "float16"
 BATCH_SIZE=16
 
 # lib/audio/whisper.py


> def transcribe_batched(audio: np.ndarray, language: str) - list[dict]:
    """Transcribe the audio into the specified language (using internal VAD to extract only speech segments, with 30-second windows processed in parallel).

    "Audio" typically refers to "a full-length stream with sections outside the target language muted" → The internal VAD
    Since silence is skipped, only the spoken language is recognized, and the timestamp remains the same as the original time.

    Returns results at the word level via `word_timestamps` → The calling party (`stt_service`) uses speaker turns and punctuation as reference points
    Resegmentation (since the layout groups segments into 30-second blocks, this restores granularity and speaker accuracy).

    Returns: [{"start", "end", "word", "seg_logprob"}, ...]  (word-level, absolute time)
             seg_logprob = avg_logprob of the segment to which the word belongs (for the hallucination filter)
    """
    if _model is None:
        load_model()
    segments_gen, _info = _batched.transcribe(
        audio,
        language=language, 
        batch_size=BATCH_SIZE, # Number of 3-second windows processed simultaneously on the GPU; set to 16
        beam_size=5,
        no_speech_threshold=0.6,
        log_prob_threshold=-1.0,
        compression_ratio_threshold=2.4,
        condition_on_previous_text=False,
        repetition_penalty=1.2,
        no_repeat_ngram_size=3,
        vad_filter=True, # Skip silent intervals (key to handling silent streams)
        word_timestamps=True,   # Word-level timestamps
    )
    words = []
    for s in segments_gen:
        for w in (s.words or []):
            words.append({
                "start": float(w.start), "end": float(w.end),
                "word": w.word, "seg_logprob": s.avg_logprob,
            })
    return words
  • Currently, only the operational process has been documented.
2026/06/25 17:15:31 INFO [stt_service.py:_transcribe_batched:204] - batched [zh]: 81 words from 7 ranges
2026/06/25 17:15:31 INFO [transcribe.py:transcribe:390] - Processing audio with duration 58:40.749
2026/06/25 17:15:34 INFO [transcribe.py:transcribe:458] - VAD filter removed 58:38.445 of audio
2026/06/25 17:15:34 INFO [stt_service.py:_transcribe_batched:204] - batched [vi]: 11 words from 1 ranges
2026/06/25 17:15:34 INFO [stt_service.py:_assemble_segments:252] - assemble: 314 segments from 2477 words
2026/06/25 17:15:34 INFO [util.py:write_json:26] - wrote json → output/2/result.json

...
  • Results can be viewed in JSON format.
  {
    "idx": 0,
    "start": "00:00:07.2",
    "end": "00:00:09.9",
    "text": "Huge spoilers—just like last time,",
    "lang": "ko",
    "speaker": "S003"
  },

Conclusion​

The STT and standardization processes, which were the objectives, have been completed.

Based on the extracted results, issues were identified in the subtitles.

We plan to perform further refinement in the future.

Reference Links)

This article presents the results of research conducted with support from the Ministry of Science and ICT and the National IT Industry Promotion Agency under the “2026 Open-Source AI and Software Development and Utilization Support Project.”