06_Project Introduction and Qwen 3.6 Broadcast Domain Implementation
Introduction
At a broadcast station, it takes several days for a professional editor to create a single “Character Roundup” video. To produce these for all five main characters in a 16-episode drama, it would take several weeks.
YouTube Shorts, Reels, TikTok—since the optimal format varies by platform, the same source material must be re-edited at least three times.
Advertising is an entirely different issue. Current VOD ad scheduling likely relies on mechanical allocation based on metadata such as time slot and genre.
Demand for K-content is exploding, yet content repurposing remains tied to manual labor. The bottleneck is clear.
In this post, I’ve outlined what we’re creating, how we’ll create it, and our future plans.
1. What Are We Creating?
A service that trains AI to understand videos, identify stories, and store them, then automatically performs compilation, remixing, and ad optimization
We’ve decided>
to adopt an OpenWeight model and adapt it to the broadcasting domain using RAG and prompt engineering.
Here’s an overview of the service.
| Planned Service | What It Does |
|---|---|
| Compilation | Automatically generates binge-watching guides, character-specific highlights, and emotional montages |
| Remix / Short-Form | Trailers, highlight clips, and quote cards; automatic conversion for each platform |
| Ad Optimization | Recommendations for optimal ad insertion points based on scene context and emotional flow |
| Batch Automation | Templatizes the above three functions for bulk content processing |
2. How It’s Built — 4 Steps
The entire pipeline consists of four steps, as follows.
| STEP | Name | One-Line Summary |
|---|---|---|
| 01 | Multimodal Video Indexing | Breaks down the video into a format readable by AI |
| 02 | Chunking · Storage | Groups scattered information into meaningful scene units and stores them |
| 03 | Video Extraction · Search | Finds the desired scene using natural language queries |
| 04 | Video Generation · Serving | Assembles the found scenes into actual output |
This seems a bit complicated, so I’ll break it down step by step.
STEP 01 — Visual-Auditory Integrated Multimodal Video Indexing
Instead of simply feeding video to the VLM, we decided to select only the parts worth showing to the VLM.
After that, we decided to optimize performance using the tools listed below.
| Section | Tool | Function |
|---|---|---|
| Video/Audio Separation | FFmpeg | Separates frames and audio streams |
| Dialogue/BGM Separation | demucs | Separates dialogue and background music into independent tracks |
| Scene Change Detection | PySceneDetect | Automatically detects cut points → Determines analysis units |
| STT | faster-whisper | Converts dialogue to text. Obtains timestamps for each word |
| BGM Analysis | librosa | Quantifies scene atmosphere using acoustic characteristics (intensity, tempo) |
| Image Analysis · Summary | Qwen3.6 | Inferred scene meaning by integrating the above results |
One thing I learned here:
VLM is an eye, not a pipeline. If you skimp on preprocessing, even the best model won’t work. Half of domain adaptation happens outside the model.
STEP 02 — Chunking, Data Construction, and Storage
So, we align visual, speech, and audio data on the same timeline and regroup them into meaningful units.
The output looks like this. (Planned to be in JSON)
Segment 47 | 00:18:20 ~ 00:18:30
─────────────────────────────────────────────
Summary: A scene in which the daughter's emotions are thrown into turmoil following her mother's confession in the hospital room
Shot Mother and daughter talking → Close-up of the daughter → Expression just before she starts crying
Dialogue "Why did you wait until now to tell me?" / "I didn't want to worry you." / "I'm scared…"
Audio: Sad piano background music, volume increases toward the end, brief silence
Emotions: Sadness 0.82 Anxiety 0.74 [Relationship Turning Point]
We embed this object into Qdrant (vector DB) and store the structured metadata in MariaDB.
It’s a hybrid structure that simultaneously supports semantic search and condition-based filtering.
STEP 03 — Video Extraction and Search
With existing metadata search, results are returned only if keywords match.
We’ve decided to adopt Langraph, which allows for query restructuring, for the service.
Since it’s a graph-based cyclic structure rather than a simple linear pipeline, it reportedly returns to the previous step on its own and retries if the results are unsatisfactory.
STEP 04 — Video Generation and Serving
The searched segments are assembled into actual video. Rather than re-encoding the video, we plan to extract only the necessary segments to maintain original quality while outputting them in platform-specific aspect ratios (16:9 / 9:16).
We decided to have LangGraph orchestrate this entire process.
3. OpenWeights in Use
As of now, we are experimenting with the following two models.
| QWEN3.6-27B | https://huggingface.co/Qwen/Qwen3.6-27B |
|---|---|
| Qwen3.6-35B-A3B | https://huggingface.co/Qwen/Qwen3.6-35B-A3B |
- System Setup and Service Flow
💡 July 2026 / Two months into the project; I’ve documented the systems and services that have been set up.
System Setup
We decided to use two GPUs.
| Instance Name | GPU Model | Number of GPUs | Total GPU Memory (VRAM) | Memory Specifications |
|---|---|---|---|---|
| g7e.2xlarge | NVIDIA RTX PRO 6000 Blackwell | 1 | 96 GB | GDDR7, bandwidth 1,597 GB/s |
| g7e.12xlarge | NVIDIA RTX PRO 6000 Blackwell | 2 | 192 GB (96 GB each) | GDDR7, supports direct GPU-to-GPU communication (P2P) |
I drew a diagram including the web, API, and database.
This diagram reflects the current status as of today and may differ at the time of final deployment.
Service Flow
These are the four agents built so far.
| Agent | Role | Key Stack |
|---|---|---|
| agent-stt | Speech recognition, speaker-specific dialogue extraction, subtitle generation | faster-whisper, demucs |
| agent-vision | Scene detection, frame analysis, visual context inference | PySceneDetect, Qwen3.6 VLM |
| agent-scenario | Narrative structure identification, emotional flow analysis, scene merging | Qwen3.6, RAG |
| agent-search | Natural language query → scene search, segment extraction | Qdrant, Embedding |
We have separated it into four independent agents. (Additionally, for STT, we are currently performing up to a second round of calibration.)
I’ve summarized the current service flow in a diagram.
5. Preview of the Next Post
In this post, I covered what we plan to build and the approach we’ll take to build it.
Starting with the next post, I’ll walk you through each step of the process.
- Speech analysis process and results
- Sharing my experience optimizing Faster-Whisper for Korean speech recognition
References
- Qwen3.6-27B: https://huggingface.co/Qwen/Qwen3.6-27B
- Qwen3.6-35B-A3B: https://huggingface.co/Qwen/Qwen3.6-35B-A3B
- vLLM: https://github.com/vllm-project/vllm
- LangGraph: https://github.com/langchain-ai/langgraph
- faster-whisper: https://github.com/SYSTRAN/faster-whisper
- PySceneDetect: https://github.com/Breakthrough/PySceneDetect
- demucs: https://github.com/adefossez/demucs
- Qdrant: https://github.com/qdrant/qdrant
- Amazon EC2 G7e: https://aws.amazon.com/ec2/instance-types/g7e/
This article presents research results conducted with support from the Ministry of Science and ICT and the National IT Industry Promotion Agency under the “2026 Open-Source AI and Software Development and Utilization Support Project.”