07_Building the RAG Pipeline
Introduction
In simple terms
, the basic RAG process consists of the
1
following three steps: Document structuring and parsing 2
. Chunking and vector embedding 3
. Fact-based retrieval and response (Generating)
For document-based RAG, the process ends with quoting a paragraph in step 3.
Since we’re dealing with broadcast video, the answer is not a paragraph but a clip segment (v_id + start~end).
Therefore, we divided Parsing, Chunking, and Retrieval into separate agent units.
RAG Pipeline = agent-scenario (indexing) + agent-search (search·generation)
Building the Scenemaker Agent
💡 The entire Scenemaker Multi-Agent system was configured as follows.
1. Parsing — Uses video as analysis data
-
Subtitles — agent-stt
- STT → Dialogue
- (Whisper, upcoming Qwen3-ASR) Currently configuring models
- STT → Dialogue
-
Screen — agent-vision
-
VLM → Screen, OCR, Actions
-
Audio → Sounds, BGM
-
2. Chunking + Embedding — agent-scenario
- Fold segments into semantic layers. Index them as vectors.
-
4-tier division: Segment → Scene → Sub-scene → Overall Plot (map-reduce)
-
After embedding, index into a single Milvus collection
-
3. Retrieval and Generation (Retrieval + Generation) — agent-search
- Receives a query and finds clip segments using LangGraph’s 6-step process.
- The result is a list of clip segments (v_id + start–end)
Table Implementation
| Process | Responsible | Task |
|---|---|---|
| Upload | ui-workspace (Next.js) | Upload to S3 + console for viewing analysis results |
| ① Subtitles | agent-stt | Audio → Dialogue·Speaker, vLLM subtitle correction |
| ② Visual Analysis | agent-vision | Analysis of 6-second segments: visuals, OCR, sound, and motion |
| ③ Indexing | agent-scenario | Scene → Act → Overall map-reduce summary + character identification + embedding |
| ④ Search | agent-search | Query via LangGraph’s 6-step process (scope → plan → retrieve → expand → select → assemble) → clip |
First, let’s just look at the flow of agent-scenario and agent-search;
detailed implementation and tuning will be covered in the following post.
Agent-Scenario
💡 The indexing flow that stores video analysis results as plot, characters, and vectors
Agent-Scenario Summary Table
| # | Step | In simple terms | What it actually does | Qwen3.6 |
|---|---|---|---|---|
| 0 | loader | Unpacking data | Merge video dialogue (t_dialogue) and video segments (t_segment) in chronological order | ❌ |
| 1 | scene | Split into scenes | Scene boundaries + summaries + tags (highlights, emotions, ads) | ✅ Every 10 minutes |
| 2 | cast | Identify characters | Guess → Confirm → Propagate (only confirm when "think" is enabled) | ✅ Per scene + 1 time |
| 3 | chapter | Group by act | Group scenes into 10-minute “acts” and summarize (reduce) | ✅ Per act |
| 4 | summary | Write overall plot | Combine acts into a single plot summary for the video (reduce) | ✅ Once |
| 5 | save | Save notes | Record in DB | ❌ |
| 6 | index | Index for search | Embedding → 1 Milvus collection | ❌ |
In a nutshell, it’s all about 6-second atoms → folding into units of meaning → converting to vectors.
The reason for folding a long video using MapReduce rather than summarizing it all at once—and the reason for logging timecodes at each step—is the same: the ultimate goal is clip production.
The “Guess → Confirm → Propagate” process in cast follows the same logic. Since a single scene isn’t enough to identify a person, we first confirm the identity across the entire video and then apply it back to each individual scene.
Agent-Search
💡 A graph illustrating the video search process
Summary table by Agent-Search node
| # | Node | In simple terms | What it actually does | Qwen3.6 |
|---|---|---|---|---|
| 1 | scope | Which video is it? | Selects only relevant episodes (40-vote chapter vector → LLM confirmation, up to 5) | ✅ 0–1 times |
| 2 | plan | Compiling a list of what to find | List of scenes (beats) to find + 2–3 search terms per scene + route determination | ✅ Once per episode |
| 3 | retrieve | Retrieve similar clips | Vector search using keywords → Remove duplicates → Top 10 candidates | ❌ |
| 4 | expand | View the context before and after | Attach a ±12-second context window to the top hits | ❌ |
| 5 | select | Is this really the right one? Where to cut? | Finalize clip range [start–end] or discard (once per material) | ✅ 1–2 times per beat |
| 6 | assemble | Merge and adjust length | Merge overlaps → Trim to target length → Sort → Final list | ❌ |
The biggest difference from document RAG is what happens after retrieval.
Since a 6-second hit isn’t a clip, we concatenate the beginning and end with expand, then use select to determine the section to cut.
If it doesn’t match, we discard it rather than forcing a fill (material is used once per beat).
The LLM is used only for scope, plan, and select; retrieve, expand, and assemble are implemented in code.
Details on how the nodes are assembled (inputs, outputs, etc.) will be covered in a future post.
Conclusion
Currently, only the pipeline has been built; actual model training and code implementation are still in progress.
In a future post, I will describe the design of a broadcasting workflow based on this pipeline.
This post presents research results supported by the Ministry of Science and ICT and the National IT Industry Promotion Agency’s “2026 Open-Source AI and Software Development and Utilization Support Project.”