Skip to main content

Designing a Langgraph-Based Broadcast Workflow

· 7 min read
sbin
sbin
The SceneMakerAI Team

Introduction​

This article details the LangGraph process for finding segments of broadcast clips using natural language queries.

It demonstrates the process of structuring the data into a clips[] JSON format.

The search process can be summarized most simply as follows:

  1. Which episode it is (scope)
  2. Compiling a list of what to search for (plan)
  3. Retrieve similar segments (retrieve)
  4. View the surrounding context (expand)
  5. Determine where to cut (select)
  6. Merge and adjust the length (assemble)

agent-search runs the fixed LangGraph in a single location.

The code implementation is as follows.

# agent-search/lib/graph/graph.py

ainvoke(request)
 ├ Selecting episodes for the scope (specifying v_ids or voting + LLM)
 ├ Plan Route + Beats (by session)
 ├ Retrieve   Multi-query vector search
 ├ expand     Hit ±12 seconds context
 ├ select     start–end confirmed (or one round of material review)
 └ assemble   Overlap Merge · budget · clips[] JSON

Requests are received via POST /api/v1/search.

service is currently being constructed only up to compilation / shortform / trailer / ad_slot.

1. scope — Which episode it is​
  • Since this is a series, we first determine “which episode we’re talking about.”
  • If v_ids exists, we use it as is. If not, we vote on the chapter vector → LLM confirms.
SCOPE_MAX_VIDEOS = 5 # Upper limit for episode selection
SCOPE_VOTE_LIMIT = 40     # Top-k chapters for voting

# lib/graph/scope.py

async def run(state: dict, deps) -> dict:
    req = state["request"]

    # ① User specifies the episode → As is
    if req.get("v_ids"):
        videos = await asyncio.to_thread(db.select_videos, req["v_ids"])
        scope = {"mode": "user", "reason": "Specify v_ids in the request", "votes": []}

    # ② Automatic Selection: Chapter Voting → LLM Finalization
    else:
        qvec = (await asyncio.to_thread(embed.embed_query, [qtext]))[0]
        votes = await asyncio.to_thread(vdb.vote_videos, deps.vdb, qvec)
        # ... The LLM determines the v_ids (up to SCOPE_MAX_VIDEOS)
        scope = {"mode": "vote+llm", "reason": ..., "votes": votes}

    return {"videos": videos, "scope": scope}
  • Sample Input/Output
Input:  query="Goal highlights from Game 7 of the Korean Series", v_ids=None

Output:  # mode=vote+llm
{
  "videos": [{"v_id": 1, "name": "Korean Series: KIA vs. SK", ...}],
  "scope": {
    "mode": "vote+llm",
    "reason": "The request terms 'Game 7 of the Korean Series' and 'scoring plays' match v_id=1 ...",
    "votes": [[1, 12], [4, 7], [3, 7], [5, 6], [6, 4], [2, 4]]
  }
}

A total of 40 votes are spread across 6 episodes. Since the top choice has only 12 votes, it’s difficult to decide based on the votes alone, so the LLM confirms one option.

2. Plan — Compiling a list of what to look for​
  • For each episode, review the chapter timeline and define search units (beats).
  • Two routes: pinpoint (1–2 iconic scenes) / structural (multiple clips from the narrative).
# lib/graph/plan.py

# route:
#   pinpoint   — Highlight clips, memorable quote cards
#   structural — Episode summaries, previews, and character highlights

# Beat 1:
#   title, queries[2~3], time_hint, want=["segment","dialogue"]
  • Input/output examples
Input:  v_id=3 Winter Sonata, preset=Emotional_Montage

Output:
{
  "route": "structural",
  "beats": [
    {
      "title": "A Walk by the Lake and a Longing Glance at the Bus Stop",
      "queries": [
        "A scene where the male and female leads take a quiet walk by the lake and gaze at each other,"
        "Two people standing awkwardly face to face at a bus stop, conscious of the stares from their friends nearby,"
        "A close-up of an exchange of glances that captures the excitement of the early episodes of *Winter Sonata*"
      ],
      "time_hint": [0, 600],
      "want": ["segment", "dialogue"],
      "v_id": 3
    }
    # ... multiple beats
  ]
}

The reason there are 2–3 queries entries: to cast a wide net for the same scene—covering visual descriptions, events, and dialogue. Vector search is cost-effective.

3. Retrieve — Fetching Similar Clips​
  • Perform Milvus vector search using the beats as queries. Do not use an LLM.
  • Merge multi-query results → Remove duplicates → Top SEARCH_TOPK (10).
  • “cast” uses a soft filter (since there are many empty “cast” rows, a hard filter would cause recall to plummet).
SEARCH_TOPK = 10

# lib/graph/retrieve.py

def search_beat(client, beat: dict, cast_filter: list[str] | None) -> list[dict]:
    queries = beat.get("queries") or [...]
    qvecs = embed.embed_query(queries)
    # Soft filter steps: [cast+event] → [event] → [cast] → [none]
    # "is_ad" is excluded during post-processing
    return sorted(...)[:SEARCH_TOPK]
  • Sample input/output
Input:  queries=["The male and female protagonists are by the lake...", ...]

Output:  candidates (partial)
[
  {
    "ref_type": "segment", "ref_id": 26,
    "start_sec": 150, "end_sec": 156,
    "text": "A man and a woman are walking along a path by the lake. Man and woman walking; woman stops; man stops",
    "cast": "",
    "distance": 0.6867   # Cosine similarity (the higher the value, the more similar)
  },
  ...
]
4. expand — View the context before and after​
  • A 6-second hit is not a clip. We append the segment and dialogue from ±12 seconds (two segment slots) before and after.
  • select provides the material to snap to the boundaries.
EXPAND_PAD_SEC = 12   # Expansion width before and after the hit
_TOP_HITS = 4 # Number of top hits to expand per beat

# lib/graph/expand.py

def expand_beat(client, beat: dict) -> list[dict]:
    ctxs = []
    for h in beat["candidates"][:_TOP_HITS]:
        s = max(0, h["start_sec"] - EXPAND_PAD_SEC)
        e = h["end_sec"] + EXPAND_PAD_SEC
        ctxs.append({
            "anchor": f"{h['ref_type']}:{h['ref_id']}",
            "rows": vdb.window(client, beat["v_id"], s, e),
        })
    return ctxs
  • Sample Input/Output
Input:  hit segment: 26 (150–156 seconds)

Output:  context (a portion of the rows in the time window)
[
  {"ref_type": "segment", "ref_id": 25, "start_sec": 144, "end_sec": 150, "text": "..."},
  {"ref_type": "segment", "ref_id": 26, "start_sec": 150, "end_sec": 156, "text": "A man and a woman are standing by the lake..."},
  {"ref_type": "segment", "ref_id": 27, "start_sec": 156, "end_sec": 162, "text": "Two people by the lake..."},
]
5. select — Finalize where to cut​
  • The LLM examines the beat, candidates, and surrounding timeline to finalize a single clip [start_sec, end_sec].
  • If irrelevant, usable=false + retry_query → Re-search and re-expand, then re-evaluate only once (REFINE_MAX=1).
REFINE_MAX = 1

# lib/graph/select.py

# Output JSON:
# {"usable": true|false, "start_sec", "end_sec",
#  "source_refs", "cast", "reason", "caption", "score",
#  "retry_query": "only when usable=false"}
  • Sample Input/Output (Success)
Input:  beat="A Walk by the Lake and a Longing Glance at the Bus Stop" + candidates + context

Output:
{
  "usable": true,
  "start_sec": 150, "end_sec": 162,
  "caption": "Two people pause during a walk by the lake, exchanging awkward yet heart-fluttering glances",
  "score": 0.92,
  "reason": "A 12-second segment connecting the 'Walk by the Lake' (26) and the 'Exchange of Glances' (27) that follows immediately"
}
  • Example Input/Output (Rejected)
Output:
{
  "usable": false,
  "score": 0.0,
  "reason": "The candidate's timeline contains only commentary and no actual scoring or batting footage",
  "retry_query": "KIA Drew's hit / KIA Yukemi's hit / ..."
}

If nothing is found, do not force a fill. That is how we prevent hallucinations in video RAG.

6. assemble — Merge and adjust length​
  • Collect only usable clips. Do not use an LLM.
  • Merge overlapping clips from the same episode → adjust to duration budget (prioritize score) → sort → final JSON.
BUDGET_TOLERANCE = 1.15   # Allowable multiplier for exceeding target_duration_sec

# lib/graph/assemble.py

# Trailers are listed in order of production; the rest are listed in order of broadcast date and time
# The output's `v_id` plus `start_sec/end_sec` is passed directly as the input to the next ffmpeg step
  • Example Input/Output
{
  "service": "compilation",
  "preset": "Emotional_Montage",
  "route": "structural",
  "total_duration_sec": 54,
  "clips": [
    {
      "v_id": 3,
      "v_name": "Winter Sonata",
      "order": 1,
      "beat": "A Walk by the Lake and a Longing Glance at the Bus Stop",
      "start_sec": 150,
      "end_sec": 162,
      "start_time": "00:02:30",
      "end_time": "00:02:42",
      "caption": "Two people pause during a walk by the lake, exchanging awkward yet heart-fluttering glances",
      "score": 0.92
    }
  ],
  "trace": {"llm_calls": 6, "retries": 0, "dropped_beats": []}
}

Intermediate results for each node are dumped as output/<req_id>/<node>.json.

Conclusion​

I’ve summarized the search flow and the process of structuring the clips[] array, which were the original goals.

The steps for cutting clips into actual video (ffmpeg serving) and integrating with the service UI will be covered next.

Reference Links)

This article presents research results conducted with support from the Ministry of Science and ICT and the National IT Industry Promotion Agency’s “2026 Open Source AI and Software Development and Utilization Support Project.”