Skip to main content

[Part 1] Benchmark for Multimodal LLM Understanding of 6-Second Clips from Korean Broadcasts [Pipeline Construction]

Testing Environment:

AWS g7e.4xlarge (96 GB VRAM) vLLM serving / Qwen3-Omni-30B-A3B-Instruct (Qwen multimodal model)

1. Project Overview​

  • Video Analysis Pipeline

    • Client → poc-vision-bench (API Server) → vLLM (Qwen3-Omni)

    • Verify whether the above flow operates normally.

      • Analysis quality, parameter tuning, and quantitative evaluation will be addressed later

  • Verification Flow:

  • The three API types verified in this section: poc-vision-bench API: .1 Status Query : /healthz

    1. Single call : /chat

    2. Batch processing : /chat/batch

2. Preliminary Investigation​

2.1. Analysis Model: Qwen3-Omni-30B-A3B-Instruct​

  • Adopted Qwen3-Omni-30B-A3B-Instruct for integrated analysis of a 6-second multimodal clip via a single invocation.
  • It is virtually the only open-source option capable of processing the four modalities—Image, Video, Audio, and Text—within a single model and supporting OpenAI-compatible vLLM serving. It features a Thinker–Talker MoE architecture. The inference core (Thinker) totals 30B / 3B active; the full checkpoint, including the Talker (speech), audio, and vision encoders, is ≈ 35B (this PoC uses only text output → Talker not used).

Model Specifications

ItemValue
ArchitectureThinker–Talker MoE (native omnimodal end-to-end)
ParametersTotal 30B for the inference core (Thinker) / 3B active · Total ≈ 35B including Talker and encoders
InputText · Image · Audio · Video
OutputText (+speech). This PoC uses text only (Talker not used)
ContextNative 32,768 tokens (16,384 used in live service)
Multilingual Support119 languages for text / 19 for voice input / 10 for voice output → Full support for Korean
LicenseApache 2.0 (Commercial use permitted)

VRAM / Production Configuration (g7e.4xlarge · 1 GPU)

ItemValueNote
GPUNVIDIA RTX PRO 6000 Blackwell × 1 (96 GB)Oregon us-west-2
BF16 Memory (Official Card)15 sec: 78.85 GB / 30 sec: 88.52 / 60 sec: 107.7496 GB is sufficient for a 6-second clip
--dtypebfloat16Original precision (not quantized, 66 GiB full checkpoint)
--gpu-memory-utilization0.85 (≈ 81.6 GB allocated)
--tensor-parallel-size1Single GPU
--max-num-seqs8More than the app’s concurrency (4), so there’s plenty of headroom

Explanation of Each Item

  • GPU - RTX PRO 6000 Blackwell × 1 (96 GB): Blackwell-generation server GPU. Running a 30B model at full precision (BF16)—including the KV cache and multimodal encoder—on a single card requires a large amount of VRAM, which the 96 GB capacity can handle.
  • BF16 Memory (Official Card): VRAM requirements by input video length as specified by the Qwen model card. The longer the video, the more video tokens there are, and the more memory is required. Since this PoC uses a 6-second clip, there is ample room within the 96 GB.
  • --dtype bfloat16 : Serving at full precision without quantization (full checkpoint ≈ 66 GiB). There is no loss in quality, but it consumes a lot of memory.
  • --gpu-memory-utilization 0.85 : The percentage of GPU memory (≈ 81.6 GB) that the vLLM reserves for weights and the KV cache. Increasing this value boosts concurrent throughput (KV cache) but raises the risk of an Out-of-Memory (OOM) error; lowering it ensures safety but reduces throughput.
  • --tensor-parallel-size 1 : The model is loaded entirely onto a single GPU rather than being split across multiple GPUs.
  • --max-num-seqs 8 : The maximum number of requests (sequences) that vLLM can process simultaneously = the internal batch size limit. This is greater than the gateway (API Server) concurrency (4), so vLLM has some headroom.
note

📍 Where are these settings located?

--dtype, --gpu-memory-utilization, --tensor-parallel-size, and --max-num-seqs are vLLM server startup arguments.

2.2. Input Method: from_video (single MP4 input) vs from_frames_audio (separate inputs)​

  • Adopted the from_video method, which passes a single 6-second MP4 file as-is as a single component via video_url (base64 data).
  • Since Qwen3-Omni natively supports integrated video and audio understanding via use_audio_in_video, the from_video method—which passes a single MP4 file as-is—is the model’s recommended input method and results in the simplest pipeline. Separate input was put on hold because it increases the number of components to four and introduces the burden of missing motion between keyframes and alignment issues.
Comparison Itemsfrom_video (Adopted)from_frames_audio (On Hold)
Input ConfigurationSingle MP4 file → video_url (data URI) 1 componentN keyframe JPEGs + 1 WAV file → 4 components
Timing AlignmentVideo and audio automatically synchronized within the containerSeparate alignment must be ensured on the client side
Preprocessing Output Size6-second MP4 (~1–3 MB per clip)3 JPG frames + WAV (~hundreds of KB per clip)
Hallucination ImpactVideo and audio alignment and context naturally preservedPossible motion gaps between keyframes

##3. Testing

Testing Method​

Verify the basic operation of Client → API Server → vLLM through the following 6 steps.

  1. Preparing Sample Data

    • Prepare test video data and split a 10-minute segment into 100 6-second clips.

    • Use ffmpeg to create clips where the audio is preserved but the video is blacked out (for voice-only analysis verification)

  2. Run the analysis server

    • Launch poc-vision-bench to receive clips and relay them to the vLLM

  3. Single Inference Call (/chat)

    1. Text only

    2. Video + prompt

    .1 Blackout only the video (same prompt as “Video + Prompt”) to verify audio is reflected

  4. Batch inference call (/chat/batch )

    • Send multiple clips (“Video + Prompt”) in a single request to verify batch processing

  5. Save results

    • Record responses and processing statistics (success/failure, elapsed time) for each call

  6. Summary and Evaluation

    • Summarize at a glance whether each API functioned properly.

3.1. Test Data​

The source data used for testing is as follows. Videos ranging from 50 minutes to 2 hours that closely resemble actual broadcast footage.

CategoryBroadcastDurationURL
newsKBS 9 News48:30https://www.youtube.com/watch?v=rX1P-jOoNmM
docuSuperfish Part 158:40https://www.youtube.com/watch?v=iNbWqC1iqKw
dramaKBS Winter Sonata1:04:52https://www.youtube.com/watch?v=irVKEhb9g8M
historical_dramaTaejo Wang Geon54:10https://www.youtube.com/watch?v=nmlE2iPWLGM
entertain"Fifteen Nights on a Business Trip" X Starship National Sports Festival Full Version1:00:06https://www.youtube.com/watch?v=6wJGpi1nkCg
baseball2009 KBO League Korean Series Game 71:55:22https://www.youtube.com/watch?v=fP1QEs1Uj5U
esports2024 LCK Summer Finals: GEN vs. HLE2:11:23https://www.youtube.com/watch?v=_A_I75nJMF8

Data Preparation Procedure

Prepare the video data from the table above.

  1. Preparing the Original Data

    • Prepare each video in the data/raw/{category}/{name} directory within the working directory.

    • Categories are as follows: baseball, docu (documentary), drama, entertain (variety), esports, hist_drama (historical drama), news

  2. Splitting into 6-Second Clips

    • Split the 00:10:00~00:20:00 section of each video into 100 clips, each 6 seconds long.

    • data/clips/{category}/{원본명}/{seq}_{start}-{end}.mp4

cd "$(git rev-parse --show-toplevel)"
TARGETS=(
  "baseball baseball.mp4"
  "docu docu.mp4"
  "drama drama.mp4"
  "entertain entertain.mp4"
  "esports lol.mp4"
  "hist_drama hist_drama.mp4"
  "news news.mp4"
)
for target in "${TARGETS[@]}"; do
  read -r CAT NAME <<< "$target"
  BASE="${NAME%.mp4}"
  SRC="data/raw/$CAT/$NAME"
  OUT="data/clips/$CAT/$BASE"; mkdir -p "$OUT"
  [ -f "$SRC" ] || { echo "! Source not found, skipping: $SRC"; continue; }
  echo "▶ Starting $CAT/$BASE split"
  for i in $(seq 0 99); do
    start=$((600 + i*6)); end=$((start + 6))
    name=$(printf "%04d_%04d-%04d" $((i+1)) "$start" "$end")
    ffmpeg -nostdin -ss "$start" -i "$SRC" -t 6 \
      -c:v libopenh264 -b:v 1500k -c:a aac -movflags +faststart \
      "$OUT/$name.mp4"
  done
done
  1. Screen Blackout (for audio-only verification)

    • Create one clip where only the screen of the first split clip is blacked out, while leaving the audio intact

    • data/blackout/{category}/{원본명}/

cd "$(git rev-parse --show-toplevel)"
TARGETS=(
  "baseball baseball.mp4"
  "docu docu.mp4"
  "drama drama.mp4"
  "entertain entertain.mp4"
  "esports lol.mp4"
  "hist_drama hist_drama.mp4"
  "news news.mp4"
)
for target in "${TARGETS[@]}"; do
  read -r CAT NAME <<< "$target"
  BASE="${NAME%.mp4}"
  OUT="data/clips/$CAT/$BASE"
  BLACK="data/blackout/$CAT/$BASE"; mkdir -p "$BLACK"
  clips=( "$OUT"/*.mp4 )
  FIRST="${clips[0]}"
  [ -f "$FIRST" ] || { echo "! No split clip, skipping: $OUT"; continue; }
  echo "▶ $CAT/$BASE blackout → $(basename "$FIRST")"
  ffmpeg -nostdin -i "$FIRST" \
    -vf "drawbox=0:0:iw:ih:color=black:t=fill" \
    -c:v libopenh264 -b:v 300k -c:a copy \
    "$BLACK/$(basename "$FIRST")"
done
note

⚡ Run in One Go

./script/prepare_data.sh <카테고리> <파일명>

  • Perform only clip splitting and blackout operations on the already acquired data/raw source (do not modify the source).

Final Test Clip Data

Category KeyGenreNumber of ClipsResolutionfpsAverage SizeRemarks
newsNews1001920×1080301.17 MBHigh proportion of subtitles and anchor commentary
docuDocumentary1001920×1080301.62 MBNarration + Nature·on-site sound
baseballBaseball Broadcast100640×36029.971.13 MBCommentator + Crowd Cheers + Scoreboard UI
entertainVariety Show1001920×108029.971.15 MBGroup conversation + subtitle effects
dramaContemporary Drama100720×48029.971.10 MBCharacter Dialogue + BGM
hist_dramaHistorical Drama1001920×108029.971.23 MBPeriod Costumes & Props + Formal Dialogue
esportsEsports1001920×1080601.35 MBGame UI Overlay + Caster + Game Sounds
Total-700--≈ 1.25 MB7 original videos (1 per genre, 10-minute window divided into 100 equal parts)
warning

🔒 Data Handling Policy

  • Videos are used solely for internal quality assessment (PoC) purposes and will not be distributed or republished externally.
  • Videos and analysis results shall not be included in the code repository.
  • Processed copies shall not be stored separately.
  • After evaluation is complete, local videos and deliverables shall be discarded in accordance with the retention policy.

3.2. Analysis Server (vLLM Frontend API Gateway)​

A lightweight server that receives analysis requests and forwards them to the vLLM. The entry points are src/app.py (PYTHONPATH=src uv run uvicorn app:app --port 8001). The interactive API documentation is provided via /docs (Swagger), /redoc, and /openapi.json.

####3.2.1. Design

  • The server poc-vision-bench is a thin gateway (FastAPI) in front of the vLLM /v1/chat/completions.

  • Inference is handled exclusively by the vLLM; the server passes the request body through without modification, adding only three elements. .1 Semaphore concurrency gate

    1. Batch NDJSON streaming (real-time verification)

    2. request_id logging (X-Request-Id header). Prompt assembly, base64 encoding, response_format schema enforcement, and response validation are all performed on the client.

  • If the gateway is set to passthrough, experimental variations (prompt, schema, fps, sampling) are modified only on the client.

  • The server guarantees only vLLM protection (concurrency cap) and multi-request efficiency (fan-out streaming).

  • Upstream calls are provided directly to vLLM inference as raw httpx, rather than via the OpenAI SDK.

3.2.2. Concurrency · Backpressure​

  • vLLM upstream calls are gated by asyncio.Semaphore(VLLM_CONCURRENCY) (default 4). Excess requests are not rejected but queued.
  • /chat and /chat/batch share the same semaphore → The number of active tasks across both routes is kept below the upper limit.
  • The semaphore is created once during the FastAPI lifespan and injected into app.state(no runtime changes).
  • After VLLMClient.chat() acquires this semaphore, it is measured by time.monotonic() → The returned value from elapsed_ms is the round-trip time (network + inference) for the vLLM call, excluding queuing time.

Server Configuration ( .env → Settings )

KeyDefaultRole
VLLM_BASE_URL-vLLM /v1 endpoint
VLLM_CONCURRENCY4Maximum concurrent calls (semaphore). Recommended: 1–8
MAX_BATCH_ITEMS128/chat/batch Maximum number of items per request
VLLM_TIMEOUT_SECONDS600sUpstream call timeout
VLLM_ACQUIRE_TIMEOUT_SECONDS300sMaximum wait time (seconds) for acquiring a semaphore permit. If exceeded, only that request is marked as failed.

####3.2.3. Batch NDJSON Streaming

  • /chat/batch receives multiple items, performs fan-out, and streams them one line at a time in completion order.

  • Since this is not in the order of input, it is matched using id; even if one or two requests fail, the rest continue to process.

  • Backpressure and deadlock prevention: The number of concurrent requests is limited by the semaphore permit. If the permit is full, new requests wait until a slot becomes available. If a disconnected request remains holding a permit, the gateway may freeze, so permits are reclaimed in two steps: .1 The streaming loop checks the connection every 0.5 seconds using request.is_disconnected() → If disconnected, it cancels any ongoing operations and immediately returns the permit

    1. For half-open connections where no disconnection signal is received, the permit is reclaimed by causing that specific request to fail via the permit acquisition timeout (VLLM_ACQUIRE_TIMEOUT_SECONDS, default 300 seconds).

Request body:

{"items": [
  {"id": "0001_0600-0606", "body": {<vLLM chat.completions body - same as /chat>}},
  {"id": "0002_0606-0612", "body": {<...>}}
]}

Response (1 line = 1 JSON object, separated by line breaks):

{"id": "0001_0600-0606", "status": 200, "elapsed_ms": 3104, "body": {<vLLM response>}}
{"id": "0002_0606-0612", "status": 500, "elapsed_ms": 0, "error": "<message>"}
FieldMeaning
idIdentifier sent by the client (usually clip_id). This differs in meaning from the body.id (chatcmpl-…) issued by vLLM
status200 = Success / vLLM 4xx·5xx as-is / 500 = Server-side exception (network disconnection, etc.)
elapsed_msTime from acquiring the semaphore until the vLLM response is complete (excluding queue wait time). 0 in case of an exception
body / errorvLLM response body on success / error message on failure
  • Constraint: len(items) ≤ MAX_BATCH_ITEMS (default 128). If exceeded, immediately return 413 (NDJSON not started, single JSON error). Include X-Batch-Total (number of received items) in the response header.

3.2.4. Server Execution​

The server is managed via script/service.sh.

./script/service.sh start # Start in the background (wait until healthz reports "OK")
./script/service.sh status     # Check PID, healthz, and port
./script/service.sh restart    # stop → start
./script/service.sh stop
  • Run directly: PYTHONPATH=src uv run uvicorn app:app --host 0.0.0.0 --port 8001

3.2.5. API Input/Output Examples​

MethodPathRoleRemarks
GET/healthzHealth CheckAlways returns 200 after the lifespan expires. Does not check whether the upstream was reached
POST/chatSingle-item passthroughvLLM body as-is → Response as-is. Returns 502 if the upstream cannot be reached
POST/chat/batchMulti-item NDJSON streamingStreams line by line in the order of completion
  1. /healthz
{"ok": true}
  1. /chat (Single)
  • Input: vLLM body assembled by the client (base64 video + prompt + strict schema). Multimodal options are separated into two keys. Frame sampling is handled by media_io_kwargs.video (fps or num_frames, vLLM I/O loader), and audio integration is handled by mm_processor_kwargs.use_audio_in_video.
{
  "model": "qwen",
  "messages": [{"role": "user", "content": [
    {"type": "video_url", "video_url": {"url": "data:video/mp4;base64,<...>"}},
    {"type": "text", "text": "<Prompt>"}
  ]}],
  "temperature": 0.3, "max_tokens": 1024,
  "response_format": {"type": "json_schema", "json_schema": {"name": "clip_analysis", "strict": true, "schema": "<AnalysisResult with 4 fields>"}},
  "media_io_kwargs": {"video": {"fps": 2.0}},
  "mm_processor_kwargs": {"use_audio_in_video": true},
  "chat_template_kwargs": {"enable_thinking": false}
}
  • Output: The vLLM response as-is, as a strict JSON string in choices[0].message.content:
{
  "id": "chatcmpl-...",
  "choices": [{"message": {"role": "assistant", "content": "<JSON below>"}, "finish_reason": "stop"}]
}

Execution: ./script/curl_examples.sh chat

  1. /chat/batch (multiple entries) Input: {items:[{id, body}, …]} (Each body = same as ②)
{"items": [
  {"id": "0001_0600-0606", "body": {"Same as ②"}},
  {"id": "0002_0606-0612", "body": {"..."}}
]}

Output: application/x-ndjson One line at a time in the order of completion (field details 3.2.3):

{"id":"0001_0600-0606","status":200,"elapsed_ms":3104,"body":{<vLLM response>}}
{"id":"0002_0606-0612","status":500,"elapsed_ms":0,"error":"<Message>"}

Execution: ./script/curl_examples.sh batch

3.3. Test Execution and Results​

Verify by actually calling the client → API server → vLLM pipeline according to the flow in §3.0. (Reproduction: experiments/01_pipeline/api_check.py )

3.3.1. Status Query (GET /healthz )​

Verify gateway availability. Always returns 200 after passing the lifespan (does not check whether the upstream vLLM was reached).

$ curl -i http://localhost:8001/healthz
HTTP/1.1 200 OK
content-type: application/json
x-request-id: 6da1b40a

{"ok":true}

3.3.2. Single Inference (POST /chat )​

  1. Text Inference

    Request
    curl -sS -X POST http://localhost:8001/chat \
      -H "Content-Type: application/json" \
      -d '{
        "model": "qwen",
        "messages": [{"role": "user", "content": [
          {"type": "text", "text": "Please write a one-sentence self-introduction in Korean."}
        ]}]
      }' | jq
    Response
    {
      "id": "chatcmpl-a4e66116bd600be3",
      "object": "chat.completion",
      "created": 1780911108,
      "prompt_routed_experts": null,
      "model": "qwen",
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "Hello, I am a native Korean speaker, and I enjoy having natural and interesting conversations on a variety of topics.",
            "refusal": null,
            "annotations": null,
            "audio": null,
            "function_call": null,
            "tool_calls": [],
            "reasoning": null
          },
          "logprobs": null,
          "finish_reason": "stop",
          "stop_reason": null,
          "token_ids": null,
          "routed_experts": null
        }
      ],
      "service_tier": null,
      "system_fingerprint": "vllm-0.21.0-955d20dc",
      "usage": {
        "prompt_tokens": 23,
        "total_tokens": 63,
        "completion_tokens": 40,
        "prompt_tokens_details": null
      },
      "prompt_logprobs": null,
      "prompt_token_ids": null,
      "prompt_text": null,
      "kv_transfer_params": null
    }

  2. Image + Prompt

    Request (curl)

    Since the video’s base64 encoding is large, build the payload as a file and send it (--data-binary). A temperature of 0.3 is recommended for the video (higher values cause out-of-class character degeneration).

    REPO_DIR=$(git rev-parse --show-toplevel)
    CLIP=${REPO_DIR}/data/clips/baseball/baseball/0001_0600-0606.mp4
    PYTHONPATH=src uv run python - "$CLIP" <<'PY'
    import base64, json, sys
    b = base64.b64encode(open(sys.argv[1],"rb").read()).decode()
    json.dump({"model":"qwen","messages":[{"role":"user","content":[
        {"type":"video_url","video_url":{"url":"data:video/mp4;base64,"+b}},
        {"type":"text","text":"Analyze the video and audio in this clip in Korean."}]}],
      "temperature":0.3,
      "mm_processor_kwargs":{"use_audio_in_video":True},
      "chat_template_kwargs":{"enable_thinking":False}}, open("/tmp/req.json","w"), ensure_ascii=False)
    PY
    curl -sS -X POST http://localhost:8001/chat -H "Content-Type: application/json" --data-binary @/tmp/req.json | jq
    Response (JSON)
    {
      "id": "chatcmpl-be4589ee14d26f36",
      "object": "chat.completion",
      "created": 1780911130,
      "prompt_routed_experts": null,
      "model": "qwen",
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "assistant>\nThis video shows a scene from a baseball game. It was filmed inside the stadium and primarily shows the positions of the pitcher, the batter, and the catcher. Both the pitcher and the catcher are wearing red uniforms, and the batter is also wearing a red uniform. Advertising boards are visible in the background, featuring ads for “Pocari Sweat” and “Super Dong.” The atmosphere in the stadium is lively, and the cheers of the crowd can be heard. \n\nThe audio is in Korean, and the voices of the game broadcasters can be heard. The broadcast commentary explains the progress of the game and mentions the performance of specific players. The comment, “Don’t you just have to give him the MVP?” is heard, which appears to be an assessment of a specific player’s outstanding performance. Additionally, the comment “Kia is mounting another counterattack” is heard, indicating that the opposing team is attempting a comeback during the game. \n\nOverall, this video captures the tense moments of a baseball game, conveying the teams’ performance and the flow of the game to the audience through the broadcast commentary.",
            "refusal": null,
            "annotations": null,
            "audio": null,
            "function_call": null,
            "tool_calls": [],
            "reasoning": null
          },
          "logprobs": null,
          "finish_reason": "stop",
          "stop_reason": null,
          "token_ids": null,
          "routed_experts": null
        }
      ],
      "service_tier": null,
      "system_fingerprint": "vllm-0.21.0-955d20dc",
      "usage": {
        "prompt_tokens": 3633,
        "total_tokens": 3974,
        "completion_tokens": 341,
        "prompt_tokens_details": null
      },
      "prompt_logprobs": null,
      "prompt_token_ids": null,
      "prompt_text": null,
      "kv_transfer_params": null
    }

  3. Blackout Video + Prompt (screen only black, audio retained)

    Request (curl)

    Completely identical to [2. Video + Prompt] above (same prompt·temperature 0.3); only the clip in CLIP is replaced with one where the screen is blacked out. If audio remains in the output even after the screen is blanked out → the model is actually processing the audio

    REPO_DIR=$(git rev-parse --show-toplevel)
    CLIP=${REPO_DIR}/data/blackout/baseball/baseball/0001_0600-0606.mp4
    PYTHONPATH=src uv run python - "$CLIP" <<'PY'
    import base64, json, sys
    b = base64.b64encode(open(sys.argv[1],"rb").read()).decode()
    json.dump({"model":"qwen","messages":[{"role":"user","content":[
        {"type":"video_url","video_url":{"url":"data:video/mp4;base64,"+b}},
        {"type":"text","text":"Analyze the visuals and audio in this video in Korean."}]}],
      "temperature":0.3,
      "mm_processor_kwargs":{"use_audio_in_video":True},
      "chat_template_kwargs":{"enable_thinking":False}}, open("/tmp/req.json","w"), ensure_ascii=False)
    PY
    curl -sS -X POST http://localhost:8001/chat -H "Content-Type: application/json" --data-binary @/tmp/req.json | jq
    Response (content)
    {
      "id": "chatcmpl-afbd596bbaba2f3b",
      "object": "chat.completion",
      "created": 1780911151,
      "prompt_routed_experts": null,
      "model": "qwen",
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "assistant\nThis video shows a scene that has been converted to black and white. There is no visual information on the screen; only a black background is visible. This appears to be either a portion of the video that has been corrupted or a scene that was intentionally converted to black and white. \n\nThe audio is in Korean, and two men are having a conversation. The first man says, “Don’t we have to give him the MVP?” and the second man agrees, saying, “That’s right.” This conversation is likely a discussion about a sports game. \n\nHowever, this dialogue is not linked to the video’s visual content, forcing viewers to rely solely on the audio to understand the video’s meaning. This could be due to a lack of visual information in the video or a deliberate strategy to direct the viewer’s attention to the audio.",
            "refusal": null,
            "annotations": null,
            "audio": null,
            "function_call": null,
            "tool_calls": [],
            "reasoning": null
          },
          "logprobs": null,
          "finish_reason": "stop",
          "stop_reason": null,
          "token_ids": null,
          "routed_experts": null
        }
      ],
      "service_tier": null,
      "system_fingerprint": "vllm-0.21.0-955d20dc",
      "usage": {
        "prompt_tokens": 3633,
        "total_tokens": 3885,
        "completion_tokens": 252,
        "prompt_tokens_details": null
      },
      "prompt_logprobs": null,
      "prompt_token_ids": null,
      "prompt_text": null,
      "kv_transfer_params": null
    }

3.3.3. Batch Inference (POST /chat/batch )​

Using the same 12 clips and identical parameters (temperature 0.3, server default samples at the time of measurement), compare the processing times for ① one clip at a time /chat sequentially versus ② /chat/batch batch processing. (Reproduction: experiments/01_pipeline/batch_throughput.py )

Code Summary for Reproduction (batch_throughput.py Core Section)
# experiments/01_pipeline/batch_throughput.py Core (Comparison of sequential vs. batch processing with the same items)
import os, base64, json, time, httpx
from pathlib import Path

_HERE = Path(__file__).resolve().parent
_DATA = Path(os.environ.get("DATA_DIR") or _HERE.parent.parent / "data")
CLIPS_ROOT = _DATA / "clips"

SVR = "http://localhost:8001"
_SCENE = CLIPS_ROOT / "baseball/baseball"
CLIPS = [str(p.relative_to(CLIPS_ROOT)) for p in sorted(_SCENE.glob("*.mp4"))[:12]]  # 12 consecutive clips (0001–0012)

def chat_body(clip):
    b64 = base64.b64encode(clip.read_bytes()).decode()
    return {"model": "qwen", "temperature": 0.3,
            "messages": [{"role": "user", "content": [
                {"type": "video_url", "video_url": {"url": f"data:video/mp4;base64,{b64}"}},
                {"type": "text", "text": "Analyze the visuals and audio in this video in Korean."}]}],
            "mm_processor_kwargs": {"use_audio_in_video": True},
            "chat_template_kwargs": {"enable_thinking": False}}

items = [{"id": Path(c).name, "body": chat_body(CLIPS_ROOT / c)} for c in CLIPS]  # Encoded once in base64 → Reused in both modes

with httpx.Client(timeout=600) as cli:
    # ① Sequence: One at a time /chat (proceed to the next after completing the previous one)
    t = time.monotonic()
    for it in items:
        cli.post(f"{SVR}/chat", json=it["body"])
    seq_ms = int((time.monotonic() - t) * 1000)

    # ② Batch: /chat/batch → NDJSON streaming in order of completion
    t = time.monotonic()
    with cli.stream("POST", f"{SVR}/chat/batch", json={"items": items}) as r:
        for line in r.iter_lines():
            if line:
                json.loads(line)  # line = {id, status, elapsed_ms, body|error}
    batch_ms = int((time.monotonic() - t) * 1000)

print(f"Sequential {seq_ms} ms · Batch {batch_ms} ms · {seq_ms / batch_ms:.2f}×")
details ModeTotal Processing TimeSuccess
Sequential (one at a time /chat )37,536 ms12/12
Batch (/chat/batch batch)22,603 ms12/12

Batch processing is faster than sequential processing (approximately 1.7 times faster; fan-out parallelism limited by gateway concurrency VLLM_CONCURRENCY=4. The multiplier varies with each execution due to empty output and burst jitter). Arrival order ≠ input order (streaming in completion order), X-Batch-Total=12. Multiple requests in a single request, streaming in completion order, and each request independent status—all function normally.

3.3.4. Summary​

§3.3 Call Results at a Glance. (Based on normal pipeline operation, not output quality or accuracy)

ItemRouteVerification DetailsKey ResultVerdict
Status CheckGET /healthzGateway availability · X-Request-IdX-Request-Id assigned✅ PASS
Single · TextPOST /chatBasic text inference behaviorNormal, 1 sentence (prompt 23 · completion 25)✅ PASS
Single·VideoPOST /chatIntegrated video and audio analysisKorean scene analysis✅ PASS
Single·BlackoutPOST /chatAudio reflected even when screen is obscured (controlled test)Black screen recognition + capture of broadcast audio✅ PASS
BatchPOST /chat/batchSimultaneous multi-item processing · Streaming in order of completionOrder of completion ≠ order of input; batch processing is approximately 1.7 times faster✅ PASS

Confirmed that all core mechanisms of the client → poc-vision-bench → vLLM pipeline (text pass-through, concurrency gate, completion-order batch streaming, and integrated audio-from-video) are functioning normally.

4. Reference Documents​

Model — Qwen3-Omni

Hardware — AWS g7e

Serving — vLLM