09_Best Practices for Optimizing Basic Inference in vLLMs

Introduction

---

GPU

 vllm serve /mnt/nvme/models/Qwen3.8-27B-FP8 \
      --served-model-name qwen \
      --port 8601 \
      --tensor-parallel-size 1 \
      --max-model-len 131072 \
      --max-num-seqs 24 \
      --gpu-memory-utilization 0.85 \
      --allowed-local-media-path /mnt/nvme/vod \
      --limit-mm-per-prompt '{"image":30}' \
      --enable-prefix-caching \
      --reasoning-parser qwen3 \
      --trust-remote-code

  vllm serve /mnt/nvme/models/Qwen3-Embedding-4B \
      --served-model-name qwen-embed \
      --port 8500 \
      --tensor-parallel-size 1 \
      --max-model-len 2048 \
      --max-num-seqs 4 \
      --gpu-memory-utilization 0.10 \
      --trust-remote-code

Inference Optimization

(1) Maximize GPU Utilization

• -gpu-memory-utilization 0.85 --max-num-seqs 24

---

(2) Reducing the GPU’s Workload

• -enable-prefix-caching ----> Prompts are sent every time, but calculations aren’t performed every time

• Disable “thinking”

• Limit answer length

• Set temperature to 0

(3)

Wrap-up

---