From 9bd64d739bb33108c091dae538f7fe4678cab2ac Mon Sep 17 00:00:00 2001 From: Yuan Luo Date: Sun, 4 Jan 2026 10:05:23 +0800 Subject: [PATCH] [VLM] Add doc for ViT CUDA Graph (#16343) Co-authored-by: luoyuan.luo --- .../cuda_graph_for_multi_modal_encoder.md | 73 +++++++++++++++++++ docs/index.rst | 1 + 2 files changed, 74 insertions(+) create mode 100644 docs/advanced_features/cuda_graph_for_multi_modal_encoder.md diff --git a/docs/advanced_features/cuda_graph_for_multi_modal_encoder.md b/docs/advanced_features/cuda_graph_for_multi_modal_encoder.md new file mode 100644 index 000000000..184995ff1 --- /dev/null +++ b/docs/advanced_features/cuda_graph_for_multi_modal_encoder.md @@ -0,0 +1,73 @@ +# Cuda Graph for Multi-Modal Encoder in SGLang + +## Motivation + +In multimodal reasoning services, the visual encoder (ViT / Vision Transformer) typically has a few characteristic traits: + +Many layers, fragmented operators: Each layer includes LN, QKV projections, attention, MLP, residual connections, etc., resulting in extremely frequent kernel launches. + +Server-side “small batch / low latency” is common: The batch size is very small (sometimes it looks like 1 after “flattening” the batch), so kernel launch overhead accounts for a large portion of end-to-end latency. + +Input token count (number of patches) varies frequently: Different image/video resolutions and different batch composition lead to different sequence lengths +S — and this is precisely the biggest obstacle for CUDA Graph (unstable shapes). + +The value of CUDA Graph: It captures a long sequence of GPU kernels with fixed shapes and fixed memory addresses into a graph; later, for the same shapes, it can replay the graph directly, dramatically reducing launch overhead and making GPU scheduling more compact. + +This led us to seek a CUDA Graph enabled feature for ViT in order to improve ViT performance. + +## Design and Restrictions + +The new CUDA Graph enabled ViT logic is built on ViTCudaGraphRunner. This runner captures the "blocks + merger + deepstack merger (optional)" part of a vision transformer into a CUDA graph and replays it for identical shapes. See the following design consideration and restrictions for more details. + +### Dynamic inputs to fit static constraints of CUDA Graph + +Variable sequence length S is very common in ViT. While CUDA Graph requires fixed shapes. The solution is to build a graph cache by S(e.g., graph_key = S). The first time create a new S, and then capture a graph; afterwards, replay it. + +If there are many distinct S values, we need to increase VRAM usage which is graph-private memory pools for many graphs. + +### Stable addresses + +Everything "parameter-like" becomes a static buffer: + +- block_input / block_ws / block_output +- cu_full_len / cu_window_len and their kk variants +- sin_cos_ws + +In this way to solve the underlying requirement: during replay, not allowed to swap tensors, can only modify tensor contents. + +### Attention backend arguments +Attention backend arguments are fixed inside the graph: + +TritonAttn expects [cu_seqlens, cu_seqlens_kk, max_len] +FA3 expects [cu_seqlens, max_len] + +max_len is frozen as an int constant. +cu_seqlens is cached into a dict during create_graph(), and its contents are not updated during subsequent replays. + +For the same graph_key = S, you not only require the input shape to match, but also require the segmentation pattern in cu_seqlens (and window seqlens) to be identical. Otherwise, attention will segment the sequence incorrectly. + +### Rotary buffer management +The feature reallocates a larger sin_cos_ws when seq_len increases. +The max_content_len is used to make sure the maximum size of the allocated rotary buffer. + + +## Command Example +You can enable CUDA Graph for ViT by setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1`, for example: +``` +SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \ +python3 -m sglang.launch_server \ + --model Qwen/Qwen3-VL-8B-Instruct +``` +Or you can run CUDA Graph for ViT together with Piecewise CUDA Graph feature by both setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` and setting `--enable-piecewise-cuda-graph`, for example: +``` +SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \ +python3 -m sglang.launch_server \ + --model Qwen/Qwen3-VL-8B-Instruct \ + --piecewise-cuda-graph-max-tokens 4096 \ + --enable-piecewise-cuda-graph \ + --piecewise-cuda-graph-compiler eager +``` + +## Known supported models +- Qwen2.5-VL (https://github.com/sgl-project/sglang/pull/14422) +- Qwen3-VL (https://github.com/sgl-project/sglang/pull/15320) diff --git a/docs/index.rst b/docs/index.rst index 832ce9172..920d15b4d 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -58,6 +58,7 @@ Its core features include: advanced_features/pd_multiplexing.md advanced_features/vlm_query.ipynb advanced_features/dp_for_multi_modal_encoder.md + advanced_features/cuda_graph_for_multi_modal_encoder.md advanced_features/sgl_model_gateway.md advanced_features/deterministic_inference.md advanced_features/observability.md