Lianmin Zheng
co-authored by
sglang-bot
sglang-bot
parent
2a5773440e
commit
7e626d12b7
@@ -1,5 +1,5 @@
|
||||
Hierarchical KV Caching (HiCache)
|
||||
======================
|
||||
=================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
@@ -7,7 +7,7 @@ You can query them by:
|
||||
curl http://localhost:30000/metrics
|
||||
```
|
||||
|
||||
See [Production Metrics](../references/production_metrics.md) for more details.
|
||||
See [Production Metrics](../references/production_metrics.md) and [Production Request Tracing](../references/production_request_trace.md) for more details.
|
||||
|
||||
## Logging
|
||||
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
|
||||
# PD Multiplexing
|
||||
|
||||
|
||||
## Server Arguments
|
||||
|
||||
| Argument | Type/Default | Description |
|
||||
|-----------------------------|-------------------------|----------------------------------------------------------|
|
||||
| `--enable-pdmux` | flag; default: disabled | Enable PD-Multiplexing (PD running on greenctx stream). |
|
||||
| `--pdmux-config-path <path>`| string path; none | Path to the PD-Multiplexing YAML config file. |
|
||||
|
||||
### YAML Configuration
|
||||
|
||||
Example configuration for an H200 (132 SMs)
|
||||
|
||||
```yaml
|
||||
# Number of SM groups to divide the GPU into.
|
||||
# Includes two default groups:
|
||||
# - Group 0: all SMs for prefill
|
||||
# - Last group: all SMs for decode
|
||||
# The number of manual divisions must be (sm_group_num - 2).
|
||||
sm_group_num: 8
|
||||
|
||||
# Optional manual divisions of SMs.
|
||||
# Each entry contains:
|
||||
# - prefill_sm: number of SMs allocated for prefill
|
||||
# - decode_sm: number of SMs allocated for decode
|
||||
# - decode_bs_threshold: minimum decode batch size to select this group
|
||||
#
|
||||
# The sum of `prefill_sm` and `decode_sm` must equal the total number of SMs.
|
||||
# If provided, the number of entries must equal (sm_group_num - 2).
|
||||
manual_divisions:
|
||||
- [112, 20, 1]
|
||||
- [104, 28, 5]
|
||||
- [96, 36, 10]
|
||||
- [80, 52, 15]
|
||||
- [64, 68, 20]
|
||||
- [56, 76, 25]
|
||||
|
||||
# Divisor for default stream index calculation.
|
||||
# Used when manual_divisions are not provided.
|
||||
# Formula:
|
||||
# stream_idx = max(
|
||||
# 1,
|
||||
# min(sm_group_num - 2,
|
||||
# decode_bs * (sm_group_num - 2) // decode_bs_divisor
|
||||
# )
|
||||
# )
|
||||
decode_bs_divisor: 36
|
||||
|
||||
# Maximum token budget for split_forward in the prefill stage.
|
||||
# Determines how many layers are executed per split_forward.
|
||||
# Formula:
|
||||
# forward_count = max(1, split_forward_token_budget // extend_num_tokens)
|
||||
split_forward_token_budget: 65536
|
||||
```
|
||||
@@ -1,4 +1,4 @@
|
||||
# DeepSeek Usage
|
||||
# DeepSeek V3/V3.1/R1 Usage
|
||||
|
||||
SGLang provides many optimizations specifically designed for the DeepSeek models, making it the inference engine recommended by the official [DeepSeek team](https://github.com/deepseek-ai/DeepSeek-V3/tree/main?tab=readme-ov-file#62-inference-with-sglang-recommended) from Day 0.
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
Popular Model Usage (DeepSeeek, GPT-OSS, Llama, Qwen, and more)
|
||||
===============================================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
deepseek_v3.md
|
||||
deepseek_v32.md
|
||||
gpt_oss.md
|
||||
llama4.md
|
||||
qwen3.md
|
||||
qwen3_vl.md
|
||||
@@ -1,4 +1,4 @@
|
||||
## Bench Serving Guide
|
||||
# Bench Serving Guide
|
||||
|
||||
This guide explains how to benchmark online serving throughput and latency using `python -m sglang.bench_serving`. It supports multiple inference backends via OpenAI-compatible and native endpoints, and produces both console metrics and optional JSONL outputs.
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
You can install SGLang using one of the methods below.
|
||||
|
||||
This page primarily applies to common NVIDIA GPU platforms.
|
||||
For other or newer platforms, please refer to the dedicated pages for [AMD GPUs](../platforms/amd_gpu.md), [Intel Xeon CPUs](../platforms/cpu_server.md), [TPU](../platforms/tpu.md), [NVIDIA DGX Spark](https://lmsys.org/blog/2025-10-13-nvidia-dgx-spark/), [NVIDIA Jetson](../platforms/nvidia_jetson.md), [Ascend NPUs](../platforms/ascend_npu.md).
|
||||
For other or newer platforms, please refer to the dedicated pages for [AMD GPUs](../platforms/amd_gpu.md), [Intel Xeon CPUs](../platforms/cpu_server.md), [TPU](../platforms/tpu.md), [NVIDIA DGX Spark](https://lmsys.org/blog/2025-11-03-gpt-oss-on-nvidia-dgx-spark/), [NVIDIA Jetson](../platforms/nvidia_jetson.md), [Ascend NPUs](../platforms/ascend_npu.md), and [Intel XPU](../platforms/xpu.md).
|
||||
|
||||
## Method 1: With pip or uv
|
||||
|
||||
@@ -35,7 +35,7 @@ pip install -e "python"
|
||||
|
||||
**Quick fixes to common problems**
|
||||
|
||||
- If you want to develop SGLang, it is recommended to use docker. Please refer to [setup docker container](../developer_guide/development_guide_using_docker.md#setup-docker-container). The docker image is `lmsysorg/sglang:dev`.
|
||||
- If you want to develop SGLang, you can try the dev docker image. Please refer to [setup docker container](../developer_guide/development_guide_using_docker.md#setup-docker-container). The docker image is `lmsysorg/sglang:dev`.
|
||||
|
||||
## Method 3: Using docker
|
||||
|
||||
|
||||
+3
-9
@@ -6,10 +6,10 @@ It is designed to deliver low-latency and high-throughput inference across a wid
|
||||
Its core features include:
|
||||
|
||||
- **Fast Backend Runtime**: Provides efficient serving with RadixAttention for prefix caching, a zero-overhead CPU scheduler, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, quantization (FP4/FP8/INT4/AWQ/GPTQ), and multi-LoRA batching.
|
||||
- **Extensive Model Support**: Supports a wide range of generative models (Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, Mistral, etc.), embedding models (e5-mistral, gte, mcdse), and reward models (Skywork), with easy extensibility for integrating new models. Compatible with most Hugging Face models and OpenAI APIs.
|
||||
- **Extensive Model Support**: Supports a wide range of generative models (Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, Mistral, etc.), embedding models (e5-mistral, gte, mcdse), reward models (Skywork), and diffusion models (WAN, Qwen-Image), with easy extensibility for integrating new models. Compatible with most Hugging Face models and OpenAI APIs.
|
||||
- **Extensive Hardware Support**: Runs on NVIDIA GPUs (GB200/B300/H100/A100/Spark), AMD GPUs (MI355/MI300), Intel Xeon CPUs, Google TPUs, Ascend NPUs, and more.
|
||||
- **Flexible Frontend Language**: Offers an intuitive interface for programming LLM applications, supporting chained generation calls, advanced prompting, control flow, multi-modal inputs, parallelism, and external interactions.
|
||||
- **Active Community**: SGLang is open-source and supported by a vibrant community with widespread industry adoption, powering over 300,000 GPUs worldwide.
|
||||
- **Active Community**: SGLang is open-source and supported by a vibrant community with widespread industry adoption, powering over 400,000 GPUs worldwide.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
@@ -26,12 +26,7 @@ Its core features include:
|
||||
basic_usage/offline_engine_api.ipynb
|
||||
basic_usage/native_api.ipynb
|
||||
basic_usage/sampling_params.md
|
||||
basic_usage/deepseek.md
|
||||
basic_usage/deepseek_v32.md
|
||||
basic_usage/gpt_oss.md
|
||||
basic_usage/llama4.md
|
||||
basic_usage/qwen3.md
|
||||
basic_usage/qwen3_vl.md
|
||||
basic_usage/popular_model_usage.rst
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
@@ -74,7 +69,6 @@ Its core features include:
|
||||
:caption: Hardware Platforms
|
||||
|
||||
platforms/amd_gpu.md
|
||||
platforms/blackwell_gpu.md
|
||||
platforms/cpu_server.md
|
||||
platforms/tpu.md
|
||||
platforms/nvidia_jetson.md
|
||||
|
||||
@@ -21,7 +21,7 @@ This error may result from kernel errors or out-of-memory issues:
|
||||
- If the server hangs during initialization or running, it can be memory issues (out of memory), network issues (nccl errors), or other bugs in sglang.
|
||||
- If it is out of memory, you might see that `avail mem` is very low during the initialization or right after initialization. In this case,
|
||||
you can try to decrease `--mem-fraction-static`, decrease `--cuda-graph-max-bs`, or decrease `--chunked-prefill-size`.
|
||||
- Other bugs, please raise a Github issue to us.
|
||||
- Other bugs, please file an issue on GitHub.
|
||||
|
||||
|
||||
## Frequently Asked Questions
|
||||
@@ -34,6 +34,6 @@ From our initial investigation, this indeterminism arises from two factors: dyna
|
||||
|
||||
To achieve more deterministic outputs in the current code, you can add `--disable-radix-cache` and send only one request at a time. The results will be mostly deterministic under this setting.
|
||||
|
||||
**Note**:
|
||||
Recently, we also introduced a deterministic mode, you can enable it with `--enable-deterministic-inference`. It might not work for all cases.
|
||||
**Update**:
|
||||
Recently, we also introduced a deterministic mode, you can enable it with `--enable-deterministic-inference`.
|
||||
Please find more details in this blog post: https://lmsys.org/blog/2025-09-22-sglang-deterministic/
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
# Learn more
|
||||
# Learn More and Join the Community
|
||||
|
||||
You can find more blogs, slides, and videos about SGLang at [https://github.com/sgl-project/sgl-learning-materials](https://github.com/sgl-project/sgl-learning-materials).
|
||||
|
||||
The latest SGLang features and updates are shared through the [LMSYS blog](https://lmsys.org/blog/).
|
||||
|
||||
The 2025 H2 roadmap can be found at this [issue](https://github.com/sgl-project/sglang/issues/7736).
|
||||
- The development roadmap: [2025 Q4](https://github.com/sgl-project/sglang/issues/12780)
|
||||
- The latest SGLang features and updates are shared through the [LMSYS blog](https://lmsys.org/blog/)
|
||||
- X (formerly Twitter): https://x.com/lmsysorg
|
||||
- LinkedIn: https://www.linkedin.com/company/sgl-project/
|
||||
- Join Slack: https://slack.sglang.ai/
|
||||
- More blogs, slides, and videos about SGLang at [https://github.com/sgl-project/sgl-learning-materials](https://github.com/sgl-project/sgl-learning-materials)
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
# Production Request Tracing
|
||||
|
||||
SGlang exports request trace data based on the OpenTelemetry Collector. You can enable tracing by adding the `--enable-trace` and configure the OpenTelemetry Collector endpoint using `--otlp-traces-endpoint` when launching the server.
|
||||
|
||||
You can find example screenshots of the visualization in https://github.com/sgl-project/sglang/issues/8965.
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
| Time | CVE ID | Credit to | Affected Versions | Severity | Impact | Description |
|
||||
|------------|--------------|------------------|---------------------------|------------|----------------------|-------------|
|
||||
| 2025-09-09 | CVE-2025-10164 | Simon Huang, pjf | ≥ 0.4.6 & ≤ 0.5.3 | Critical | Remote Code Execution | A security flaw exists in lmsys sglang versions ≥ 0.4.6 and ≤ 0.5.3. The vulnerability arises from the use of unsafe pickle deserialization of the `serialized_named_tensors` parameter in the `/update_weights_from_tensor` API endpoint, which could allow a remote attacker to execute arbitrary code on the server by sending a specially crafted payload. |
|
||||
Reference in New Issue
Block a user