chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 14 Packages Projects Releases Wiki Activity

[Doc] Default lora backend: csgmv (#16444)

Browse Source
This commit is contained in:
Huapeng Zhou
2026-01-05 12:45:49 +08:00
committed by GitHub
parent c63e9cb29e
commit 078270473a
2 changed files with 4 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/lora.ipynb
+3 -4
View File
@@ -534,11 +534,10 @@
"\n",
"SGLang supports two LoRA backends that you can choose from using the `--lora-backend` argument:\n",
"\n",
"- `triton`: Default basic Triton-based backend.\n",
"- `csgmv`: Chunked SGMV backend optimized for high concurrency scenarios.\n",
"- `triton`: Basic Triton-based backend.\n",
"- `csgmv`: Default chunked SGMV backend optimized for high concurrency scenarios.\n",
"\n",
"The `csgmv` backend was recently introduced to improve performance especially at high-concurrency scenarios. Our benchmark shows that it achieves 20% to 80% latency improvements over the basic triton backend.\n",
"Currently it is at preview phase, we expect to make it our the default LoRA backend in future release. Before that, you can adopt it by manually setting the `--lora-backend` server config."
"The `csgmv` backend was recently introduced to improve performance especially at high-concurrency scenarios. Our benchmark shows that it achieves 20% to 80% latency improvements over the basic triton backend."
]
},
{
docs/advanced_features/server_arguments.md
+1 -1
View File
@@ -231,7 +231,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s
| `--max-loras-per-batch` | Maximum number of adapters for a running batch, including base-only requests. | `8` | Type: int |
| `--max-loaded-loras` | If specified, limits the maximum number of LoRA adapters loaded in CPU memory at a time. Must be ≥ `--max-loras-per-batch`. | `None` | Type: int |
| `--lora-eviction-policy` | LoRA adapter eviction policy when the GPU memory pool is full. | `lru` | `lru`, `fifo` |
| `--lora-backend` | Choose the kernel backend for multi-LoRA serving. | `triton` | `triton`, `csgmv` |
| `--lora-backend` | Choose the kernel backend for multi-LoRA serving. | `csgmv` | `triton`, `csgmv` |
| `--max-lora-chunk-size` | Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when `--lora-backend` is `csgmv`. Larger values may improve performance. | `16` | `16`, `32`, `64`, `128` |
## Kernel Backends (Attention, Sampling, Grammar, GEMM)