[diffusion] docs: consolidate diffusion documentation into docs (#18095)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: JiaxinD <djx2048@gmail.com>
This commit is contained in:
qianyue76
@@ -16,11 +16,11 @@ SGLang Diffusion has the following features:
|
||||
|
||||
### AMD/ROCm Support
|
||||
|
||||
SGLang Diffusion supports AMD Instinct GPUs through ROCm. On AMD platforms, we use the Triton attention backend and leverage AITER kernels for optimized layernorm and other operations. See the [ROCm installation guide](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/docs/install_rocm.md) for setup instructions.
|
||||
SGLang Diffusion supports AMD Instinct GPUs through ROCm. On AMD platforms, we use the Triton attention backend and leverage AITER kernels for optimized layernorm and other operations. See the [installation guide](https://github.com/sgl-project/sglang/tree/main/docs/diffusion/installation.md) for setup instructions.
|
||||
|
||||
### Moore Threads/MUSA Support
|
||||
|
||||
SGLang Diffusion supports Moore Threads GPUs (MTGPU) through the MUSA software stack. On MUSA platforms, we use the Torch SDPA backend for attention. See the [MUSA installation guide](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/docs/install_musa.md) for setup instructions.
|
||||
SGLang Diffusion supports Moore Threads GPUs (MTGPU) through the MUSA software stack. On MUSA platforms, we use the Torch SDPA backend for attention. See the [installation guide](https://github.com/sgl-project/sglang/tree/main/docs/diffusion/installation.md) for setup instructions.
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -28,9 +28,7 @@ SGLang Diffusion supports Moore Threads GPUs (MTGPU) through the MUSA software s
|
||||
uv pip install 'sglang[diffusion]' --prerelease=allow
|
||||
```
|
||||
|
||||
For more installation methods (e.g. pypi, uv, docker), check [install.md](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/docs/install.md).
|
||||
* ROCm/AMD users should follow the [ROCm quickstart](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/docs/install_rocm.md) that includes the additional kernel builds and attention backend settings we validated on MI300X.
|
||||
* MUSA/Moore Threads users should follow the [MUSA quickstart](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/docs/install_musa.md) that includes the attention backend settings we validated on MTT S5000.
|
||||
For more installation methods (e.g. pypi, uv, docker, ROCm/AMD, MUSA/Moore Threads), check [install.md](https://github.com/sgl-project/sglang/tree/main/docs/diffusion/installation.md).
|
||||
|
||||
## Inference
|
||||
|
||||
@@ -82,11 +80,11 @@ sglang generate \
|
||||
--save-output
|
||||
```
|
||||
|
||||
For more usage examples (e.g. OpenAI compatible API, server mode), check [cli.md](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/docs/cli.md).
|
||||
For more usage examples (e.g. OpenAI compatible API, server mode), check [cli.md](https://github.com/sgl-project/sglang/tree/main/docs/diffusion/cli.md).
|
||||
|
||||
## Contributing
|
||||
|
||||
All contributions are welcome. The contribution guide is available [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/docs/contributing.md).
|
||||
All contributions are welcome. The contribution guide is available [here](https://github.com/sgl-project/sglang/tree/main/docs/diffusion/contributing.md).
|
||||
|
||||
## Acknowledgement
|
||||
|
||||
|
||||
@@ -1,128 +0,0 @@
|
||||
# Attention Backends
|
||||
|
||||
This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them.
|
||||
|
||||
## Overview
|
||||
|
||||
Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`.
|
||||
|
||||
Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`) and therefore applies to any model component using these layers (e.g. diffusion transformer / DiT and encoders).
|
||||
|
||||
When using the diffusers backend, `--attention-backend` is passed through to diffusers'
|
||||
`set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`).
|
||||
|
||||
- **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA.
|
||||
- **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA.
|
||||
- **MPS**: always uses PyTorch SDPA.
|
||||
|
||||
## Backend options
|
||||
|
||||
For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
|
||||
|
||||
| CLI value | Enum value | Notes |
|
||||
|---|---|---|
|
||||
| `fa` / `fa3` / `fa4` | `FA` | FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.__post_init__`). |
|
||||
| `torch_sdpa` | `TORCH_SDPA` | PyTorch `scaled_dot_product_attention`. |
|
||||
| `sliding_tile_attn` | `SLIDING_TILE_ATTN` | Sliding Tile Attention (STA). Requires `st_attn`. Configure via `--attention-backend-config`. |
|
||||
| `sage_attn` | `SAGE_ATTN` | Requires `sageattention`. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120 (compute capability 8.0/8.6/8.9/9.0/12.0); see upstream `setup.py`: https://github.com/thu-ml/SageAttention/blob/main/setup.py. |
|
||||
| `sage_attn_3` | `SAGE_ATTN_3` | Requires SageAttention3 installed per upstream instructions. |
|
||||
| `video_sparse_attn` | `VIDEO_SPARSE_ATTN` | Requires `vsa`. Configure `sparsity` via `--attention-backend-config`. |
|
||||
| `vmoba_attn` | `VMOBA_ATTN` | Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. |
|
||||
| `aiter` | `AITER` | Requires `aiter`. |
|
||||
|
||||
## Selection priority
|
||||
|
||||
The selection order in `runtime/layers/attention/selector.py` is:
|
||||
|
||||
1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)`
|
||||
2. CLI `--attention-backend` (`ServerArgs.attention_backend`)
|
||||
3. Auto selection (platform capability, dtype, and installed packages)
|
||||
|
||||
## Configuration
|
||||
|
||||
Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts:
|
||||
- A path to a JSON or YAML configuration file.
|
||||
- A JSON string (e.g., `'{"sparsity": 0.5}'`).
|
||||
- Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`).
|
||||
|
||||
### Supported Configuration Parameters
|
||||
|
||||
#### Sliding Tile Attention (`sliding_tile_attn`)
|
||||
|
||||
| Parameter | Type | Description | Default |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `mask_strategy_file_path` | `str` | **Required.** Path to the mask strategy JSON file. | - |
|
||||
| `sta_mode` | `str` | Mode of STA. | `STA_inference` |
|
||||
| `skip_time_steps` | `int` | Number of steps to use full attention before switching to sparse attention. | `15` |
|
||||
|
||||
#### Video Sparse Attention (`video_sparse_attn`)
|
||||
|
||||
| Parameter | Type | Description | Default |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `sparsity` | `float` | Validation sparsity (0.0 - 1.0). | `0.0` |
|
||||
|
||||
#### V-MoBA (`vmoba_attn`)
|
||||
|
||||
| Parameter | Type | Description | Default |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `temporal_chunk_size` | `int` | Chunk size for temporal dimension. | - |
|
||||
| `temporal_topk` | `int` | Top-K tokens to select in temporal dimension. | - |
|
||||
| `spatial_chunk_size` | `list[int]` | Chunk size for spatial dimension (H, W). | - |
|
||||
| `spatial_topk` | `int` | Top-K tokens to select in spatial dimension. | - |
|
||||
| `st_chunk_size` | `list[int]` | Chunk size for spatiotemporal dimension (T, H, W). | - |
|
||||
| `st_topk` | `int` | Top-K tokens to select in spatiotemporal dimension. | - |
|
||||
| `moba_select_mode` | `str` | Selection mode (e.g., `threshold`). | `threshold` |
|
||||
| `moba_threshold` | `float` | Threshold value for selection. | `0.25` |
|
||||
| `moba_threshold_type` | `str` | Type of thresholding (e.g., `query_head`). | `query_head` |
|
||||
| `first_full_step` | `int` | Number of initial steps to use full attention. | `12` |
|
||||
| `first_full_layer` | `int` | Number of initial layers to use full attention. | `0` |
|
||||
| `temporal_layer` | `int` | Number of temporal layers. | `1` |
|
||||
| `spatial_layer` | `int` | Number of spatial layers. | `1` |
|
||||
| `st_layer` | `int` | Number of spatiotemporal layers. | `1` |
|
||||
|
||||
## Platform support matrix
|
||||
|
||||
| Backend | CUDA | ROCm | MPS | Notes |
|
||||
|---|---:|---:|---:|---|
|
||||
| `fa` | ✅ | ✅ | ❌ | CUDA requires SM80+ and fp16/bf16. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. |
|
||||
| `torch_sdpa` | ✅ | ✅ | ✅ | Most compatible option across platforms. |
|
||||
| `sliding_tile_attn` | ✅ | ❌ | ❌ | CUDA-only. Requires `st_attn`. Configure via `--attention-backend-config`. |
|
||||
| `sage_attn` | ✅ | ❌ | ❌ | CUDA-only (optional dependency). |
|
||||
| `sage_attn_3` | ✅ | ❌ | ❌ | CUDA-only (optional dependency). |
|
||||
| `video_sparse_attn` | ✅ | ❌ | ❌ | CUDA-only. Requires `vsa`. Configure `sparsity` via `--attention-backend-config`. |
|
||||
| `vmoba_attn` | ✅ | ❌ | ❌ | CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. |
|
||||
| `aiter` | ✅ | ❌ | ❌ | Requires `aiter`. |
|
||||
|
||||
## Usage
|
||||
|
||||
### Select a backend via CLI
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path <MODEL_PATH_OR_ID> \
|
||||
--prompt "..." \
|
||||
--attention-backend fa
|
||||
```
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path <MODEL_PATH_OR_ID> \
|
||||
--prompt "..." \
|
||||
--attention-backend torch_sdpa
|
||||
```
|
||||
|
||||
### Using Sliding Tile Attention (STA)
|
||||
|
||||
```bash
|
||||
# Pass the mask strategy file path via config
|
||||
sglang generate \
|
||||
--model-path <MODEL_PATH_OR_ID> \
|
||||
--prompt "..." \
|
||||
--attention-backend sliding_tile_attn \
|
||||
--attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json"
|
||||
```
|
||||
|
||||
### Notes for ROCm / MPS
|
||||
|
||||
- ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment.
|
||||
- MPS: the platform implementation always uses `torch_sdpa`.
|
||||
243
python/sglang/multimodal_gen/docs/cache/cache_dit.md
vendored
243
python/sglang/multimodal_gen/docs/cache/cache_dit.md
vendored
@@ -1,243 +0,0 @@
|
||||
# Cache-DiT Acceleration
|
||||
|
||||
> **Note**: This is one of two caching strategies available in SGLang.
|
||||
> For an overview of all caching options, see [caching.md](caching.md).
|
||||
> For TeaCache documentation, see [teacache.md](teacache.md).
|
||||
|
||||
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to **1.69x inference speedup** with minimal quality loss.
|
||||
|
||||
## Overview
|
||||
|
||||
**Cache-DiT** uses intelligent caching strategies to skip redundant computation in the denoising loop:
|
||||
|
||||
- **DBCache (Dual Block Cache)**: Dynamically decides when to cache transformer blocks based on residual differences
|
||||
- **TaylorSeer**: Uses Taylor expansion for calibration to optimize caching decisions
|
||||
- **SCM (Step Computation Masking)**: Step-level caching control for additional speedup
|
||||
|
||||
## Basic Usage
|
||||
|
||||
Enable Cache-DiT by exporting the environment variable and using `sglang generate` or `sglang serve` :
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A beautiful sunset over the mountains"
|
||||
```
|
||||
|
||||
## Diffusers Backend Configuration
|
||||
|
||||
Cache-DiT supports loading acceleration configs from a custom YAML file. For
|
||||
diffusers pipelines, pass the YAML/JSON path via `--cache-dit-config`. This
|
||||
flow requires cache-dit >= 1.2.0 (`cache_dit.load_configs`).
|
||||
|
||||
### Single GPU inference
|
||||
|
||||
Define a `config.yaml` file that contains:
|
||||
|
||||
```yaml
|
||||
cache_config:
|
||||
max_warmup_steps: 8
|
||||
warmup_interval: 2
|
||||
max_cached_steps: -1
|
||||
max_continuous_cached_steps: 2
|
||||
Fn_compute_blocks: 1
|
||||
Bn_compute_blocks: 0
|
||||
residual_diff_threshold: 0.12
|
||||
enable_taylorseer: true
|
||||
taylorseer_order: 1
|
||||
```
|
||||
|
||||
Then apply the config with:
|
||||
|
||||
```bash
|
||||
sglang generate --backend diffusers \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--cache-dit-config config.yaml \
|
||||
--prompt "A beautiful sunset over the mountains"
|
||||
```
|
||||
|
||||
### Distributed inference
|
||||
|
||||
Define a `parallel_config.yaml` file that contains:
|
||||
|
||||
```yaml
|
||||
cache_config:
|
||||
max_warmup_steps: 8
|
||||
warmup_interval: 2
|
||||
max_cached_steps: -1
|
||||
max_continuous_cached_steps: 2
|
||||
Fn_compute_blocks: 1
|
||||
Bn_compute_blocks: 0
|
||||
residual_diff_threshold: 0.12
|
||||
enable_taylorseer: true
|
||||
taylorseer_order: 1
|
||||
parallelism_config:
|
||||
ulysses_size: auto
|
||||
parallel_kwargs:
|
||||
attention_backend: native
|
||||
extra_parallel_modules: ["text_encoder", "vae"]
|
||||
```
|
||||
|
||||
`ulysses_size: auto` means cache-dit will auto-detect the world_size. Otherwise,
|
||||
set it to a specific integer (e.g., `4`).
|
||||
|
||||
Then apply the distributed config with:
|
||||
|
||||
```bash
|
||||
sglang generate --backend diffusers \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--cache-dit-config parallel_config.yaml \
|
||||
--prompt "A futuristic cityscape at sunset"
|
||||
```
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### DBCache Parameters
|
||||
|
||||
DBCache controls block-level caching behavior:
|
||||
|
||||
| Parameter | Env Variable | Default | Description |
|
||||
|-----------|---------------------------|---------|------------------------------------------|
|
||||
| Fn | `SGLANG_CACHE_DIT_FN` | 1 | Number of first blocks to always compute |
|
||||
| Bn | `SGLANG_CACHE_DIT_BN` | 0 | Number of last blocks to always compute |
|
||||
| W | `SGLANG_CACHE_DIT_WARMUP` | 4 | Warmup steps before caching starts |
|
||||
| R | `SGLANG_CACHE_DIT_RDT` | 0.24 | Residual difference threshold |
|
||||
| MC | `SGLANG_CACHE_DIT_MC` | 3 | Maximum continuous cached steps |
|
||||
|
||||
### TaylorSeer Configuration
|
||||
|
||||
TaylorSeer improves caching accuracy using Taylor expansion:
|
||||
|
||||
| Parameter | Env Variable | Default | Description |
|
||||
|-----------|-------------------------------|---------|---------------------------------|
|
||||
| Enable | `SGLANG_CACHE_DIT_TAYLORSEER` | false | Enable TaylorSeer calibrator |
|
||||
| Order | `SGLANG_CACHE_DIT_TS_ORDER` | 1 | Taylor expansion order (1 or 2) |
|
||||
|
||||
### Combined Configuration Example
|
||||
|
||||
DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters
|
||||
simultaneously:
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang generate --model-path black-forest-labs/FLUX.1-dev \
|
||||
--prompt "A curious raccoon in a forest"
|
||||
```
|
||||
|
||||
### SCM (Step Computation Masking)
|
||||
|
||||
SCM provides step-level caching control for additional speedup. It decides which denoising steps to compute fully and
|
||||
which to use cached results.
|
||||
|
||||
#### SCM Presets
|
||||
|
||||
SCM is configured with presets:
|
||||
|
||||
| Preset | Compute Ratio | Speed | Quality |
|
||||
|----------|---------------|----------|------------|
|
||||
| `none` | 100% | Baseline | Best |
|
||||
| `slow` | ~75% | ~1.3x | High |
|
||||
| `medium` | ~50% | ~2x | Good |
|
||||
| `fast` | ~35% | ~3x | Acceptable |
|
||||
| `ultra` | ~25% | ~4x | Lower |
|
||||
|
||||
##### Usage
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_SCM_PRESET=medium \
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A futuristic cityscape at sunset"
|
||||
```
|
||||
|
||||
#### Custom SCM Bins
|
||||
|
||||
For fine-grained control over which steps to compute vs cache:
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \
|
||||
SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A futuristic cityscape at sunset"
|
||||
```
|
||||
|
||||
#### SCM Policy
|
||||
|
||||
| Policy | Env Variable | Description |
|
||||
|-----------|---------------------------------------|---------------------------------------------|
|
||||
| `dynamic` | `SGLANG_CACHE_DIT_SCM_POLICY=dynamic` | Adaptive caching based on content (default) |
|
||||
| `static` | `SGLANG_CACHE_DIT_SCM_POLICY=static` | Fixed caching pattern |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
All Cache-DiT parameters can be set via the following environment variables:
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
|-------------------------------------|---------|------------------------------------------|
|
||||
| `SGLANG_CACHE_DIT_ENABLED` | false | Enable Cache-DiT acceleration |
|
||||
| `SGLANG_CACHE_DIT_FN` | 1 | First N blocks to always compute |
|
||||
| `SGLANG_CACHE_DIT_BN` | 0 | Last N blocks to always compute |
|
||||
| `SGLANG_CACHE_DIT_WARMUP` | 4 | Warmup steps before caching |
|
||||
| `SGLANG_CACHE_DIT_RDT` | 0.24 | Residual difference threshold |
|
||||
| `SGLANG_CACHE_DIT_MC` | 3 | Max continuous cached steps |
|
||||
| `SGLANG_CACHE_DIT_TAYLORSEER` | false | Enable TaylorSeer calibrator |
|
||||
| `SGLANG_CACHE_DIT_TS_ORDER` | 1 | TaylorSeer order (1 or 2) |
|
||||
| `SGLANG_CACHE_DIT_SCM_PRESET` | none | SCM preset (none/slow/medium/fast/ultra) |
|
||||
| `SGLANG_CACHE_DIT_SCM_POLICY` | dynamic | SCM caching policy |
|
||||
| `SGLANG_CACHE_DIT_SCM_COMPUTE_BINS` | not set | Custom SCM compute bins |
|
||||
| `SGLANG_CACHE_DIT_SCM_CACHE_BINS` | not set | Custom SCM cache bins |
|
||||
|
||||
## Supported Models
|
||||
|
||||
SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:
|
||||
|
||||
| Model Family | Example Models |
|
||||
|--------------|-----------------------------|
|
||||
| Wan | Wan2.1, Wan2.2 |
|
||||
| Flux | FLUX.1-dev, FLUX.2-dev |
|
||||
| Z-Image | Z-Image-Turbo |
|
||||
| Qwen | Qwen-Image, Qwen-Image-Edit |
|
||||
| Hunyuan | HunyuanVideo |
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Start with defaults**: The default parameters work well for most models
|
||||
2. **Use TaylorSeer**: It typically improves both speed and quality
|
||||
3. **Tune R threshold**: Lower values = better quality, higher values = faster
|
||||
4. **SCM for extra speed**: Use `medium` preset for good speed/quality balance
|
||||
5. **Warmup matters**: Higher warmup = more stable caching decisions
|
||||
|
||||
## Limitations
|
||||
|
||||
- **SGLang-native pipelines**: Distributed support (TP/SP) is not yet validated; Cache-DiT will be automatically
|
||||
disabled when `world_size > 1`.
|
||||
- **SCM minimum steps**: SCM requires >= 8 inference steps to be effective
|
||||
- **Model support**: Only models registered in Cache-DiT's BlockAdapterRegister are supported
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Distributed environment warning
|
||||
|
||||
```
|
||||
WARNING: cache-dit is disabled in distributed environment (world_size=N)
|
||||
```
|
||||
|
||||
This is expected behavior. Cache-DiT currently only supports single-GPU inference.
|
||||
|
||||
### SCM disabled for low step count
|
||||
|
||||
For models with < 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache
|
||||
acceleration still works.
|
||||
|
||||
## References
|
||||
|
||||
- [Cache-Dit](https://github.com/vipshop/cache-dit)
|
||||
- [SGLang Diffusion](../README.md)
|
||||
@@ -1,60 +0,0 @@
|
||||
# Caching Acceleration for Diffusion Models
|
||||
|
||||
SGLang provides multiple caching acceleration strategies for Diffusion Transformer (DiT) models. These strategies can significantly reduce inference time by skipping redundant computation.
|
||||
|
||||
## Overview
|
||||
|
||||
SGLang supports two complementary caching approaches:
|
||||
|
||||
| Strategy | Scope | Mechanism | Best For |
|
||||
|----------|-------|-----------|----------|
|
||||
| **Cache-DiT** | Block-level | Skip individual transformer blocks dynamically | Advanced, higher speedup |
|
||||
| **TeaCache** | Timestep-level | Skip entire denoising steps based on L1 similarity | Simple, built-in |
|
||||
|
||||
|
||||
|
||||
## Cache-DiT
|
||||
|
||||
[Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with
|
||||
advanced strategies like DBCache and TaylorSeer. It can achieve up to **1.69x speedup**.
|
||||
|
||||
See [cache_dit.md](cache_dit.md) for detailed configuration.
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A beautiful sunset over the mountains"
|
||||
```
|
||||
|
||||
### Key Features
|
||||
|
||||
- **DBCache**: Dynamic block-level caching based on residual differences
|
||||
- **TaylorSeer**: Taylor expansion-based calibration for optimized caching
|
||||
- **SCM**: Step-level computation masking for additional speedup
|
||||
|
||||
## TeaCache
|
||||
|
||||
TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
|
||||
|
||||
See [teacache.md](teacache.md) for detailed documentation.
|
||||
|
||||
### Quick Overview
|
||||
|
||||
- Tracks L1 distance between modulated inputs across timesteps
|
||||
- When accumulated distance is below threshold, reuses cached residual
|
||||
- Supports CFG with separate positive/negative caches
|
||||
|
||||
### Supported Models
|
||||
|
||||
- Wan (wan2.1, wan2.2)
|
||||
- Hunyuan (HunyuanVideo)
|
||||
- Z-Image
|
||||
|
||||
For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled.
|
||||
|
||||
## References
|
||||
|
||||
- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
|
||||
- [TeaCache Paper](https://arxiv.org/abs/2411.14324)
|
||||
@@ -1,84 +0,0 @@
|
||||
# TeaCache Acceleration
|
||||
|
||||
> **Note**: This is one of two caching strategies available in SGLang.
|
||||
> For an overview of all caching options, see [caching.md](caching.md).
|
||||
|
||||
TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
|
||||
|
||||
## Overview
|
||||
|
||||
TeaCache works by:
|
||||
1. Tracking the L1 distance between modulated inputs across consecutive timesteps
|
||||
2. Accumulating the rescaled L1 distance over steps
|
||||
3. When accumulated distance is below a threshold, reusing the cached residual
|
||||
4. Supporting CFG (Classifier-Free Guidance) with separate positive/negative caches
|
||||
|
||||
## How It Works
|
||||
|
||||
### L1 Distance Tracking
|
||||
|
||||
At each denoising step, TeaCache computes the relative L1 distance between the current and previous modulated inputs:
|
||||
|
||||
```
|
||||
rel_l1 = |current - previous|.mean() / |previous|.mean()
|
||||
```
|
||||
|
||||
This distance is then rescaled using polynomial coefficients and accumulated:
|
||||
|
||||
```
|
||||
accumulated += poly(coefficients)(rel_l1)
|
||||
```
|
||||
|
||||
### Cache Decision
|
||||
|
||||
- If `accumulated >= threshold`: Force computation, reset accumulator
|
||||
- If `accumulated < threshold`: Skip computation, use cached residual
|
||||
|
||||
### CFG Support
|
||||
|
||||
For models that support CFG cache separation (Wan, Hunyuan, Z-Image), TeaCache maintains separate caches for positive and negative branches:
|
||||
- `previous_modulated_input` / `previous_residual` for positive branch
|
||||
- `previous_modulated_input_negative` / `previous_residual_negative` for negative branch
|
||||
|
||||
For models that don't support CFG separation (Flux, Qwen), TeaCache is automatically disabled when CFG is enabled.
|
||||
|
||||
## Configuration
|
||||
|
||||
TeaCache is configured via `TeaCacheParams` in the sampling parameters:
|
||||
|
||||
```python
|
||||
from sglang.multimodal_gen.configs.sample.teacache import TeaCacheParams
|
||||
|
||||
params = TeaCacheParams(
|
||||
teacache_thresh=0.1, # Threshold for accumulated L1 distance
|
||||
coefficients=[1.0, 0.0, 0.0], # Polynomial coefficients for L1 rescaling
|
||||
)
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|-----------|------|-------------|
|
||||
| `teacache_thresh` | float | Threshold for accumulated L1 distance. Lower = more caching, faster but potentially lower quality |
|
||||
| `coefficients` | list[float] | Polynomial coefficients for L1 rescaling. Model-specific tuning |
|
||||
|
||||
### Model-Specific Configurations
|
||||
|
||||
Different models may have different optimal configurations. The coefficients are typically tuned per-model to balance speed and quality.
|
||||
|
||||
## Supported Models
|
||||
|
||||
TeaCache is built into the following model families:
|
||||
|
||||
| Model Family | CFG Cache Separation | Notes |
|
||||
|--------------|---------------------|-------|
|
||||
| Wan (wan2.1, wan2.2) | Yes | Full support |
|
||||
| Hunyuan (HunyuanVideo) | Yes | To be supported |
|
||||
| Z-Image | Yes | To be supported |
|
||||
| Flux | No | To be supported |
|
||||
| Qwen | No | To be supported |
|
||||
|
||||
|
||||
## References
|
||||
|
||||
- [TeaCache: Accelerating Diffusion Models with Temporal Similarity](https://arxiv.org/abs/2411.14324)
|
||||
@@ -1,30 +0,0 @@
|
||||
|
||||
## Perf baseline generation script
|
||||
|
||||
`python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py` starts a local diffusion server, issues requests for selected test cases, aggregates stage/denoise-step/E2E timings from the perf log, and writes the results back to the `scenarios` section of `perf_baselines.json`.
|
||||
|
||||
### Usage
|
||||
|
||||
Update a single case:
|
||||
|
||||
```bash
|
||||
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --case qwen_image_t2i
|
||||
```
|
||||
|
||||
Select by regex:
|
||||
|
||||
```bash
|
||||
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --match 'qwen_image_.*'
|
||||
```
|
||||
|
||||
Run all keys from the baseline file `scenarios`:
|
||||
|
||||
```bash
|
||||
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --all-from-baseline
|
||||
```
|
||||
|
||||
Specify input/output paths and timeout:
|
||||
|
||||
```bash
|
||||
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --baseline python/sglang/multimodal_gen/test/server/perf_baselines.json --out /tmp/perf_baselines.json --timeout 600
|
||||
```
|
||||
@@ -1,274 +0,0 @@
|
||||
# SGLang diffusion CLI Inference
|
||||
|
||||
The SGLang-diffusion CLI provides a quick way to access the inference pipeline for image and video generation.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A working SGLang diffusion installation and the `sglang` CLI available in `$PATH`.
|
||||
- Python 3.11+ if you plan to use the OpenAI Python SDK.
|
||||
|
||||
|
||||
## Supported Arguments
|
||||
|
||||
### Server Arguments
|
||||
|
||||
- `--model-path {MODEL_PATH}`: Path to the model or model ID
|
||||
- `--vae-path {VAE_PATH}`: Path to a custom VAE model or HuggingFace model ID (e.g., `fal/FLUX.2-Tiny-AutoEncoder`). If not specified, the VAE will be loaded from the main model path.
|
||||
- `--lora-path {LORA_PATH}`: Path to a LoRA adapter (local path or HuggingFace model ID). If not specified, LoRA will not be applied.
|
||||
- `--lora-nickname {NAME}`: Nickname for the LoRA adapter. (default: `default`).
|
||||
- `--num-gpus {NUM_GPUS}`: Number of GPUs to use
|
||||
- `--tp-size {TP_SIZE}`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
|
||||
- `--sp-degree {SP_SIZE}`: Sequence parallelism size (typically should match the number of GPUs)
|
||||
- `--ulysses-degree {ULYSSES_DEGREE}`: The degree of DeepSpeed-Ulysses-style SP in USP
|
||||
- `--ring-degree {RING_DEGREE}`: The degree of ring attention-style SP in USP
|
||||
- `--attention-backend {BACKEND}`: Attention backend to use. For SGLang-native pipelines use `fa`, `torch_sdpa`, `sage_attn`, etc. For diffusers pipelines use diffusers backend names like `flash`, `_flash_3_hub`, `sage`, `xformers`.
|
||||
- `--attention-backend-config {CONFIG}`: Configuration for the attention backend. Can be a JSON string (e.g., '{"k": "v"}'), a path to a JSON/YAML file, or key=value pairs (e.g., "k=v,k2=v2").
|
||||
- `--cache-dit-config {PATH}`: Path to a Cache-DiT YAML/JSON config (diffusers backend only)
|
||||
- `--dit-precision {DTYPE}`: Precision for the DiT model (currently supports fp32, fp16, and bf16).
|
||||
|
||||
|
||||
### Sampling Parameters
|
||||
|
||||
- `--prompt {PROMPT}`: Text description for the video you want to generate
|
||||
- `--num-inference-steps {STEPS}`: Number of denoising steps
|
||||
- `--negative-prompt {PROMPT}`: Negative prompt to guide generation away from certain concepts
|
||||
- `--seed {SEED}`: Random seed for reproducible generation
|
||||
|
||||
|
||||
#### Image/Video Configuration
|
||||
|
||||
- `--height {HEIGHT}`: Height of the generated output
|
||||
- `--width {WIDTH}`: Width of the generated output
|
||||
- `--num-frames {NUM_FRAMES}`: Number of frames to generate
|
||||
- `--fps {FPS}`: Frames per second for the saved output, if this is a video-generation task
|
||||
|
||||
|
||||
#### Output Options
|
||||
|
||||
- `--output-path {PATH}`: Directory to save the generated video
|
||||
- `--save-output`: Whether to save the image/video to disk
|
||||
- `--return-frames`: Whether to return the raw frames
|
||||
|
||||
### Using Configuration Files
|
||||
|
||||
Instead of specifying all parameters on the command line, you can use a configuration file:
|
||||
|
||||
```bash
|
||||
sglang generate --config {CONFIG_FILE_PATH}
|
||||
```
|
||||
|
||||
The configuration file should be in JSON or YAML format with the same parameter names as the CLI options. Command-line arguments take precedence over settings in the configuration file, allowing you to override specific values while keeping the rest from the configuration file.
|
||||
|
||||
Example configuration file (config.json):
|
||||
|
||||
```json
|
||||
{
|
||||
"model_path": "FastVideo/FastHunyuan-diffusers",
|
||||
"prompt": "A beautiful woman in a red dress walking down a street",
|
||||
"output_path": "outputs/",
|
||||
"num_gpus": 2,
|
||||
"sp_size": 2,
|
||||
"tp_size": 1,
|
||||
"num_frames": 45,
|
||||
"height": 720,
|
||||
"width": 1280,
|
||||
"num_inference_steps": 6,
|
||||
"seed": 1024,
|
||||
"fps": 24,
|
||||
"precision": "bf16",
|
||||
"vae_precision": "fp16",
|
||||
"vae_tiling": true,
|
||||
"vae_sp": true,
|
||||
"vae_config": {
|
||||
"load_encoder": false,
|
||||
"load_decoder": true,
|
||||
"tile_sample_min_height": 256,
|
||||
"tile_sample_min_width": 256
|
||||
},
|
||||
"text_encoder_precisions": [
|
||||
"fp16",
|
||||
"fp16"
|
||||
],
|
||||
"mask_strategy_file_path": null,
|
||||
"enable_torch_compile": false
|
||||
}
|
||||
```
|
||||
|
||||
Or using YAML format (config.yaml):
|
||||
|
||||
```yaml
|
||||
model_path: "FastVideo/FastHunyuan-diffusers"
|
||||
prompt: "A beautiful woman in a red dress walking down a street"
|
||||
output_path: "outputs/"
|
||||
num_gpus: 2
|
||||
sp_size: 2
|
||||
tp_size: 1
|
||||
num_frames: 45
|
||||
height: 720
|
||||
width: 1280
|
||||
num_inference_steps: 6
|
||||
seed: 1024
|
||||
fps: 24
|
||||
precision: "bf16"
|
||||
vae_precision: "fp16"
|
||||
vae_tiling: true
|
||||
vae_sp: true
|
||||
vae_config:
|
||||
load_encoder: false
|
||||
load_decoder: true
|
||||
tile_sample_min_height: 256
|
||||
tile_sample_min_width: 256
|
||||
text_encoder_precisions:
|
||||
- "fp16"
|
||||
- "fp16"
|
||||
mask_strategy_file_path: null
|
||||
enable_torch_compile: false
|
||||
```
|
||||
|
||||
|
||||
To see all the options, you can use the `--help` flag:
|
||||
|
||||
```bash
|
||||
sglang generate --help
|
||||
```
|
||||
|
||||
## Serve
|
||||
|
||||
Launch the SGLang diffusion HTTP server and interact with it using the OpenAI SDK and curl.
|
||||
|
||||
### Start the server
|
||||
|
||||
Use the following command to launch the server:
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
)
|
||||
|
||||
sglang serve "${SERVER_ARGS[@]}"
|
||||
```
|
||||
|
||||
- **--model-path**: Which model to load. The example uses `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`.
|
||||
- **--port**: HTTP port to listen on (the default here is `30010`).
|
||||
|
||||
For detailed API usage, including Image, Video Generation and LoRA management, please refer to the [OpenAI API Documentation](openai_api.md).
|
||||
|
||||
### Cloud Storage Support
|
||||
|
||||
SGLang diffusion supports automatically uploading generated images and videos to S3-compatible cloud storage (e.g., AWS S3, MinIO, Alibaba Cloud OSS, Tencent Cloud COS).
|
||||
|
||||
When enabled, the server follows a **Generate -> Upload -> Delete** workflow:
|
||||
1. The artifact is generated to a temporary local file.
|
||||
2. The file is immediately uploaded to the configured S3 bucket in a background thread.
|
||||
3. Upon successful upload, the local file is deleted.
|
||||
4. The API response returns the public URL of the uploaded object.
|
||||
|
||||
#### Configuration
|
||||
|
||||
Cloud storage is enabled via environment variables. Note that `boto3` must be installed separately (`pip install boto3`) to use this feature.
|
||||
|
||||
```bash
|
||||
# Enable S3 storage
|
||||
export SGLANG_CLOUD_STORAGE_TYPE=s3
|
||||
export SGLANG_S3_BUCKET_NAME=my-bucket
|
||||
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
|
||||
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
|
||||
|
||||
# Optional: Custom endpoint for MinIO/OSS/COS
|
||||
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
|
||||
```
|
||||
|
||||
See [Environment Variables Documentation](environment_variables.md) for more details.
|
||||
|
||||
## Generate
|
||||
|
||||
Run a one-off generation task without launching a persistent server.
|
||||
|
||||
To use it, pass both server arguments and sampling parameters in one command, after the `generate` subcommand, for example:
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
)
|
||||
|
||||
SAMPLING_ARGS=(
|
||||
--prompt "A curious raccoon"
|
||||
--save-output
|
||||
--output-path outputs
|
||||
--output-file-name "A curious raccoon.mp4"
|
||||
)
|
||||
|
||||
sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
|
||||
|
||||
# Or, users can set `SGLANG_CACHE_DIT_ENABLED` env as `true` to enable cache acceleration
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
|
||||
```
|
||||
|
||||
Once the generation task has finished, the server will shut down automatically.
|
||||
|
||||
> [!NOTE]
|
||||
> The HTTP server-related arguments are ignored in this subcommand.
|
||||
|
||||
## Diffusers Backend
|
||||
|
||||
SGLang diffusion supports a **diffusers backend** that allows you to run any diffusers-compatible model through SGLang's infrastructure using vanilla diffusers pipelines. This is useful for running models without native SGLang implementations or models with custom pipeline classes.
|
||||
|
||||
### Arguments
|
||||
|
||||
| Argument | Values | Description |
|
||||
|----------|--------|-------------|
|
||||
| `--backend` | `auto` (default), `sglang`, `diffusers` | `auto`: prefer native SGLang, fallback to diffusers. `sglang`: force native (fails if unavailable). `diffusers`: force vanilla diffusers pipeline. |
|
||||
| `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines. See [diffusers attention backends](https://huggingface.co/docs/diffusers/main/en/optimization/attention_backends). |
|
||||
| `--trust-remote-code` | flag | Required for models with custom pipeline classes (e.g., Ovis). |
|
||||
| `--vae-tiling` | flag | Enable VAE tiling for large image support (decodes tile-by-tile). |
|
||||
| `--vae-slicing` | flag | Enable VAE slicing for lower memory usage (decodes slice-by-slice). |
|
||||
| `--dit-precision` | `fp16`, `bf16`, `fp32` | Precision for the diffusion transformer. |
|
||||
| `--vae-precision` | `fp16`, `bf16`, `fp32` | Precision for the VAE. |
|
||||
|
||||
### Example: Running Ovis-Image-7B
|
||||
|
||||
[Ovis-Image-7B](https://huggingface.co/AIDC-AI/Ovis-Image-7B) is a 7B text-to-image model optimized for high-quality text rendering.
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path AIDC-AI/Ovis-Image-7B \
|
||||
--backend diffusers \
|
||||
--trust-remote-code \
|
||||
--diffusers-attention-backend flash \
|
||||
--prompt "A serene Japanese garden with cherry blossoms" \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--num-inference-steps 30 \
|
||||
--save-output \
|
||||
--output-path outputs \
|
||||
--output-file-name ovis_garden.png
|
||||
```
|
||||
|
||||
### Extra Diffusers Arguments
|
||||
|
||||
For pipeline-specific parameters not exposed via CLI, use `diffusers_kwargs` in a config file:
|
||||
|
||||
```json
|
||||
{
|
||||
"model_path": "AIDC-AI/Ovis-Image-7B",
|
||||
"backend": "diffusers",
|
||||
"prompt": "A beautiful landscape",
|
||||
"diffusers_kwargs": {
|
||||
"cross_attention_kwargs": {"scale": 0.5}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
sglang generate --config config.json
|
||||
```
|
||||
@@ -1,59 +0,0 @@
|
||||
# Contributing to SGLang Diffusion
|
||||
|
||||
This guide outlines the requirements for contributing to the SGLang Diffusion module (`sglang.multimodal_gen`).
|
||||
|
||||
## 1. Commit Message Convention
|
||||
|
||||
We follow a structured commit message format to maintain a clean history.
|
||||
|
||||
**Format:**
|
||||
```text
|
||||
[diffusion] <scope>: <subject>
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
- `[diffusion] cli: add --perf-dump-path argument`
|
||||
- `[diffusion] scheduler: fix deadlock in batch processing`
|
||||
- `[diffusion] model: support Stable Diffusion 3.5`
|
||||
|
||||
**Rules:**
|
||||
- **Prefix**: Always start with `[diffusion]`.
|
||||
- **Scope** (Optional): `cli`, `scheduler`, `model`, `pipeline`, `docs`, etc.
|
||||
- **Subject**: Imperative mood, short and clear (e.g., "add feature" not "added feature").
|
||||
|
||||
## 2. Performance Reporting
|
||||
|
||||
For PRs that impact **latency**, **throughput**, or **memory usage**, you **should** provide a performance comparison report.
|
||||
|
||||
### How to Generate a Report
|
||||
|
||||
1. **Baseline**: run the benchmark (for a single generation task)
|
||||
```bash
|
||||
$ sglang generate --model-path <model> --prompt "A benchmark prompt" --perf-dump-path baseline.json
|
||||
```
|
||||
|
||||
2. **New**: run the same benchmark, without modifying any server_args or sampling_params
|
||||
```bash
|
||||
$ sglang generate --model-path <model> --prompt "A benchmark prompt" --perf-dump-path new.json
|
||||
```
|
||||
|
||||
3. **Compare**: run the compare script, which will print a Markdown table to the console
|
||||
```bash
|
||||
$ python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json new.json [new2.json ...]
|
||||
### Performance Comparison Report
|
||||
...
|
||||
```
|
||||
4. **Paste**: paste the table into the PR description
|
||||
|
||||
## 3. CI-Based Change Protection
|
||||
|
||||
Consider adding tests to the `pr-test` or `nightly-test` suites to safeguard your changes, especially for PRs that:
|
||||
|
||||
- support a new model
|
||||
- add a testcase for this new model to `testcase_configs.py`
|
||||
- support or fix important features
|
||||
- significantly improve performance
|
||||
|
||||
Please run the according testcase, then update/add the baseline to `perf_baselines.json` by following the instruction in console if applicable.
|
||||
|
||||
See [test](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/test) for examples
|
||||
@@ -1,36 +0,0 @@
|
||||
## Caching Acceleration
|
||||
|
||||
These variables configure caching acceleration for Diffusion Transformer (DiT) models.
|
||||
SGLang supports multiple caching strategies - see [caching documentation](cache/caching.md) for an overview.
|
||||
|
||||
### Cache-DiT Configuration
|
||||
|
||||
See [cache-dit documentation](cache/cache_dit.md) for detailed configuration.
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
|-------------------------------------|---------|------------------------------------------|
|
||||
| `SGLANG_CACHE_DIT_ENABLED` | false | Enable Cache-DiT acceleration |
|
||||
| `SGLANG_CACHE_DIT_FN` | 1 | First N blocks to always compute |
|
||||
| `SGLANG_CACHE_DIT_BN` | 0 | Last N blocks to always compute |
|
||||
| `SGLANG_CACHE_DIT_WARMUP` | 4 | Warmup steps before caching |
|
||||
| `SGLANG_CACHE_DIT_RDT` | 0.24 | Residual difference threshold |
|
||||
| `SGLANG_CACHE_DIT_MC` | 3 | Max continuous cached steps |
|
||||
| `SGLANG_CACHE_DIT_TAYLORSEER` | false | Enable TaylorSeer calibrator |
|
||||
| `SGLANG_CACHE_DIT_TS_ORDER` | 1 | TaylorSeer order (1 or 2) |
|
||||
| `SGLANG_CACHE_DIT_SCM_PRESET` | none | SCM preset (none/slow/medium/fast/ultra) |
|
||||
| `SGLANG_CACHE_DIT_SCM_POLICY` | dynamic | SCM caching policy |
|
||||
| `SGLANG_CACHE_DIT_SCM_COMPUTE_BINS` | not set | Custom SCM compute bins |
|
||||
| `SGLANG_CACHE_DIT_SCM_CACHE_BINS` | not set | Custom SCM cache bins |
|
||||
|
||||
## Cloud Storage
|
||||
|
||||
These variables configure S3-compatible cloud storage for automatically uploading generated images and videos.
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
|---------------------------------|---------|--------------------------------------------------------|
|
||||
| `SGLANG_CLOUD_STORAGE_TYPE` | not set | Set to `s3` to enable cloud storage |
|
||||
| `SGLANG_S3_BUCKET_NAME` | not set | The name of the S3 bucket |
|
||||
| `SGLANG_S3_ENDPOINT_URL` | not set | Custom endpoint URL (for MinIO, OSS, etc.) |
|
||||
| `SGLANG_S3_REGION_NAME` | us-east-1 | AWS region name |
|
||||
| `SGLANG_S3_ACCESS_KEY_ID` | not set | AWS Access Key ID |
|
||||
| `SGLANG_S3_SECRET_ACCESS_KEY` | not set | AWS Secret Access Key |
|
||||
@@ -1,56 +0,0 @@
|
||||
# Install SGLang-diffusion
|
||||
|
||||
You can install sglang-diffusion using one of the methods below.
|
||||
|
||||
This page primarily applies to common NVIDIA GPU platforms.
|
||||
* For AMD Instinct/ROCm environments see the dedicated [ROCm quickstart](install_rocm.md), which lists the exact steps (including kernel builds) we used to validate sgl-diffusion on MI300X.
|
||||
* For Moore Threads GPU (MTGPU) with the MUSA software stack, see the [MUSA quickstart](install_musa.md), which lists the exact steps we used to validate sgl-diffusion on MTT S5000.
|
||||
|
||||
## Method 1: With pip or uv
|
||||
|
||||
It is recommended to use uv for a faster installation:
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install "sglang[diffusion]" --prerelease=allow
|
||||
```
|
||||
|
||||
## Method 2: From source
|
||||
|
||||
```bash
|
||||
# Use the latest release branch
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
|
||||
# Install the Python packages
|
||||
pip install --upgrade pip
|
||||
pip install -e "python[diffusion]"
|
||||
|
||||
# With uv
|
||||
uv pip install -e "python[diffusion]" --prerelease=allow
|
||||
```
|
||||
|
||||
## Method 3: Using Docker
|
||||
|
||||
The Docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang), built from the [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/Dockerfile).
|
||||
Replace `<secret>` below with your HuggingFace Hub [token](https://huggingface.co/docs/hub/en/security-tokens).
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:dev \
|
||||
zsh -c '\
|
||||
echo "Installing diffusion dependencies..." && \
|
||||
pip install -e "python[diffusion]" && \
|
||||
echo "Starting SGLang-Diffusion..." && \
|
||||
sglang generate \
|
||||
--model-path black-forest-labs/FLUX.1-dev \
|
||||
--prompt "A logo With Bold Large text: SGL Diffusion" \
|
||||
--save-output \
|
||||
'
|
||||
```
|
||||
@@ -1,24 +0,0 @@
|
||||
# MUSA Quickstart for SGLang-Diffusion
|
||||
|
||||
This page covers installation and usage of SGLang-Diffusion on Moore Threads GPU (MTGPU) with the MUSA software stack.
|
||||
|
||||
## Install from Source
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
|
||||
# Install the Python packages
|
||||
pip install --upgrade pip
|
||||
rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
|
||||
pip install -e "python[all_musa]"
|
||||
```
|
||||
|
||||
## Quick Test
|
||||
|
||||
```bash
|
||||
sglang generate --model-path black-forest-labs/FLUX.1-dev \
|
||||
--prompt "A logo With Bold Large text: SGL Diffusion" \
|
||||
--save-output
|
||||
```
|
||||
@@ -1,9 +0,0 @@
|
||||
# ROCm quickstart for sgl-diffusion
|
||||
|
||||
```bash
|
||||
docker run --device=/dev/kfd --device=/dev/dri --ipc=host \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env HF_TOKEN=<secret> \
|
||||
lmsysorg/sglang:v0.5.5.post2-rocm700-mi30x \
|
||||
sglang generate --model-path black-forest-labs/FLUX.1-dev --prompt "A logo With Bold Large text: SGL Diffusion" --save-output
|
||||
```
|
||||
@@ -1,391 +0,0 @@
|
||||
# SGLang Diffusion OpenAI API
|
||||
|
||||
The SGLang diffusion HTTP server implements an OpenAI-compatible API for image and video generation, as well as LoRA adapter management.
|
||||
|
||||
## Serve
|
||||
|
||||
Launch the server using the `sglang serve` command.
|
||||
|
||||
### Start the server
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
--port 30010
|
||||
)
|
||||
|
||||
sglang serve "${SERVER_ARGS[@]}"
|
||||
```
|
||||
|
||||
- **--model-path**: Path to the model or model ID.
|
||||
- **--port**: HTTP port to listen on (default: `30000`).
|
||||
|
||||
#### Get Model Information
|
||||
|
||||
**Endpoint:** `GET /models`
|
||||
|
||||
Returns information about the model served by this server, including model path, task type, pipeline configuration, and precision settings.
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -sS -X GET "http://localhost:30010/models"
|
||||
```
|
||||
|
||||
**Response Example:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
|
||||
"task_type": "T2V",
|
||||
"pipeline_name": "wan_pipeline",
|
||||
"pipeline_class": "WanPipeline",
|
||||
"num_gpus": 4,
|
||||
"dit_precision": "bf16",
|
||||
"vae_precision": "fp16"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Image Generation
|
||||
|
||||
The server implements an OpenAI-compatible Images API under the `/v1/images` namespace.
|
||||
|
||||
#### Create an image
|
||||
|
||||
**Endpoint:** `POST /v1/images/generations`
|
||||
|
||||
**Python Example (b64_json response):**
|
||||
|
||||
```python
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")
|
||||
|
||||
img = client.images.generate(
|
||||
prompt="A calico cat playing a piano on stage",
|
||||
size="1024x1024",
|
||||
n=1,
|
||||
response_format="b64_json",
|
||||
)
|
||||
|
||||
image_bytes = base64.b64decode(img.data[0].b64_json)
|
||||
with open("output.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
```
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/generations" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-d '{
|
||||
"prompt": "A calico cat playing a piano on stage",
|
||||
"size": "1024x1024",
|
||||
"n": 1,
|
||||
"response_format": "b64_json"
|
||||
}'
|
||||
```
|
||||
|
||||
> **Note**
|
||||
> The `response_format=url` option is not supported for `POST /v1/images/generations` and will return a `400` error.
|
||||
|
||||
#### Edit an image
|
||||
|
||||
**Endpoint:** `POST /v1/images/edits`
|
||||
|
||||
This endpoint accepts a multipart form upload with input images and a text prompt. The server can return either a base64-encoded image or a URL to download the image.
|
||||
|
||||
**Curl Example (b64_json response):**
|
||||
|
||||
```bash
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/edits" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-F "image=@local_input_image.png" \
|
||||
-F "url=image_url.jpg" \
|
||||
-F "prompt=A calico cat playing a piano on stage" \
|
||||
-F "size=1024x1024" \
|
||||
-F "response_format=b64_json"
|
||||
```
|
||||
|
||||
**Curl Example (URL response):**
|
||||
|
||||
```bash
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/edits" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-F "image=@local_input_image.png" \
|
||||
-F "url=image_url.jpg" \
|
||||
-F "prompt=A calico cat playing a piano on stage" \
|
||||
-F "size=1024x1024" \
|
||||
-F "response_format=url"
|
||||
```
|
||||
|
||||
#### Download image content
|
||||
|
||||
When `response_format=url` is used with `POST /v1/images/edits`, the API returns a relative URL like `/v1/images/<IMAGE_ID>/content`.
|
||||
|
||||
**Endpoint:** `GET /v1/images/{image_id}/content`
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -sS -L "http://localhost:30010/v1/images/<IMAGE_ID>/content" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-o output.png
|
||||
```
|
||||
|
||||
### Video Generation
|
||||
|
||||
The server implements a subset of the OpenAI Videos API under the `/v1/videos` namespace.
|
||||
|
||||
#### Create a video
|
||||
|
||||
**Endpoint:** `POST /v1/videos`
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")
|
||||
|
||||
video = client.videos.create(
|
||||
prompt="A calico cat playing a piano on stage",
|
||||
size="1280x720"
|
||||
)
|
||||
print(f"Video ID: {video.id}, Status: {video.status}")
|
||||
```
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -sS -X POST "http://localhost:30010/v1/videos" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-d '{
|
||||
"prompt": "A calico cat playing a piano on stage",
|
||||
"size": "1280x720"
|
||||
}'
|
||||
```
|
||||
|
||||
#### List videos
|
||||
|
||||
**Endpoint:** `GET /v1/videos`
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python
|
||||
videos = client.videos.list()
|
||||
for item in videos.data:
|
||||
print(item.id, item.status)
|
||||
```
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -sS -X GET "http://localhost:30010/v1/videos" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890"
|
||||
```
|
||||
|
||||
#### Download video content
|
||||
|
||||
**Endpoint:** `GET /v1/videos/{video_id}/content`
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python
|
||||
import time
|
||||
|
||||
# Poll for completion
|
||||
while True:
|
||||
page = client.videos.list()
|
||||
item = next((v for v in page.data if v.id == video_id), None)
|
||||
if item and item.status == "completed":
|
||||
break
|
||||
time.sleep(5)
|
||||
|
||||
# Download content
|
||||
resp = client.videos.download_content(video_id=video_id)
|
||||
with open("output.mp4", "wb") as f:
|
||||
f.write(resp.read())
|
||||
```
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -sS -L "http://localhost:30010/v1/videos/<VIDEO_ID>/content" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-o output.mp4
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### LoRA Management
|
||||
|
||||
The server supports dynamic loading, merging, and unmerging of LoRA adapters.
|
||||
|
||||
**Important Notes:**
|
||||
- Mutual Exclusion: Only one LoRA can be *merged* (active) at a time
|
||||
- Switching: To switch LoRAs, you must first `unmerge` the current one, then `set` the new one
|
||||
- Caching: The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has little cost
|
||||
|
||||
#### Set LoRA Adapter
|
||||
|
||||
Loads one or more LoRA adapters and merges their weights into the model. Supports both single LoRA (backward compatible) and multiple LoRA adapters.
|
||||
|
||||
**Endpoint:** `POST /v1/set_lora`
|
||||
|
||||
**Parameters:**
|
||||
- `lora_nickname` (string or list of strings, required): A unique identifier for the LoRA adapter(s). Can be a single string or a list of strings for multiple LoRAs
|
||||
- `lora_path` (string or list of strings/None, optional): Path to the `.safetensors` file(s) or Hugging Face repo ID(s). Required for the first load; optional if re-activating a cached nickname. If a list, must match the length of `lora_nickname`
|
||||
- `target` (string or list of strings, optional): Which transformer(s) to apply the LoRA to. If a list, must match the length of `lora_nickname`. Valid values:
|
||||
- `"all"` (default): Apply to all transformers
|
||||
- `"transformer"`: Apply only to the primary transformer (high noise for Wan2.2)
|
||||
- `"transformer_2"`: Apply only to transformer_2 (low noise for Wan2.2)
|
||||
- `"critic"`: Apply only to the critic model
|
||||
- `strength` (float or list of floats, optional): LoRA strength for merge, default 1.0. If a list, must match the length of `lora_nickname`. Values < 1.0 reduce the effect, values > 1.0 amplify the effect
|
||||
|
||||
**Single LoRA Example:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": "lora_name",
|
||||
"lora_path": "/path/to/lora.safetensors",
|
||||
"target": "all",
|
||||
"strength": 0.8
|
||||
}'
|
||||
```
|
||||
|
||||
**Multiple LoRA Example:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": ["lora_1", "lora_2"],
|
||||
"lora_path": ["/path/to/lora1.safetensors", "/path/to/lora2.safetensors"],
|
||||
"target": ["transformer", "transformer_2"],
|
||||
"strength": [0.8, 1.0]
|
||||
}'
|
||||
```
|
||||
|
||||
**Multiple LoRA with Same Target:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": ["style_lora", "character_lora"],
|
||||
"lora_path": ["/path/to/style.safetensors", "/path/to/character.safetensors"],
|
||||
"target": "all",
|
||||
"strength": [0.7, 0.9]
|
||||
}'
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> When using multiple LoRAs:
|
||||
> - All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length
|
||||
> - If `target` or `strength` is a single value, it will be applied to all LoRAs
|
||||
> - Multiple LoRAs applied to the same target will be merged in order
|
||||
|
||||
|
||||
#### Merge LoRA Weights
|
||||
|
||||
Manually merges the currently set LoRA weights into the base model.
|
||||
|
||||
> [!NOTE]
|
||||
> `set_lora` automatically performs a merge, so this is typically only needed if you have manually unmerged but want to re-apply the same LoRA without calling `set_lora` again.*
|
||||
|
||||
**Endpoint:** `POST /v1/merge_lora_weights`
|
||||
|
||||
**Parameters:**
|
||||
- `target` (string, optional): Which transformer(s) to merge. One of "all" (default), "transformer", "transformer_2", "critic"
|
||||
- `strength` (float, optional): LoRA strength for merge, default 1.0. Values < 1.0 reduce the effect, values > 1.0 amplify the effect
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/merge_lora_weights \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"strength": 0.8}'
|
||||
```
|
||||
|
||||
|
||||
#### Unmerge LoRA Weights
|
||||
|
||||
Unmerges the currently active LoRA weights from the base model, restoring it to its original state. This **must** be called before setting a different LoRA.
|
||||
|
||||
**Endpoint:** `POST /v1/unmerge_lora_weights`
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/unmerge_lora_weights \
|
||||
-H "Content-Type: application/json"
|
||||
```
|
||||
|
||||
#### List LoRA Adapters
|
||||
|
||||
Returns loaded LoRA adapters and current application status per module.
|
||||
|
||||
**Endpoint:** `GET /v1/list_loras`
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -sS -X GET "http://localhost:30010/v1/list_loras"
|
||||
```
|
||||
|
||||
**Response Example:**
|
||||
|
||||
```json
|
||||
{
|
||||
"loaded_adapters": [
|
||||
{ "nickname": "lora_a", "path": "/weights/lora_a.safetensors" },
|
||||
{ "nickname": "lora_b", "path": "/weights/lora_b.safetensors" }
|
||||
],
|
||||
"active": {
|
||||
"transformer": [
|
||||
{
|
||||
"nickname": "lora2",
|
||||
"path": "tarn59/pixel_art_style_lora_z_image_turbo",
|
||||
"merged": true,
|
||||
"strength": 1.0
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
- If LoRA is not enabled for the current pipeline, the server will return an error.
|
||||
- `num_lora_layers_with_weights` counts only layers that have LoRA weights applied for the active adapter.
|
||||
|
||||
### Example: Switching LoRAs
|
||||
|
||||
1. Set LoRA A:
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_a", "lora_path": "path/to/A"}'
|
||||
```
|
||||
2. Generate with LoRA A...
|
||||
3. Unmerge LoRA A:
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/unmerge_lora_weights
|
||||
```
|
||||
4. Set LoRA B:
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_b", "lora_path": "path/to/B"}'
|
||||
```
|
||||
5. Generate with LoRA B...
|
||||
@@ -1,136 +0,0 @@
|
||||
# Profiling Multimodal Generation
|
||||
|
||||
This guide covers profiling techniques for multimodal generation pipelines in SGLang.
|
||||
|
||||
## PyTorch Profiler
|
||||
|
||||
PyTorch Profiler provides detailed kernel execution time, call stack, and GPU utilization metrics.
|
||||
|
||||
### Denoising Stage Profiling
|
||||
|
||||
Profile the denoising stage with sampled timesteps (default: 5 steps after 1 warmup step):
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
|
||||
--seed 0 \
|
||||
--profile
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `--profile`: Enable profiling for the denoising stage
|
||||
- `--num-profiled-timesteps N`: Number of timesteps to profile after warmup (default: 5)
|
||||
- Smaller values reduce trace file size
|
||||
- Example: `--num-profiled-timesteps 10` profiles 10 steps after 1 warmup step
|
||||
|
||||
### Full Pipeline Profiling
|
||||
|
||||
Profile all pipeline stages (text encoding, denoising, VAE decoding, etc.):
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
|
||||
--seed 0 \
|
||||
--profile \
|
||||
--profile-all-stages
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `--profile-all-stages`: Used with `--profile`, profile all pipeline stages instead of just denoising
|
||||
|
||||
### Output Location
|
||||
|
||||
By default, trace files are saved in the ./logs/ directory.
|
||||
|
||||
The exact output file path will be shown in the console output, for example:
|
||||
|
||||
```bash
|
||||
[mm-dd hh:mm:ss] Saved profiler traces to: /sgl-workspace/sglang/logs/mocked_fake_id_for_offline_generate-5_steps-global-rank0.trace.json.gz
|
||||
```
|
||||
|
||||
### View Traces
|
||||
|
||||
Load and visualize trace files at:
|
||||
- https://ui.perfetto.dev/ (recommended)
|
||||
- chrome://tracing (Chrome only)
|
||||
|
||||
For large trace files, reduce `--num-profiled-timesteps` or avoid using `--profile-all-stages`.
|
||||
|
||||
|
||||
### `--perf-dump-path` (Stage/Step Timing Dump)
|
||||
|
||||
Besides profiler traces, you can also dump a lightweight JSON report that contains:
|
||||
- stage-level timing breakdown for the full pipeline
|
||||
- step-level timing breakdown for the denoising stage (per diffusion step)
|
||||
|
||||
This is useful to quickly identify which stage dominates end-to-end latency, and whether denoising steps have uniform runtimes (and if not, which step has an abnormal spike).
|
||||
|
||||
The dumped JSON contains a `denoise_steps_ms` field formatted as an array of objects, each with a `step` key (the step index) and a `duration_ms` key.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path <MODEL_PATH_OR_ID> \
|
||||
--prompt "<PROMPT>" \
|
||||
--perf-dump-path perf.json
|
||||
```
|
||||
|
||||
## Nsight Systems
|
||||
|
||||
Nsight Systems provides low-level CUDA profiling with kernel details, register usage, and memory access patterns.
|
||||
|
||||
### Installation
|
||||
|
||||
See the [SGLang profiling guide](https://github.com/sgl-project/sglang/blob/main/docs/developer_guide/benchmark_and_profiling.md#profile-with-nsight) for installation instructions.
|
||||
|
||||
### Basic Profiling
|
||||
|
||||
Profile the entire pipeline execution:
|
||||
|
||||
```bash
|
||||
nsys profile \
|
||||
--trace-fork-before-exec=true \
|
||||
--cuda-graph-trace=node \
|
||||
--force-overwrite=true \
|
||||
-o QwenImage \
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
|
||||
--seed 0
|
||||
```
|
||||
|
||||
### Targeted Stage Profiling
|
||||
|
||||
Use `--delay` and `--duration` to capture specific stages and reduce file size:
|
||||
|
||||
```bash
|
||||
nsys profile \
|
||||
--trace-fork-before-exec=true \
|
||||
--cuda-graph-trace=node \
|
||||
--force-overwrite=true \
|
||||
--delay 10 \
|
||||
--duration 30 \
|
||||
-o QwenImage_denoising \
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
|
||||
--seed 0
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `--delay N`: Wait N seconds before starting capture (skip initialization overhead)
|
||||
- `--duration N`: Capture for N seconds (focus on specific stages)
|
||||
- `--force-overwrite`: Overwrite existing output files
|
||||
|
||||
## Notes
|
||||
|
||||
- **Reduce trace size**: Use `--num-profiled-timesteps` with smaller values or `--delay`/`--duration` with Nsight Systems
|
||||
- **Stage-specific analysis**: Use `--profile` alone for denoising stage, add `--profile-all-stages` for full pipeline
|
||||
- **Multiple runs**: Profile with different prompts and resolutions to identify bottlenecks across workloads
|
||||
|
||||
## FAQ
|
||||
|
||||
- If you are profiling `sglang generate` with Nsight Systems and find that the generated profiler file did not capture any CUDA kernels, you can resolve this issue by increasing the model's inference steps to extend the execution time.
|
||||
@@ -1,78 +0,0 @@
|
||||
# Compatibility Matrix
|
||||
|
||||
The table below shows every supported model and the optimizations supported for them.
|
||||
|
||||
The symbols used have the following meanings:
|
||||
|
||||
- ✅ = Full compatibility
|
||||
- ❌ = No compatibility
|
||||
- ⭕ = Does not apply to this model
|
||||
|
||||
## Models x Optimization
|
||||
|
||||
The `HuggingFace Model ID` can be passed directly to `from_pretrained()` methods, and sglang-diffusion will use the
|
||||
optimal
|
||||
default parameters when initializing and generating videos.
|
||||
|
||||
### Video Generation Models
|
||||
|
||||
| Model Name | Hugging Face Model ID | Resolutions | TeaCache | Sliding Tile Attn | Sage Attn | Video Sparse Attention (VSA) | Sparse Linear Attention（SLA）| Sage Sparse Linear Attention（SageSLA）|
|
||||
|:-----------------------------|:--------------------------------------------------|:--------------------|:--------:|:-----------------:|:---------:|:----------------------------:|:----------------------------:|:-----------------------------------------------:|
|
||||
| FastWan2.1 T2V 1.3B | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers` | 480p | ⭕ | ⭕ | ⭕ | ✅ | ❌ | ❌ |
|
||||
| FastWan2.2 TI2V 5B Full Attn | `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` | 720p | ⭕ | ⭕ | ⭕ | ✅ | ❌ | ❌ |
|
||||
| Wan2.2 TI2V 5B | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | 720p | ⭕ | ⭕ | ✅ | ⭕ | ❌ | ❌ |
|
||||
| Wan2.2 T2V A14B | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 480p<br>720p | ❌ | ❌ | ✅ | ⭕ | ❌ | ❌ |
|
||||
| Wan2.2 I2V A14B | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | 480p<br>720p | ❌ | ❌ | ✅ | ⭕ | ❌ | ❌ |
|
||||
| HunyuanVideo | `hunyuanvideo-community/HunyuanVideo` | 720×1280<br>544×960 | ❌ | ✅ | ✅ | ⭕ | ❌ | ❌ |
|
||||
| FastHunyuan | `FastVideo/FastHunyuan-diffusers` | 720×1280<br>544×960 | ❌ | ✅ | ✅ | ⭕ | ❌ | ❌ |
|
||||
| Wan2.1 T2V 1.3B | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | 480p | ✅ | ✅ | ✅ | ⭕ | ❌ | ❌ |
|
||||
| Wan2.1 T2V 14B | `Wan-AI/Wan2.1-T2V-14B-Diffusers` | 480p, 720p | ✅ | ✅ | ✅ | ⭕ | ❌ | ❌ |
|
||||
| Wan2.1 I2V 480P | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` | 480p | ✅ | ✅ | ✅ | ⭕ | ❌ | ❌ |
|
||||
| Wan2.1 I2V 720P | `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers` | 720p | ✅ | ✅ | ✅ | ⭕ | ❌ | ❌ |
|
||||
| TurboWan2.1 T2V 1.3B | `IPostYellow/TurboWan2.1-T2V-1.3B-Diffusers` | 480p | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ |
|
||||
| TurboWan2.1 T2V 14B | `IPostYellow/TurboWan2.1-T2V-14B-Diffusers` | 480p | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ |
|
||||
| TurboWan2.1 T2V 14B 720P | `IPostYellow/TurboWan2.1-T2V-14B-720P-Diffusers` | 720p | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ |
|
||||
| TurboWan2.2 I2V A14B | `IPostYellow/TurboWan2.2-I2V-A14B-Diffusers` | 720p | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ |
|
||||
|
||||
**Note**: <br>
|
||||
1.Wan2.2 TI2V 5B has some quality issues when performing I2V generation. We are working on fixing this issue.<br>
|
||||
2.SageSLA Based on SpargeAttn. Install it first with `pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation`
|
||||
|
||||
### Image Generation Models
|
||||
|
||||
| Model Name | HuggingFace Model ID | Resolutions |
|
||||
|:-----------------|:----------------------------------------|:---------------|
|
||||
| FLUX.1-dev | `black-forest-labs/FLUX.1-dev` | Any resolution |
|
||||
| FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | Any resolution |
|
||||
| FLUX.2-Klein | `black-forest-labs/FLUX.2-klein-4B` | Any resolution |
|
||||
| Z-Image-Turbo | `Tongyi-MAI/Z-Image-Turbo` | Any resolution |
|
||||
| GLM-Image | `zai-org/GLM-Image` | Any resolution |
|
||||
| Qwen Image | `Qwen/Qwen-Image` | Any resolution |
|
||||
| Qwen Image 2512 | `Qwen/Qwen-Image-2512` | Any resolution |
|
||||
| Qwen Image Edit | `Qwen/Qwen-Image-Edit` | Any resolution |
|
||||
|
||||
## Verified LoRA Examples
|
||||
|
||||
This section lists example LoRAs that have been explicitly tested and verified with each base model in the **SGLang Diffusion** pipeline.
|
||||
|
||||
> Important: \
|
||||
> LoRAs that are not listed here are not necessarily incompatible.
|
||||
> In practice, most standard LoRAs are expected to work, especially those following common Diffusers or SD-style conventions.
|
||||
> The entries below simply reflect configurations that have been manually validated by the SGLang team.
|
||||
|
||||
### Verified LoRAs by Base Model
|
||||
|
||||
| Base Model | Supported LoRAs |
|
||||
|:-----------------|:----------------|
|
||||
| Wan2.2 | `lightx2v/Wan2.2-Distill-Loras`<br>`Cseti/wan2.2-14B-Arcane_Jinx-lora-v1` |
|
||||
| Wan2.1 | `lightx2v/Wan2.1-Distill-Loras` |
|
||||
| Z-Image-Turbo | `tarn59/pixel_art_style_lora_z_image_turbo`<br>`wcde/Z-Image-Turbo-DeJPEG-Lora` |
|
||||
| Qwen-Image | `lightx2v/Qwen-Image-Lightning`<br>`flymy-ai/qwen-image-realism-lora`<br>`prithivMLmods/Qwen-Image-HeadshotX`<br>`starsfriday/Qwen-Image-EVA-LoRA` |
|
||||
| Qwen-Image-Edit | `ostris/qwen_image_edit_inpainting`<br>`lightx2v/Qwen-Image-Edit-2511-Lightning` |
|
||||
| Flux | `dvyio/flux-lora-simple-illustration`<br>`XLabs-AI/flux-furry-lora`<br>`XLabs-AI/flux-RealismLora` |
|
||||
|
||||
## Special requirements
|
||||
|
||||
### Sliding Tile Attention
|
||||
|
||||
- Currently, only Hopper GPUs (H100s) are supported.
|
||||
@@ -1,107 +0,0 @@
|
||||
# How to Support New Diffusion Models
|
||||
|
||||
This document explains how to add support for new diffusion models in SGLang diffusion.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
SGLang diffusion is engineered for both performance and flexibility, built upon a modular pipeline architecture. This
|
||||
design allows developers to easily construct complex, customized pipelines for various diffusion models by combining and
|
||||
reusing different components.
|
||||
|
||||
At its core, the architecture revolves around two key concepts, as highlighted in our [blog post](https://lmsys.org/blog/2025-11-07-sglang-diffusion/#architecture):
|
||||
|
||||
- **`ComposedPipeline`**: This class orchestrates a series of `PipelineStage`s to define the complete generation process for a specific model. It acts as the main entry point for a model and manages the data flow between the different stages of the diffusion process.
|
||||
- **`PipelineStage`**: Each stage is a modular component that encapsulates a common function within the diffusion process. Examples include prompt encoding, the denoising loop, or VAE decoding. These stages are designed to be self-contained and reusable across different pipelines.
|
||||
|
||||
## Key Components for Implementation
|
||||
|
||||
To add support for a new diffusion model, you will primarily need to define or configure the following components:
|
||||
|
||||
1. **`PipelineConfig`**: This is a dataclass that holds all the static configurations for your model pipeline. It includes paths to model components (like UNet, VAE, text encoders), precision settings (e.g., `fp16`, `bf16`), and other model-specific architectural parameters. Each model typically has its own subclass of `PipelineConfig`.
|
||||
|
||||
2. **`SamplingParams`**: This dataclass defines the parameters that control the generation process at runtime. These are the user-provided inputs for a generation request, such as the `prompt`, `negative_prompt`, `guidance_scale`, `num_inference_steps`, `seed`, output dimensions (`height`, `width`), etc.
|
||||
|
||||
3. **`ComposedPipeline` (not a config)**: This is the central class where you define the structure of your model's generation pipeline. You will create a new class that inherits from `ComposedPipelineBase` and, within it, instantiate and chain together the necessary `PipelineStage`s in the correct order. See `ComposedPipelineBase` and `PipelineStage` base definitions:
|
||||
- [`ComposedPipelineBase`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines/composed_pipeline_base.py)
|
||||
- [`PipelineStage`]( https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines/stages/base.py)
|
||||
- [Central registry (models/config mapping)](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/registry.py)
|
||||
|
||||
4. **Modules (components referenced by the pipeline)**: Each pipeline references a set of modules that are loaded from the model repository (e.g., Diffusers `model_index.json`) and assembled via the registry/loader. Common modules include:
|
||||
- `text_encoder`: Encodes text prompts into embeddings
|
||||
- `tokenizer`: Tokenizes raw text input for the text encoder(s).
|
||||
- `processor`: Preprocesses images and extracts features; often used in image-to-image tasks.
|
||||
- `image_encoder`: Specialized image feature extractor (may be distinct from or combined with `processor`).
|
||||
- `dit/transformer`: The core denoising network (DiT/UNet architecture) operating in latent space.
|
||||
- `scheduler`: Controls the timestep schedule and denoising dynamics throughout inference.
|
||||
- `vae`: Variational Autoencoder for encoding/decoding between pixel space and latent space.
|
||||
|
||||
## Available Pipeline Stages
|
||||
|
||||
You can build your custom `ComposedPipeline` by combining the following available stages as your will. Each stage is responsible for a specific part of the generation process.
|
||||
|
||||
| Stage Class | Description |
|
||||
| -------------------------------- | ------------------------------------------------------------------------------------------------------- |
|
||||
| `InputValidationStage` | Validates the user-provided `SamplingParams` to ensure they are correct before starting the pipeline. |
|
||||
| `TextEncodingStage` | Encodes text prompts into embeddings using one or more text encoders. |
|
||||
| `ImageEncodingStage` | Encodes input images into embeddings, often used in image-to-image tasks. |
|
||||
| `ImageVAEEncodingStage` | Specifically encodes an input image into the latent space using a Variational Autoencoder (VAE). |
|
||||
| `ConditioningStage` | Prepares the conditioning tensors (e.g., from text or image embeddings) for the denoising loop. |
|
||||
| `TimestepPreparationStage` | Prepares the scheduler's timesteps for the diffusion process. |
|
||||
| `LatentPreparationStage` | Creates the initial noisy latent tensor that will be denoised. |
|
||||
| `DenoisingStage` | Executes the main denoising loop, iteratively applying the model (e.g., UNet) to refine the latents. |
|
||||
| `DecodingStage` | Decodes the final latent tensor from the denoising loop back into pixel space (e.g., an image) using the VAE. |
|
||||
| `DmdDenoisingStage` | A specialized denoising stage for certain model architectures. |
|
||||
| `CausalDMDDenoisingStage` | A specialized causal denoising stage for specific video models. |
|
||||
|
||||
## Example: Implementing `Qwen-Image-Edit`
|
||||
|
||||
To illustrate the process, let's look at how `Qwen-Image-Edit` is implemented. The typical implementation order is:
|
||||
|
||||
1. **Analyze Required Modules**:
|
||||
- Study the target model's components by examining its `model_index.json` or Diffusers implementation to identify required modules:
|
||||
- `processor`: Image preprocessing and feature extraction
|
||||
- `scheduler`: Diffusion timestep scheduling
|
||||
- `text_encoder`: Text-to-embedding conversion
|
||||
- `tokenizer`: Text tokenization for the encoder
|
||||
- `transformer`: Core DiT denoising network
|
||||
- `vae`: Variational autoencoder for latent encoding/decoding
|
||||
|
||||
2. **Create Configs**:
|
||||
- **PipelineConfig**: [`QwenImageEditPipelineConfig`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/configs/pipelines/qwen_image.py) defines model-specific parameters, precision settings, preprocessing functions, and latent shape calculations.
|
||||
- **SamplingParams**: [`QwenImageSamplingParams`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/configs/sample/qwenimage.py) sets runtime defaults like `num_frames=1`, `guidance_scale=4.0`, `num_inference_steps=50`.
|
||||
|
||||
3. **Implement Model Components**:
|
||||
- Adapt or implement specific model components in the appropriate directories:
|
||||
- **DiT/Transformer**: Implement in [`runtime/models/dits/`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/dits/) - e.g., [`qwen_image.py`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/dits/qwen_image.py) for Qwen's DiT architecture
|
||||
- **Encoders**: Implement in [`runtime/models/encoders/`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/encoders/) - e.g., text encoders like [`qwen2_5vl.py`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/encoders/qwen2_5vl.py)
|
||||
- **VAEs**: Implement in [`runtime/models/vaes/`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/vaes/) - e.g., [`autoencoder_kl_qwenimage.py`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/vaes/autoencoder_kl_qwenimage.py)
|
||||
- **Schedulers**: Implement in [`runtime/models/schedulers/`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/schedulers/) if needed
|
||||
- These components handle the core model logic, attention mechanisms, and data transformations specific to the target diffusion model.
|
||||
|
||||
4. **Define Pipeline Class**:
|
||||
- The [`QwenImageEditPipeline`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/architectures/basic/qwen_image/qwen_image.py) class inherits from `ComposedPipelineBase` and orchestrates stages sequentially.
|
||||
- Declare required modules via `_required_config_modules` and implement the pipeline stages:
|
||||
|
||||
```python
|
||||
class QwenImageEditPipeline(ComposedPipelineBase):
|
||||
pipeline_name = "QwenImageEditPipeline" # Matches Diffusers model_index.json
|
||||
_required_config_modules = ["processor", "scheduler", "text_encoder", "tokenizer", "transformer", "vae"]
|
||||
|
||||
def create_pipeline_stages(self, server_args: ServerArgs):
|
||||
"""Set up pipeline stages sequentially."""
|
||||
self.add_stage(stage_name="input_validation_stage", stage=InputValidationStage())
|
||||
self.add_stage(stage_name="prompt_encoding_stage_primary", stage=ImageEncodingStage(...))
|
||||
self.add_stage(stage_name="image_encoding_stage_primary", stage=ImageVAEEncodingStage(...))
|
||||
self.add_stage(stage_name="timestep_preparation_stage", stage=TimestepPreparationStage(...))
|
||||
self.add_stage(stage_name="latent_preparation_stage", stage=LatentPreparationStage(...))
|
||||
self.add_stage(stage_name="conditioning_stage", stage=ConditioningStage())
|
||||
self.add_stage(stage_name="denoising_stage", stage=DenoisingStage(...))
|
||||
self.add_stage(stage_name="decoding_stage", stage=DecodingStage(...))
|
||||
```
|
||||
The pipeline is constructed by adding stages in order. `Qwen-Image-Edit` uses `ImageEncodingStage` (for prompt and image processing) and `ImageVAEEncodingStage` (for latent extraction) before standard denoising and decoding.
|
||||
|
||||
5. **Register Configs**:
|
||||
- Register the configs in the central registry ([`registry.py`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/registry.py)) via `_register_configs` to enable automatic loading and instantiation for the model. Modules are automatically loaded and injected based on the config and repository structure.
|
||||
|
||||
By following this pattern of defining configurations and composing pipelines, you can integrate new diffusion models
|
||||
into SGLang with ease.
|
||||
Reference in New Issue
Block a user