[diffusion] add .claude and update contributing with attitude towards vibe-pr (#19511)
This commit is contained in:
Mick
@@ -0,0 +1,81 @@
|
||||
---
|
||||
name: diffusion-perf
|
||||
description: Measure and compare sglang-diffusion performance. Use when benchmarking a model, comparing before/after performance, or generating a perf report for a PR.
|
||||
user-invocable: true
|
||||
allowed-tools: Bash, Read
|
||||
argument-hint: <model-path> [--prompt "..."] [--baseline baseline.json]
|
||||
---
|
||||
|
||||
# Diffusion Performance Measurement
|
||||
|
||||
Measure sglang-diffusion e2e latency via `--perf-dump-path`, then extract or compare results from the JSON dump.
|
||||
|
||||
## JSON dump structure
|
||||
|
||||
`--perf-dump-path` writes a JSON file with:
|
||||
|
||||
```json
|
||||
{
|
||||
"total_duration_ms": 14959.11,
|
||||
"steps": [
|
||||
{"name": "TextEncodingStage", "duration_ms": 611.83},
|
||||
{"name": "DenoisingStage", "duration_ms": 14289.46}
|
||||
],
|
||||
"denoise_steps_ms": [
|
||||
{"step": 0, "duration_ms": 240.5},
|
||||
{"step": 1, "duration_ms": 279.1}
|
||||
],
|
||||
"commit_hash": "abc123",
|
||||
"timestamp": "...",
|
||||
"memory_checkpoints": {}
|
||||
}
|
||||
```
|
||||
|
||||
Key fields:
|
||||
- `total_duration_ms` — e2e walltime (warmup excluded when `--warmup` is used)
|
||||
- `steps` — per-stage breakdown
|
||||
- `denoise_steps_ms` — per denoising step timing
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Single measurement
|
||||
|
||||
```bash
|
||||
sglang generate --model-path $MODEL --prompt "$PROMPT" --warmup --perf-dump-path result.json
|
||||
```
|
||||
|
||||
Then read `total_duration_ms` from `result.json`.
|
||||
|
||||
### 2. Before/after comparison
|
||||
|
||||
```bash
|
||||
# Baseline (on main branch or before changes)
|
||||
sglang generate --model-path $MODEL --prompt "$PROMPT" --warmup --perf-dump-path baseline.json
|
||||
|
||||
# New (after changes)
|
||||
sglang generate --model-path $MODEL --prompt "$PROMPT" --warmup --perf-dump-path new.json
|
||||
|
||||
# Compare — outputs a Markdown table suitable for PR descriptions
|
||||
python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json new.json
|
||||
```
|
||||
|
||||
### 3. Extracting a single number
|
||||
|
||||
To get e2e latency in seconds from a dump:
|
||||
|
||||
```bash
|
||||
python3 -c "import json; print(f\"{json.load(open('result.json'))['total_duration_ms']/1000:.2f}\")"
|
||||
```
|
||||
|
||||
## Arguments
|
||||
|
||||
If `$ARGUMENTS` is provided, parse it as:
|
||||
- First positional arg → `--model-path`
|
||||
- `--prompt "..."` → generation prompt (default: `"A curious raccoon"`)
|
||||
- `--baseline <file>` → if given, run comparison against this baseline file
|
||||
|
||||
## Notes
|
||||
|
||||
- Always use `--warmup` for accurate timing (excludes CUDA warmup from measurement).
|
||||
- Keep `--prompt` and all server/sampling args identical between baseline and new runs.
|
||||
- For PR descriptions, paste the output of `compare_perf.py` directly.
|
||||
108
python/sglang/multimodal_gen/.claude/CLAUDE.md
Normal file
108
python/sglang/multimodal_gen/.claude/CLAUDE.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# CLAUDE.md — sglang-diffusion (multimodal_gen)
|
||||
|
||||
## What is this?
|
||||
|
||||
SGLang's diffusion/multimodal generation subsystem. Separate from the LLM runtime (`srt`). Supports 20+ image/video diffusion models (Wan, FLUX, HunyuanVideo, LTX, Qwen-Image, etc.) with distributed inference, LoRA, and multiple attention backends.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# One-shot generation
|
||||
sglang generate --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers --prompt "A curious raccoon" --save-output
|
||||
|
||||
# Start server
|
||||
sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers --num-gpus 4
|
||||
|
||||
# Python API
|
||||
from sglang import DiffGenerator
|
||||
gen = DiffGenerator.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers")
|
||||
result = gen.generate(sampling_params_kwargs={"prompt": "A curious raccoon"})
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
CLI / Python API / HTTP Server (FastAPI + OpenAI-compatible)
|
||||
↓ ZMQ
|
||||
Scheduler (request queue, batching, dispatch)
|
||||
↓ multiprocessing pipes
|
||||
GPU Worker(s) → ComposedPipeline (stages: TextEncode → Denoise → Decode)
|
||||
```
|
||||
|
||||
### Key Directories
|
||||
|
||||
```
|
||||
runtime/
|
||||
├── entrypoints/ # CLI (generate/serve), HTTP server, Python API (DiffGenerator)
|
||||
├── managers/ # scheduler.py, gpu_worker.py
|
||||
├── pipelines_core/ # ComposedPipelineBase, stages/, schedule_batch.py (Req/OutputBatch)
|
||||
├── pipelines/ # Model-specific pipelines (wan, flux, hunyuan, ltx, qwen_image, ...)
|
||||
├── models/ # encoders/, dits/, vaes/, schedulers/
|
||||
├── layers/ # attention/, lora/, quantization/
|
||||
├── loader/ # Model loading, weight utils
|
||||
├── server_args.py # ServerArgs (all CLI/config params)
|
||||
└── distributed/ # Multi-GPU (TP, SP: ulysses/ring)
|
||||
configs/
|
||||
├── pipeline_configs/ # Per-model pipeline configs
|
||||
├── sample/ # SamplingParams
|
||||
└── models/ # DiT, VAE, Encoder configs
|
||||
```
|
||||
|
||||
### Key Classes
|
||||
|
||||
| Class | Location | Purpose |
|
||||
|-------|----------|---------|
|
||||
| `DiffGenerator` | `runtime/entrypoints/diffusion_generator.py` | Python API entry point |
|
||||
| `ComposedPipelineBase` | `runtime/pipelines_core/composed_pipeline_base.py` | Pipeline orchestrator (stages) |
|
||||
| `Scheduler` | `runtime/managers/scheduler.py` | ZMQ event loop, request dispatch |
|
||||
| `GPUWorker` | `runtime/managers/gpu_worker.py` | GPU inference worker |
|
||||
| `Req` / `OutputBatch` | `runtime/pipelines_core/schedule_batch.py` | Request/output containers |
|
||||
| `ServerArgs` | `runtime/server_args.py` | All config params |
|
||||
| `SamplingParams` | `configs/sample/sampling_params.py` | Generation params |
|
||||
| `PipelineConfig` | `configs/pipeline_configs/base.py` | Model structure config |
|
||||
|
||||
### Key Functions
|
||||
|
||||
| Function | Module | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `build_pipeline()` | `runtime/pipelines_core/__init__.py` | Instantiate pipeline from model_path |
|
||||
| `get_model_info()` | `registry.py` | Resolve pipeline + config classes |
|
||||
| `launch_server()` | `runtime/launch_server.py` | Start multi-process server |
|
||||
|
||||
## Adding a New Model
|
||||
|
||||
1. Create pipeline in `runtime/pipelines/` extending `ComposedPipelineBase`
|
||||
2. Define stages via `create_pipeline_stages()` (TextEncoding → Denoising → Decoding)
|
||||
3. Add config in `configs/pipeline_configs/`
|
||||
4. Register in `registry.py` via `register_configs()`
|
||||
|
||||
## Multi-GPU
|
||||
|
||||
```bash
|
||||
# Sequence parallelism (video frames across GPUs)
|
||||
sglang serve --model-path ... --num-gpus 4 --ulysses-degree 2 --ring-degree 2
|
||||
|
||||
# Tensor parallelism (model layers across GPUs)
|
||||
sglang serve --model-path ... --num-gpus 2 --tp-size 2
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Tests live in test/ subdirectory
|
||||
python -m pytest python/sglang/multimodal_gen/test/
|
||||
|
||||
# No need to pre-download models — auto-downloaded at runtime
|
||||
# Dependencies assumed already installed via `pip install -e "python[diffusion]"`
|
||||
```
|
||||
|
||||
## Perf Measurement
|
||||
|
||||
Look for `Pixel data generated successfully in xxxx seconds` in console output. With warmup enabled, use the line containing `warmup excluded` for accurate timing.
|
||||
|
||||
## Env Vars
|
||||
|
||||
Defined in `envs.py` (300+ vars). Key ones:
|
||||
- `SGLANG_DIFFUSION_ATTENTION_BACKEND` — attention backend override
|
||||
- `SGLANG_CACHE_DIT_ENABLED` — enable Cache-DiT acceleration
|
||||
- `SGLANG_CLOUD_STORAGE_TYPE` — cloud output storage (s3, etc.)
|
||||
Reference in New Issue
Block a user