diff --git a/docs/diffusion/contributing.md b/docs/diffusion/contributing.md index cc83b1b56..7de656100 100644 --- a/docs/diffusion/contributing.md +++ b/docs/diffusion/contributing.md @@ -2,6 +2,14 @@ This guide outlines the requirements for contributing to the SGLang Diffusion module (`sglang.multimodal_gen`). +## On AI-Assisted ("Vibe Coding") PRs + +Vibe-coded PRs are welcome — we judge code quality, not how it was produced. The bar is the same for all PRs: + +- **No over-commenting.** If the name says it all, skip the docstring. +- **No over-catching.** Don't guard against errors that virtually never happen in practice. +- **Test before submitting.** AI-generated code can be subtly wrong — verify correctness end-to-end. + ## Commit Message Convention We follow a structured commit message format to maintain a clean history. diff --git a/python/sglang/multimodal_gen/.claude/.skills/diffusion-perf/SKILL.md b/python/sglang/multimodal_gen/.claude/.skills/diffusion-perf/SKILL.md new file mode 100644 index 000000000..76ae5ff8f --- /dev/null +++ b/python/sglang/multimodal_gen/.claude/.skills/diffusion-perf/SKILL.md @@ -0,0 +1,81 @@ +--- +name: diffusion-perf +description: Measure and compare sglang-diffusion performance. Use when benchmarking a model, comparing before/after performance, or generating a perf report for a PR. +user-invocable: true +allowed-tools: Bash, Read +argument-hint: [--prompt "..."] [--baseline baseline.json] +--- + +# Diffusion Performance Measurement + +Measure sglang-diffusion e2e latency via `--perf-dump-path`, then extract or compare results from the JSON dump. + +## JSON dump structure + +`--perf-dump-path` writes a JSON file with: + +```json +{ + "total_duration_ms": 14959.11, + "steps": [ + {"name": "TextEncodingStage", "duration_ms": 611.83}, + {"name": "DenoisingStage", "duration_ms": 14289.46} + ], + "denoise_steps_ms": [ + {"step": 0, "duration_ms": 240.5}, + {"step": 1, "duration_ms": 279.1} + ], + "commit_hash": "abc123", + "timestamp": "...", + "memory_checkpoints": {} +} +``` + +Key fields: +- `total_duration_ms` — e2e walltime (warmup excluded when `--warmup` is used) +- `steps` — per-stage breakdown +- `denoise_steps_ms` — per denoising step timing + +## Workflow + +### 1. Single measurement + +```bash +sglang generate --model-path $MODEL --prompt "$PROMPT" --warmup --perf-dump-path result.json +``` + +Then read `total_duration_ms` from `result.json`. + +### 2. Before/after comparison + +```bash +# Baseline (on main branch or before changes) +sglang generate --model-path $MODEL --prompt "$PROMPT" --warmup --perf-dump-path baseline.json + +# New (after changes) +sglang generate --model-path $MODEL --prompt "$PROMPT" --warmup --perf-dump-path new.json + +# Compare — outputs a Markdown table suitable for PR descriptions +python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json new.json +``` + +### 3. Extracting a single number + +To get e2e latency in seconds from a dump: + +```bash +python3 -c "import json; print(f\"{json.load(open('result.json'))['total_duration_ms']/1000:.2f}\")" +``` + +## Arguments + +If `$ARGUMENTS` is provided, parse it as: +- First positional arg → `--model-path` +- `--prompt "..."` → generation prompt (default: `"A curious raccoon"`) +- `--baseline ` → if given, run comparison against this baseline file + +## Notes + +- Always use `--warmup` for accurate timing (excludes CUDA warmup from measurement). +- Keep `--prompt` and all server/sampling args identical between baseline and new runs. +- For PR descriptions, paste the output of `compare_perf.py` directly. diff --git a/python/sglang/multimodal_gen/.claude/CLAUDE.md b/python/sglang/multimodal_gen/.claude/CLAUDE.md new file mode 100644 index 000000000..26cb86ae6 --- /dev/null +++ b/python/sglang/multimodal_gen/.claude/CLAUDE.md @@ -0,0 +1,108 @@ +# CLAUDE.md — sglang-diffusion (multimodal_gen) + +## What is this? + +SGLang's diffusion/multimodal generation subsystem. Separate from the LLM runtime (`srt`). Supports 20+ image/video diffusion models (Wan, FLUX, HunyuanVideo, LTX, Qwen-Image, etc.) with distributed inference, LoRA, and multiple attention backends. + +## Quick Start + +```bash +# One-shot generation +sglang generate --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers --prompt "A curious raccoon" --save-output + +# Start server +sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers --num-gpus 4 + +# Python API +from sglang import DiffGenerator +gen = DiffGenerator.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers") +result = gen.generate(sampling_params_kwargs={"prompt": "A curious raccoon"}) +``` + +## Architecture + +``` +CLI / Python API / HTTP Server (FastAPI + OpenAI-compatible) + ↓ ZMQ +Scheduler (request queue, batching, dispatch) + ↓ multiprocessing pipes +GPU Worker(s) → ComposedPipeline (stages: TextEncode → Denoise → Decode) +``` + +### Key Directories + +``` +runtime/ +├── entrypoints/ # CLI (generate/serve), HTTP server, Python API (DiffGenerator) +├── managers/ # scheduler.py, gpu_worker.py +├── pipelines_core/ # ComposedPipelineBase, stages/, schedule_batch.py (Req/OutputBatch) +├── pipelines/ # Model-specific pipelines (wan, flux, hunyuan, ltx, qwen_image, ...) +├── models/ # encoders/, dits/, vaes/, schedulers/ +├── layers/ # attention/, lora/, quantization/ +├── loader/ # Model loading, weight utils +├── server_args.py # ServerArgs (all CLI/config params) +└── distributed/ # Multi-GPU (TP, SP: ulysses/ring) +configs/ +├── pipeline_configs/ # Per-model pipeline configs +├── sample/ # SamplingParams +└── models/ # DiT, VAE, Encoder configs +``` + +### Key Classes + +| Class | Location | Purpose | +|-------|----------|---------| +| `DiffGenerator` | `runtime/entrypoints/diffusion_generator.py` | Python API entry point | +| `ComposedPipelineBase` | `runtime/pipelines_core/composed_pipeline_base.py` | Pipeline orchestrator (stages) | +| `Scheduler` | `runtime/managers/scheduler.py` | ZMQ event loop, request dispatch | +| `GPUWorker` | `runtime/managers/gpu_worker.py` | GPU inference worker | +| `Req` / `OutputBatch` | `runtime/pipelines_core/schedule_batch.py` | Request/output containers | +| `ServerArgs` | `runtime/server_args.py` | All config params | +| `SamplingParams` | `configs/sample/sampling_params.py` | Generation params | +| `PipelineConfig` | `configs/pipeline_configs/base.py` | Model structure config | + +### Key Functions + +| Function | Module | Purpose | +|----------|--------|---------| +| `build_pipeline()` | `runtime/pipelines_core/__init__.py` | Instantiate pipeline from model_path | +| `get_model_info()` | `registry.py` | Resolve pipeline + config classes | +| `launch_server()` | `runtime/launch_server.py` | Start multi-process server | + +## Adding a New Model + +1. Create pipeline in `runtime/pipelines/` extending `ComposedPipelineBase` +2. Define stages via `create_pipeline_stages()` (TextEncoding → Denoising → Decoding) +3. Add config in `configs/pipeline_configs/` +4. Register in `registry.py` via `register_configs()` + +## Multi-GPU + +```bash +# Sequence parallelism (video frames across GPUs) +sglang serve --model-path ... --num-gpus 4 --ulysses-degree 2 --ring-degree 2 + +# Tensor parallelism (model layers across GPUs) +sglang serve --model-path ... --num-gpus 2 --tp-size 2 +``` + +## Testing + +```bash +# Tests live in test/ subdirectory +python -m pytest python/sglang/multimodal_gen/test/ + +# No need to pre-download models — auto-downloaded at runtime +# Dependencies assumed already installed via `pip install -e "python[diffusion]"` +``` + +## Perf Measurement + +Look for `Pixel data generated successfully in xxxx seconds` in console output. With warmup enabled, use the line containing `warmup excluded` for accurate timing. + +## Env Vars + +Defined in `envs.py` (300+ vars). Key ones: +- `SGLANG_DIFFUSION_ATTENTION_BACKEND` — attention backend override +- `SGLANG_CACHE_DIT_ENABLED` — enable Cache-DiT acceleration +- `SGLANG_CLOUD_STORAGE_TYPE` — cloud output storage (s3, etc.)