diff --git a/test/README.md b/test/README.md index 17de6bc35..225d2b245 100644 --- a/test/README.md +++ b/test/README.md @@ -32,14 +32,94 @@ python3 test_choices.py - Ensure you added `unittest.main()` for unittest and `sys.exit(pytest.main([__file__]))` for pytest in the scripts. The CI run them via `python3 test_file.py`. - The CI will run some suites such as `per-commit-1-gpu`, `per-commit-2-gpu`, and `nightly-1-gpu` automatically. If you need special setup or custom test groups, you may modify the workflows in [`.github/workflows/`](https://github.com/sgl-project/sglang/tree/main/.github/workflows). -## CI Registry Quick Peek +## CI Registry System -Tests in `test/registered/` declare CI metadata via lightweight markers: +Tests in `test/registered/` use a registry-based CI system for flexible backend/schedule configuration. + +### Registration Functions ```python -from sglang.test.ci.ci_register import register_cuda_ci +from sglang.test.ci.ci_register import ( + register_cuda_ci, + register_amd_ci, + register_cpu_ci, + register_npu_ci, +) -register_cuda_ci(est_time=80, suite="stage-a-test-1") +# Per-commit test (small 1-gpu, runs on 5090) +register_cuda_ci(est_time=80, suite="stage-b-test-small-1-gpu") + +# Per-commit test (large 1-gpu, runs on H100) +register_cuda_ci(est_time=120, suite="stage-b-test-large-1-gpu") + +# Per-commit test (2-gpu) +register_cuda_ci(est_time=200, suite="stage-b-test-large-2-gpu") + +# Nightly-only test +register_cuda_ci(est_time=200, suite="nightly-1-gpu", nightly=True) + +# Multi-backend test +register_cuda_ci(est_time=80, suite="stage-b-test-small-1-gpu") +register_amd_ci(est_time=120, suite="stage-a-test-1") + +# Temporarily disabled test +register_cuda_ci(est_time=80, suite="stage-b-test-small-1-gpu", disabled="flaky - see #12345") +``` + +### Choosing Between 1-GPU Suites (5090 vs H100) + +When adding 1-GPU tests, choose the appropriate suite based on hardware compatibility: + +| Suite | Runner | GPU | When to Use | +|-------|--------|-----|-------------| +| `stage-b-test-small-1-gpu` | `1-gpu-5090` | RTX 5090 (32GB, SM120) | 5090-compatible tests (preferred) | +| `stage-b-test-large-1-gpu` | `1-gpu-runner` | H100 (80GB, SM90) | Large models or 5090-incompatible tests | + +**Use `stage-b-test-small-1-gpu` (5090) whenever possible** - this is the preferred suite for most 1-GPU tests. + +**Use `stage-b-test-large-1-gpu` (H100) if ANY of these apply:** + +1. **Architecture incompatibility (SM120/Blackwell)**: + - FA3 attention backend (requires SM≤90) + - MLA with FA3 backend + - FP8/MXFP4 quantization (not supported on SM120) + - Certain Triton kernels (shared memory limits) + +2. **Memory requirements**: + - Models >30B params or large MoE + - Tests requiring >32GB VRAM + +3. **Known 5090 failures**: + - Weight update/sync tests + - Certain spec decoding tests + +If a test cannot run on 5090 due to any of the above, use `stage-b-test-large-1-gpu` which runs on H100. + +### Available Suites + +**Per-Commit (CUDA)**: +- Stage A: `stage-a-test-1` (locked), `stage-a-test-2`, `stage-a-test-cpu` +- Stage B: `stage-b-test-small-1-gpu` (5090), `stage-b-test-large-1-gpu` (H100), `stage-b-test-large-2-gpu` +- Stage C: `stage-c-test-large-4-gpu`, `stage-c-test-large-4-gpu-b200`, `stage-c-test-large-8-gpu-b200` + +**Per-Commit (AMD)**: +- `stage-a-test-1`, `stage-b-test-small-1-gpu-amd`, `stage-b-test-large-2-gpu-amd` + +**Nightly**: +- `nightly-1-gpu`, `nightly-2-gpu`, `nightly-4-gpu`, `nightly-8-gpu`, etc. + +### Running Tests with run_suite.py + +```bash +# Run per-commit tests +python test/run_suite.py --hw cuda --suite stage-b-test-small-1-gpu + +# Run nightly tests +python test/run_suite.py --hw cuda --suite nightly-1-gpu --nightly + +# With auto-partitioning (for parallel CI jobs) +python test/run_suite.py --hw cuda --suite stage-b-test-small-1-gpu \ + --auto-partition-id 0 --auto-partition-size 4 ``` ## Writing Elegant Test Cases