diff --git a/.github/MAINTAINER.md b/.github/MAINTAINER.md index 7476d5ab7..634f11cdf 100644 --- a/.github/MAINTAINER.md +++ b/.github/MAINTAINER.md @@ -25,13 +25,13 @@ __Note__: Difference between Merge Oncall and Codeowner - The Codeowner is a passive protection role provided by GitHub; it prevents accidental changes to critical code. - The list of Merge Oncalls is attached below. The list of Codeowners is in the [CODEOWNERS](./CODEOWNERS) file. -__Note__: The permissions to trigger CI tests are defined separately according to these [rules](https://docs.sglang.ai/developer_guide/contribution_guide.html#how-to-trigger-ci-tests). +__Note__: The permissions to trigger CI tests are defined separately according to these [rules](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests). ## Pull Request Merge Process 1. The author submits a pull request (PR) and fills out the PR checklist. 2. A bot assigns this PR to a Merge Oncall and @-mentions them. At the same time, GitHub will automatically request reviews from Codeowners. -3. Someone tags the PR with a `run-ci` label ([help](https://docs.sglang.ai/developer_guide/contribution_guide.html#how-to-trigger-ci-tests)). Then the author can trigger CI by pushing new commits. +3. Someone tags the PR with a `run-ci` label ([help](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests)). Then the author can trigger CI by pushing new commits. 4. The Merge Oncall coordinates the review (e.g., asking people to review) and approves the PR; the Codeowners also approve the PR. If the assigned Merge Oncall is not responsive, the author can ping other related Merge Oncalls and Reviewers in the list below. 5. The code can now be merged: - **Ideal case:** For each modified file, one Codeowner has approved the PR. The PR has also passed the required CI tests. Then, anyone with write permission can merge the PR. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 940807b88..7696bc51b 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -18,9 +18,9 @@ ## Checklist -- [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). -- [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). -- [ ] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). -- [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed). -- [ ] Follow the SGLang code style [guidance](https://docs.sglang.ai/developer_guide/contribution_guide.html#code-style-guidance). +- [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). +- [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). +- [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). +- [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). +- [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). - [ ] Work with maintainers to merge your PR. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process) diff --git a/README.md b/README.md index a9cd859fd..3ae194934 100644 --- a/README.md +++ b/README.md @@ -14,8 +14,9 @@ | [**Blog**](https://lmsys.org/blog/) | [**Documentation**](https://docs.sglang.io/) -| [**Join Slack**](https://slack.sglang.io/) | [**Roadmap**](https://roadmap.sglang.io/) +| [**Join Slack**](https://slack.sglang.io/) +| [**Weekly Dev Meeting**](https://meet.sglang.io/) | [**Slides**](https://github.com/sgl-project/sgl-learning-materials?tab=readme-ov-file#slides) | ## News @@ -60,11 +61,11 @@ Its core features include: - **Active Community**: SGLang is open-source and supported by a vibrant community with widespread industry adoption, powering over 400,000 GPUs worldwide. ## Getting Started -- [Install SGLang](https://docs.sglang.ai/get_started/install.html) -- [Quick Start](https://docs.sglang.ai/basic_usage/send_request.html) -- [Backend Tutorial](https://docs.sglang.ai/basic_usage/openai_api_completions.html) -- [Frontend Tutorial](https://docs.sglang.ai/references/frontend/frontend_tutorial.html) -- [Contribution Guide](https://docs.sglang.ai/developer_guide/contribution_guide.html) +- [Install SGLang](https://docs.sglang.io/get_started/install.html) +- [Quick Start](https://docs.sglang.io/basic_usage/send_request.html) +- [Backend Tutorial](https://docs.sglang.io/basic_usage/openai_api_completions.html) +- [Frontend Tutorial](https://docs.sglang.io/references/frontend/frontend_tutorial.html) +- [Contribution Guide](https://docs.sglang.io/developer_guide/contribution_guide.html) ## Benchmark and Performance Learn more in the release blogs: [v0.2 blog](https://lmsys.org/blog/2024-07-25-sglang-llama3/), [v0.3 blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/), [v0.4 blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/), [Large-scale expert parallelism](https://lmsys.org/blog/2025-05-05-large-scale-ep/), [GB200 rack-scale parallelism](https://lmsys.org/blog/2025-09-25-gb200-part-2/). diff --git a/benchmark/deepseek_v3/README.md b/benchmark/deepseek_v3/README.md index c0dbc6db3..dfd537586 100644 --- a/benchmark/deepseek_v3/README.md +++ b/benchmark/deepseek_v3/README.md @@ -4,7 +4,7 @@ The SGLang and DeepSeek teams collaborated to get DeepSeek V3 FP8 running on NVI Special thanks to Meituan's Search & Recommend Platform Team and Baseten's Model Performance Team for implementing the model, and DataCrunch for providing GPU resources. -For optimizations made on the DeepSeek series models regarding SGLang, please refer to [DeepSeek Model Optimizations in SGLang](https://docs.sglang.ai/basic_usage/deepseek.html). +For optimizations made on the DeepSeek series models regarding SGLang, please refer to [DeepSeek Model Optimizations in SGLang](https://docs.sglang.io/basic_usage/deepseek.html). ## Installation & Launch diff --git a/docs/README.md b/docs/README.md index 2edb2c619..2cad1335c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -51,5 +51,5 @@ pre-commit run --all-files - Keep in mind the documentation execution time when writing interactive Jupyter notebooks. Each interactive notebook will be run and compiled against every commit to ensure they are runnable, so it is important to apply some tips to reduce the documentation compilation time: - Use small models (e.g., `qwen/qwen2.5-0.5b-instruct`) for most cases to reduce server launch time. - Reuse the launched server as much as possible to reduce server launch time. -- Do not use absolute links (e.g., `https://docs.sglang.ai/get_started/install.html`). Always prefer relative links (e.g., `../get_started/install.md`). +- Do not use absolute links (e.g., `https://docs.sglang.io/get_started/install.html`). Always prefer relative links (e.g., `../get_started/install.md`). - Follow the existing examples to learn how to launch a server, send a query and other common styles. diff --git a/docs/advanced_features/hicache_design.md b/docs/advanced_features/hicache_design.md index 226617d4d..b775cef5a 100644 --- a/docs/advanced_features/hicache_design.md +++ b/docs/advanced_features/hicache_design.md @@ -99,7 +99,7 @@ However, because GPU KV computation is naturally performed layer by layer, the G ### Integration with PD-Disaggregation Deployment Mode -SGLang supports a PD (Prefill-Decode) disaggregation deployment mode through the Mooncake TransferEngine (for details, see [this doc](https://docs.sglang.ai/advanced_features/pd_disaggregation.html)). In the PD-disaggregation deployment mode, HiCache can be enabled on both the prefill nodes and decode nodes to optimize prefill performance. If enabled on decode nodes, the decode output will also be written back to L3. +SGLang supports a PD (Prefill-Decode) disaggregation deployment mode through the Mooncake TransferEngine (for details, see [this doc](https://docs.sglang.io/advanced_features/pd_disaggregation.html)). In the PD-disaggregation deployment mode, HiCache can be enabled on both the prefill nodes and decode nodes to optimize prefill performance. If enabled on decode nodes, the decode output will also be written back to L3. ### Unified Interfaces and Rich L3 Storage Backends diff --git a/docs/advanced_features/pd_disaggregation.md b/docs/advanced_features/pd_disaggregation.md index 2c74b77d8..ec437ecb6 100644 --- a/docs/advanced_features/pd_disaggregation.md +++ b/docs/advanced_features/pd_disaggregation.md @@ -19,7 +19,7 @@ Currently, we support Mooncake and NIXL as the transfer engine. ## Profiling in PD Disaggregation Mode -When you need to profile prefill or decode workers in PD disaggregation mode, please refer to the [Profile In PD Disaggregation Mode](https://docs.sglang.ai/developer_guide/benchmark_and_profiling.html#profile-in-pd-disaggregation-mode) section in the Benchmark and Profiling guide. Due to torch profiler limitations, prefill and decode workers must be profiled separately using dedicated command-line options. +When you need to profile prefill or decode workers in PD disaggregation mode, please refer to the [Profile In PD Disaggregation Mode](https://docs.sglang.io/developer_guide/benchmark_and_profiling.html#profile-in-pd-disaggregation-mode) section in the Benchmark and Profiling guide. Due to torch profiler limitations, prefill and decode workers must be profiled separately using dedicated command-line options. ## Router Integration diff --git a/docs/advanced_features/server_arguments.md b/docs/advanced_features/server_arguments.md index ed68bf965..197e5f298 100644 --- a/docs/advanced_features/server_arguments.md +++ b/docs/advanced_features/server_arguments.md @@ -51,7 +51,7 @@ You can find all arguments by `python3 -m sglang.launch_server --help` python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --chunked-prefill-size 4096 ``` -- To enable `torch.compile` acceleration, add `--enable-torch-compile`. It accelerates small models on small batch sizes. By default, the cache path is located at `/tmp/torchinductor_root`, you can customize it using environment variable `TORCHINDUCTOR_CACHE_DIR`. For more details, please refer to [PyTorch official documentation](https://pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html) and [Enabling cache for torch.compile](https://docs.sglang.ai/references/torch_compile_cache.html). +- To enable `torch.compile` acceleration, add `--enable-torch-compile`. It accelerates small models on small batch sizes. By default, the cache path is located at `/tmp/torchinductor_root`, you can customize it using environment variable `TORCHINDUCTOR_CACHE_DIR`. For more details, please refer to [PyTorch official documentation](https://pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html) and [Enabling cache for torch.compile](https://docs.sglang.io/references/torch_compile_cache.html). - To enable torchao quantization, add `--torchao-config int4wo-128`. It supports other [quantization strategies (INT8/FP8)](https://github.com/sgl-project/sglang/blob/v0.3.6/python/sglang/srt/server_args.py#L671) as well. - To enable fp8 weight quantization, add `--quantization fp8` on a fp16 checkpoint or directly load a fp8 checkpoint without specifying any arguments. - To enable fp8 kv cache quantization, add `--kv-cache-dtype fp8_e5m2`. diff --git a/docs/basic_usage/deepseek_v3.md b/docs/basic_usage/deepseek_v3.md index b364c733f..285a318fe 100644 --- a/docs/basic_usage/deepseek_v3.md +++ b/docs/basic_usage/deepseek_v3.md @@ -153,7 +153,7 @@ python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --tru The precompilation process typically takes around 10 minutes to complete. ### Multi-token Prediction -**Description**: SGLang implements DeepSeek V3 Multi-Token Prediction (MTP) based on [EAGLE speculative decoding](https://docs.sglang.ai/advanced_features/speculative_decoding.html#EAGLE-Decoding). With this optimization, the decoding speed can be improved by **1.8x** for batch size 1 and **1.5x** for batch size 32 respectively on H200 TP8 setting. +**Description**: SGLang implements DeepSeek V3 Multi-Token Prediction (MTP) based on [EAGLE speculative decoding](https://docs.sglang.io/advanced_features/speculative_decoding.html#EAGLE-Decoding). With this optimization, the decoding speed can be improved by **1.8x** for batch size 1 and **1.5x** for batch size 32 respectively on H200 TP8 setting. **Usage**: Add arguments `--speculative-algorithm`, `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` to enable this feature. For example: @@ -176,7 +176,7 @@ python3 -m sglang.launch_server \ ### Reasoning Content for DeepSeek R1 & V3.1 -See [Reasoning Parser](https://docs.sglang.ai/advanced_features/separate_reasoning.html) and [Thinking Parameter for DeepSeek V3.1](https://docs.sglang.ai/basic_usage/openai_api_completions.html#Example:-DeepSeek-V3-Models). +See [Reasoning Parser](https://docs.sglang.io/advanced_features/separate_reasoning.html) and [Thinking Parameter for DeepSeek V3.1](https://docs.sglang.io/basic_usage/openai_api_completions.html#Example:-DeepSeek-V3-Models). ### Function calling for DeepSeek Models diff --git a/docs/basic_usage/deepseek_v32.md b/docs/basic_usage/deepseek_v32.md index 2ef73fad7..229c7c5a1 100644 --- a/docs/basic_usage/deepseek_v32.md +++ b/docs/basic_usage/deepseek_v32.md @@ -59,7 +59,7 @@ python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 - B200: `flashmla_auto` prefill attention (short-seq prefill uses MHA via TRT-LLM ragged), `flashmla_kv` decode attention, `fp8_e4m3` kv cache dtype. `flashmla_auto` enables automatic selection of either `flashmla_sparse` or `flashmla_kv` kernel for prefill based on KV cache dtype, hardware, and heuristics. When FP8 KV cache is enabled and `total_kv_tokens < total_q_tokens * 512`, it uses the `flashmla_sparse` kernel; otherwise, it falls back to the `flashmla_kv` kernel. The heuristics may need to be tuned if the performance of either the `flashmla_sparse` or `flashmla_kv` kernel changes significantly. ## Multi-token Prediction -SGLang implements Multi-Token Prediction (MTP) for DeepSeek V3.2 based on [EAGLE speculative decoding](https://docs.sglang.ai/advanced_features/speculative_decoding.html#EAGLE-Decoding). With this optimization, the decoding speed can be improved significantly on small batch sizes. Please look at [this PR](https://github.com/sgl-project/sglang/pull/11652) for more information. +SGLang implements Multi-Token Prediction (MTP) for DeepSeek V3.2 based on [EAGLE speculative decoding](https://docs.sglang.io/advanced_features/speculative_decoding.html#EAGLE-Decoding). With this optimization, the decoding speed can be improved significantly on small batch sizes. Please look at [this PR](https://github.com/sgl-project/sglang/pull/11652) for more information. Example usage: ```bash @@ -70,7 +70,7 @@ python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp ## Function Calling and Reasoning Parser -The usage of function calling and reasoning parser is the same as DeepSeek V3.1. Please refer to [Reasoning Parser](https://docs.sglang.ai/advanced_features/separate_reasoning.html) and [Tool Parser](https://docs.sglang.ai/advanced_features/tool_parser.html) documents. +The usage of function calling and reasoning parser is the same as DeepSeek V3.1. Please refer to [Reasoning Parser](https://docs.sglang.io/advanced_features/separate_reasoning.html) and [Tool Parser](https://docs.sglang.io/advanced_features/tool_parser.html) documents. ## PD Disaggregation diff --git a/docs/basic_usage/llama4.md b/docs/basic_usage/llama4.md index b91720818..1a2338a3f 100644 --- a/docs/basic_usage/llama4.md +++ b/docs/basic_usage/llama4.md @@ -36,7 +36,7 @@ python3 -m sglang.launch_server \ ### EAGLE Speculative Decoding -**Description**: SGLang has supported Llama 4 Maverick (400B) with [EAGLE speculative decoding](https://docs.sglang.ai/advanced_features/speculative_decoding.html#EAGLE-Decoding). +**Description**: SGLang has supported Llama 4 Maverick (400B) with [EAGLE speculative decoding](https://docs.sglang.io/advanced_features/speculative_decoding.html#EAGLE-Decoding). **Usage**: Add arguments `--speculative-draft-model-path`, `--speculative-algorithm`, `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` to enable this feature. For example: diff --git a/docs/basic_usage/qwen3.md b/docs/basic_usage/qwen3.md index c68a304b0..5e6773c0d 100644 --- a/docs/basic_usage/qwen3.md +++ b/docs/basic_usage/qwen3.md @@ -15,7 +15,7 @@ python3 -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct --tp 4 - `--mamba-ssm-dtype`: `bfloat16` or `float32`, use `bfloat16` to save mamba cache size and `float32` to get more accurate results. The default setting is `float32`. ### EAGLE Speculative Decoding -**Description**: SGLang has supported Qwen3-Next models with [EAGLE speculative decoding](https://docs.sglang.ai/advanced_features/speculative_decoding.html#EAGLE-Decoding). +**Description**: SGLang has supported Qwen3-Next models with [EAGLE speculative decoding](https://docs.sglang.io/advanced_features/speculative_decoding.html#EAGLE-Decoding). **Usage**: Add arguments `--speculative-algorithm`, `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` to enable this feature. For example: diff --git a/docs/platforms/amd_gpu.md b/docs/platforms/amd_gpu.md index ea093175d..4b9474940 100644 --- a/docs/platforms/amd_gpu.md +++ b/docs/platforms/amd_gpu.md @@ -100,7 +100,7 @@ The steps below show how to build and use an image. --port 30000 ``` -4. To verify the utility, you can run a benchmark in another terminal or refer to [other docs](https://docs.sglang.ai/basic_usage/openai_api_completions.html) to send requests to the engine. +4. To verify the utility, you can run a benchmark in another terminal or refer to [other docs](https://docs.sglang.io/basic_usage/openai_api_completions.html) to send requests to the engine. ```bash drun sglang_image \ diff --git a/docs/platforms/cpu_server.md b/docs/platforms/cpu_server.md index 5b86c8288..71be9f6f0 100644 --- a/docs/platforms/cpu_server.md +++ b/docs/platforms/cpu_server.md @@ -173,7 +173,7 @@ python -m sglang.bench_serving -h ``` Additionally, the requests can be formed with -[OpenAI Completions API](https://docs.sglang.ai/basic_usage/openai_api_completions.html) +[OpenAI Completions API](https://docs.sglang.io/basic_usage/openai_api_completions.html) and sent via the command line (e.g. using `curl`) or via your own script. ## Example: Running DeepSeek-V3.1-Terminus diff --git a/docs/platforms/nvidia_jetson.md b/docs/platforms/nvidia_jetson.md index 7451cfbd0..ba3b68ae8 100644 --- a/docs/platforms/nvidia_jetson.md +++ b/docs/platforms/nvidia_jetson.md @@ -49,7 +49,7 @@ python -m sglang.launch_server \ ``` The quantization and limited context length (`--dtype half --context-length 8192`) are due to the limited computational resources in [Nvidia jetson kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/). A detailed explanation can be found in [Server Arguments](../advanced_features/server_arguments.md). -After launching the engine, refer to [Chat completions](https://docs.sglang.ai/basic_usage/openai_api_completions.html#Usage) to test the usability. +After launching the engine, refer to [Chat completions](https://docs.sglang.io/basic_usage/openai_api_completions.html#Usage) to test the usability. * * * * * Running quantization with TorchAO ------------------------------------- diff --git a/docs/platforms/xpu.md b/docs/platforms/xpu.md index 099cc413e..88fa1552c 100644 --- a/docs/platforms/xpu.md +++ b/docs/platforms/xpu.md @@ -88,5 +88,5 @@ python -m sglang.bench_serving -h ``` Additionally, the requests can be formed with -[OpenAI Completions API](https://docs.sglang.ai/basic_usage/openai_api_completions.html) +[OpenAI Completions API](https://docs.sglang.io/basic_usage/openai_api_completions.html) and sent via the command line (e.g. using `curl`) or via your own script. diff --git a/docs/references/frontend/frontend_tutorial.ipynb b/docs/references/frontend/frontend_tutorial.ipynb index 1fb48972f..166f8cacc 100644 --- a/docs/references/frontend/frontend_tutorial.ipynb +++ b/docs/references/frontend/frontend_tutorial.ipynb @@ -385,7 +385,7 @@ "## Multi-modal Generation\n", "\n", "You may use SGLang frontend language to define multi-modal prompts.\n", - "See [here](https://docs.sglang.ai/supported_models/generative_models.html) for supported models." + "See [here](https://docs.sglang.io/supported_models/generative_models.html) for supported models." ] }, { diff --git a/docs/references/learn_more.md b/docs/references/learn_more.md index e61c24f22..f0d6ffb8b 100644 --- a/docs/references/learn_more.md +++ b/docs/references/learn_more.md @@ -1,8 +1,9 @@ # Learn More and Join the Community -- The development roadmap: https://roadmap.sglang.io +- The development roadmap: [https://roadmap.sglang.io](https://roadmap.sglang.io) +- Join weekly public development meeting: [https://meet.sglang.io](https://meet.sglang.io) +- Join Slack: [https://slack.sglang.io/](https://slack.sglang.io/) +- Follow on X (formerly Twitter): [https://x.com/lmsysorg](https://x.com/lmsysorg) +- Follow on LinkedIn: [https://www.linkedin.com/company/sgl-project/](https://www.linkedin.com/company/sgl-project/) - The latest SGLang features and updates are shared through the [LMSYS blog](https://lmsys.org/blog/) -- X (formerly Twitter): https://x.com/lmsysorg -- LinkedIn: https://www.linkedin.com/company/sgl-project/ -- Join Slack: https://slack.sglang.io/ - More blogs, slides, and videos about SGLang at [https://github.com/sgl-project/sgl-learning-materials](https://github.com/sgl-project/sgl-learning-materials) diff --git a/docs/references/multi_node_deployment/multi_node.md b/docs/references/multi_node_deployment/multi_node.md index b9d492c62..e6e5b5344 100644 --- a/docs/references/multi_node_deployment/multi_node.md +++ b/docs/references/multi_node_deployment/multi_node.md @@ -30,7 +30,7 @@ python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instr ## DeepSeek V3/R1 -Please refer to [DeepSeek documents for reference](https://docs.sglang.ai/basic_usage/deepseek.html#running-examples-on-multi-node). +Please refer to [DeepSeek documents for reference](https://docs.sglang.io/basic_usage/deepseek.html#running-examples-on-multi-node). ## Multi-Node Inference on SLURM @@ -95,6 +95,6 @@ echo "[INFO] $HEAD_NODE:30000 is ready to accept connections" wait ``` -Then, you can test the server by sending requests following other [documents](https://docs.sglang.ai/basic_usage/openai_api_completions.html). +Then, you can test the server by sending requests following other [documents](https://docs.sglang.io/basic_usage/openai_api_completions.html). Thanks for [aflah02](https://github.com/aflah02) for providing the example, based on his [blog post](https://aflah02.substack.com/p/multi-node-llm-inference-with-sglang). diff --git a/docs/supported_models/multimodal_language_models.md b/docs/supported_models/multimodal_language_models.md index 90c877518..8aa642d53 100644 --- a/docs/supported_models/multimodal_language_models.md +++ b/docs/supported_models/multimodal_language_models.md @@ -11,7 +11,7 @@ python3 -m sglang.launch_server \ --port 30000 \ ``` -> See the [OpenAI APIs section](https://docs.sglang.ai/basic_usage/openai_api_vision.html) for how to send multimodal requests. +> See the [OpenAI APIs section](https://docs.sglang.io/basic_usage/openai_api_vision.html) for how to send multimodal requests. ## Supported models diff --git a/docs/supported_models/support_new_models.md b/docs/supported_models/support_new_models.md index 511a8f398..b71e06c47 100644 --- a/docs/supported_models/support_new_models.md +++ b/docs/supported_models/support_new_models.md @@ -79,7 +79,7 @@ ONLY_RUN=Qwen/Qwen2-1.5B python3 -m unittest test_generation_models.TestGenerati ### Benchmark -- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow https://docs.sglang.ai/developer_guide/benchmark_and_profiling.html to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer). +- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow https://docs.sglang.io/developer_guide/benchmark_and_profiling.html to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer). - **(Optional) Other evals**: If you ran other evals, please note the results in PR description. ## Port a Model from vLLM to SGLang diff --git a/examples/runtime/README.md b/examples/runtime/README.md index 09344d466..8b623fc34 100644 --- a/examples/runtime/README.md +++ b/examples/runtime/README.md @@ -16,12 +16,12 @@ The below examples will mostly need you to start a server in a separate terminal ## Engine -The `engine` folder contains that examples that show how to use [Offline Engine API](https://docs.sglang.ai/basic_usage/offline_engine_api.html#Offline-Engine-API) for common workflows. +The `engine` folder contains that examples that show how to use [Offline Engine API](https://docs.sglang.io/basic_usage/offline_engine_api.html#Offline-Engine-API) for common workflows. * `custom_server.py`: An example how to deploy a custom server. * `embedding.py`: An example how to extract embeddings. * `launch_engine.py`: An example how to launch the Engine. -* `offline_batch_inference_eagle.py`: An example how to perform speculative decoding using [EAGLE](https://docs.sglang.ai/advanced_features/speculative_decoding.html). +* `offline_batch_inference_eagle.py`: An example how to perform speculative decoding using [EAGLE](https://docs.sglang.io/advanced_features/speculative_decoding.html). * `offline_batch_inference_torchrun.py`: An example how to perform inference using [torchrun](https://pytorch.org/docs/stable/elastic/run.html). * `offline_batch_inference_vlm.py`: An example how to use VLMs with the engine. * `offline_batch_inference.py`: An example how to use the engine to perform inference on a batch of examples. diff --git a/python/pyproject_cpu.toml b/python/pyproject_cpu.toml index ed542a8a6..71c98953e 100644 --- a/python/pyproject_cpu.toml +++ b/python/pyproject_cpu.toml @@ -1,4 +1,4 @@ -# https://docs.sglang.ai/platforms/cpu_server.html +# https://docs.sglang.io/platforms/cpu_server.html [build-system] requires = ["setuptools>=61.0", "wheel"] build-backend = "setuptools.build_meta" diff --git a/python/pyproject_other.toml b/python/pyproject_other.toml index dcf88b103..e81c0f22b 100755 --- a/python/pyproject_other.toml +++ b/python/pyproject_other.toml @@ -84,7 +84,7 @@ srt_hip = [ "wave-lang==3.8.2", ] -# https://docs.sglang.ai/platforms/ascend_npu.html +# https://docs.sglang.io/platforms/ascend_npu.html srt_npu = ["sglang[runtime_common]"] # For Intel Gaudi(device : hpu) follow the installation guide diff --git a/python/sglang/srt/mem_cache/storage/mooncake_store/README.md b/python/sglang/srt/mem_cache/storage/mooncake_store/README.md index a70dbe56c..dc2be6db4 100644 --- a/python/sglang/srt/mem_cache/storage/mooncake_store/README.md +++ b/python/sglang/srt/mem_cache/storage/mooncake_store/README.md @@ -6,7 +6,7 @@ Related documentation: * [Quick Start: SGLang HiCache with Mooncake Backend](https://kvcache-ai.github.io/Mooncake/getting_started/examples/sglang-integration/hicache-quick-start.html) * [Complete Guide: SGLang HiCache with Mooncake Backend](https://kvcache-ai.github.io/Mooncake/getting_started/examples/sglang-integration/hicache-integration-v1.html) * [Mooncake x SGLang HiCache System Design](https://kvcache-ai.github.io/Mooncake/design/hicache-design.html) -* [HiCache System Design and Optimization](https://docs.sglang.ai/advanced_features/hicache_design.html) +* [HiCache System Design and Optimization](https://docs.sglang.io/advanced_features/hicache_design.html) * [SGLang HiCache with Mooncake Backend Benchmark](https://kvcache-ai.github.io/Mooncake/performance/sglang-hicache-benchmark-results-v1.html) ## About Mooncake @@ -252,7 +252,7 @@ In particular, for the `global segment size`, if at least one `store service` in **HiCache Related Parameters for SGLang Server** -For a comprehensive overview of HiCache-related parameters, please refer to [this document](https://docs.sglang.ai/advanced_features/hicache_design.html#related-parameters). +For a comprehensive overview of HiCache-related parameters, please refer to [this document](https://docs.sglang.io/advanced_features/hicache_design.html#related-parameters). Note that, for `--hicache-mem-layout {layer_first,page_first,page_first_direct}`, which specifies the memory layout for the host memory pool, `page_first` or `page_first_direct` are required if use Mooncake backend. diff --git a/python/sglang/srt/sampling/sampling_params.py b/python/sglang/srt/sampling/sampling_params.py index e367a4865..34e5252c8 100644 --- a/python/sglang/srt/sampling/sampling_params.py +++ b/python/sglang/srt/sampling/sampling_params.py @@ -28,7 +28,7 @@ class SamplingParams: The sampling parameters. See docs/backend/sampling_params.md or - https://docs.sglang.ai/backend/sampling_params.html + https://docs.sglang.io/backend/sampling_params.html for the documentation. """ diff --git a/sgl-router/README.md b/sgl-router/README.md index e1f89173a..780df2444 100644 --- a/sgl-router/README.md +++ b/sgl-router/README.md @@ -39,7 +39,7 @@ High-performance model routing control and data plane for large-scale LLM deploy - Prometheus metrics and structured tracing for every stage of routing. ## Documentation -- **User Guide**: [docs.sglang.ai/advanced_features/router.html](https://docs.sglang.ai/advanced_features/router.html) +- **User Guide**: [docs.sglang.io/advanced_features/router.html](https://docs.sglang.io/advanced_features/router.html) - Additional guides, API references, and deployment patterns are continuously updated alongside SGLang releases. ## Installation diff --git a/sgl-router/tests/metrics_aggregator_test.rs b/sgl-router/tests/metrics_aggregator_test.rs index fa3a78897..b8797180f 100644 --- a/sgl-router/tests/metrics_aggregator_test.rs +++ b/sgl-router/tests/metrics_aggregator_test.rs @@ -91,7 +91,7 @@ valid_metric{source="worker2"} 123 fn test_real() { let pack1 = MetricPack { labels: vec![("source".to_string(), "worker1".to_string())], - // https://docs.sglang.ai/references/production_metrics.html + // https://docs.sglang.io/references/production_metrics.html metrics_text: r###"# HELP sglang:prompt_tokens_total Number of prefill tokens processed. # TYPE sglang:prompt_tokens_total counter sglang:prompt_tokens_total{model_name="meta-llama/Llama-3.1-8B-Instruct"} 8.128902e+06