diff --git a/docs/advanced_features/dp_for_multi_modal_encoder.md b/docs/advanced_features/dp_for_multi_modal_encoder.md index 62057f958..a100e0688 100644 --- a/docs/advanced_features/dp_for_multi_modal_encoder.md +++ b/docs/advanced_features/dp_for_multi_modal_encoder.md @@ -4,7 +4,7 @@ A typical VLM architecture involves two main components: an multi-modal encoder Most VLMs utilize a Vision Transformer (ViT) as their multi-modal encoder, it is responsible for processing visual data, extracting features (objects, colors, textures, etc.), and transforming them into a format that can be understood by the model. -The text deocoder is based on LLM. It processes textual data and generates output based on the encoded visual features. +The text decoder is based on LLM. It processes textual data and generates output based on the encoded visual features. However, since the size of ViT is very small compared to language decoders, there is relatively little gain from TP. On the other hand, TP incurs significant communication diff --git a/docs/advanced_features/server_arguments.md b/docs/advanced_features/server_arguments.md index 0f1a327c9..cdb5c5122 100644 --- a/docs/advanced_features/server_arguments.md +++ b/docs/advanced_features/server_arguments.md @@ -212,9 +212,9 @@ Please consult the documentation below and [server_args.py](https://github.com/s | `--admin-api-key` | Set **admin API key** for administrative/control endpoints (e.g., weights update, cache flush, `/get_server_info`). Endpoints marked as admin-only require `Authorization: Bearer ` when this is set. | `None` | Type: str | | `--served-model-name` | Override the model name returned by the v1/models endpoint in OpenAI API server. | `None` | Type: str | | `--weight-version` | Version identifier for the model weights. Defaults to 'default' if not specified. | `default` | Type: str | -| `--chat-template` | The buliltin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server. | `None` | Type: str | +| `--chat-template` | The builtin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server. | `None` | Type: str | | `--hf-chat-template-name` | When the HuggingFace tokenizer has multiple chat templates (e.g., 'default', 'tool_use', 'rag'), specify which named template to use. If not set, the first available template is used. | `None` | Type: str | -| `--completion-template` | The buliltin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently. | `None` | Type: str | +| `--completion-template` | The builtin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently. | `None` | Type: str | | `--file-storage-path` | The path of the file storage in backend. | `sglang_storage` | Type: str | | `--enable-cache-report` | Return number of cached tokens in usage.prompt_tokens_details for each openai request. | `False` | bool flag (set to enable) | | `--reasoning-parser` | Specify the parser for reasoning models. Supported parsers: [deepseek-r1, deepseek-v3, glm45, gpt-oss, kimi, qwen3, qwen3-thinking, step3]. | `None` | `deepseek-r1`, `deepseek-v3`, `glm45`, `gpt-oss`, `kimi`, `qwen3`, `qwen3-thinking`, `step3` | diff --git a/docs/basic_usage/deepseek_v3.md b/docs/basic_usage/deepseek_v3.md index a321eb09c..0fd06baed 100644 --- a/docs/basic_usage/deepseek_v3.md +++ b/docs/basic_usage/deepseek_v3.md @@ -223,7 +223,7 @@ Sample Request: ``` curl "http://127.0.0.1:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --d '{"temperature": 0, "max_tokens": 100, "model": "deepseek-ai/DeepSeek-V3-0324", "tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of an city, the user should supply a city first", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city, e.g. Beijing"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "Hows the weather like in Qingdao today"}]}' +-d '{"temperature": 0, "max_tokens": 100, "model": "deepseek-ai/DeepSeek-V3-0324", "tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of a city, the user should supply a city first", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city, e.g. Beijing"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "How'\''s the weather like in Qingdao today"}]}' ``` Expected Response @@ -236,7 +236,7 @@ Sample Streaming Request: ``` curl "http://127.0.0.1:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --d '{"temperature": 0, "max_tokens": 100, "model": "deepseek-ai/DeepSeek-V3-0324","stream":true,"tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of an city, the user should supply a city first", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city, e.g. Beijing"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "Hows the weather like in Qingdao today"}]}' +-d '{"temperature": 0, "max_tokens": 100, "model": "deepseek-ai/DeepSeek-V3-0324","stream":true,"tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of a city, the user should supply a city first", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city, e.g. Beijing"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "How'\''s the weather like in Qingdao today"}]}' ``` Expected Streamed Chunks (simplified for clarity): ``` diff --git a/docs/basic_usage/glmv.md b/docs/basic_usage/glmv.md index c56b6ecd5..ad36cea26 100644 --- a/docs/basic_usage/glmv.md +++ b/docs/basic_usage/glmv.md @@ -133,4 +133,4 @@ python -m sglang.launch_server \ In SGLang, we can implement thinking budget with `CustomLogitProcessor`. -Launch a server with `--enable-custom-logit-processor` flag on. and using `Glm4MoeThinkingBudgetLogitProcessor` in the request likes `GLM-4.6` example in [glm45.md](./glm45.md). +Launch a server with the `--enable-custom-logit-processor` flag. Then, use `Glm4MoeThinkingBudgetLogitProcessor` in the request, similar to the `GLM-4.6` example in [glm45.md](./glm45.md). diff --git a/docs/basic_usage/gpt_oss.md b/docs/basic_usage/gpt_oss.md index f74ba40d9..9f81da4f4 100644 --- a/docs/basic_usage/gpt_oss.md +++ b/docs/basic_usage/gpt_oss.md @@ -105,7 +105,7 @@ print(response.output_text) # Test python tool response = client.responses.create( model="openai/gpt-oss-120b", - instructions="You are a helfpul assistant, you could use python tool to execute code.", + instructions="You are a helpful assistant, you could use python tool to execute code.", input="Use python tool to calculate the sum of 29138749187 and 29138749187", # 58,277,498,374 tools=tools ) @@ -115,7 +115,7 @@ print(response.output_text) # Test browser tool response = client.responses.create( model="openai/gpt-oss-120b", - instructions="You are a helfpul assistant, you could use browser to search the web", + instructions="You are a helpful assistant, you could use browser to search the web", input="Search the web for the latest news about Nvidia stock price", tools=tools ) diff --git a/docs/basic_usage/sampling_params.md b/docs/basic_usage/sampling_params.md index a1848d41d..23415f9af 100644 --- a/docs/basic_usage/sampling_params.md +++ b/docs/basic_usage/sampling_params.md @@ -74,7 +74,7 @@ Please refer to our dedicated guide on [constrained decoding](../advanced_featur | json_schema | `Optional[str] = None` | JSON schema for structured outputs. | | regex | `Optional[str] = None` | Regex for structured outputs. | | ebnf | `Optional[str] = None` | EBNF for structured outputs. | -| structural_tag | `Optional[str] = None` | The structal tag for structured outputs. | +| structural_tag | `Optional[str] = None` | The structural tag for structured outputs. | ### Other options diff --git a/docs/developer_guide/bench_serving.md b/docs/developer_guide/bench_serving.md index b2f8568e2..5a67723c8 100644 --- a/docs/developer_guide/bench_serving.md +++ b/docs/developer_guide/bench_serving.md @@ -332,7 +332,7 @@ python3 -m sglang.bench_serving \ python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ - --model mode-name \ + --model model-name \ --dataset-name mooncake \ --mooncake-slowdown-factor 1.0 \ --mooncake-num-rounds 1000 \ diff --git a/docs/developer_guide/benchmark_and_profiling.md b/docs/developer_guide/benchmark_and_profiling.md index 728bcba3a..b694705a8 100644 --- a/docs/developer_guide/benchmark_and_profiling.md +++ b/docs/developer_guide/benchmark_and_profiling.md @@ -43,7 +43,7 @@ python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile ``` -Please make sure that the `SGLANG_TORCH_PROFILER_DIR` should be set at both server and client side, otherwise the trace file cannot be generated correctly . A secure way will be setting `SGLANG_TORCH_PROFILER_DIR` in the `.*rc` file of shell (e.g. `~/.bashrc` for bash shells). +The `SGLANG_TORCH_PROFILER_DIR` environment variable must be set on both the server and client side; otherwise, the trace file will not be generated correctly. A secure way to do this is by setting it in your shell's resource file (e.g., `~/.bashrc` for bash). For more details, please refer to [Bench Serving Guide](./bench_serving.md). diff --git a/docs/developer_guide/development_guide_using_docker.md b/docs/developer_guide/development_guide_using_docker.md index e38947902..a833011c6 100644 --- a/docs/developer_guide/development_guide_using_docker.md +++ b/docs/developer_guide/development_guide_using_docker.md @@ -55,7 +55,7 @@ Some useful volumes to mount are: 1. **Huggingface model cache**: mounting model cache can avoid re-download every time docker restarts. Default location on Linux is `~/.cache/huggingface/`. 2. **SGLang repository**: code changes in the SGLang local repository will be automatically synced to the .devcontainer. -Example 1: Monting local cache folder `/opt/dlami/nvme/.cache` but not the SGLang repo. Use this when you prefer to manually transfer local code changes to the devcontainer. +Example 1: Mounting local cache folder `/opt/dlami/nvme/.cache` but not the SGLang repo. Use this when you prefer to manually transfer local code changes to the devcontainer. ```bash docker run -itd --shm-size 32g --gpus all -v /opt/dlami/nvme/.cache:/root/.cache --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh docker exec -it sglang_zhyncs /bin/zsh diff --git a/docs/developer_guide/setup_github_runner.md b/docs/developer_guide/setup_github_runner.md index 67d1b0be0..6a0e92cdc 100644 --- a/docs/developer_guide/setup_github_runner.md +++ b/docs/developer_guide/setup_github_runner.md @@ -1,4 +1,4 @@ -# Set Up Self-Hosted Runners for GitHub Action +# Set Up Self-Hosted Runners for GitHub Actions ## Add a Runner diff --git a/docs/platforms/amd_gpu.md b/docs/platforms/amd_gpu.md index 2b3f4ae72..473a9c568 100644 --- a/docs/platforms/amd_gpu.md +++ b/docs/platforms/amd_gpu.md @@ -1,6 +1,6 @@ # AMD GPUs -This document describes how run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues). +This document describes how to run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues). ## System Configuration diff --git a/docs/platforms/ascend_contribution_guide.md b/docs/platforms/ascend_contribution_guide.md index db3431260..a5e8f020f 100644 --- a/docs/platforms/ascend_contribution_guide.md +++ b/docs/platforms/ascend_contribution_guide.md @@ -6,7 +6,7 @@ Welcome to **SGLang**! We appreciate your interest in contributing. This guide p ### Prepare Environment -Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](../platforms/ascend_npu.md) to install the necessary dependencies. we recommend [using docker](../platforms/ascend_npu.md#method-2-using-docker-image) to build the environment. +Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](../platforms/ascend_npu.md) to install the necessary dependencies. We recommend [using docker](../platforms/ascend_npu.md#method-2-using-docker-image) to build the environment. ### Fork and clone the repository diff --git a/docs/platforms/ascend_npu_best_practice.md b/docs/platforms/ascend_npu_best_practice.md index caa182fe3..51ff6ca59 100644 --- a/docs/platforms/ascend_npu_best_practice.md +++ b/docs/platforms/ascend_npu_best_practice.md @@ -1,6 +1,6 @@ # Best Practice on Ascend NPU -This section describes the best practice data of mainstream LLM models such as DeepSeek and Qwen on the Ascend Npu. If +This section describes the best practice data of mainstream LLM models such as DeepSeek and Qwen on the Ascend NPU. If you encounter issues or have any questions, please [open an issue](https://github.com/sgl-project/sglang/issues). ## DeepSeek Series Models diff --git a/docs/references/environment_variables.md b/docs/references/environment_variables.md index 2e0eebc19..3b470b55d 100644 --- a/docs/references/environment_variables.md +++ b/docs/references/environment_variables.md @@ -44,7 +44,7 @@ SGLang supports various environment variables that can be used to configure its | `SGLANG_MM_PRECOMPUTE_HASH` | Enable precomputing of hash values for MultimodalDataItem | `false` | | `SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH` | Enable NCCL for gathering when preparing mlp sync batch under overlap scheduler (without this flag gloo is used for gathering) | `false` | | `SGLANG_SYMM_MEM_PREALLOC_GB_SIZE` | Size of preallocated GPU buffer (in GB) for NCCL symmetric memory pool to limit memory fragmentation. Only have an effect when server arg `--enable-symm-mem` is set. | `4` | -| `SGLANG_CUSTOM_ALLREDUCE_ALGO` | The algorithom of custom all-reduce. Set to `oneshot` or `1stage` to force use one-shot. Set to `twoshot` or `2stage` to force use two-shot. | `` | +| `SGLANG_CUSTOM_ALLREDUCE_ALGO` | The algorithm of custom all-reduce. Set to `oneshot` or `1stage` to force use one-shot. Set to `twoshot` or `2stage` to force use two-shot. | `` | ## DeepGEMM Configuration (Advanced Optimization) diff --git a/docs/references/multi_node_deployment/lws_pd/lws_pd_deploy.md b/docs/references/multi_node_deployment/lws_pd/lws_pd_deploy.md index 368aee34b..419474a4e 100644 --- a/docs/references/multi_node_deployment/lws_pd/lws_pd_deploy.md +++ b/docs/references/multi_node_deployment/lws_pd/lws_pd_deploy.md @@ -638,7 +638,7 @@ kubectl apply -f p.yaml kubectl apply -f d.yaml ``` -At this point, we have completed the deployment of the 1P1D SGlang engine part. +At this point, we have completed the deployment of the 1P1D SGLang engine part. To allow our users to directly experience the model API, we still need a load balancer to handle sequential calls between prefill and decode. Different companies implement LBs differently, and the community will also officially release a new LB component written in Rust in the near future. diff --git a/docs/references/post_training_integration.md b/docs/references/post_training_integration.md index 5e82f8374..4dddf5905 100644 --- a/docs/references/post_training_integration.md +++ b/docs/references/post_training_integration.md @@ -6,7 +6,7 @@ What makes SGLang essential for post-training? - Open-To-Use Refit Functionality: diverse method for colocate or disaggregate - Easy To Postpone Generation: enable partial rollout and dedicated rollout control -- Fine-Grained Engine Sleep And Wake Up: facilitate maxium-powered rollout and training +- Fine-Grained Engine Sleep And Wake Up: facilitate maximum-powered rollout and training - Training Serving Alignment: ensure the performance consistency in training and serving - Load Balancing Router: cache-aware load-balancing for high-throughput rollout - Deterministic Inference: ensure zero kl divergence between rollout and training @@ -28,4 +28,4 @@ These capabilities, combined with native integration support across major framew ## Collaboration -Due to the privacy of the design parternes, we cannot list the companies that adopt SGLang for post-training. However, we are happy to share the details with you if you are interested and trust the choice among 10+ top companies and frontier labs across US and China. If you are interested in integrating SGLang with your training framework or need technical support, we're here to help! Reach out to us at **rl_team@lmsys.org** for partnerships, integration guidance, and custom feature development. +Due to the privacy of the design partners, we cannot list the companies that adopt SGLang for post-training. However, we are happy to share the details with you if you are interested and trust the choice among 10+ top companies and frontier labs across US and China. If you are interested in integrating SGLang with your training framework or need technical support, we're here to help! Reach out to us at **rl_team@lmsys.org** for partnerships, integration guidance, and custom feature development. diff --git a/docs/references/production_request_trace.md b/docs/references/production_request_trace.md index 2d19570c2..a60c68b9e 100644 --- a/docs/references/production_request_trace.md +++ b/docs/references/production_request_trace.md @@ -1,6 +1,6 @@ # Production Request Tracing -SGlang exports request trace data based on the OpenTelemetry Collector. You can enable tracing by adding the `--enable-trace` and configure the OpenTelemetry Collector endpoint using `--otlp-traces-endpoint` when launching the server. +SGLang exports request trace data based on the OpenTelemetry Collector. You can enable tracing by adding the `--enable-trace` and configure the OpenTelemetry Collector endpoint using `--otlp-traces-endpoint` when launching the server. You can find example screenshots of the visualization in https://github.com/sgl-project/sglang/issues/8965. diff --git a/docs/supported_models/mindspore_models.md b/docs/supported_models/mindspore_models.md index 9c1160034..6085e84d3 100644 --- a/docs/supported_models/mindspore_models.md +++ b/docs/supported_models/mindspore_models.md @@ -32,9 +32,9 @@ pip install -e . Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example. -### Offline infer +### Offline inference -Use the following script for offline infer: +Use the following script for offline inference: ```python import sglang as sgl