diff --git a/benchmark/deepseek_v3/README.md b/benchmark/deepseek_v3/README.md index 59459180b..7690410e8 100644 --- a/benchmark/deepseek_v3/README.md +++ b/benchmark/deepseek_v3/README.md @@ -168,6 +168,30 @@ If you have two H100 nodes, the usage is similar to the aforementioned H20. > **Note that the launch command here does not enable Data Parallelism Attention or `torch.compile` Optimization**. For optimal performance, please refer to the command options in [Performance Optimization Options](#option_args). +### Example: Serving with one B200 node + +There is one B200 node with 4 (for FP4) GPUs or 8 (for FP4 or FP8) GPUs. Both FP4 and FP8 models are supported for DeepSeek R1. The flags to achieve optimal performance for each are slightly different. + +#### FP4 + +If using 4 GPUs: + +```bash +python3 -m sglang.launch_server --model-path nvidia/DeepSeek-R1-0528-FP4-V2 --host 0.0.0.0 --port 8000 --tensor-parallel-size=4 --cuda-graph-max-bs 256 --max-running-requests 256 --mem-fraction-static 0.85 --ep-size 4 --scheduler-recv-interval 30 --enable-symm-mem --stream-interval 10 +``` + +If using 8 GPUs: + +```bash +python3 -m sglang.launch_server --model-path nvidia/DeepSeek-R1-0528-FP4-V2 --host 0.0.0.0 --port 8000 --tensor-parallel-size=8 --cuda-graph-max-bs 256 --max-running-requests 256 --mem-fraction-static 0.85 --ep-size 8 --scheduler-recv-interval 30 --enable-symm-mem --stream-interval 10 +``` + +#### FP8 + +``` +SGLANG_ENABLE_JIT_DEEPGEMM=false SGLANG_ENABLE_FLASHINFER_GEMM=true bash python3 -m sglang.launch_server --model-path=deepseek-ai/DeepSeek-R1-0528 --host=0.0.0.0 --port=8000 --tensor-parallel-size=8 --cuda-graph-max-bs 128 --max-running-requests 128 --mem-fraction-static 0.82 --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 32768 --max-prefill-tokens 32768 --scheduler-recv-interval 30 --stream-interval 30 +``` + ### Example: Serving with two H200\*8 nodes and docker There are two H200 nodes, each with 8 GPUs. The first node's IP is `192.168.114.10`, and the second node's IP is `192.168.114.11`. Configure the endpoint to expose it to another Docker container using `--host 0.0.0.0` and `--port 40000`, and set up communications with `--dist-init-addr 192.168.114.10:20000`. diff --git a/docs/basic_usage/deepseek_v3.md b/docs/basic_usage/deepseek_v3.md index 8396b2f50..1e406ee90 100644 --- a/docs/basic_usage/deepseek_v3.md +++ b/docs/basic_usage/deepseek_v3.md @@ -12,6 +12,7 @@ To run DeepSeek V3.1/V3/R1 models, the recommended settings are as follows: | Weight Type | Configuration | |------------|-------------------| | **Full precision [FP8](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528)**
*(recommended)* | 8 x H200 | +| | 8 x B200 | | | 8 x MI300X | | | 2 x 8 x H100/800/20 | | | Xeon 6980P CPU | @@ -27,7 +28,7 @@ To run DeepSeek V3.1/V3/R1 models, the recommended settings are as follows: | **Quantized weights ([AWQ](https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ))** | 8 x H100/800/20 | | | 8 x A100/A800 | | **Quantized weights ([MXFP4](https://huggingface.co/amd/DeepSeek-R1-MXFP4-Preview))** | 8, 4 x MI355X/350X | -| **Quantized weights ([NVFP4](https://huggingface.co/nvidia/DeepSeek-R1-0528-NVFP4))** | 8, 4 x B200 | +| **Quantized weights ([NVFP4](https://huggingface.co/nvidia/DeepSeek-R1-0528-NVFP4-v2))** | 8, 4 x B200 |