[CPU] improve numa memory binding (#19666)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
parent
2f4babe32b
commit
cbea9f6909
@@ -192,13 +192,31 @@ Notes:
|
||||
the system will automatically utilize the first `n` SNCs.
|
||||
Note that `n` cannot exceed the total SNC number, doing so will result in an error.
|
||||
|
||||
To specify the cores to be used, we need to explicitly set the environment variable `SGLANG_CPU_OMP_THREADS_BIND`.
|
||||
For example, if we want to run the SGLang service using the first 40 cores of each SNC on a Xeon® 6980P server,
|
||||
`SGLANG_CPU_OMP_THREADS_BIND` allows explicit control of CPU cores for each tensor parallel (TP) rank.
|
||||
|
||||
**example 1**: Run SGLang service with TP=6, using the first 40 cores of each SNC on a Xeon® 6980P server,
|
||||
which has 43-43-42 cores on the 3 SNCs of a socket, we should set:
|
||||
|
||||
```bash
|
||||
export SGLANG_CPU_OMP_THREADS_BIND="0-39|43-82|86-125|128-167|171-210|214-253"
|
||||
```
|
||||
This configuration is equivalent to:
|
||||
- rank 0: `numactl -C 0-39 -m 0`
|
||||
- rank 1: `numactl -C 43-82 -m 1`
|
||||
- rank 2: `numactl -C 86-125 -m 2`
|
||||
- rank 3: `numactl -C 128-167 -m 3`
|
||||
- rank 4: `numactl -C 171-210 -m 4`
|
||||
- rank 5: `numactl -C 214-253 -m 5`
|
||||
|
||||
|
||||
**example 2**: Run SGLang service with TP=2, using 96 cores cross 3 SNCs on a Xeon® 6972P server,
|
||||
which has 32-32-32 cores on the 3 SNCs in a socket, we should set:
|
||||
```bash
|
||||
export SGLANG_CPU_OMP_THREADS_BIND="0-95|96-191"
|
||||
```
|
||||
This configuration is equivalent to:
|
||||
- rank 0: `numactl -C 0-95 -m 0-2`
|
||||
- rank 1: `numactl -C 96-191 -m 3-5`
|
||||
|
||||
Please beware that with SGLANG_CPU_OMP_THREADS_BIND set,
|
||||
the available memory amounts of the ranks may not be determined in prior.
|
||||
|
||||
Reference in New Issue
Block a user