chenchenghao/sglang

Files

History

Lianmin Zheng a9e90b4bce [Minor] Fix styles for overlap mode (#2068 )

2024-11-17 19:49:20 -08:00

..

Unify the model type checking (#1905 )

2024-11-03 12:25:39 -08:00

[Minor] Fix styles for overlap mode (#2068 )

2024-11-17 19:49:20 -08:00

fix: align enable_overlap_scheduler naming between code and docs (#2038 )

2024-11-15 03:39:10 -08:00

__init__.py

Provide an offline engine API (#1567 )

2024-10-06 20:27:03 -07:00

api.py

Update links (#1805 )

2024-10-26 04:46:01 -07:00

bench_latency.py

Add Tensor Parallel to torch_native_llama (#1876 )

2024-11-15 21:26:00 -08:00

bench_offline_throughput.py

Fix illegal memory access in overlap mode & Use more fused triton kernels for building meta data (#2051 )

2024-11-16 16:14:23 -08:00

bench_server_latency.py

Add a watch dog thread (#1816 )

2024-10-27 02:00:50 -07:00

bench_serving.py

Fix the default arguments of bench_offline_throughput.py & simplify detokenizer manager (#2042 )

2024-11-15 05:02:44 -08:00

check_env.py

misc: add hypervisor vendor (#1165 )

2024-08-21 02:14:51 +10:00

global_config.py

Improve the user control of new_token_ratio (#1811 )

2024-10-26 16:39:41 -07:00

launch_server_llavavid.py

Update vllm to 0.6.3 (#1711 ) (#1720 )

2024-10-19 20:45:41 -07:00

launch_server.py

Add a watch dog thread (#1816 )

2024-10-27 02:00:50 -07:00

README.md

Update readme and workflow (#1716 )

2024-10-19 13:01:44 -07:00

utils.py

fix black in pre-commit (#1940 )

2024-11-08 07:42:47 +08:00

version.py

Release v0.3.5.post2 (#2046 )

2024-11-15 06:54:00 -08:00

README.md

Code Structures

lang: The frontend language.
srt: The backend engine for running local models. (SRT = SGLang Runtime).
test: The test utilities.
api.py: The public APIs.
bench_latency.py: Benchmark the latency of running a single static batch.
bench_server_latency.py: Benchmark the latency of serving a single batch with a real server.
bench_serving.py: Benchmark online serving with dynamic requests.
global_config.py: The global configs and constants.
launch_server.py: The entry point for launching the local server.
utils.py: Common utilities.