 Byron Hsu
|
5636d16dec
|
[CI] Add logs to debug TestOpenAIServer.test_completion_stream (#17471)
|
2026-02-02 02:38:08 -08:00 |
|
|
Byron Hsuandroot
|
339915ce2b
|
[logprob] Fix logprob + streaming for long concurrent decode by caching already processed logprob (#17005)
Co-authored-by: root <root@memx-cge-29-sr1.xpop.twttr.net>
|
2026-01-13 12:39:14 -08:00 |
|
|
Byron Hsu
|
edbeaf3b88
|
[MM][style] rename inputs_embeds to input_embeds for consistency (#14240)
|
2025-12-01 11:36:51 -08:00 |
|
|
Byron Hsu
|
0825d7f4c6
|
[piecewise] Refactor VLM to support input embed buffer and remove external embedder hack (#14155)
|
2025-11-30 21:43:09 -08:00 |
|
|
Byron Hsu
|
96be97bfff
|
Minor PD style fix (#7215)
|
2025-06-15 16:12:12 -07:00 |
|
|
Byron Hsu
|
88f9c347b2
|
[PD] use int32 for kv indices & get num_reserved_decode_tokens from server_args (#7214)
|
2025-06-15 11:51:03 -07:00 |
|
|
Byron Hsu
|
db0cc57e75
|
[PD] Support decode retract and update decode.py (#7196)
|
2025-06-14 19:48:05 -07:00 |
|
|
Byron Hsu
|
7d316991b2
|
[PD] Update prefill.py (#7190)
|
2025-06-14 15:59:54 -07:00 |
|
|
Byron Hsu
|
c2b16795b5
|
Add decode req pool (#6980)
|
2025-06-09 21:23:36 -07:00 |
|
|
Byron Hsu
|
2d831c6ef9
|
[PD] Support structured output (#6560)
|
2025-05-23 21:49:00 -07:00 |
|
|
Byron Hsu
|
8233cc10fd
|
[PD] Support logprob & Add failure test (#6558)
|
2025-05-23 14:29:20 -07:00 |
|
|
Byron HsuandSangBin Cho
|
d2e0881a34
|
[PD] support spec decode (#6507)
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
|
2025-05-23 12:03:05 -07:00 |
|
|
Byron Hsu
|
0a4fc73b48
|
[PD] Fix failure abort (#6535)
|
2025-05-22 20:32:03 -07:00 |
|
|
Byron Hsu
|
3bde101099
|
[PD] Abort request if transfer fails (#6504)
|
2025-05-21 21:44:25 -07:00 |
|
|
Byron Hsu
|
7513558074
|
[PD] Add doc and simplify sender.send (#6019)
|
2025-05-21 21:22:21 -07:00 |
|
|
Byron Hsu
|
bf98d2e377
|
[PD] Support prefill overlap + Ensure no race condition (#5609)
|
2025-04-21 12:12:56 -07:00 |
|
|
Byron Hsu
|
e65b9f21e3
|
[PD] Support decode overlap schedule (#5608)
|
2025-04-21 12:06:16 -07:00 |
|
|
Byron Hsu
|
deded17f38
|
[PD] Fix edge case and simplify large page size + chunked prefill (#5589)
|
2025-04-21 10:27:02 -07:00 |
|
|
Byron Hsu
|
c951d312ed
|
[PD] Fix large page size + chunk prefill (#5588)
|
2025-04-20 17:21:54 -07:00 |
|
|
Byron Hsu
|
ab4b5606e4
|
[PD] Support page size > 1 (#5561)
|
2025-04-19 21:54:27 -07:00 |
|
|
Byron Hsu
|
a9499885e9
|
[PD] Add transfer backend abstraction (#5328)
|
2025-04-14 01:39:39 +08:00 |
|
|
Byron HsuandLiangsheng Yin
|
6d3b35fae9
|
[PD] Simplify mini LB (#4911)
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
|
2025-04-08 09:42:34 -07:00 |
|
|
|
c7c7dbebbe
|
[PD] Release initial code (#4654)
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Ying1123 <sqy1415@gmail.com>
Co-authored-by: merrymercy <lianminzheng@gmail.com>
Co-authored-by: makro
Co-authored-by: dhou-xai
|
2025-03-21 14:47:47 -07:00 |
|
|
Byron Hsu
|
8cc300f536
|
Fix router test (#4483)
|
2025-03-16 22:49:47 -07:00 |
|
|
Byron Hsu
|
734daedd8f
|
[fix] Clamp logprob with dtype min to prevent -inf (#3224)
|
2025-01-31 17:04:04 +08:00 |
|
|
Byron Hsu
|
20453cef62
|
[test] Lower number of top logprobs to get rid of -inf (#3212)
|
2025-01-30 18:01:23 +08:00 |
|
|
Byron Hsuandzhyncs
|
988d0a4bfc
|
[kernel] Use sgl_kernel rope (#3169)
Co-authored-by: zhyncs <me@zhyncs.com>
|
2025-01-28 14:33:11 +08:00 |
|
|
Byron Hsu
|
27aeb4b7d8
|
[test] deduplicate test_session_control (#3183)
|
2025-01-28 13:17:06 +08:00 |
|
|
Byron Hsu
|
514f37c32b
|
[kernel] Fix position ids in rope (#3173)
|
2025-01-27 17:09:51 +08:00 |
|
|
Byron Hsu
|
741fccd7bf
|
Bump sgl kernel to 0.0.2.post19 (#3167)
|
2025-01-27 15:36:07 +08:00 |
|
|
Byron Hsu
|
fb11a43981
|
[kernel] Integrate flashinfer's rope with higher precision and better perf (#3134)
|
2025-01-27 15:28:00 +08:00 |
|
|
Byron Hsu
|
3ed0a547b2
|
[router] Fix twine uploading (#3095)
|
2025-01-23 21:01:01 -08:00 |
|
|
Byron Hsu
|
8d8ef8497e
|
bump router to 0.1.4 (#3094)
|
2025-01-23 20:32:43 -08:00 |
|
|
Byron Hsu
|
9a0cc2e90e
|
[router] Forward all request headers from router to workers (#3070)
|
2025-01-23 20:30:31 -08:00 |
|
|
Byron Hsu
|
5de50653cd
|
[router] make error actionable (#3063)
|
2025-01-22 17:56:21 -08:00 |
|
|
Byron Hsu
|
c0bf9bf15c
|
[devcontainer] add non-root user (#2989)
|
2025-01-22 17:47:54 -08:00 |
|
|
Byron Hsu
|
3ad4cd4915
|
bump router to 0.1.3 (#3020)
|
2025-01-20 14:38:06 -08:00 |
|
|
Byron Hsu
|
3a8428ecaa
|
[router] Expose worker startup interval (#3019)
|
2025-01-20 14:36:54 -08:00 |
|
|
Byron Hsu
|
0311ce8e1c
|
[router] Expose worker startup secs & Return error instead of panic for router init (#3016)
|
2025-01-20 12:45:13 -08:00 |
|
|
Byron HsuandYineng Zhang
|
b5caa22dfb
|
[kernel] port rope cuda kernel to sgl-kernel (#2993)
Co-authored-by: Yineng Zhang <me@zhyncs.com>
|
2025-01-20 20:58:51 +08:00 |
|
|
Byron Hsu
|
4719c1d04a
|
[router] Fix sgl router path for release (#2980)
|
2025-01-19 01:11:06 -08:00 |
|
|
Byron Hsu
|
ef18b0eda2
|
[router] Allow empty worker list for sglang.launch_router (#2979)
|
2025-01-19 01:05:23 -08:00 |
|
|
Byron Hsu
|
53cc91e504
|
[devcontainer] Fix mount and GPU & Support rust dev (#2978)
|
2025-01-19 16:34:01 +08:00 |
|
|
Byron Hsu
|
2d60a5ee75
|
Update v0.1.0.md
|
2024-12-11 13:48:18 -08:00 |
|
|
Byron Hsu
|
2e4a5907c9
|
[router] Release router 0.1.0 with dynamic scaling and fault tolerance (#2455)
|
2024-12-11 13:42:35 -08:00 |
|
|
Byron Hsu
|
c0ee46fe10
|
[router] Update doc for dynamic scaling and fault tolerance (#2454)
|
2024-12-11 13:11:42 -08:00 |
|
|
Byron Hsu
|
864bf2ba00
|
[router] remove main.rs because only lib.rs is used for py binding (#2453)
|
2024-12-11 12:13:19 -08:00 |
|
|
Byron Hsu
|
a4cca7fc53
|
[router] Add retries based fault tolerance (#2452)
|
2024-12-11 12:13:08 -08:00 |
|
|
Byron Hsu
|
0fb88aaa77
|
[router] Use borrow if possible to save cost (#2441)
|
2024-12-11 01:38:50 -08:00 |
|
|
Byron Hsu
|
d4de9a6235
|
[router] Refactor: decouple select and send stage (#2440)
|
2024-12-11 00:51:21 -08:00 |
|
|
Byron Hsu
|
27f7bed7a7
|
reduce watchdog interval to 5s (#2410)
|
2024-12-08 21:17:31 -08:00 |
|
|
Byron Hsu
|
6387098f5f
|
[router] add health checking in router init (#2393)
|
2024-12-08 17:17:37 -08:00 |
|
|
Byron Hsu
|
2a717c5078
|
[Router] fix interrupt from terminal (#2413)
|
2024-12-08 16:58:41 -08:00 |
|
|
Byron Hsu
|
a1e697b25b
|
[router] Improve cleanup logic (#2411)
|
2024-12-08 15:24:02 -08:00 |
|
|
Byron Hsu
|
ef995dae1e
|
[router] Health check on worker before adding to the router (#2392)
|
2024-12-07 15:39:54 -08:00 |
|
|
Byron Hsu
|
c36736c841
|
[router] Add remove worker api (#2380)
|
2024-12-06 17:16:03 -08:00 |
|
|
Byron Hsu
|
1bf9e34745
|
[router] add remove tenant method in the radix tree (#2379)
|
2024-12-06 11:53:15 -08:00 |
|
|
Byron Hsu
|
499c85f131
|
[Router] remove duplicate char count (#2378)
|
2024-12-06 11:26:07 -08:00 |
|
|
Byron Hsu
|
67b657945a
|
[router] support /add_worker api (#2369)
|
2024-12-06 01:17:04 -08:00 |
|
|
Byron Hsu
|
64fceab8af
|
[router] use 2-gpu-runner (#2368)
|
2024-12-06 14:13:57 +08:00 |
|
|
Byron Hsu
|
0495796517
|
[router] Copy license when publishing & bump version (#2339)
|
2024-12-03 10:27:43 -08:00 |
|
|
Byron Hsu
|
92d1253e58
|
Bump sglang-router to 0.0.10 for env name change (#2226)
|
2024-11-27 11:23:32 -08:00 |
|
|
Byron Hsu
|
0b46b951ae
|
Fix rust warning (#2208)
|
2024-11-26 15:00:41 -08:00 |
|
|
Byron Hsu
|
2763c0a73a
|
Bump router to 0.0.9 with better logging (#2207)
|
2024-11-26 13:30:28 -08:00 |
|
|
Byron Hsu
|
1f76fc6e3f
|
[router] Rust e2e test (#2184)
|
2024-11-25 16:02:03 -08:00 |
|
|
Byron Hsu
|
4d62bca542
|
[router] Replace print with logger (#2183)
|
2024-11-25 13:36:02 -08:00 |
|
|
Byron Hsu
|
a866b65e1d
|
Bump rust router to 0.0.8
|
2024-11-24 23:17:38 -08:00 |
|
|
Byron Hsu
|
4b0a1c9365
|
Replace prob based with threshold based load balancing (#2170)
|
2024-11-24 23:17:11 -08:00 |
|
|
Byron Hsu
|
84a1698d67
|
Update release-pypi-router.yml
|
2024-11-23 17:35:25 -08:00 |
|
|
Byron Hsu
|
32293a299c
|
Improve sglang router (#2148)
|
2024-11-23 17:34:24 -08:00 |
|
|
Byron Hsu
|
7921690893
|
add prefix match for certain tenant (#2147)
|
2024-11-23 15:35:02 -08:00 |
|
|
Byron Hsu
|
bbb81c2457
|
Add more api routes (completion, health, etc) to the router (#2146)
|
2024-11-23 15:10:26 -08:00 |
|
|
Byron Hsu
|
52f58fc42a
|
fix dp_rank env (#2144)
|
2024-11-23 11:46:21 -08:00 |
|
|
Byron Hsu
|
145c0ddc2d
|
update router doc (#2143)
|
2024-11-23 11:01:04 -08:00 |
|
|
Byron Hsu
|
505d7f71a6
|
Bump sglang-router to 0.0.5 (#2142)
|
2024-11-23 08:35:46 -08:00 |
|
|
Byron Hsu
|
cbedd1db1d
|
[router] cache-aware load-balancing router v1 (#2114)
|
2024-11-23 08:34:48 -08:00 |
|
|
Byron Hsu
|
30af7dfb34
|
[router] add base_gpu_id server args & merged radix tree python reference (#2115)
|
2024-11-21 17:13:33 -08:00 |
|
|
Byron Hsu
|
602ebc661d
|
remove sglang folder in rust (#2010)
|
2024-11-11 20:45:52 -08:00 |
|
|
Byron Hsu
|
86c37d010a
|
fix sglang_router not found (#2005)
|
2024-11-11 15:20:14 -08:00 |
|
|
Byron Hsu
|
3e33574374
|
run rust test on ubuntu instead of 1-gpu-runner (#2003)
|
2024-11-11 14:46:08 -08:00 |
|
|
Byron Hsu
|
0d94f1dd03
|
Bump router to 0.0.3 (#2004)
|
2024-11-11 14:42:22 -08:00 |
|
|
Byron Hsu
|
e728258d34
|
release router from py38 to py312 (#2002)
|
2024-11-11 14:30:25 -08:00 |
|
|
Byron Hsu
|
239eafbd2e
|
Fix rust unit test and pypi token (#2001)
|
2024-11-11 14:18:21 -08:00 |
|
|
Byron Hsu
|
00ffde206f
|
setup router python binding ci (#1999)
|
2024-11-11 12:19:32 -08:00 |
|
|
Byron Hsu
|
f9633fa9b9
|
[rust] cache-aware DP - approx tree (#1934)
|
2024-11-10 21:57:32 -08:00 |
|
|
Byron Hsu
|
8169c6f4ef
|
Add gen-shared-prefix dataset in bench_serving (#1990)
|
2024-11-11 08:39:56 +08:00 |
|
|
Byron Hsu
|
96766101b4
|
[rust] refactor server and router (#1922)
|
2024-11-06 00:02:02 -08:00 |
|
|
Byron Hsu
|
463d56bf44
|
Update CODEOWNERS (#1916)
|
2024-11-05 09:13:41 +08:00 |
|
|
Byron HsuandLianmin Zheng
|
530ff541cf
|
[router] Impl radix tree and set up CI (#1893)
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
|
2024-11-04 10:56:52 -08:00 |
|
|
Byron Hsu
|
438526a814
|
Refactor tokenizer manager (#1846)
|
2024-10-30 21:32:18 -07:00 |
|
|
Byron Hsu
|
a7a0a6886b
|
Make decode log interval configurable (#1847)
|
2024-10-30 19:59:20 -07:00 |
|
|
Byron Hsu
|
680cad2023
|
fix get_memory_pool_size deadlock for DP (#1830)
|
2024-10-28 23:07:14 -07:00 |
|
|
Byron Hsu
|
0a24eb850a
|
Fix update_weights deadlock for DP (#1825)
|
2024-10-28 12:02:23 -07:00 |
|
|
Byron Hsu
|
3839be2913
|
[Router] Add a rust-based router (#1790)
|
2024-10-28 09:49:48 -07:00 |
|
|
Byron Hsu
|
6fcd6d7d6d
|
Support token ids in engine.generate (#1820)
|
2024-10-27 14:02:34 -07:00 |
|
|
Byron Hsu
|
c26507484f
|
fix int conversion for SGLANG_CPU_COUNT (#1803)
|
2024-10-26 00:09:44 -07:00 |
|
|
Byron Hsu
|
17536e7e3d
|
Fix edge case for truncated (#1747)
|
2024-10-23 00:00:25 -04:00 |
|
|
Byron Hsu
|
cd0be7489f
|
[doc] improve engine doc and add to readme (#1670)
|
2024-10-14 19:56:21 -07:00 |
|
|
Byron Hsu
|
56503d9bc9
|
[1/N] Remove CacheConfig import in all model files (#1658)
|
2024-10-14 09:06:34 -07:00 |
|
|
Byron Hsu
|
c3f2fc5a7a
|
[doc] Add engine section in backend.md (#1656)
|
2024-10-13 00:33:58 -07:00 |
|