Commit Graph
100 Commits
Author SHA1 Message Date
Simo Lin d736e0b65e [router] add grpc router pd mode for chat and generate (#11140) 2025-10-04 06:58:28 -07:00
Simo Lin ffd03a9bd3 [router] fix get load response parsing (#11213) 2025-10-04 06:58:02 -07:00
Simo Lin 96fe2d0f15 [router] add pd service in grpc router for pd (#11120) 2025-10-01 08:09:21 -07:00
Simo Lin 5a290a5644 [router][grpc-server] Fix gRPC server shutdown (#11094) 2025-09-30 07:12:12 -04:00
Simo LinandChang Su 33b3c0f85f [router] grpc router generate endpoint support (#11070)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-09-29 22:07:53 -07:00
Simo Lin 816b3a433a [router] add n to generate sampling params (#11069) 2025-09-29 07:37:43 -07:00
Simo Lin 2572886367 [router] add harmony tool parser base structure and interface (#11036) 2025-09-28 19:46:38 -07:00
Simo Lin 336e9a6058 [router] migrate to rust python module for pythonic parser (#11033) 2025-09-28 14:48:59 -04:00
Simo Lin 5519766a4d [router] fix chat template loading and tokenizer path (#10999) 2025-09-27 23:54:12 -04:00
Simo Lin aae7ead2d0 [router] remove old/oudated/useless comments across code base (#10968) 2025-09-26 10:48:50 -07:00
Simo Lin a7fe6e10a1 [router] remove old/oudated/useless comments (#10967) 2025-09-26 09:45:15 -07:00
Simo Lin be059b83d6 [router] grpc router regular mode import cleanup (#10963) 2025-09-26 04:06:59 -07:00
Simo Lin 5d4fe1ceee [router] add move grpc worker management from router to worker manager (#10960) 2025-09-26 03:57:57 -07:00
Simo Lin 1b011e68dc [router] move grpc client from router to worker and builder (#10958) 2025-09-26 03:13:47 -07:00
Simo Lin 5c0efa562b [router]fix code owner syntax error (#10956) 2025-09-26 03:07:18 -07:00
Simo Lin 1e57b9472d [router] add grpc client get and set (#10955) 2025-09-26 03:07:05 -07:00
Simo Lin 6c2c467d77 [router] update owners for router components (#10927) 2025-09-25 23:46:46 -07:00
Simo Lin f7bab41a29 [router] change log level to warning (#10926) 2025-09-25 17:32:59 -07:00
Simo Lin d511b2d905 [router] consolidate worker load monitoring (#10894) 2025-09-25 09:59:30 -04:00
Simo Lin 458c0219a6 [router] simplify tokenizer dev doc (#10895) 2025-09-24 22:15:56 -07:00
Simo Lin e738703547 [router] consolidate worker get loads (#10880) 2025-09-24 22:13:31 -04:00
Simo Lin 7a06ef984d [router] consolidate health endpoints and flush cache (#10876) 2025-09-24 15:23:21 -07:00
Simo Lin b24b2e7ed7 [router] use dashmap for radix tree instead of hash for multi model (#10814) 2025-09-23 11:25:53 -07:00
Simo Linandkey4ng 98c3b04ff2 [router] responses api POST and GET with local storage (#10581)
Co-authored-by: key4ng <rukeyang@gmail.com>
2025-09-23 09:12:02 -07:00
Simo Lin ddab4fc7c7 [router] fix cache aware routing strategy and lock contention (#10773) 2025-09-23 08:53:49 -07:00
Simo Lin c3a1d7759f [router] remove pd router draining channel (#10767) 2025-09-22 20:49:33 -07:00
Simo LinandChang Su 89971c4c3c [router] refactor router and worker management 4/n (#10756)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-09-22 18:35:10 -07:00
Simo Lin 7ca1bea63d [router] update ci so only execute benchmarks when labels are added (#10757) 2025-09-22 13:23:07 -07:00
Simo Lin 97c3823931 [router] refactor router and worker management 3/n (#10727) 2025-09-22 12:17:50 -07:00
Simo Lin 1d1ce62495 [router] refactor router and worker management 2.5/n (#10677) 2025-09-19 20:54:40 -07:00
Simo Lin 00eb5eb721 [router] refactor router and worker management 2/n (#10666) 2025-09-19 12:37:57 -07:00
Simo Lin 36efd5be8a [router] refactor router and worker management 1/n (#10664) 2025-09-19 06:19:57 -07:00
Simo Lin 873d858b28 [router] refactor worker to builder pattern 5/n (#10653) 2025-09-19 05:43:23 -04:00
Simo Lin 4f2055ad56 [router] refactor worker to builder pattern 4/n (#10650) 2025-09-18 23:49:10 -07:00
Simo Lin ac2a723bb3 [router] refactor worker to builder pattern 3/n (#10647) 2025-09-18 22:52:57 -07:00
Simo Lin 780d6a22cd [router] refactor worker to builder pattern 2/n (#10633) 2025-09-18 21:47:56 -07:00
Simo Lin 5291f32d75 [router] refactor worker to builder pattern 1/n (#10628) 2025-09-18 13:25:40 -07:00
Simo Lin 16e9335998 [router] add router db connector for responses api (#10487) 2025-09-15 22:04:56 -07:00
Simo Lin 7eccbe992d [router] fix service discovery and mcp ut (#10449) 2025-09-14 21:07:23 -07:00
Simo Lin 7c5a0a1b77 [router] add not implemented functions for multi model trait (#10394) 2025-09-12 16:44:18 -07:00
Simo Lin 2f173ea074 [router] allow one router to support different model families and serving mode (#10244) 2025-09-12 16:18:27 -07:00
Simo Lin 07bcad7fb7 [bug] fix router ci syntax error (#10390) 2025-09-12 14:39:15 -07:00
Simo Lin 8c86595c93 [router] enable sccache in ci and local build (#10099) 2025-09-12 09:43:48 -07:00
Simo Lin db37422c92 [router] move to mcp sdk instead (#10057) 2025-09-05 18:03:46 -07:00
Simo Lin bde73ee43f [router] add rust cache in benchmark ci (#10080) 2025-09-05 09:59:36 -07:00
Simo Lin bbf261ae4a [router] fix grpc connection mode detection (#9999) 2025-09-03 21:36:16 -07:00
Simo Lin 4f8a982d52 [router] clean up dependency injector to use ctx (#10000) 2025-09-03 21:35:51 -07:00
Simo Lin d966b902af [router] move tokenizer, reasoning, tool initialization to server (#9996) 2025-09-03 19:35:13 -07:00
Simo Lin 9491d6e554 [router] include rust benchamrks (#9932) 2025-09-02 09:32:09 -07:00
Simo Lin 92d79646e5 [router] add reasoning parser readme (#9837) 2025-08-30 21:06:23 -07:00
Simo Lin 5343058875 [router] grpc router bootstraps (#9759) 2025-08-28 12:07:06 -07:00
Simo Lin 07c9d8fba2 [router] add llama3.2 multi json streaming parser (#9735) 2025-08-28 05:57:13 -07:00
Simo Lin e1f7cf57dc [router] additional llama32 parser unit test and multi json support (#9732) 2025-08-27 20:34:11 -07:00
Simo Lin 2bb9d454b5 [router] additional pythonic parser unit test (#9730) 2025-08-27 19:55:59 -07:00
Simo LinandChang Su 07ee0ab750 [router] add gpt-oss and glm4 tool parser (#9703)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-27 11:26:00 -07:00
Simo LinandChang Su 5c06dcb75a [router] add kimi-k2 tool parser (#9702)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-27 11:04:55 -07:00
Simo LinandChang Su 6f6beca49d [router] add step3 tool parser (#9695)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-27 10:44:52 -07:00
Simo LinandChang Su 6e4e1c8cdc [router] add deepseek tool parser (#9694)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-27 06:18:24 -07:00
Simo Lin 9768c50d90 [router] restructure tool parser module folder (#9693) 2025-08-27 06:05:53 -07:00
Simo Linandfzyzcjy 3578eb1e9b [router] address worker load tracking consistency (#9523)
Co-authored-by: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com>
2025-08-26 06:40:51 -07:00
Simo LinandChang Su e2e378caba [router] add ut for mistral, llama, pythonic, and streaming tool parser (#9632)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-25 22:02:15 -07:00
Simo LinandChang Su dc1decc6af [router] add llama tool parser (#9629)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-25 20:43:36 -07:00
Simo LinandChang Su 03680f33be [router] add pythonic parser (#9628)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-25 20:40:06 -07:00
Simo LinandChang Su d4c5e53401 [router] add qwen tool parser (#9623)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-25 20:32:05 -07:00
Simo LinandChang Su 817c62a077 [router] add mistral tool parser (#9622)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-25 20:09:51 -07:00
Simo Lin f556ac8bd8 [router] add json tool parser (#9516) 2025-08-22 12:13:04 -07:00
Simo Lin 49f9d02538 [router] tokenizer arch doc (#9513) 2025-08-22 09:52:33 -07:00
Simo Lin 0f8cee8cd3 [router] fix router load guard tracking for streaming (#9491) 2025-08-21 22:48:29 -07:00
Simo Lin 78ae175866 [router] add tokenizer benchmark (#9427) 2025-08-21 11:09:39 -07:00
Simo LinandChang Su 5fbad308cd [router] add tokenizer chat template support (#9370)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-19 20:14:02 -07:00
Simo Lin b45f753cba [router] adds reasoning parser pooling and thread-safe (#9360) 2025-08-19 13:35:39 -07:00
Simo LinandChang Su 0b95a01a8f [router] add tiktokenizer and sequence in router (#9354)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-19 10:46:28 -07:00
Simo Lin 94959237bf [router] add dsr1, kimi, and qwen reasoning parser (#9353) 2025-08-19 10:15:24 -07:00
Simo LinandChang Su 6e316588f8 [router] add reasoning parser base structure (#9310)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-18 09:26:09 -07:00
Simo LinandChang Su 24247b4168 [router] add tokenizer metrics (#9307)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-18 09:25:51 -07:00
Simo LinandChang Su d08663eec1 [router] tokenizer factory, hf tokenizer, and stop sequence detector (#9293)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-17 22:38:38 -07:00
Simo Lin ff0cf51c8e [router] introducing tokenizer trait (#9287) 2025-08-17 16:30:01 -07:00
Simo Lin b341b7dbce [router] introduce prefill response draining for http compliance (#9281) 2025-08-17 14:23:04 -07:00
Simo Lin bb10e3a1c3 [router] fix pd prefill http request complinace issue (#9237) 2025-08-16 22:36:45 -07:00
Simo Lin da53e13cbb [router] preserve original worker response header in router (#9236) 2025-08-15 11:01:47 -07:00
Simo Lin 21b8846066 [router] allow more health check configuration (#9198) 2025-08-15 08:07:45 -07:00
Simo Lin 6337d9057c [router] optimize Rust compilation and development workflow (#9133) 2025-08-13 05:14:25 -07:00
Simo Lin 1ce30dd13e [router] update router documentation (#9121) 2025-08-12 13:16:34 -07:00
Simo Lin 9d68bdb240 [router] Add Rust Binary Entrypoint for SGLang Router (#9089) 2025-08-11 21:37:36 -07:00
Simo Lin 38907fe639 refactor(pd-router): extract common patterns to reduce code duplication (#9081) 2025-08-11 13:32:31 -07:00
Simo Lin 6f81a710f7 [pd-router] add retry and circuit breakfor for pd router (#9051) 2025-08-11 05:53:26 -07:00
Simo Lin 067068f271 [router] regular router circuit breaker (#8997) 2025-08-10 21:19:30 -07:00
Simo Lin ce9cf35327 [router] update pyo3 version to 0.25.1 (#9022) 2025-08-10 06:45:51 -07:00
Simo Lin 473400e452 [router] upgrade kube version to latest (#9018) 2025-08-09 22:49:45 -07:00
Simo Lin dd665f967f [router] upgrade rand to latest version (#9017) 2025-08-09 22:49:30 -07:00
Simo Lin 3817a37d87 [router] upgrade to latest sgl kernel for router ci (#9019) 2025-08-09 21:49:18 -07:00
Simo Lin 7b7e56150e [router] fix radix tree integration issues in PD router (#8982) 2025-08-08 14:47:51 -07:00
Simo Lin 1a8706c8b9 [router] reduce contention, fix double-count race (#8978) 2025-08-08 14:29:14 -07:00
Simo Lin a59cbea92d [router] harden retries + metrics; fix streaming load; header filtering (#8972) 2025-08-08 13:10:14 -07:00
Simo Lin 61a4680494 [router] router circuit breaker core (#8941) 2025-08-08 09:20:22 -07:00
Simo Lin 2e901e892f [router] dedicated prefill HTTP client and request-path optimizations (#8923) 2025-08-07 17:31:45 -07:00
Simo Lin 500b15c960 [router] upgrade router version to 0.1.9 (#8844) 2025-08-07 09:29:12 -07:00
Simo Lin 16a4c66d25 [router] update pd router ci summary step with new threshold (#8916) 2025-08-07 07:15:38 -07:00
Simo Lin 89e6521c61 [router] re-enable pd router benchmark CI (#8912) 2025-08-07 06:29:36 -07:00
Simo Lin a69b637014 [router] fix req handling order, improve serialization, remove retry (#8888) 2025-08-06 23:24:39 -07:00