Commit Graph
19 Commits
Author SHA1 Message Date
Simo Lin b4ce7a6d71 [model-gateway] cache_aware eliminate String allocations in hot path (#16209) 2025-12-31 00:26:10 -08:00
Simo Lin 8c6f865afb [model-gateway]: optimize prefix_match with zero-copy tenant and deferred char count (#16099) 2025-12-29 11:32:57 -08:00
fzyzcjy 26c5091217 Tiny extract PeriodicTask in router (#15988) 2025-12-28 14:07:13 +08:00
Simo Lin 0e25aa439b [model-gateway] optimize radix tree memory and reduce allocations (#15933) 2025-12-27 02:47:30 -08:00
Simo Lin caa95c7eb4 [model-gateway] update ManualPolicy with header-based routing (#15847) 2025-12-25 13:04:43 -08:00
fzyzcjy 45adad37d0 Add manual routing policy for router (#15586) 2025-12-25 11:27:36 +08:00
fzyzcjy 1ba897f330 Tiny refactor select_workers API for future passing more information (#15596) 2025-12-25 10:31:10 +08:00
Simo Lin aa6ac96635 [model-gateway] Fix tokenizer caching and improve error handling (#15695) 2025-12-23 10:33:38 -08:00
Arthur Cheng dd620987d1 [model-gateway] Replace tokenizer with tokenizer registry for dynamic tokenizer loading in gRPC router (#12968) 2025-12-23 07:58:47 -08:00
Simo Lin 3518b33178 [model-gateway] Remove legacy RouterMetrics and Rename SmgMetrics to Metrics and smg_labels to metrics_labels (#15160) 2025-12-15 04:57:42 -08:00
Simo Lin 7d8e42c979 [model-gateway] Fix metric emission gaps and name mismatch (#15093) 2025-12-13 20:16:30 -08:00
Simo Lin 9ad02b799d [Perf] Optimize radix tree for cache-aware load balancin (#14758) 2025-12-09 15:35:29 -08:00
fzyzcjy 817daba062 Tiny extract select_worker_min_load (#14648) 2025-12-08 21:22:10 -08:00
fzyzcjy 2e3946d889 Fix cache-aware router should pick min load instead of min tenant size (#14650) 2025-12-08 13:33:34 -08:00
Simo Lin d69ecc19b8 [model-gateway] reducing cpu overhead in various of places (#14658) 2025-12-08 09:44:40 -08:00
fzyzcjy c08b780fe0 Super tiny remove unused select_worker_pair (#14609) 2025-12-07 20:32:38 -08:00
fzyzcjy 85d0ccfac0 Tiny fix missing policy decision recording (#14605) 2025-12-07 19:24:27 -08:00
Simo Lin aff1238ef2 [model-gateway] reorganize metrics, logging, and otel to its own module (#14590) 2025-12-07 16:50:39 -08:00
Simo Lin 49dfa1d891 [model-gateway] change sgl-router to sgl-model-gateway (#14312) 2025-12-05 12:04:48 -08:00