Commit Graph
56 Commits
Author SHA1 Message Date
Simo Lin 846953d9f1 [model-gateway] Add tokenize/detokenize HTTP endpoints and tokenizer management (#15702) 2025-12-23 17:32:07 -08:00
Simo Lin aa6ac96635 [model-gateway] Fix tokenizer caching and improve error handling (#15695) 2025-12-23 10:33:38 -08:00
ratish 5f3a47d8a7 [model-gateway]: add gRPC router embeddings endpoint implementation (#15273) 2025-12-23 09:12:22 -08:00
Praneth Paruchuri 80ae2229d3 [model-gateway] Optimize router selection with lock-free snapshots (#15672) 2025-12-23 09:09:30 -08:00
Arthur Cheng dd620987d1 [model-gateway] Replace tokenizer with tokenizer registry for dynamic tokenizer loading in gRPC router (#12968) 2025-12-23 07:58:47 -08:00
fzyzcjy b736a1525a Tiny add back missing router per attempt response metric (#15621) 2025-12-22 09:59:10 -08:00
fzyzcjy 6984837d82 Fix router gRPC mode launch error caused by async loading (#15368) 2025-12-22 09:39:01 -08:00
Simo Lin 2142881b7e [model-gateway] return 503 when all workers are circuit-broken (#15611) 2025-12-22 04:06:09 -10:00
Simo Lin 1167867e8d [model-gateway] add retry support to OpenAI router chat endpoint (#15589) 2025-12-21 20:28:51 -08:00
Simo Lin 122c250336 [model-gateway] add retry and circuit breaker support to gRPC routers (#15585) 2025-12-21 17:12:36 -08:00
Simo Lin e1dcd0df3e [model-gateway] minor code clean up (#15578) 2025-12-21 11:00:32 -10:00
Haoran Wang ef001fb811 [model-gateway] /parse/easoning and parse/function_call for sgl-model-gateway (#15568) 2025-12-21 07:39:24 -10:00
Simo Lin 5529ab5895 [model-gateway] Implement RAII load guard with response body attachment (#15507) 2025-12-19 19:14:52 -08:00
kun-llfl 3c116d5e5a [router] bugfix: cache_aware in grpc inbalance forward (#15473)
Signed-off-by: Kun(llfl) <i@imux.top>
2025-12-19 16:25:46 -08:00
Simo Lin ba72e759ca [model-gateway] Improve logging across core modules (#15497) 2025-12-19 13:35:28 -08:00
Simo Lin 53e151945a [model-gateway] optimize worker registry and reduce lock contention in grpc client fetch (#15336) 2025-12-17 08:49:10 -10:00
Simo Lin d747147a26 [model-gateway] reduce cpu overhead (#15316) 2025-12-17 09:42:01 -08:00
Simo Lin 3518b33178 [model-gateway] Remove legacy RouterMetrics and Rename SmgMetrics to Metrics and smg_labels to metrics_labels (#15160) 2025-12-15 04:57:42 -08:00
Simo Lin 7b9156c773 [model-gateway] add mcp and discovery metrics (#15156) 2025-12-14 22:54:53 -08:00
Simo Lin bd9c3a47d6 [model-gateway] Add streaming metrics for harmony gRPC router (#15147) 2025-12-14 20:54:43 -08:00
Simo Lin fb96669ff9 [model-gateway] Add Layer 3 worker metrics (smg_worker_*) (#15130) 2025-12-14 20:25:46 -08:00
fzyzcjy 62b3fdae43 Fix cache aware wrong routing caused by incorrect load tracking (#15101) 2025-12-15 10:03:54 +08:00
Simo Lin 0612175ce1 [model-gateway] add streaming metrics (TTFT, TPOT, tokens, duration) for gRPC router (#15125) 2025-12-14 14:39:10 -08:00
Simo Lin b11af135eb [model-gateway] feat(metrics): implement Layer 2 router metrics (smg_router_*) (#15124) 2025-12-14 10:34:12 -08:00
Simo Lin 74ea45cc4b [model-gateway] optimize metric labels to avoid unnecessary allocations (#15095) 2025-12-13 21:10:45 -08:00
Simo Lin e048ee90fc [model-gateway] Simplify error response creation (#15079) 2025-12-13 10:28:24 -08:00
fzyzcjy 993278b488 Fix double decrease load (#15060) 2025-12-13 22:12:35 +08:00
fzyzcjy 9e9d910744 Fix load metric not updated when using guard (#15059) 2025-12-13 22:11:52 +08:00
fzyzcjy bf17e769fe Add sgl_router_attempt_http_responses_total for single attempt information (#15037) 2025-12-13 13:29:18 +08:00
fzyzcjy 9a5d6a84ab Add error code in prometheus metrics and add X-SMG-Error-Code header (#15036) 2025-12-13 13:28:32 +08:00
fzyzcjy 31c23e5fe3 Provide more fine grained error reason for reqwest error (#15032) 2025-12-13 13:27:06 +08:00
fzyzcjy 06617a9ec8 Tiny change http router response format to unify (#15031) 2025-12-13 13:25:45 +08:00
fzyzcjy e79ca95961 Tiny unify grpc existing error responses into new format (#15030) 2025-12-13 13:25:10 +08:00
fzyzcjy 9d3b411c56 Add code field and unify error responses for router (#15028) 2025-12-13 13:21:34 +08:00
fzyzcjy ec242f516e Super tiny extract route_typed_request_once (#14951) 2025-12-12 11:10:10 -08:00
fzyzcjy edb172e927 Super tiny refactor error.rs logic (#14949) 2025-12-12 14:48:18 +08:00
fzyzcjy a7fa31ff17 Super tiny move error.rs (#14944) 2025-12-12 14:25:30 +08:00
Praneth Paruchuri 1834401e7c [model-gateway] optimize worker selection (#14894) 2025-12-11 21:15:12 -08:00
fzyzcjy e4873d0439 Super tiny remove sgl_router_active_workers (#14891) 2025-12-11 20:00:55 -08:00
Jimmy 2c4d376d23 [SMG][DS32][fix] support dsv32, add role developer (#14307) 2025-12-11 08:09:09 -08:00
Simo Lin cd1c1fa5bc [model-gateway] fix annotation error and code formating (#14910) 2025-12-11 07:06:32 -08:00
Simo Lin 5b0872d2d7 [model-gateway] code clean up on oai router in responses (#14852) 2025-12-11 06:55:09 -08:00
fzyzcjy ba88f1cace Tiny clean router load report logic (#14889) 2025-12-11 06:50:32 -08:00
Simo Lin bcc5483ed7 [model-gateway] fix import order in oai conversation (#14851) 2025-12-10 15:36:42 -08:00
Simo Lin ccf2602773 [model-gateway] code clean up on oai router (#14850) 2025-12-10 15:34:27 -08:00
Simo Lin c032b559a8 [model-gateway] adds default implementations to RouterTrait in mod.rs (#14841) 2025-12-10 11:46:35 -08:00
fzyzcjy 3d82c0f17f [model-gateway] support engine response http status statistics in router (#14712) 2025-12-10 08:30:53 -08:00
Simo Lin edde5e5d40 [model-gateway] add OTEL integration to grpc router (#14671) 2025-12-08 14:45:05 -08:00
Simo Lin 8550822d6b [model-gateway] Optimize memory usage in HTTP router (#14667) 2025-12-08 13:10:31 -08:00
Simo Lin 39f9a9c2a5 [model-gateway] reduce cpu overhead in grpc router (#14663) 2025-12-08 11:54:56 -08:00