Commit Graph
100 Commits
Author SHA1 Message Date
Simo Lin 69412ccb1f [model-gateway] add model gateway multi-arch docker build, test and document docker image (#15544) 2025-12-20 11:56:59 -08:00
Simo Lin d3885d4ba6 Fix docker gateway image name and add latest tag (#15542) 2025-12-20 11:05:03 -08:00
Simo Lin 5529ab5895 [model-gateway] Implement RAII load guard with response body attachment (#15507) 2025-12-19 19:14:52 -08:00
Simo Lin 74a3349bea [ci] remove rust benchmark in unit test ci (#15510) 2025-12-19 18:59:41 -08:00
Simo Lin 71a602883a [model-gateway] simplify workflow engine backoff and reduce duplicate reads (#15505) 2025-12-19 16:23:02 -08:00
Simo Lin 61405b3d00 [model-gateway] Run workflow event subscribers concurrently (#15504) 2025-12-19 15:49:51 -08:00
Simo Lin 0adfc42b99 [model-gateway] Optimize workflow engine with pre-computed dependency graph (#15503) 2025-12-19 15:44:57 -08:00
Simo Lin ba72e759ca [model-gateway] Improve logging across core modules (#15497) 2025-12-19 13:35:28 -08:00
Simo Lin bd16244d93 [model-gateway] Improve logging in policies module (#15496) 2025-12-19 13:02:57 -08:00
Simo Lin b5eb0214ca [model-gateway] Improve logging in data_connector module (#15495) 2025-12-19 12:57:18 -08:00
Simo Lin d72e908ba5 [model-gateway] refactor: extract common graceful shutdown code before TLS branch (#15494) 2025-12-19 12:56:33 -08:00
Simo Lin 50cad014dc [model-gateway] fix graceful shutdown for TLS/Non-TLS server (#15491) 2025-12-19 12:35:50 -08:00
Simo Lin 70607e55e8 [model-gateway] Replace PolicyRegistry RwLock with DashMap for lock-free policy lookups (#15361) 2025-12-18 00:11:20 -08:00
Simo Lin 53e151945a [model-gateway] optimize worker registry and reduce lock contention in grpc client fetch (#15336) 2025-12-17 08:49:10 -10:00
Simo Lin d747147a26 [model-gateway] reduce cpu overhead (#15316) 2025-12-17 09:42:01 -08:00
Simo Lin 3518b33178 [model-gateway] Remove legacy RouterMetrics and Rename SmgMetrics to Metrics and smg_labels to metrics_labels (#15160) 2025-12-15 04:57:42 -08:00
Simo Lin 7b9156c773 [model-gateway] add mcp and discovery metrics (#15156) 2025-12-14 22:54:53 -08:00
Simo Lin bd9c3a47d6 [model-gateway] Add streaming metrics for harmony gRPC router (#15147) 2025-12-14 20:54:43 -08:00
Simo Lin 1e641ee4a7 [model-gateway] upgrade axum and axum server (#15146) 2025-12-14 20:54:01 -08:00
Simo Lin fb96669ff9 [model-gateway] Add Layer 3 worker metrics (smg_worker_*) (#15130) 2025-12-14 20:25:46 -08:00
Simo Lin 0612175ce1 [model-gateway] add streaming metrics (TTFT, TPOT, tokens, duration) for gRPC router (#15125) 2025-12-14 14:39:10 -08:00
Simo Lin b11af135eb [model-gateway] feat(metrics): implement Layer 2 router metrics (smg_router_*) (#15124) 2025-12-14 10:34:12 -08:00
Simo Lin f9bceea064 [model-gateway] Implement Layer 1 HTTP metrics instrumentation (#15121) 2025-12-14 09:39:15 -08:00
Simo Lin 47633c191f [model-gateway] Add new SMG metrics architecture with 6 layers (#15106) 2025-12-14 08:37:19 -08:00
Simo Lin 74ea45cc4b [model-gateway] optimize metric labels to avoid unnecessary allocations (#15095) 2025-12-13 21:10:45 -08:00
Simo Lin 0fa044ad63 [model-gateway] Add circuit breaker and discovery watcher metrics (#15094) 2025-12-13 20:47:04 -08:00
Simo Lin 7d8e42c979 [model-gateway] Fix metric emission gaps and name mismatch (#15093) 2025-12-13 20:16:30 -08:00
Simo Lin 20ce9938b5 [model-gateway] Remove unused TokenizerMetrics to reduce CPU overhead (#15087) 2025-12-13 18:54:18 -08:00
Simo Lin fd37cc5d38 [model-gateway] Refactor worker steps and add update workflow (#15085) 2025-12-13 15:00:16 -08:00
Simo Lin 5d9c6bac07 [bug] fix grpc secheduler launcher breaking change (#15080) 2025-12-13 10:36:49 -08:00
Simo Lin e048ee90fc [model-gateway] Simplify error response creation (#15079) 2025-12-13 10:28:24 -08:00
Simo Lin 9a56273ad2 [model-gateway] refactor: unify worker management into modular workflow structure (#15010) 2025-12-12 15:25:35 -08:00
Simo Lin 526fd0082f [model-gateway] refactor: workflow engine cleanup and minor optimization (#15001) 2025-12-12 09:03:28 -08:00
Simo Lin 56d0ad47f4 [model-gateway] fix: handle workflow deadlock and optimize cycle detection (#15000) 2025-12-12 08:53:45 -08:00
Simo Lin 306e5b8d0b [model-gateway] feat: add DAG parallel execution support and workflow optimization (#14999) 2025-12-12 08:31:12 -08:00
Simo Lin 10c68f6236 [model-gateway] refactor: extract workflow engine to src/workflow module (#14996) 2025-12-12 06:29:12 -08:00
Simo Lin 6f99dc97f7 [model-gateway] fix imports and delete unused code (#14911) 2025-12-11 07:42:47 -08:00
Simo Lin cd1c1fa5bc [model-gateway] fix annotation error and code formating (#14910) 2025-12-11 07:06:32 -08:00
Simo Lin 5b0872d2d7 [model-gateway] code clean up on oai router in responses (#14852) 2025-12-11 06:55:09 -08:00
Simo Lin bcc5483ed7 [model-gateway] fix import order in oai conversation (#14851) 2025-12-10 15:36:42 -08:00
Simo Lin ccf2602773 [model-gateway] code clean up on oai router (#14850) 2025-12-10 15:34:27 -08:00
Simo Lin c032b559a8 [model-gateway] adds default implementations to RouterTrait in mod.rs (#14841) 2025-12-10 11:46:35 -08:00
Simo Lin 2543666c8e [model-gateway] add anthropic message api spec (#14834) 2025-12-10 08:26:51 -08:00
Simo Lin 390406c46c [model-gateway] release gateway 0.2.4 (#14763) 2025-12-09 16:34:29 -08:00
Simo Lin 9ad02b799d [Perf] Optimize radix tree for cache-aware load balancin (#14758) 2025-12-09 15:35:29 -08:00
Simo Lin 73df7a4e8d [SMG] perf: optimize tokenizer for reduced CPU and memory overhead (#14752) 2025-12-09 11:41:13 -08:00
Simo Lin 8b98bb768c [model-gateway] optimize core modules (#14751) 2025-12-09 10:54:07 -08:00
Simo Lin af60cad05d [ci][smg] fix docker release ci and add it to pr test (#14683) 2025-12-08 20:56:38 -08:00
Simo Lin edde5e5d40 [model-gateway] add OTEL integration to grpc router (#14671) 2025-12-08 14:45:05 -08:00
Simo Lin 8550822d6b [model-gateway] Optimize memory usage in HTTP router (#14667) 2025-12-08 13:10:31 -08:00
Simo Lin 7bf16c6339 [model-gateway] fix WASM arbitrary file read security vol (#14664) 2025-12-08 12:11:16 -08:00
Simo Lin 39f9a9c2a5 [model-gateway] reduce cpu overhead in grpc router (#14663) 2025-12-08 11:54:56 -08:00
Simo Lin d69ecc19b8 [model-gateway] reducing cpu overhead in various of places (#14658) 2025-12-08 09:44:40 -08:00
Simo Lin 2970f22917 [model-gateway] fix WASM unbounded request/response body read vuln (#14612) 2025-12-07 20:47:25 -08:00
Simo Lin 8fbf7dd56f [model-gateway] refactor otel to be more efficient (#14604) 2025-12-07 20:09:25 -08:00
Simo Lin a4ffd665c0 [model-gateway] fix WASM memory limit per module (#14600) 2025-12-07 19:15:19 -08:00
Simo Lin aff1238ef2 [model-gateway] reorganize metrics, logging, and otel to its own module (#14590) 2025-12-07 16:50:39 -08:00
Simo Lin 5e2cda6158 [model-gateway] Fixed WASM Security Vulnerability - Execution Timeout (#14588) 2025-12-07 16:05:53 -08:00
Simo Lin b0bbc7f53d [model-gateway] extra accumulator and tool handler in oai router (#14587) 2025-12-07 15:52:12 -08:00
Simo Lin d30d6b368f [bug] fix notebook to include new keys from model_info (#14528) 2025-12-05 20:44:03 -08:00
Simo Lin 959a17466d [model-gateway] fix code owner for wasm (#14516) 2025-12-05 13:48:25 -08:00
Simo Lin b72f0268c7 [model-gateway] fix left over sgl-router names in wasm (#14514) 2025-12-05 13:16:58 -08:00
Simo Lin 09376fd738 [model-gateway] fix logs in smg workflow (#14513) 2025-12-05 12:50:50 -08:00
Simo Lin aed835e32d [model-gateway] fix left over sgl-router names to sgl-model-gateway (#14512) 2025-12-05 12:41:48 -08:00
Simo Lin 49dfa1d891 [model-gateway] change sgl-router to sgl-model-gateway (#14312) 2025-12-05 12:04:48 -08:00
Simo Lin cf9a774cf8 [model-gateway] fix server info comment (#14508) 2025-12-05 10:09:58 -08:00
Simo Linandkey4ng 1569fc7f49 [model-gateway] reorganized conversation handler (#14507)
Co-authored-by: key4ng <rukeyang@gmail.com>
2025-12-05 09:31:27 -08:00
Simo Linandkey4ng 05284378d6 [model-gateway] move conversation to first class routing (#14506)
Co-authored-by: key4ng <rukeyang@gmail.com>
2025-12-05 08:44:13 -08:00
Simo Lin 7235a7fbe9 [misc] add model arch and type to server info and use it for harmony (#14456) 2025-12-05 06:51:00 -08:00
Simo LinandChang Su c1006fd8a1 [model-gateway] grpc to leverage event type (#14450)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-12-04 14:30:19 -08:00
Simo LinandChang Su 29c6c2ea9d [model-gateway] add mistral 3 image processor (#14445)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-12-04 13:16:04 -08:00
Simo Linandkey4ng eb85fa6daf [model-gateway] move all responses api event from oai to proto (#14446)
Co-authored-by: key4ng <rukeyang@gmail.com>
2025-12-04 13:11:03 -08:00
Simo Linandkey4ng d8faf2f361 [model-gateway] move oai header util to router header util (#14441)
Co-authored-by: key4ng <rukeyang@gmail.com>
2025-12-04 11:02:50 -08:00
Simo Linandkey4ng 7f3308bc68 [model-gateway] extract conversation out of oai router (#14440)
Co-authored-by: key4ng <rukeyang@gmail.com>
2025-12-04 10:16:14 -08:00
Simo Lin fdc2ef58db [model-gateway] add llama4 vision image processor (#14438) 2025-12-04 09:52:36 -08:00
Simo Linandkey4ng b01fc161eb [model-gateway] introduce request ctx for oai router (#14434)
Co-authored-by: key4ng <rukeyang@gmail.com>
2025-12-04 08:31:44 -08:00
Simo Lin 11d33c0e8f [model-gateway] add phi4 vision image processor (#14430) 2025-12-04 07:40:33 -08:00
Simo Lin fa78c44a3d [model-gateway] introduce provider in openai router (#14394) 2025-12-03 22:47:20 -08:00
Simo Lin 654a78f9c8 [model-gateway] add phi3 vision image processor (#14381) 2025-12-03 22:32:40 -08:00
Simo Lin d42c167bfd [model-gateway] add qwen3_vl model image processor (#14377) 2025-12-03 17:16:37 -08:00
Simo Lin 388151053d [model-gateway] use worker crate in openai router (#14330) 2025-12-03 13:36:32 -08:00
Simo Lin 8ab5d8b40d [model-gateway] add qwen2.5_vl model image processor (#14375) 2025-12-03 12:51:30 -08:00
Simo Lin 03575ce38d [model-gateway] add qwen2_vl model image processor and tests (#14374) 2025-12-03 12:34:06 -08:00
Simo Lin abf6272bdd [model-gateway] add llava model image processor and tests (#14371) 2025-12-03 11:18:00 -08:00
Simo Lin 58ac3f317b [model-gateway] add image processor and transformer structure (#14344) 2025-12-03 00:04:06 -08:00
Simo Lin d5ea8c7176 [model-gateway] multimodality initialization (#13350) 2025-12-02 19:41:40 -08:00
Simo Lin 5dad1ff1b6 [model-gateway] add workflow for external model providers (#14323) 2025-12-02 16:47:06 -08:00
Simo Lin 51a86ce69e [model-gateway] change rust package name to sgl-model-gateway instead (#14283) 2025-12-02 00:06:54 -08:00
Simo Lin 383689e3ad [model-gateway] fix version output (#14276) 2025-12-01 22:51:57 -08:00
Simo Lin 73df525382 [model-gateway] include smg version command in py binding (#14274) 2025-12-01 21:21:02 -08:00
Simo Lin cd4151abc7 [model-gateway] add audio and moderation in model card (#14263) 2025-12-01 18:36:38 -08:00
Simo Lin 57f933fd7d [model-gateway] Migrate Worker trait to model-aware methods (#14250) 2025-12-01 14:32:09 -08:00
Simo Lin e7d6027e4a [model-gateway] add ModelCard support to WorkerMetadata (#14243) 2025-12-01 13:04:09 -08:00
Simo Lin 9325f94534 [model-gateway] add ModelCard and ProviderType for model configuration (#14237) 2025-12-01 09:47:51 -08:00
Simo Lin 92ca629582 [model-gateway] add ModelType bitflags and Endpoint enum for worker (#14230) 2025-12-01 08:14:44 -08:00
Simo Lin 79b389da6b [model-gateway] refactor oai router 1/n (#14228) 2025-12-01 07:16:35 -08:00
Simo Lin e9e90460ed [model-gateway] allow refill rate to be zero (#14030) 2025-11-26 17:15:27 -08:00
Simo Lin 69a03bc3f7 [ci] allow manual label to trigger ci in rust, change ci order (#14016) 2025-11-26 11:38:50 -08:00
Simo Lin 7e964b5198 [ci] mark skip as success instead of failure (#14014) 2025-11-26 10:39:38 -08:00
Simo Lin a0d9f6cdd8 [model-gateway] fix xpu ci (#14012) 2025-11-26 10:33:59 -08:00