Commit Graph
100 Commits
Author SHA1 Message Date
Simo Lin c8dc4d2d1f [smg][ci] migrate enable_thinking tests to new infrastructure (#16739) 2026-01-08 06:57:45 -08:00
Simo Lin f52ae586b6 Remove migrated e2e_grpc/basic tests (#16738) 2026-01-08 06:30:45 -08:00
Simo Lin 2f8a36347c [smg][ci] migrate chat completions tests to new infrastructure and build wheel once and share via artifact (#16709) 2026-01-08 06:29:23 -08:00
Simo Lin 4c46ecde80 [smg][ci] delete old responses api ci (#16695) 2026-01-07 18:24:12 -08:00
Simo Lin a08dc5aa10 [smg][ci] rename 3rd models from cloud backend and delete dead code (#16692) 2026-01-07 18:19:44 -08:00
109fe03ad1 [smg][ci] Migrate Response API e2e tests to shared infrastructure (#16680)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
Co-authored-by: key4ng <rukeyang@gmail.com>
2026-01-07 17:40:03 -08:00
Simo Lin 6037267f5b [smg][ci] Add thread safety to ModelPool and GPUAllocator (#16674) 2026-01-07 13:25:41 -08:00
Simo Lin 7385834c8d Add reference counting to ModelInstance for parallel test safety (#16672) 2026-01-07 08:28:15 -08:00
Simo Lin c356ed03dd refactor(e2e): unify RouterInstance into Gateway class, split conftest.py into modular fixtures (#16671) 2026-01-07 07:50:28 -08:00
Simo Lin 55b7936582 refactor(e2e_test): fix smg ci e2e test code quality (#16664) 2026-01-07 06:59:49 -08:00
Simo Lin 8729ad5e6c fix(e2e_test): remove dead code and fix type annotations (#16661) 2026-01-07 06:35:15 -08:00
Simo Lin e432057381 [smg][ci] preserve model launch order with test collected (#16618) 2026-01-07 06:16:59 -08:00
Simo Lin d8b8198192 [smg][ci]: migrate benchmarks to e2e_test/benchmarks/, use parent conftest (#16597) 2026-01-06 20:15:20 -08:00
Simo Lin dce8b0606c refactor(e2e): keep only benchmark tests in e2e_http, remove redundant tests (#16594) 2026-01-06 15:05:08 -08:00
Simo Lin d415d22daa refactor(e2e): remove old embedding tests migrated to e2e_test/embeddings (#16592) 2026-01-06 14:07:45 -08:00
Simo Lin a49b9a6420 [model-gateway] add embedding tests (#16583) 2026-01-06 13:24:37 -08:00
Simo Lin d57d8e7e5f [smg] clean up logs in mcp (should be info instead warn) (#16591) 2026-01-06 12:44:32 -08:00
Simo LinandChang Su a3656cbb96 [ci] fix url strips in smg ci (#16548)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2026-01-06 09:23:58 -08:00
Simo Lin 402a0bd6dc [model-gateway] Add model scope support and LRU eviction for GPU-constrained environments (#16525) 2026-01-05 18:28:07 -08:00
Simo Lin 4221b7c573 [model-gateway] refactor e2e test infrastructure and add router CI (#16513) 2026-01-05 13:50:59 -08:00
Simo Lin 2b4d6d813b [model-gateway][e2e_test]: Create directory structure and backends config (#16469) 2026-01-05 08:01:53 -08:00
Simo Lin 7f6a678f8f [model-gateway] add GPU allocator and model pool infrastructure for parallel E2E tests (#16460) 2026-01-04 23:30:32 -08:00
Simo Lin 399ca037b1 [bug] fix lint blocking ci (#16464) 2026-01-04 22:57:33 -08:00
Simo Lin e6fe092dcc [model-gateway] rename py_test to e2e_test (#16454) 2026-01-04 22:24:34 -08:00
Simo Lin e53160bb31 [model-gateway] reorganize integration tests into logical subdirectories (#16451) 2026-01-04 21:49:20 -08:00
Simo Lin 2181bc9e51 [model-gateway] Delete Python integration_mock tests (#16448) 2026-01-04 21:34:06 -08:00
Simo Lin f84487af59 [model-gateway] : Rust integration tests for integration_mock replacement (#16441) 2026-01-04 21:01:47 -08:00
Simo Lin 0ff3747ca1 [model-gateway]: move unit tests to bindings/python/tests/ (#16430) 2026-01-04 18:10:35 -08:00
Simo Lin 4436dc0f6c [model-gateway] improve lock contention and allocation in middleware (#16405) 2026-01-04 10:40:07 -08:00
Simo Lin cf6800f6cc [model-gateway] optimize Vec and HashMap allocations in responses api (#16406) 2026-01-04 10:07:55 -08:00
Simo Lin 65b0b5b248 fix(logging): use Display format for model_id instead of Debug (#16337) 2026-01-03 00:21:39 -08:00
Simo Lin 31ed68e7c1 [model-gateway] address embedding similarity threshold in ut (#16321) 2026-01-02 16:46:55 -08:00
Simo LinandChang Su d401d23876 [model-gateway] Add embedding correctness test comparing against HuggingFace (#16092)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2026-01-02 10:58:20 -08:00
Simo Lin 851845579c perf(tree): optimize input_char_count calculation (#16238) 2025-12-31 15:29:37 -08:00
Simo Lin ace6f3005f ci(benchmark): disable sccache summary annotations and fix tree output (#16233) 2025-12-31 11:05:21 -08:00
Simo Lin 60d7279c46 test(tree): add comprehensive unit tests and fix input_char_count bug (#16228) 2025-12-31 07:31:39 -08:00
Simo Lin b4ce7a6d71 [model-gateway] cache_aware eliminate String allocations in hot path (#16209) 2025-12-31 00:26:10 -08:00
Simo Lin 9940c6f592 [model-gateway]: ASCII byte comparison and probabilistic timestamp updates (#16181) 2025-12-30 12:28:01 -08:00
Simo LinandChang Su 4bc2f2e0bc [model-gateway][docs] Add Classification API documentation (#16182)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-12-30 10:53:31 -08:00
Simo Lin 8c6f865afb [model-gateway]: optimize prefix_match with zero-copy tenant and deferred char count (#16099) 2025-12-29 11:32:57 -08:00
Simo LinandChang Su f39382c6ca [model-gateway] Fix duplicate classify prefix in response ID (#16101)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-12-29 11:07:21 -08:00
Simo LinandChang Su 2ff289e2dc [model-gateway] Generate UUID-based request IDs for embedding/classify (#16100)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-12-29 10:49:58 -08:00
Simo LinandChang Su b2a3f0551a [model-gateway] Wire classify pipeline to gRPC router (#16098)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-12-29 10:44:45 -08:00
Simo Lin 684e148e38 [model-gateway] Optimize INSERT with leaf-only timestamp updates (#16097) 2025-12-29 10:43:20 -08:00
Simo LinandChang Su f44c4b3715 [model-gateway] Add classify pipeline stages and protocol types (#16094)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-12-29 10:23:36 -08:00
Simo Lin ac78f96ec5 [model-gateway] Optimize radix tree timestamp updates for multi-tenant scaling (#16093) 2025-12-29 10:08:19 -08:00
Simo Lin 7f5879985c [model-gateway] Improve tree benchmark with realistic multi-tenant scenarios (#14838) 2025-12-29 08:37:24 -08:00
Simo LinandChang Su 162d1cf9be [model-gateway] Add classification model support infrastructure (#16061)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-12-29 08:34:05 -08:00
Simo Lin c31f62722c [model-gateway] fix tokenizer to match transformers special token handling (#16087) 2025-12-29 08:13:03 -08:00
Simo Lin de2799f3f5 [docs] Fix non-clickable ToC links in model gateway documentation (#16054) 2025-12-28 23:35:37 -08:00
Simo Lin 097330909f [model-gateway] update WorkerRegistryStats with connection mode and circuit breaker info (#16046) 2025-12-28 23:14:57 -08:00
Simo Lin a44d0079f8 [ci] update genai bench to 0.0.3 for pd testing (#16051) 2025-12-28 23:14:44 -08:00
Simo Lin ec8c831d5a [model-gateway]: optimize metrics for minimal CPU and memory overhead (#16041) 2025-12-28 22:28:08 -08:00
Simo Lin f13949e5a9 [model-gateway] perf: optimize observability logging for minimal CPU/memory overhead (#16039) 2025-12-28 21:20:46 -08:00
Simo Lin c236a3fdf0 [model-gateway][CI] Display benchmark results in GitHub Actions summary (#16037) 2025-12-28 20:57:27 -08:00
Simo Linandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 41a1d16b02 [model-gateway] Organize CLI arguments into logical groups for better --help output (#16035)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-28 20:25:53 -08:00
Simo Lin 9884c9fd4c [model-gateway] Organize Rust CLI arguments into logical groups for better --help output (#16036) 2025-12-28 20:25:38 -08:00
Simo Lin 3645ed0f73 [model-gateway] Add PrefixHash load balancing policy for KV cache-aware routing (#15935) 2025-12-27 05:58:05 -05:00
Simo Lin 0e25aa439b [model-gateway] optimize radix tree memory and reduce allocations (#15933) 2025-12-27 02:47:30 -08:00
Simo Lin 171912a9e3 [model-gateway] Add consistent hashing for ManualPolicy routing (#15907) 2025-12-26 22:53:03 -08:00
Simo Lin 4edee6954a [model-gateway] add JWT/OIDC authentication for control plane APIs (#15850) 2025-12-26 18:30:00 -08:00
Simo Lin acddb8e0ec Revert embedding integration tests from 5f3a47d8a (#15914) 2025-12-26 20:29:03 -05:00
Simo Lin caa95c7eb4 [model-gateway] update ManualPolicy with header-based routing (#15847) 2025-12-25 13:04:43 -08:00
Simo Lin 5454d2a7ae [model-gateway] release smg 0.3.0 (#15781) 2025-12-24 13:23:57 -08:00
Simo Lin 643aeefe6d [model-gateway] Fix logging module name, parse endpoint context, and tokenizer factory (#15782) 2025-12-24 10:54:48 -08:00
Simo Lin b27b5a834e [bug] fix code formatting which blocks ci (#15780) 2025-12-24 09:14:13 -08:00
Simo Lin 2f7c629235 [model-gateway] Fix IGW routing and optimize RouterManager (#15741) 2025-12-24 11:28:53 -05:00
Simo Lin 9665574937 [docs] major SGL Model Gateway documentation update (#15715) 2025-12-23 20:26:09 -08:00
Simo Lin 846953d9f1 [model-gateway] Add tokenize/detokenize HTTP endpoints and tokenizer management (#15702) 2025-12-23 17:32:07 -08:00
Simo Lin aa6ac96635 [model-gateway] Fix tokenizer caching and improve error handling (#15695) 2025-12-23 10:33:38 -08:00
Simo Lin 2142881b7e [model-gateway] return 503 when all workers are circuit-broken (#15611) 2025-12-22 04:06:09 -10:00
Simo Lin 1167867e8d [model-gateway] add retry support to OpenAI router chat endpoint (#15589) 2025-12-21 20:28:51 -08:00
Simo Lin 796969caf0 Optimize Rust CI builds with proper sccache configuration (#15581) 2025-12-21 17:32:13 -08:00
Simo Lin 122c250336 [model-gateway] add retry and circuit breaker support to gRPC routers (#15585) 2025-12-21 17:12:36 -08:00
Simo Lin b9d786059f [model-gateway] refactor WorkerManager with fan_out helper and thin handlers (#15583) 2025-12-21 15:29:37 -08:00
Simo Lin 1354063a48 [model-gateway] add WorkerService abstraction for worker business logic (#15580) 2025-12-21 14:35:35 -08:00
Simo Lin e1dcd0df3e [model-gateway] minor code clean up (#15578) 2025-12-21 11:00:32 -10:00
Simo Lin 69412ccb1f [model-gateway] add model gateway multi-arch docker build, test and document docker image (#15544) 2025-12-20 11:56:59 -08:00
Simo Lin d3885d4ba6 Fix docker gateway image name and add latest tag (#15542) 2025-12-20 11:05:03 -08:00
Simo Lin 5529ab5895 [model-gateway] Implement RAII load guard with response body attachment (#15507) 2025-12-19 19:14:52 -08:00
Simo Lin 74a3349bea [ci] remove rust benchmark in unit test ci (#15510) 2025-12-19 18:59:41 -08:00
Simo Lin 71a602883a [model-gateway] simplify workflow engine backoff and reduce duplicate reads (#15505) 2025-12-19 16:23:02 -08:00
Simo Lin 61405b3d00 [model-gateway] Run workflow event subscribers concurrently (#15504) 2025-12-19 15:49:51 -08:00
Simo Lin 0adfc42b99 [model-gateway] Optimize workflow engine with pre-computed dependency graph (#15503) 2025-12-19 15:44:57 -08:00
Simo Lin ba72e759ca [model-gateway] Improve logging across core modules (#15497) 2025-12-19 13:35:28 -08:00
Simo Lin bd16244d93 [model-gateway] Improve logging in policies module (#15496) 2025-12-19 13:02:57 -08:00
Simo Lin b5eb0214ca [model-gateway] Improve logging in data_connector module (#15495) 2025-12-19 12:57:18 -08:00
Simo Lin d72e908ba5 [model-gateway] refactor: extract common graceful shutdown code before TLS branch (#15494) 2025-12-19 12:56:33 -08:00
Simo Lin 50cad014dc [model-gateway] fix graceful shutdown for TLS/Non-TLS server (#15491) 2025-12-19 12:35:50 -08:00
Simo Lin 70607e55e8 [model-gateway] Replace PolicyRegistry RwLock with DashMap for lock-free policy lookups (#15361) 2025-12-18 00:11:20 -08:00
Simo Lin 53e151945a [model-gateway] optimize worker registry and reduce lock contention in grpc client fetch (#15336) 2025-12-17 08:49:10 -10:00
Simo Lin d747147a26 [model-gateway] reduce cpu overhead (#15316) 2025-12-17 09:42:01 -08:00
Simo Lin 3518b33178 [model-gateway] Remove legacy RouterMetrics and Rename SmgMetrics to Metrics and smg_labels to metrics_labels (#15160) 2025-12-15 04:57:42 -08:00
Simo Lin 7b9156c773 [model-gateway] add mcp and discovery metrics (#15156) 2025-12-14 22:54:53 -08:00
Simo Lin bd9c3a47d6 [model-gateway] Add streaming metrics for harmony gRPC router (#15147) 2025-12-14 20:54:43 -08:00
Simo Lin 1e641ee4a7 [model-gateway] upgrade axum and axum server (#15146) 2025-12-14 20:54:01 -08:00
Simo Lin fb96669ff9 [model-gateway] Add Layer 3 worker metrics (smg_worker_*) (#15130) 2025-12-14 20:25:46 -08:00
Simo Lin 0612175ce1 [model-gateway] add streaming metrics (TTFT, TPOT, tokens, duration) for gRPC router (#15125) 2025-12-14 14:39:10 -08:00
Simo Lin b11af135eb [model-gateway] feat(metrics): implement Layer 2 router metrics (smg_router_*) (#15124) 2025-12-14 10:34:12 -08:00
Simo Lin f9bceea064 [model-gateway] Implement Layer 1 HTTP metrics instrumentation (#15121) 2025-12-14 09:39:15 -08:00