chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[docs] major SGL Model Gateway documentation update (#15715)

Browse Source
This commit is contained in:
Simo Lin
2025-12-23 20:26:09 -08:00
committed by GitHub
parent ac320a6f04
commit 9665574937
4 changed files with 1882 additions and 490 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/router.md
-479
View File
@@ -1,479 +0,0 @@
# SGLang Model Gateway (formerly SGLang Router)
SGLang Model Gateway is a high-performance model-routing gateway for large-scale LLM deployments. It centralizes worker lifecycle management, balances traffic across heterogeneous protocols (HTTP, gRPC, OpenAI-compatible), and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows. The router is deeply optimized for the SGLang serving runtime, but can route to any OpenAI-compatible backend.
---
## Table of Contents
1. [Overview](#overview)
2. [Architecture](#architecture)
- [Control Plane](#control-plane)
- [Data Plane](#data-plane)
- [Storage & Privacy](#storage--privacy)
3. [Deployment Modes](#deployment-modes)
- [Co-launch Router + Workers](#co-launch-router--workers)
- [Separate Launch (HTTP)](#separate-launch-http)
- [gRPC Launch](#grpc-launch)
- [Prefill/Decode Disaggregation](#prefilldecode-disaggregation)
- [OpenAI Backend Proxy](#openai-backend-proxy)
4. [Worker Lifecycle & Dynamic Scaling](#worker-lifecycle--dynamic-scaling)
5. [Reliability & Flow Control](#reliability--flow-control)
6. [Load Balancing Policies](#load-balancing-policies)
7. [Service Discovery (Kubernetes)](#service-discovery-kubernetes)
8. [Security & Authentication](#security--authentication)
9. [History & Data Connectors](#history--data-connectors)
10. [MCP & Advanced Tooling](#mcp--advanced-tooling)
11. [API Surface](#api-surface)
12. [Configuration Reference](#configuration-reference)
13. [Observability](#observability)
14. [Troubleshooting](#troubleshooting)
---
## Overview
- **Unified control plane** for registering, monitoring, and orchestrating regular, prefill, and decode workers across heterogeneous model fleets.
- **Multi-protocol data plane** that routes traffic across HTTP, PD (prefill/decode), gRPC, and OpenAI-compatible backends with shared reliability primitives.
- **Industry-first gRPC pipeline** with native Rust tokenization, reasoning parsers, and tool-call execution for high-throughput, OpenAI-compatible serving; supports both single-stage and PD topologies.
- **Inference Gateway Mode (`--enable-igw`)** dynamically instantiates multiple router stacks (HTTP regular/PD, gRPC) and applies per-model policies for multi-tenant deployments.
- **Conversation & responses connectors** centralize chat history inside the router so the same context can be reused across models and MCP loops without leaking data to upstream vendors (memory, none, Oracle ATP).
- **Enterprise privacy**: agentic multi-turn `/v1/responses`, native MCP client (STDIO/HTTP/SSE/Streamable), and history storage all operate within the router boundary.
- **Reliability core**: retries with jitter, worker-scoped circuit breakers, token-bucket rate limiting with queuing, background health checks, and cache-aware load monitoring.
- **Observability**: Prometheus metrics, structured tracing, request ID propagation, and detailed job queue stats.
---
## Architecture
### Control Plane
- **Worker Manager** discovers capabilities (`/get_server_info`, `/get_model_info`), tracks load, and registers/removes workers in the shared registry.
- **Job Queue** serializes add/remove requests and exposes status (`/workers/{worker_id}`) so clients can track onboarding progress.
- **Load Monitor** feeds cache-aware and power-of-two policies with live worker load statistics.
- **Health Checker** continuously probes workers and updates readiness, circuit breaker state, and router metrics.
### Data Plane
- **HTTP routers** (regular & PD) implement `/generate`, `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/rerank`, and associated admin endpoints.
- **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing.
- **OpenAI router** proxies OpenAI-compatible endpoints to external vendors (OpenAI, xAI, etc.) while keeping chat history and multi-turn orchestration local.
### Storage & Privacy
- Conversation and response history is stored at the router tier (memory, none, or Oracle ATP). The same history can power multiple models or MCP loops without sending data to upstream vendors.
- `/v1/responses` agentic flows, MCP sessions, and conversation APIs share the same storage layer, enabling compliance for regulated workloads.
---
## Deployment Modes
### Co-launch Router + Workers
Launch the router and a fleet of SGLang workers in one process (ideal for single-node or quick starts). The CLI accepts two namespaces of arguments:
- **Worker arguments** (no prefix) configure the SGLang runtime (`--model`, `--tp-size`, `--dp-size`, `--grpc-mode`, etc.).
- **Router arguments** are prefixed with `--router-` and map directly to `launch_router` flags (`--router-policy`, `--router-model-path`, `--router-log-level`, ...).
```bash
python -m sglang_router.launch_server \
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
--dp-size 4 \
--host 0.0.0.0 \
--port 30000
```
Comprehensive example:
```bash
python3 -m sglang_router.launch_server \
--host 0.0.0.0 \
--port 8080 \
--model meta-llama/Llama-3.1-8B-Instruct \
--tp-size 1 \
--dp-size 8 \
--grpc-mode \
--log-level debug \
--router-prometheus-port 10001 \
--router-tool-call-parser llama \
--router-health-success-threshold 2 \
--router-health-check-timeout-secs 6000 \
--router-health-check-interval-secs 60 \
--router-model-path meta-llama/Llama-3.1-8B-Instruct \
--router-policy round_robin \
--router-log-level debug
```
### Separate Launch (HTTP)
Run workers independently and point the router at their HTTP endpoints.
```bash
# Worker nodes
python -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --port 8000
python -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --port 8001
# Router node
python -m sglang_router.launch_router \
--worker-urls http://worker1:8000 http://worker2:8001 \
--policy cache_aware \
--host 0.0.0.0 --port 30000
```
### gRPC Launch
Use SRT gRPC workers to unlock the highest throughput and access native reasoning/tool pipelines.
```bash
# Workers expose gRPC endpoints
python -m sglang.launch_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--grpc-mode \
--port 20000
# Router
python -m sglang_router.launch_router \
--worker-urls grpc://127.0.0.1:20000 \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--reasoning-parser deepseek-r1 \
--tool-call-parser json \
--host 0.0.0.0 --port 8080
```
> gRPC router supports both single-stage and PD serving. Provide `--tokenizer-path` or `--model-path` (HF repo or local directory) plus optional `--chat-template`.
### Prefill/Decode Disaggregation
Split prefill and decode workers for PD-aware caching and balancing. Specifying `--policy A` is equivalent to `--prefill-policy A --decode-policy A`.
```bash
python -m sglang_router.launch_router \
--pd-disaggregation \
--prefill http://prefill1:30001 9001 \
--decode http://decode1:30011 \
--prefill-policy cache_aware \
--decode-policy power_of_two
```
### OpenAI Backend Proxy
Proxy OpenAI-compatible endpoints (OpenAI, xAI, etc.) while keeping history and MCP sessions local.
```bash
python -m sglang_router.launch_router \
--backend openai \
--worker-urls https://api.openai.com \
--history-backend memory
```
> OpenAI backend mode expects exactly one `--worker-urls` entry per router instance.
---
## Worker Lifecycle & Dynamic Scaling
Add or remove workers at runtime using the REST APIs. Jobs are queued and tracked for eventual consistency.
```bash
# Add a worker (HTTP or gRPC)
curl -X POST http://localhost:30000/workers \
-H "Content-Type: application/json" \
-d '{"url":"grpc://0.0.0.0:31000","worker_type":"regular"}'
# Inspect registry
curl http://localhost:30000/workers
# Remove a worker (RESTful: delete by UUID)
# Tip: POST /workers returns a JSON body containing worker_id and a Location header.
WORKER_ID="$(curl -s http://localhost:30000/workers | jq -r '.workers[0].id')"
curl -X DELETE "http://localhost:30000/workers/${WORKER_ID}"
```
Legacy endpoints (`/add_worker`, `/remove_worker`, `/list_workers`) remain available but will be deprecated. `/workers/{worker_id}` returns both registry data and queued job status.
---
## Reliability & Flow Control
### Retries
```bash
python -m sglang_router.launch_router \
--worker-urls http://worker1:8000 http://worker2:8001 \
--retry-max-retries 5 \
--retry-initial-backoff-ms 50 \
--retry-max-backoff-ms 30000 \
--retry-backoff-multiplier 1.5 \
--retry-jitter-factor 0.2
```
### Circuit Breaker
```bash
python -m sglang_router.launch_router \
--worker-urls http://worker1:8000 http://worker2:8001 \
--cb-failure-threshold 5 \
--cb-success-threshold 2 \
--cb-timeout-duration-secs 30 \
--cb-window-duration-secs 60
```
### Rate Limiting & Queuing
```bash
python -m sglang_router.launch_router \
--worker-urls http://worker1:8000 http://worker2:8001 \
--max-concurrent-requests 256 \
--rate-limit-tokens-per-second 512 \
--queue-size 128 \
--queue-timeout-secs 30
```
Requests beyond the concurrency limit wait in a FIFO queue (up to `queue-size`). A `429` is returned when the queue is full; `408` is returned when `queue-timeout-secs` expires.
---
## Load Balancing Policies
| Policy | Description | Usage |
|--------------------|--------------------------------------------------------------------------------------------------|-------------------------------|
| `random` | Uniform random selection. | `--policy random` |
| `round_robin` | Cycles through workers in order. | `--policy round_robin` |
| `power_of_two` | Samples two workers and picks the lighter one (requires Load Monitor). | `--policy power_of_two` |
| `cache_aware` | Default policy; combines cache locality with load balancing, falling back to shortest queue. | `--policy cache_aware` + tuning flags |
Key tuning flags:
```bash
--cache-threshold 0.5 \
--balance-abs-threshold 32 \
--balance-rel-threshold 1.5 \
--eviction-interval-secs 120 \
--max-tree-size 67108864
```
---
## Service Discovery (Kubernetes)
Enable automatic worker discovery via Kubernetes pod selectors.
```bash
python -m sglang_router.launch_router \
--service-discovery \
--selector app=sglang-worker role=inference \
--service-discovery-namespace production \
--service-discovery-port 8000
```
PD deployments can specify `--prefill-selector` and `--decode-selector` plus the `sglang.ai/bootstrap-port` annotation for prefill bootstrap ports. Ensure RBAC grants `get/list/watch` on pods.
---
## Security & Authentication
- **Router API key (`--api-key`)**: clients must supply `Authorization: Bearer <key>`.
- **Worker API keys**: when adding workers dynamically, include `api_key` in the payload; workers listed via CLI inherit the router key.
- **Full-stack auth**: start router with `--api-key`, then add workers with their own keys:
```bash
curl -H "Authorization: Bearer router-key" \
-X POST http://localhost:30000/workers \
-H "Content-Type: application/json" \
-d '{"url":"http://worker:8000","api_key":"worker-key"}'
```
- **Privacy**: All conversation history, `/v1/responses` state, and MCP sessions stay inside the router. Nothing is persisted at remote model vendors unless explicitly proxied.
---
## History & Data Connectors
| Backend | Description | Usage |
|---------|-------------|-------|
| `memory` (default) | In-memory storage for quick prototyping. | `--history-backend memory` |
| `none` | No persistence; APIs operate but store nothing. | `--history-backend none` |
| `oracle` | Oracle Autonomous Database-backed storage (pooled connections). | `--history-backend oracle` |
| `postgres` | PostgreSQL Database-backed storage (pooled connections). | `--history-backend postgres` |
Oracle configuration (choose DSN *or* TNS alias):
Install the Oracle Instant Client and set `LD_LIBRARY_PATH` accordingly.
Choose **one** connection method:
```bash
# Option 1: Full connection descriptor
export ATP_DSN="(description=(address=(protocol=tcps)(port=1522)(host=adb.region.oraclecloud.com))(connect_data=(service_name=service_name)))"
# Option 2: TNS alias (requires wallet)
export ATP_TNS_ALIAS="sglroutertestatp_high"
export ATP_WALLET_PATH="/path/to/wallet"
```
Provide database credentials and optional pool sizing:
```bash
export ATP_USER="admin"
export ATP_PASSWORD="secret"
export ATP_POOL_MIN=4
export ATP_POOL_MAX=32
python -m sglang_router.launch_router \
--backend openai \
--worker-urls https://api.openai.com \
--history-backend oracle
```
> History backends currently apply to OpenAI router mode. gRPC parity for `/v1/responses` is on the roadmap.
---
## MCP & Advanced Tooling
- Native MCP client supports **STDIO**, **HTTP**, **SSE**, and **Streamable** transports—no external config files required.
- Tool-call parsers cover JSON, Pythonic, XML, and custom schemas with streaming/non-streaming execution loops.
- Reasoning parsers ship for DeepSeek-R1, Qwen3, Step-3, GLM4, Llama families, Kimi K2, GPT-OSS, Mistral, and more (`src/reasoning_parser`).
- Tokenizer factory accepts HuggingFace IDs, local directories, and explicit `tokenizer.json` files with chat template overrides (`src/tokenizer`).
Use CLI flags to select parsers:
```bash
--reasoning-parser deepseek-r1 \
--tool-call-parser json \
--chat-template /path/to/template.json
```
---
## API Surface
| Method | Path | Description |
|-----------------------|------------------------------------------|------------------------------------------------|
| `POST` | `/generate` | SGLang generate API. |
| `POST` | `/v1/chat/completions` | OpenAI-compatible chat (streaming/tool calls). |
| `POST` | `/v1/completions` | OpenAI-compatible text completions. |
| `POST` | `/v1/responses` | Create background responses (agentic loops). |
| `GET` | `/v1/responses/{id}` | Retrieve stored responses. |
| `POST` | `/v1/embeddings` | Forward embedding requests. |
| `POST` | `/v1/rerank` | Ranking endpoint (`/rerank` synonym). |
| `POST` | `/v1/conversations` | Create conversation metadata. |
| `GET`/`POST`/`DELETE` | `/v1/conversations/{id}` | Get/update/delete conversation. |
| `GET`/`POST` | `/v1/conversations/{id}/items` | List or append conversation items. |
| `GET`/`DELETE` | `/v1/conversations/{id}/items/{item_id}` | Inspect/delete conversation item. |
| `GET` | `/workers` | List registered workers with health/load. |
| `POST` | `/workers` | Queue worker registration. |
| `GET`/`PUT`/`DELETE` | `/workers/{worker_id}` | Get/update/remove a worker by UUID. |
| `POST` | `/flush_cache` | Flush worker caches (HTTP workers). |
| `GET` | `/get_loads` | Retrieve worker load snapshot. |
| `GET` | `/liveness` / `/readiness` / `/health` | Health probes. |
---
## Configuration Reference
### Core Settings
| Parameter | Type | Default | Description |
|-----------------------------|------|-------------|--------------------------------------------------------------------------|
| `--host` | str | 127.0.0.1 | Router host. |
| `--port` | int | 30000 | Router port. |
| `--worker-urls` | list | [] | Worker URLs (HTTP or gRPC). |
| `--policy` | str | cache_aware | Routing policy (`random`, `round_robin`, `cache_aware`, `power_of_two`). |
| `--max-concurrent-requests` | int | -1 | Concurrency limit (-1 disables rate limiting). |
| `--request-timeout-secs` | int | 600 | Request timeout. |
| `--max-payload-size` | int | 256MB | Maximum request payload. |
### Cache-Aware Tuning
| Parameter | Type | Default | Description |
|----------------------------|-------|----------|-----------------------------|
| `--cache-threshold` | float | 0.3 | Minimum prefix match ratio. |
| `--balance-abs-threshold` | int | 64 | Absolute load threshold. |
| `--balance-rel-threshold` | float | 1.5 | Relative load ratio. |
| `--eviction-interval-secs` | int | 120 | Cache eviction cadence. |
| `--max-tree-size` | int | 67108864 | Max nodes in cache tree. |
### Fault Tolerance
| Parameter | Type | Default | Description |
|------------------------------|-------|---------|----------------------------------|
| `--retry-max-retries` | int | 5 | Max retries. |
| `--retry-initial-backoff-ms` | int | 50 | Initial backoff (ms). |
| `--retry-max-backoff-ms` | int | 30000 | Max backoff (ms). |
| `--retry-backoff-multiplier` | float | 1.5 | Backoff multiplier. |
| `--retry-jitter-factor` | float | 0.2 | Retry jitter (0.0-1.0). |
| `--disable-retries` | flag | False | Disable retries. |
| `--cb-failure-threshold` | int | 5 | Failures before opening circuit. |
| `--cb-success-threshold` | int | 2 | Successes to close circuit. |
| `--cb-timeout-duration-secs` | int | 30 | Cooldown period. |
| `--cb-window-duration-secs` | int | 60 | Window size. |
| `--disable-circuit-breaker` | flag | False | Disable circuit breaker. |
### Prefill/Decode
| Parameter | Type | Default | Description |
|-----------------------------------|------|---------|------------------------------------------|
| `--pd-disaggregation` | flag | False | Enable PD mode. |
| `--prefill` | list | [] | Prefill URLs + optional bootstrap ports. |
| `--decode` | list | [] | Decode URLs. |
| `--prefill-policy` | str | None | Override policy for prefill nodes. |
| `--decode-policy` | str | None | Override policy for decode nodes. |
| `--worker-startup-timeout-secs` | int | 600 | Worker init timeout. |
| `--worker-startup-check-interval` | int | 30 | Polling interval. |
### Kubernetes Discovery
| Parameter | Type | Description |
|--------------------------------------------|------|--------------------------------------------------------------------|
| `--service-discovery` | flag | Enable discovery. |
| `--selector key=value ...` | list | Label selectors (regular mode). |
| `--prefill-selector` / `--decode-selector` | list | Label selectors for PD mode. |
| `--service-discovery-namespace` | str | Namespace to watch. |
| `--service-discovery-port` | int | Worker port (default 80). |
| `--bootstrap-port-annotation` | str | Prefill bootstrap annotation (default `sglang.ai/bootstrap-port`). |
---
## Observability
Enable Prometheus metrics:
```bash
python -m sglang_router.launch_router \
--worker-urls http://worker1:8000 http://worker2:8001 \
--prometheus-host 0.0.0.0 \
--prometheus-port 29000
```
Key metrics:
| Metric | Type | Description |
|--------|------|-------------|
| `sgl_router_requests_total` | Counter | Total requests by endpoint/method. |
| `sgl_router_processed_requests_total` | Counter | Requests processed per worker. |
| `sgl_router_active_workers` | Gauge | Healthy worker count. |
| `sgl_router_running_requests` | Gauge | In-flight requests per worker. |
| `sgl_router_cache_hits_total` / `misses_total` | Counter | Cache-aware routing hits/misses. |
| `sgl_router_generate_duration_seconds` | Histogram | Request latency distribution. |
Enable request ID propagation:
```bash
python -m sglang_router.launch_router \
--worker-urls http://worker1:8000 \
--request-id-headers x-request-id x-trace-id
```
Enable opentelmetry tracing:
```bash
python -m sglang_router.launch_router \
--worker-urls http://worker1:8000 \
--enable-trace \
--otlp-traces-endpoint 0.0.0.0:4317
```
---
## Troubleshooting
1. **Workers never ready**
Increase `--worker-startup-timeout-secs` or ensure health probes respond before router startup.
2. **Load imbalance / hot workers**
Inspect `sgl_router_processed_requests_total` and tune cache-aware thresholds (`--balance-*`, `--cache-threshold`).
3. **Circuit breaker flapping**
Increase `--cb-failure-threshold` or extend the timeout/window durations. Consider temporarily disabling retries.
4. **Queue overflow (429)**
Increase `--queue-size` or reduce client concurrency. Ensure `--max-concurrent-requests` matches downstream capacity.
5. **Memory growth**
Reduce `--max-tree-size` or lower `--eviction-interval-secs` for more aggressive cache pruning.
6. **Debugging**
```bash
python -m sglang_router.launch_router \
--worker-urls http://worker1:8000 \
--log-level debug \
--log-dir ./router_logs
```
---
SGLang Model Gateway continues to evolve alongside the SGLang runtime. Keep CLI flags, integrations, and documentation aligned when adopting new features or contributing improvements.
docs/advanced_features/sgl_model_gateway.md
+1654
View File
File diff suppressed because it is too large Load Diff
docs/index.rst
+1 -1
View File
@@ -52,7 +52,7 @@ Its core features include:
advanced_features/pd_multiplexing.md
advanced_features/vlm_query.ipynb
advanced_features/dp_for_multi_modal_encoder.md
advanced_features/router.md
advanced_features/sgl_model_gateway.md
advanced_features/deterministic_inference.md
advanced_features/observability.md
advanced_features/checkpoint_engine.md
sgl-model-gateway/README.md
+227 -10
View File
@@ -31,15 +31,17 @@ High-performance model routing control and data plane for large-scale LLM deploy
- Multi-model HTTP serving and inference gateway routing with model-specific policies.
- Prefill/decode disaggregation, including bootstrap port handling and cache-aware merging.
- gRPC routing with fully Rust tokenizer loading, reasoning parser selection, and tool parser integration for OpenAI-compatible endpoints—supporting streaming and non-streaming modes across DeepSeek, Llama, Kimi K2, Qwen, GPT-OSS, Mistral, Step-3, GLM4, GLM4.7 and other reasoning-capable models.
- OpenAI-compatible `/v1/chat/completions`, `/v1/responses`, `/v1/conversations`, `/v1/embeddings`, and `/v1/rerank` endpoints.
- OpenAI-compatible `/v1/chat/completions`, `/v1/responses`, `/v1/conversations`, `/v1/embeddings`, `/v1/rerank`, `/v1/classify` endpoints.
- **Tokenization APIs**: HTTP endpoints for tokenize (`/v1/tokenize`) and detokenize (`/v1/detokenize`) with batch support; tokenizer management APIs for dynamic registration.
- **Parser endpoints**: Reasoning parser (`/parse/reasoning`) and function call parser (`/parse/function_call`) for separating reasoning content and extracting tool calls.
- Native MCP client integration supporting all MCP transport protocols (STDIO, HTTP, SSE, and Streamable) for tool execution loops.
- Pluggable history connectors: in-memory, disabled, or Oracle ATP (with pooling and credential support).
- Pluggable history connectors: in-memory, disabled, Oracle ATP, or PostgreSQL (with pooling and credential support).
- Reliability controls: retry with jitter, worker-scoped circuit breakers, token bucket limiter with optional queue, and cache flush APIs.
- Service discovery for regular and PD workloads with independent selectors.
- Prometheus metrics and structured tracing for every stage of routing.
- **Comprehensive observability**: 40+ Prometheus metrics across HTTP, router, worker, circuit breaker, retry, discovery, MCP, and database layers; OpenTelemetry tracing with OTLP export; structured logging with request ID propagation.
## Documentation
- **User Guide**: [docs.sglang.io/advanced_features/router.html](https://docs.sglang.io/advanced_features/router.html)
- **User Guide**: [docs.sglang.io/advanced_features/sgl_model_gateway.html](https://docs.sglang.io/advanced_features/sgl_model_gateway.html)
- Additional guides, API references, and deployment patterns are continuously updated alongside SGLang releases.
## Installation
@@ -476,11 +478,115 @@ The HTTP router exposes the full OpenAI-compatible surface area (`/generate`, `/
| `POST /v1/responses` | Create background responses, returns response IDs. |
| `GET /v1/responses/{id}` | Retrieve stored responses. |
| Conversation endpoints (`/v1/conversations`, `/v1/conversations/{id}`, `/v1/conversations/{id}/items`) | Manage chat history. |
| `POST /v1/embeddings` | Forward embedding requests. |
| `POST /v1/embeddings` | Forward embedding requests (HTTP and gRPC). |
| `POST /v1/rerank`, `POST /rerank` | Ranking APIs. |
| `POST /v1/classify` | Text classification endpoint. |
Public health endpoints (`/liveness`, `/readiness`, `/health`, `/health_generate`) reflect registry state; readiness ensures PD workers are paired and IGW has at least one healthy route.
### Tokenization Endpoints
The gateway provides HTTP endpoints for text tokenization, designed to mirror the SGLang Python tokenization API with support for batch operations.
| Endpoint | Method | Description |
|-------------------------------|----------|-------------------------------------------------------|
| `POST /v1/tokenize` | `POST` | Tokenize text to token IDs (single or batch). |
| `POST /v1/detokenize` | `POST` | Convert token IDs back to text (single or batch). |
| `POST /v1/tokenizers` | `POST` | Register a new tokenizer (async, returns job status). |
| `GET /v1/tokenizers` | `GET` | List all registered tokenizers. |
| `GET /v1/tokenizers/{id}` | `GET` | Get tokenizer info by UUID. |
| `GET /v1/tokenizers/{id}/status` | `GET` | Check async tokenizer loading status. |
| `DELETE /v1/tokenizers/{id}` | `DELETE` | Remove a tokenizer from the registry. |
**Tokenize Request:**
```json
{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Hello, world!"
}
```
**Batch Tokenize Request:**
```json
{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": ["Hello", "World", "How are you?"]
}
```
**Tokenize Response:**
```json
{
"tokens": [15339, 11, 1917, 0],
"count": 4,
"char_count": 13
}
```
**Detokenize Request:**
```json
{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"tokens": [15339, 11, 1917, 0],
"skip_special_tokens": true
}
```
**Add Tokenizer (async registration):**
```bash
# Register from HuggingFace
curl -X POST http://localhost:30000/v1/tokenizers \
-H "Content-Type: application/json" \
-d '{"name": "llama3", "source": "meta-llama/Llama-3.1-8B-Instruct"}'
# Check status
curl http://localhost:30000/v1/tokenizers/{tokenizer_id}/status
```
### Parser Endpoints
The gateway provides admin endpoints for parsing reasoning content and function calls from LLM outputs.
| Endpoint | Method | Description |
|--------------------------|--------|--------------------------------------------------------|
| `POST /parse/reasoning` | `POST` | Separate reasoning (`<think>`) from normal text. |
| `POST /parse/function_call` | `POST` | Parse function/tool calls from text. |
**Separate Reasoning Request:**
```json
{
"text": "<think>Let me analyze this step by step...</think>The answer is 42.",
"parser": "deepseek-r1"
}
```
**Response:**
```json
{
"normal_text": "The answer is 42.",
"reasoning_text": "Let me analyze this step by step..."
}
```
**Supported Reasoning Parsers:**
- `deepseek-r1` - DeepSeek-R1 (initial reasoning mode)
- `qwen3` - Qwen-3 models
- `qwen3-thinking` / `qwen-thinking` - Qwen thinking variant
- `kimi` - Kimi K2 with Unicode tokens
- `glm45` / `glm47` - GLM-4.5/4.6/4.7 models
- `step3` - Step-3 models
- `minimax` - MiniMax models
**Function Call Parsing:**
```json
{
"text": "{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}",
"parser": "json"
}
```
Supported tool parsers: `json`, `python`, `xml`.
## Conversations, Responses, and Data Connectors
- `--history-backend memory` (default) stores responses and conversations in-process.
- `--history-backend none` disables persistence while keeping APIs.
@@ -566,11 +672,67 @@ Only one of `--oracle-dsn` or `--oracle-tns-alias` should be supplied.
Per-model overrides are available in PD mode (`--prefill-policy`, `--decode-policy`) and IGW mode via the worker registry.
## Observability
- **Logging**: Structured tracing through `tracing` with optional file sink (`--log-dir`) and `--log-level` (`debug`, `info`, `warn`, `error`).
- **Prometheus Metrics**: Enable with `--prometheus-host`/`--prometheus-port` (defaults to `0.0.0.0:29000`). Metrics cover request latency, retry behavior, circuit breaker states, worker health/load, queue depth, PD pipeline stats, tokenizer timings, and MCP activity.
- **Request IDs**: Configurable headers via `--request-id-headers`; responses include `x-request-id`.
- **CORS**: Set `--cors-allowed-origins` for browser access.
- **Request Tracing via OpenTelemetry**: Enable with `--enable-trace` and set opentelemetry collector endpoint with `--otlp-traces-endpoint <ip>:<port>`.
### Logging
Structured tracing through `tracing` with optional file sink (`--log-dir`) and `--log-level` (`debug`, `info`, `warn`, `error`).
### Prometheus Metrics
Enable with `--prometheus-host`/`--prometheus-port` (defaults to `0.0.0.0:29000`).
**Metric Categories (40+ metrics):**
| Layer | Metric Prefix | Description |
|-------|---------------|-------------|
| HTTP | `smg_http_*` | Request counts, duration, active connections, rate limiting |
| Router | `smg_router_*` | Requests by model/endpoint, latency, errors, upstream responses |
| Inference | `smg_router_ttft/tpot/tokens_*` | Time to first token, time per output token, token counts (gRPC) |
| Worker | `smg_worker_*` | Pool size, active connections, health checks, selection events |
| Circuit Breaker | `smg_worker_cb_*` | State (closed/open/half-open), transitions, outcomes |
| Retry | `smg_worker_retries_*` | Retry attempts, exhausted retries, backoff duration |
| Discovery | `smg_discovery_*` | K8s registrations, sync duration, workers discovered |
| MCP | `smg_mcp_*` | Tool calls, duration, active servers, iterations |
| Database | `smg_db_*` | Operations, duration, connections, items stored |
**Key Metrics:**
- `smg_router_ttft_seconds` - Time to first token histogram (gRPC mode)
- `smg_router_tpot_seconds` - Time per output token histogram (gRPC mode)
- `smg_router_tokens_total` - Total input/output tokens by model
- `smg_router_generation_duration_seconds` - End-to-end generation time
- `smg_worker_cb_state` - Circuit breaker state gauge (0=closed, 1=open, 2=half-open)
**Duration Buckets:**
1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s, 15s, 30s, 45s, 60s, 90s, 120s, 180s, 240s
### OpenTelemetry Tracing
Enable distributed tracing with OTLP export:
```bash
python -m sglang_router.launch_router \
--worker-urls http://worker1:8000 \
--enable-trace \
--otlp-traces-endpoint localhost:4317
```
**Features:**
- OTLP/gRPC exporter (default port 4317)
- W3C Trace Context propagation for HTTP and gRPC
- Batch span processing (500ms delay, 64 span batch size)
- Custom filtering to reduce noise (only exports relevant spans)
- Trace context injection into upstream worker requests
**Configuration:**
- `--enable-trace` - Enable OpenTelemetry tracing
- `--otlp-traces-endpoint <host:port>` - OTLP collector endpoint
### Request ID Propagation
Configure headers for request ID extraction:
```bash
--request-id-headers x-request-id x-trace-id x-correlation-id
```
Responses include `x-request-id` header for correlation.
### CORS
Set `--cors-allowed-origins` for browser access.
## Security
@@ -608,6 +770,61 @@ curl -X POST "http://localhost:8080/add_worker?url=http://worker3:8000&api_key=w
- Router logs a warning when a worker is registered without a key while the router expects authentication.
- When router and workers share the same key, still include the key when invoking dynamic registration APIs.
### TLS (HTTPS) for Gateway Server
Enable TLS to serve the gateway over HTTPS:
```bash
python3 -m sglang_router.launch_router \
--worker-urls http://worker1:8000 \
--tls-cert-path /path/to/server.crt \
--tls-key-path /path/to/server.key
```
| Parameter | Description |
|-----------|-------------|
| `--tls-cert-path` | Path to server certificate (PEM format) |
| `--tls-key-path` | Path to server private key (PEM format) |
Both parameters must be provided together. The gateway uses rustls with the ring crypto provider for TLS termination. If TLS is not configured, the gateway falls back to plain HTTP.
### mTLS for Worker Communication
Enable mutual TLS (mTLS) for secure communication with workers in HTTP mode:
```bash
python3 -m sglang_router.launch_router \
--worker-urls https://worker1:8443 https://worker2:8443 \
--client-cert-path /path/to/client.crt \
--client-key-path /path/to/client.key \
--ca-cert-path /path/to/ca.crt
```
| Parameter | Description |
|-----------|-------------|
| `--client-cert-path` | Path to client certificate for mTLS (PEM format) |
| `--client-key-path` | Path to client private key for mTLS (PEM format) |
| `--ca-cert-path` | Path to CA certificate for verifying worker TLS (PEM format) |
**Key Points:**
- Client certificate and key must be provided together
- Multiple CA certificates can be added with multiple `--ca-cert-path` flags
- Uses rustls backend when TLS is configured
- Single HTTP client is created for all workers (assumes single security domain)
- TCP keepalive (30 seconds) is enabled for long-lived connections
**Full TLS Example (Gateway HTTPS + Worker mTLS):**
```bash
python3 -m sglang_router.launch_router \
--worker-urls https://worker1:8443 https://worker2:8443 \
--tls-cert-path /etc/certs/server.crt \
--tls-key-path /etc/certs/server.key \
--client-cert-path /etc/certs/client.crt \
--client-key-path /etc/certs/client.key \
--ca-cert-path /etc/certs/ca.crt \
--api-key "secure-api-key"
```
## Development & Testing
```bash
# Build Rust components (debug mode, fast)