From de2799f3f5702977876a06fa84678cc9ad0abd65 Mon Sep 17 00:00:00 2001 From: Simo Lin Date: Sun, 28 Dec 2025 23:35:37 -0800 Subject: [PATCH] [docs] Fix non-clickable ToC links in model gateway documentation (#16054) --- docs/advanced_features/sgl_model_gateway.md | 40 ++++++++++----------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/advanced_features/sgl_model_gateway.md b/docs/advanced_features/sgl_model_gateway.md index c3891e601..4db449008 100644 --- a/docs/advanced_features/sgl_model_gateway.md +++ b/docs/advanced_features/sgl_model_gateway.md @@ -10,38 +10,38 @@ SGLang Model Gateway is a high-performance model-routing gateway for large-scale 2. [Architecture](#architecture) - [Control Plane](#control-plane) - [Data Plane](#data-plane) - - [Storage & Privacy](#storage--privacy) + - [Storage and Privacy](#storage-and-privacy) 3. [Installation](#installation) 4. [Quick Start](#quick-start) 5. [Deployment Modes](#deployment-modes) - - [Co-launch Router + Workers](#co-launch-router--workers) + - [Co-launch Router and Workers](#co-launch-router-and-workers) - [Separate Launch (HTTP)](#separate-launch-http) - [gRPC Launch](#grpc-launch) - - [Prefill/Decode Disaggregation](#prefilldecode-disaggregation) + - [Prefill-Decode Disaggregation](#prefill-decode-disaggregation) - [OpenAI Backend Proxy](#openai-backend-proxy) - [Multi-Model Inference Gateway](#multi-model-inference-gateway) 6. [API Reference](#api-reference) - [Inference Endpoints](#inference-endpoints) - [Tokenization Endpoints](#tokenization-endpoints) - [Parser Endpoints](#parser-endpoints) - - [Conversation & Response APIs](#conversation--response-apis) + - [Conversation and Response APIs](#conversation-and-response-apis) - [Worker Management APIs](#worker-management-apis) - - [Admin & Health Endpoints](#admin--health-endpoints) + - [Admin and Health Endpoints](#admin-and-health-endpoints) 7. [Load Balancing Policies](#load-balancing-policies) -8. [Reliability & Flow Control](#reliability--flow-control) +8. [Reliability and Flow Control](#reliability-and-flow-control) - [Retries](#retries) - [Circuit Breaker](#circuit-breaker) - - [Rate Limiting & Queuing](#rate-limiting--queuing) + - [Rate Limiting and Queuing](#rate-limiting-and-queuing) - [Health Checks](#health-checks) 9. [Reasoning Parser Integration](#reasoning-parser-integration) 10. [Tool Call Parsing](#tool-call-parsing) 11. [Tokenizer Management](#tokenizer-management) 12. [MCP Integration](#mcp-integration) 13. [Service Discovery (Kubernetes)](#service-discovery-kubernetes) -14. [History & Data Connectors](#history--data-connectors) +14. [History and Data Connectors](#history-and-data-connectors) 15. [WASM Middleware](#wasm-middleware) 16. [Language Bindings](#language-bindings) -17. [Security & Authentication](#security--authentication) +17. [Security and Authentication](#security-and-authentication) - [TLS (HTTPS) for Gateway Server](#tls-https-for-gateway-server) - [mTLS for Worker Communication](#mtls-for-worker-communication) 18. [Observability](#observability) @@ -49,7 +49,7 @@ SGLang Model Gateway is a high-performance model-routing gateway for large-scale - [OpenTelemetry Tracing](#opentelemetry-tracing) - [Logging](#logging) 19. [Production Recommendations](#production-recommendations) - - [Security](#security-1) + - [Security Best Practices](#security-best-practices) - [High Availability](#high-availability) - [Performance](#performance) - [Kubernetes Deployment](#kubernetes-deployment) @@ -88,7 +88,7 @@ SGLang Model Gateway is a high-performance model-routing gateway for large-scale - **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing, including embeddings. - **OpenAI router** proxies OpenAI-compatible endpoints to external vendors (OpenAI, xAI, etc.) while keeping chat history and multi-turn orchestration local. -### Storage & Privacy +### Storage and Privacy - Conversation and response history is stored at the router tier (memory, none, Oracle ATP, or PostgreSQL). The same history can power multiple models or MCP loops without sending data to upstream vendors. - `/v1/responses` agentic flows, MCP sessions, and conversation APIs share the same storage layer, enabling compliance for regulated workloads. @@ -170,7 +170,7 @@ python -m sglang_router.launch_router \ ## Deployment Modes -### Co-launch Router + Workers +### Co-launch Router and Workers Launch the router and a fleet of SGLang workers in one process: @@ -238,7 +238,7 @@ python -m sglang_router.launch_router \ The gRPC router supports both regular HTTP-equivalent serving and PD (prefill/decode) serving. Provide `--tokenizer-path` or `--model-path` (HuggingFace ID or local directory) whenever connection mode resolves to gRPC. -### Prefill/Decode Disaggregation +### Prefill-Decode Disaggregation Split prefill and decode workers for PD-aware caching and balancing: @@ -420,7 +420,7 @@ The gateway provides admin endpoints for parsing reasoning content and function } ``` -### Conversation & Response APIs +### Conversation and Response APIs | Method | Path | Description | |--------|------|-------------| @@ -487,7 +487,7 @@ Response: } ``` -### Admin & Health Endpoints +### Admin and Health Endpoints | Method | Path | Description | |--------|------|-------------| @@ -537,7 +537,7 @@ Response: --- -## Reliability & Flow Control +## Reliability and Flow Control ### Retries @@ -590,7 +590,7 @@ python -m sglang_router.launch_router \ - **Open**: Failing, requests rejected immediately - **Half-Open**: Testing recovery, limited requests allowed -### Rate Limiting & Queuing +### Rate Limiting and Queuing ```bash python -m sglang_router.launch_router \ @@ -796,7 +796,7 @@ Prefill pods can expose bootstrap ports via the `sglang.ai/bootstrap-port` annot --- -## History & Data Connectors +## History and Data Connectors | Backend | Description | Usage | |---------|-------------|-------| @@ -1042,7 +1042,7 @@ go test -tags=integration -v ./... --- -## Security & Authentication +## Security and Authentication ### Router API Key @@ -1210,7 +1210,7 @@ Responses include `x-request-id` header for correlation. This section provides guidance for deploying SGLang Model Gateway in production environments. -### Security +### Security Best Practices **Always enable TLS in production:**