diff --git a/docs/advanced_features/sgl_model_gateway.md b/docs/advanced_features/sgl_model_gateway.md index 4db449008..2f15c6bd3 100644 --- a/docs/advanced_features/sgl_model_gateway.md +++ b/docs/advanced_features/sgl_model_gateway.md @@ -24,6 +24,7 @@ SGLang Model Gateway is a high-performance model-routing gateway for large-scale - [Inference Endpoints](#inference-endpoints) - [Tokenization Endpoints](#tokenization-endpoints) - [Parser Endpoints](#parser-endpoints) + - [Classification API](#classification-api) - [Conversation and Response APIs](#conversation-and-response-apis) - [Worker Management APIs](#worker-management-apis) - [Admin and Health Endpoints](#admin-and-health-endpoints) @@ -85,7 +86,7 @@ SGLang Model Gateway is a high-performance model-routing gateway for large-scale ### Data Plane - **HTTP routers** (regular & PD) implement `/generate`, `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/rerank`, `/v1/classify`, `/v1/tokenize`, `/v1/detokenize`, and associated admin endpoints. -- **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing, including embeddings. +- **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing, including embeddings and classification. - **OpenAI router** proxies OpenAI-compatible endpoints to external vendors (OpenAI, xAI, etc.) while keeping chat history and multi-turn orchestration local. ### Storage and Privacy @@ -420,6 +421,59 @@ The gateway provides admin endpoints for parsing reasoning content and function } ``` +### Classification API + +The `/v1/classify` endpoint provides text classification using sequence classification models (e.g., `Qwen2ForSequenceClassification`, `BertForSequenceClassification`). + +#### Request + +```bash +curl http://localhost:30000/v1/classify \ + -H "Content-Type: application/json" \ + -d '{ + "model": "jason9693/Qwen2.5-1.5B-apeach", + "input": "I love this product!" + }' +``` + +#### Response + +```json +{ + "id": "classify-a1b2c3d4-5678-90ab-cdef-1234567890ab", + "object": "list", + "created": 1767034308, + "model": "jason9693/Qwen2.5-1.5B-apeach", + "data": [ + { + "index": 0, + "label": "positive", + "probs": [0.12, 0.88], + "num_classes": 2 + } + ], + "usage": { + "prompt_tokens": 6, + "completion_tokens": 0, + "total_tokens": 6 + } +} +``` + +#### Response Fields + +| Field | Description | +|-------|-------------| +| `label` | Predicted class label (from model's `id2label` config, or `LABEL_N` fallback) | +| `probs` | Probability distribution over all classes (softmax of logits) | +| `num_classes` | Number of classification classes | + +#### Notes + +- Classification reuses the embedding backend—the scheduler returns logits which are converted to probabilities via softmax +- Labels come from the model's HuggingFace config (`id2label` field); models without this mapping use generic labels (`LABEL_0`, `LABEL_1`, etc.) +- Both HTTP and gRPC routers support classification + ### Conversation and Response APIs | Method | Path | Description | diff --git a/sgl-model-gateway/README.md b/sgl-model-gateway/README.md index 60be371d1..04d96a414 100644 --- a/sgl-model-gateway/README.md +++ b/sgl-model-gateway/README.md @@ -482,6 +482,53 @@ The HTTP router exposes the full OpenAI-compatible surface area (`/generate`, `/ | `POST /v1/rerank`, `POST /rerank` | Ranking APIs. | | `POST /v1/classify` | Text classification endpoint. | +### Classification API + +The `/v1/classify` endpoint provides text classification using sequence classification models (e.g., `Qwen2ForSequenceClassification`, `BertForSequenceClassification`). + +**Request:** +```bash +curl http://localhost:30000/v1/classify \ + -H "Content-Type: application/json" \ + -d '{ + "model": "jason9693/Qwen2.5-1.5B-apeach", + "input": "I love this product!" + }' +``` + +**Response:** +```json +{ + "id": "classify-a1b2c3d4-5678-90ab-cdef-1234567890ab", + "object": "list", + "created": 1767034308, + "model": "jason9693/Qwen2.5-1.5B-apeach", + "data": [ + { + "index": 0, + "label": "positive", + "probs": [0.12, 0.88], + "num_classes": 2 + } + ], + "usage": { + "prompt_tokens": 6, + "completion_tokens": 0, + "total_tokens": 6 + } +} +``` + +**Fields:** +- `label`: Predicted class label (from model's `id2label` config, or `LABEL_N` fallback) +- `probs`: Probability distribution over all classes (softmax of logits) +- `num_classes`: Number of classification classes + +**Notes:** +- Classification reuses the embedding backend—the scheduler returns logits which are converted to probabilities via softmax +- Labels come from the model's HuggingFace config (`id2label` field); models without this mapping use generic labels (`LABEL_0`, `LABEL_1`, etc.) +- Both HTTP and gRPC routers support classification + Public health endpoints (`/liveness`, `/readiness`, `/health`, `/health_generate`) reflect registry state; readiness ensures PD workers are paired and IGW has at least one healthy route. ### Tokenization Endpoints