[model-gateway][docs] Add Classification API documentation (#16182)

Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-12-30 10:53:31 -08:00
parent d17b9e6392
commit 4bc2f2e0bc
2 changed files with 102 additions and 1 deletions
--- a/docs/advanced_features/sgl_model_gateway.md
+++ b/docs/advanced_features/sgl_model_gateway.md
@@ -24,6 +24,7 @@ SGLang Model Gateway is a high-performance model-routing gateway for large-scale
   - [Inference Endpoints](#inference-endpoints)
   - [Tokenization Endpoints](#tokenization-endpoints)
   - [Parser Endpoints](#parser-endpoints)
+   - [Classification API](#classification-api)
   - [Conversation and Response APIs](#conversation-and-response-apis)
   - [Worker Management APIs](#worker-management-apis)
   - [Admin and Health Endpoints](#admin-and-health-endpoints)
@@ -85,7 +86,7 @@ SGLang Model Gateway is a high-performance model-routing gateway for large-scale
 ### Data Plane

 - **HTTP routers** (regular & PD) implement `/generate`, `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/rerank`, `/v1/classify`, `/v1/tokenize`, `/v1/detokenize`, and associated admin endpoints.
- **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing, including embeddings.
+- **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing, including embeddings and classification.
 - **OpenAI router** proxies OpenAI-compatible endpoints to external vendors (OpenAI, xAI, etc.) while keeping chat history and multi-turn orchestration local.

 ### Storage and Privacy
@@ -420,6 +421,59 @@ The gateway provides admin endpoints for parsing reasoning content and function
 }
 ```

+### Classification API
+
+The `/v1/classify` endpoint provides text classification using sequence classification models (e.g., `Qwen2ForSequenceClassification`, `BertForSequenceClassification`).
+
+#### Request
+
+```bash
+curl http://localhost:30000/v1/classify \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "jason9693/Qwen2.5-1.5B-apeach",
+    "input": "I love this product!"
+  }'
+```
+
+#### Response
+
+```json
+{
+  "id": "classify-a1b2c3d4-5678-90ab-cdef-1234567890ab",
+  "object": "list",
+  "created": 1767034308,
+  "model": "jason9693/Qwen2.5-1.5B-apeach",
+  "data": [
+    {
+      "index": 0,
+      "label": "positive",
+      "probs": [0.12, 0.88],
+      "num_classes": 2
+    }
+  ],
+  "usage": {
+    "prompt_tokens": 6,
+    "completion_tokens": 0,
+    "total_tokens": 6
+  }
+}
+```
+
+#### Response Fields
+
+| Field | Description |
+|-------|-------------|
+| `label` | Predicted class label (from model's `id2label` config, or `LABEL_N` fallback) |
+| `probs` | Probability distribution over all classes (softmax of logits) |
+| `num_classes` | Number of classification classes |
+
+#### Notes
+
+- Classification reuses the embedding backend—the scheduler returns logits which are converted to probabilities via softmax
+- Labels come from the model's HuggingFace config (`id2label` field); models without this mapping use generic labels (`LABEL_0`, `LABEL_1`, etc.)
+- Both HTTP and gRPC routers support classification
+
 ### Conversation and Response APIs

 | Method | Path | Description |
--- a/sgl-model-gateway/README.md
+++ b/sgl-model-gateway/README.md
@@ -482,6 +482,53 @@ The HTTP router exposes the full OpenAI-compatible surface area (`/generate`, `/
 | `POST /v1/rerank`, `POST /rerank`                                                | Ranking APIs.                                              |
 | `POST /v1/classify`                                                              | Text classification endpoint.                              |

+### Classification API
+
+The `/v1/classify` endpoint provides text classification using sequence classification models (e.g., `Qwen2ForSequenceClassification`, `BertForSequenceClassification`).
+
+**Request:**
+```bash
+curl http://localhost:30000/v1/classify \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "jason9693/Qwen2.5-1.5B-apeach",
+    "input": "I love this product!"
+  }'
+```
+
+**Response:**
+```json
+{
+  "id": "classify-a1b2c3d4-5678-90ab-cdef-1234567890ab",
+  "object": "list",
+  "created": 1767034308,
+  "model": "jason9693/Qwen2.5-1.5B-apeach",
+  "data": [
+    {
+      "index": 0,
+      "label": "positive",
+      "probs": [0.12, 0.88],
+      "num_classes": 2
+    }
+  ],
+  "usage": {
+    "prompt_tokens": 6,
+    "completion_tokens": 0,
+    "total_tokens": 6
+  }
+}
+```
+
+**Fields:**
+- `label`: Predicted class label (from model's `id2label` config, or `LABEL_N` fallback)
+- `probs`: Probability distribution over all classes (softmax of logits)
+- `num_classes`: Number of classification classes
+
+**Notes:**
+- Classification reuses the embedding backend—the scheduler returns logits which are converted to probabilities via softmax
+- Labels come from the model's HuggingFace config (`id2label` field); models without this mapping use generic labels (`LABEL_0`, `LABEL_1`, etc.)
+- Both HTTP and gRPC routers support classification
+
 Public health endpoints (`/liveness`, `/readiness`, `/health`, `/health_generate`) reflect registry state; readiness ensures PD workers are paired and IGW has at least one healthy route.

 ### Tokenization Endpoints