sglang/docs/backend/native_api.ipynb

Native APIs

Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce these following APIs:

• /generate
• /get_server_args
• /get_model_info
• /health
• /health_generate
• /flush_cache
• /get_memory_pool_size
• /update_weights
• /encode

We mainly use requests to test these APIs in the following examples. You can also use curl.

Launch A Server

In [ ]:

from sglang.utils import (
    execute_shell_command,
    wait_for_server,
    terminate_process,
    print_highlight,
)

server_process = execute_shell_command(
    """
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --port=30010
"""
)

wait_for_server("http://localhost:30010")

Generate

Generate completions. This is similar to the /v1/completions in OpenAI API. Detailed parameters can be found in the sampling parameters.

In [ ]:

import requests

url = "http://localhost:30010/generate"
data = {"text": "What is the capital of France?"}

response = requests.post(url, json=data)
print_highlight(response.json())

Get Server Args

Get the arguments of a server.

In [ ]:

url = "http://localhost:30010/get_server_args"

response = requests.get(url)
print_highlight(response.json())

Get Model Info

Get the information of the model.

• model_path: The path/name of the model.
• is_generation: Whether the model is used as generation model or embedding model.

In [ ]:

url = "http://localhost:30010/get_model_info"

response = requests.get(url)
response_json = response.json()
print_highlight(response_json)
assert response_json["model_path"] == "meta-llama/Llama-3.2-1B-Instruct"
assert response_json["is_generation"] is True
assert response_json.keys() == {"model_path", "is_generation"}

Health Check

• /health: Check the health of the server.
• /health_generate: Check the health of the server by generating one token.

In [ ]:

url = "http://localhost:30010/health_generate"

response = requests.get(url)
print_highlight(response.text)

In [ ]:

url = "http://localhost:30010/health"

response = requests.get(url)
print_highlight(response.text)

Flush Cache

Flush the radix cache. It will be automatically triggered when the model weights are updated by the /update_weights API.

In [ ]:

# flush cache

url = "http://localhost:30010/flush_cache"

response = requests.post(url)
print_highlight(response.text)

Get Memory Pool Size

Get the memory pool size in number of tokens.

In [ ]:

# get_memory_pool_size

url = "http://localhost:30010/get_memory_pool_size"

response = requests.get(url)
print_highlight(response.text)

Update Weights

Update model weights without restarting the server. Use for continuous evaluation during training. Only applicable for models with the same architecture and parameter size.

In [ ]:

# successful update with same architecture and size

url = "http://localhost:30010/update_weights"
data = {"model_path": "meta-llama/Llama-3.2-1B"}

response = requests.post(url, json=data)
print_highlight(response.text)
assert response.json()["success"] == True
assert response.json()["message"] == "Succeeded to update model weights."
assert response.json().keys() == {"success", "message"}

In [ ]:

# failed update with different parameter size

url = "http://localhost:30010/update_weights"
data = {"model_path": "meta-llama/Llama-3.2-3B"}

response = requests.post(url, json=data)
response_json = response.json()
print_highlight(response_json)
assert response_json["success"] == False
assert response_json["message"] == (
    "Failed to update weights: The size of tensor a (2048) must match "
    "the size of tensor b (3072) at non-singleton dimension 1.\n"
    "Rolling back to original weights."
)

Encode

Encode text into embeddings. Note that this API is only available for embedding models and will raise an error for generation models. Therefore, we launch a new server to server an embedding model.

In [ ]:

terminate_process(server_process)

embedding_process = execute_shell_command(
    """
python -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-7B-instruct \
    --port 30020 --host 0.0.0.0 --is-embedding
"""
)

wait_for_server("http://localhost:30020")

In [ ]:

# successful encode for embedding model

url = "http://localhost:30020/encode"
data = {"model": "Alibaba-NLP/gte-Qwen2-7B-instruct", "text": "Once upon a time"}

response = requests.post(url, json=data)
response_json = response.json()
print_highlight(f"Text embedding (first 10): {response_json['embedding'][:10]}")

In [43]:

terminate_process(embedding_process)