sglang/docs/backend/native_api.ipynb

Native APIs

Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce these following APIs:

• /generate (text generation model)
• /get_model_info
• /get_server_info
• /health
• /health_generate
• /flush_cache
• /update_weights
• /encode(embedding model)
• /classify(reward model)

We mainly use requests to test these APIs in the following examples. You can also use curl.

Launch A Server

In [ ]:

from sglang.utils import (
    execute_shell_command,
    wait_for_server,
    terminate_process,
    print_highlight,
)

import requests

server_process = execute_shell_command(
    """
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --port=30010
"""
)

wait_for_server("http://localhost:30010")

Generate (text generation model)

Generate completions. This is similar to the /v1/completions in OpenAI API. Detailed parameters can be found in the sampling parameters.

In [ ]:

url = "http://localhost:30010/generate"
data = {"text": "What is the capital of France?"}

response = requests.post(url, json=data)
print_highlight(response.json())

Get Model Info

Get the information of the model.

• model_path: The path/name of the model.
• is_generation: Whether the model is used as generation model or embedding model.

In [ ]:

url = "http://localhost:30010/get_model_info"

response = requests.get(url)
response_json = response.json()
print_highlight(response_json)
assert response_json["model_path"] == "meta-llama/Llama-3.2-1B-Instruct"
assert response_json["is_generation"] is True
assert response_json.keys() == {"model_path", "is_generation"}

Get Server Info

Gets the server information including CLI arguments, token limits, and memory pool sizes.

• Note: get_server_info merges the following deprecated endpoints:
  • get_server_args
  • get_memory_pool_size
  • get_max_total_num_tokens

In [ ]:

# get_server_info

url = "http://localhost:30010/get_server_info"

response = requests.get(url)
print_highlight(response.text)

Health Check

• /health: Check the health of the server.
• /health_generate: Check the health of the server by generating one token.

In [ ]:

url = "http://localhost:30010/health_generate"

response = requests.get(url)
print_highlight(response.text)

In [ ]:

url = "http://localhost:30010/health"

response = requests.get(url)
print_highlight(response.text)

Flush Cache

Flush the radix cache. It will be automatically triggered when the model weights are updated by the /update_weights API.

In [ ]:

# flush cache

url = "http://localhost:30010/flush_cache"

response = requests.post(url)
print_highlight(response.text)

Update Weights

Update model weights without restarting the server. Use for continuous evaluation during training. Only applicable for models with the same architecture and parameter size.

In [ ]:

# successful update with same architecture and size

url = "http://localhost:30010/update_weights"
data = {"model_path": "meta-llama/Llama-3.2-1B"}

response = requests.post(url, json=data)
print_highlight(response.text)
assert response.json()["success"] is True
assert response.json()["message"] == "Succeeded to update model weights."
assert response.json().keys() == {"success", "message"}

In [ ]:

# failed update with different parameter size

url = "http://localhost:30010/update_weights"
data = {"model_path": "meta-llama/Llama-3.2-3B"}

response = requests.post(url, json=data)
response_json = response.json()
print_highlight(response_json)
assert response_json["success"] is False
assert response_json["message"] == (
    "Failed to update weights: The size of tensor a (2048) must match "
    "the size of tensor b (3072) at non-singleton dimension 1.\n"
    "Rolling back to original weights."
)

Encode (embedding model)

Encode text into embeddings. Note that this API is only available for embedding models and will raise an error for generation models. Therefore, we launch a new server to server an embedding model.

In [ ]:

terminate_process(server_process)

embedding_process = execute_shell_command(
    """
python -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-7B-instruct \
    --port 30020 --host 0.0.0.0 --is-embedding
"""
)

wait_for_server("http://localhost:30020")

In [ ]:

# successful encode for embedding model

url = "http://localhost:30020/encode"
data = {"model": "Alibaba-NLP/gte-Qwen2-7B-instruct", "text": "Once upon a time"}

response = requests.post(url, json=data)
response_json = response.json()
print_highlight(f"Text embedding (first 10): {response_json['embedding'][:10]}")

Classify (reward model)

SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations.

In [ ]:

terminate_process(embedding_process)

# Note that SGLang now treats embedding models and reward models as the same type of models.
# This will be updated in the future.

reward_process = execute_shell_command(
    """
python -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --port 30030 --host 0.0.0.0 --is-embedding
"""
)

wait_for_server("http://localhost:30030")

In [ ]:

from transformers import AutoTokenizer

PROMPT = (
    "What is the range of the numeric output of a sigmoid node in a neural network?"
)

RESPONSE1 = "The output of a sigmoid node is bounded between -1 and 1."
RESPONSE2 = "The output of a sigmoid node is bounded between 0 and 1."

CONVS = [
    [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE1}],
    [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE2}],
]

tokenizer = AutoTokenizer.from_pretrained("Skywork/Skywork-Reward-Llama-3.1-8B-v0.2")
prompts = tokenizer.apply_chat_template(CONVS, tokenize=False)

url = "http://localhost:30030/classify"
data = {"model": "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2", "text": prompts}

responses = requests.post(url, json=data).json()
for response in responses:
    print_highlight(f"reward: {response['embedding'][0]}")

In [ ]:

terminate_process(reward_process)