sglang/docs/start/send_request.ipynb

Quick Start: Sending Requests

This notebook provides a quick-start guide for using SGLang after installation.

Launch a server

This code block is equivalent to executing

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 --host 0.0.0.0

in your terminal and wait for the server to be ready.

In [7]:

from sglang.utils import (
    execute_shell_command,
    wait_for_server,
    terminate_process,
    print_highlight,
)

server_process = execute_shell_command(
"""
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 --host 0.0.0.0
"""
)

wait_for_server("http://localhost:30000")

Send a Request

Once the server is up, you can send test requests using curl. The server implements the OpenAI-compatible API.

In [9]:

import subprocess

curl_command = """
curl http://localhost:30000/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -H "Authorization: Bearer None" \\
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "What is an LLM? Tell me in one sentence."
      }
    ]
  }'
"""

response = subprocess.check_output(curl_command, shell=True).decode()

print_highlight(response)

Using OpenAI Python Client

You can use the OpenAI Python API library to send requests.

In [3]:

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant"},
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print_highlight(response)

Using Native Generation APIs

You can also use the native /generate endpoint. It provides more flexiblity. An API reference is available at Sampling Parameters.

In [5]:

import requests

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "The capital of France is",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 32,
        },
    },
)

print_highlight(response.json())

In [6]:

terminate_process(server_process)