sglang/docs/send_request.ipynb

Quick Start

Launch a server

This code uses subprocess.Popen to start an SGLang server process, equivalent to executing

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 --host 0.0.0.0 --log-level warning

in your command line and wait for the server to be ready.

In [1]:

from sglang.utils import execute_shell_command, wait_for_server, terminate_process


server_process = execute_shell_command("""
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 --host 0.0.0.0 --log-level warning
""")

wait_for_server("http://localhost:30000")
print("Server is ready. Proceeding with the next steps.")

Server is ready. Proceeding with the next steps.

Send a Request

Once the server is running, you can send test requests using curl.

In [2]:

!curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer None" \
  -d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is a LLM?"}]}'

{"id":"449710eb827c49c99b82ce187e912c2a","object":"chat.completion","created":1729962606,"model":"meta-llama/Meta-Llama-3.1-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"LLM stands for Large Language Model. It's a type of artificial intelligence (AI) designed to process and generate human-like language. These models are trained on vast amounts of text data, allowing them to learn patterns, relationships, and context within language.\n\nLarge language models use various techniques, such as deep learning and natural language processing, to analyze and understand the input text. They can then use this understanding to generate coherent and context-specific text, such as:\n\n1. Responses to questions or prompts\n2. Summaries of long pieces of text\n3. Creative writing, like stories or poetry\n4. Translation of text from one language to another\n\nSome popular examples of LLMs include:\n\n1. Chatbots: Virtual assistants that can understand and respond to user input\n2. Virtual assistants: Like Siri, Alexa, or Google Assistant\n3. Language translation tools: Such as Google Translate\n4. Writing assistants: Like Grammarly or Language Tool\n\nThe key characteristics of LLMs include:\n\n1. **Scalability**: They can process large amounts of text data\n2. **Flexibility**: They can be fine-tuned for specific tasks or domains\n3. **Contextual understanding**: They can recognize context and nuances in language\n4. **Creativity**: They can generate original text or responses\n\nHowever, LLMs also have limitations and potential drawbacks:\n\n1. **Bias**: They can perpetuate existing biases in the training data\n2. **Misinformation**: They can spread misinformation or false information\n3. **Dependence on data quality**: The quality of the training data directly affects the model's performance\n\nOverall, LLMs are powerful tools that can be used in various applications, from language translation and writing assistance to chatbots and virtual assistants."},"logprobs":null,"finish_reason":"stop","matched_stop":128009}],"usage":{"prompt_tokens":47,"total_tokens":408,"completion_tokens":361,"prompt_tokens_details":null}}

Using OpenAI Compatible API

SGLang supports OpenAI-compatible APIs. Here are Python examples:

In [3]:

import openai

# Always assign an api_key, even if not specified during server initialization.
# Setting an API key during server initialization is strongly recommended.

client = openai.Client(
    base_url="http://127.0.0.1:30000/v1", api_key="None"
)

# Chat completion example

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant"},
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)
print(response)

ChatCompletion(id='6bbf20fed17940739eb5cd5d685fa29a', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Here are 3 countries and their capitals:\n\n1. **Country:** Japan\n**Capital:** Tokyo\n\n2. **Country:** Australia\n**Capital:** Canberra\n\n3. **Country:** Brazil\n**Capital:** Brasília', refusal=None, role='assistant', function_call=None, tool_calls=None), matched_stop=128009)], created=1729962608, model='meta-llama/Meta-Llama-3.1-8B-Instruct', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=46, prompt_tokens=49, total_tokens=95, prompt_tokens_details=None))

In [4]:

terminate_process(server_process)