sglang/docs/backend/openai_api_vision.ipynb

OpenAI APIs - Vision

SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the OpenAI API Reference. This tutorial covers the vision APIs for vision language models.

SGLang supports vision language models such as Llama 3.2, LLaVA-OneVision, and QWen-VL2

• meta-llama/Llama-3.2-11B-Vision-Instruct
• lmms-lab/llava-onevision-qwen2-72b-ov-chat
• Qwen/Qwen2-VL-7B-Instruct

Launch A Server

This code block is equivalent to executing

python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
  --port 30000 --chat-template llama_3_vision

in your terminal and wait for the server to be ready.

Remember to add --chat-template llama_3_vision to specify the vision chat template, otherwise the server only supports text. We need to specify --chat-template for vision language models because the chat template provided in Hugging Face tokenizer only supports text.

In [ ]:

from sglang.utils import (
    execute_shell_command,
    wait_for_server,
    terminate_process,
    print_highlight,
)

embedding_process = execute_shell_command(
    """
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
    --port=30000 --chat-template=llama_3_vision
"""
)

wait_for_server("http://localhost:30000")

Using cURL

Once the server is up, you can send test requests using curl or requests.

In [ ]:

import subprocess

curl_command = """
curl -s http://localhost:30000/v1/chat/completions \
  -d '{
    "model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What’s in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
            }
          }
        ]
      }
    ],
    "max_tokens": 300
  }'
"""

response = subprocess.check_output(curl_command, shell=True).decode()
print_highlight(response)

Using Python Requests

In [ ]:

import requests

url = "http://localhost:30000/v1/chat/completions"

data = {
    "model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
                    },
                },
            ],
        }
    ],
    "max_tokens": 300,
}

response = requests.post(url, json=data)
print_highlight(response.text)

Using OpenAI Python Client

In [ ]:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-11B-Vision-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in this image?",
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
                    },
                },
            ],
        }
    ],
    max_tokens=300,
)

print_highlight(response.choices[0].message.content)

Multiple-Image Inputs

The server also supports multiple images and interleaved text and images if the model supports it.

In [ ]:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-11B-Vision-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true",
                    },
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png",
                    },
                },
                {
                    "type": "text",
                    "text": "I have two very different images. They are not related at all. "
                    "Please describe the first image in one sentence, and then describe the second image in another sentence.",
                },
            ],
        }
    ],
    temperature=0,
)

print_highlight(response.choices[0].message.content)

In [ ]:

terminate_process(embedding_process)

Chat Template

As mentioned before, if you do not specify a vision model's --chat-template, the server uses Hugging Face's default template, which only supports text.

We list popular vision models with their chat templates:

• meta-llama/Llama-3.2-Vision uses llama_3_vision.
• Qwen/Qwen2-VL-7B-Instruct uses qwen2-vl.
• LlaVA-OneVision uses chatml-llava.
• LLaVA-NeXT uses chatml-llava.
• Llama3-LLaVA-NeXT uses llava_llama_3.
• LLaVA-v1.5 / 1.6 uses vicuna_v1.1.