sglang/docs/backend/speculative_decoding.ipynb

Speculative Decoding

SGLang now provides an EAGLE-based speculative decoding option. The implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines.

Note: Currently, Speculative Decoding in SGLang does not support radix cache.

Performance Highlights

• Official EAGLE code (SafeAILab/EAGLE): ~200 tokens/s
• Standard SGLang Decoding: ~156 tokens/s
• EAGLE Decoding in SGLang: ~297 tokens/s
• EAGLE Decoding in SGLang (w/ torch.compile): ~316 tokens/s

All benchmarks below were run on a single H100.

EAGLE Decoding

To enable EAGLE-based speculative decoding, specify the draft model (--speculative-draft) and the relevant EAGLE parameters:

In [ ]:

# EAGLE decoding
from sglang.utils import (
    execute_shell_command,
    wait_for_server,
    terminate_process,
    print_highlight,
)

server_process = execute_shell_command(
    """
python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf  --speculative-algo EAGLE \
    --speculative-draft lmzheng/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 5 \
    --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.7 --port=30020
"""
)

wait_for_server("http://localhost:30020")

In [ ]:

import openai

client = openai.Client(base_url="http://127.0.0.1:30020/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print_highlight(f"Response: {response}")

In [ ]:

terminate_process(server_process)

EAGLE Decoding with torch.compile

You can also enable torch.compile for further optimizations and optionally set --cuda-graph-max-bs:

In [ ]:

server_process = execute_shell_command(
    """
python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf  --speculative-algo EAGLE \
    --speculative-draft lmzheng/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 5 \
        --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.7 \
            --enable-torch-compile --cuda-graph-max-bs 2 --port=30020
"""
)

wait_for_server("http://localhost:30020")

Benchmark Script

The following code example shows how to measure the decoding speed when generating tokens:

In [ ]:

import time
import requests

tic = time.time()
response = requests.post(
    "http://localhost:30020/generate",
    json={
        "text": "[INST] Give me a simple FastAPI server. Show the python code. [/INST]",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 256,
        },
    },
)
latency = time.time() - tic
ret = response.json()
completion_text = ret["text"]
speed = ret["meta_info"]["completion_tokens"] / latency

print_highlight(completion_text)
print_highlight(f"speed: {speed:.2f} token/s")

In [ ]:

terminate_process(server_process)