sglang/docs/backend/lora.ipynb

LoRA Serving

SGLang enables the use of LoRA adapters with a base model. By incorporating techniques from S-LoRA and Punica, SGLang can efficiently support multiple LoRA adapters for different sequences within a single batch of inputs.

Arguments for LoRA Serving

The following server arguments are relevant for multi-LoRA serving:

• lora_paths: A mapping from each adaptor's name to its path, in the form of {name}={path} {name}={path}.

• max_loras_per_batch: Maximum number of adaptors used by each batch. This argument can affect the amount of GPU memory reserved for multi-LoRA serving, so it should be set to a smaller value when memory is scarce. Defaults to be 8.

• lora_backend: The backend of running GEMM kernels for Lora modules. It can be one of triton or flashinfer, and set to triton by default. For better performance and stability, we recommend using the Triton LoRA backend. In the future, faster backend built upon Cutlass or Cuda kernels will be added.

• max_lora_rank: The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in --lora-paths. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup.

• lora_target_modules: The union set of all target modules where LoRA should be applied (e.g., q_proj, k_proj, gate_proj). If not specified, it will be automatically inferred from the adapters provided in --lora-paths. This argument is needed when you expect to dynamically load adapters of different target modules after server startup.

• tp_size: LoRA serving along with Tensor Parallelism is supported by SGLang. tp_size controls the number of GPUs for tensor parallelism. More details on the tensor sharding strategy can be found in S-Lora paper.

From client side, the user needs to provide a list of strings as input batch, and a list of adaptor names that each input sequence corresponds to.

Usage

Serving Single Adaptor

In [ ]:

from sglang.test.test_utils import is_in_ci

if is_in_ci():
    from patch import launch_server_cmd
else:
    from sglang.utils import launch_server_cmd

from sglang.utils import wait_for_server, terminate_process

import json
import requests

In [ ]:

server_process, port = launch_server_cmd(
    """
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
    --max-loras-per-batch 1 --lora-backend triton \
    --disable-radix-cache
"""
)

wait_for_server(f"http://localhost:{port}")

In [ ]:

url = f"http://127.0.0.1:{port}"
json_data = {
    "text": [
        "List 3 countries and their capitals.",
        "AI is a field of computer science focused on",
    ],
    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
    # The first input uses lora0, and the second input uses the base model
    "lora_path": ["lora0", None],
}
response = requests.post(
    url + "/generate",
    json=json_data,
)
print(f"Output 0: {response.json()[0]['text']}")
print(f"Output 1: {response.json()[1]['text']}")

In [ ]:

terminate_process(server_process)

Serving Multiple Adaptors

In [ ]:

server_process, port = launch_server_cmd(
    """
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
    lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16 \
    --max-loras-per-batch 2 --lora-backend triton \
    --disable-radix-cache
"""
)

wait_for_server(f"http://localhost:{port}")

In [ ]:

url = f"http://127.0.0.1:{port}"
json_data = {
    "text": [
        "List 3 countries and their capitals.",
        "AI is a field of computer science focused on",
    ],
    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
    # The first input uses lora0, and the second input uses lora1
    "lora_path": ["lora0", "lora1"],
}
response = requests.post(
    url + "/generate",
    json=json_data,
)
print(f"Output 0: {response.json()[0]['text']}")
print(f"Output 1: {response.json()[1]['text']}")

In [ ]:

terminate_process(server_process)

Dynamic LoRA loading

Basic Usage

Instead of specifying all adapters during server startup via --lora-paths. You can also load & unload LoRA adapters dynamically via the /load_lora_adapter and /unload_lora_adapter API.

(Please note that, currently we still require you to specify at least one adapter in --lora-paths to enable the LoRA feature, this limitation will be lifted soon.)

In [ ]:

server_process, port = launch_server_cmd(
    """
    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --lora-paths lora0=philschmid/code-llama-3-1-8b-text-to-sql-lora \
    --cuda-graph-max-bs 2 \
    --max-loras-per-batch 2 --lora-backend triton \
    --disable-radix-cache
    """
)

url = f"http://127.0.0.1:{port}"
wait_for_server(url)

In [ ]:

response = requests.post(
    url + "/load_lora_adapter",
    json={
        "lora_name": "lora1",
        "lora_path": "Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16",
    },
)

if response.status_code == 200:
    print("LoRA adapter loaded successfully.", response.json())
else:
    print("Failed to load LoRA adapter.", response.json())

In [ ]:

response = requests.post(
    url + "/generate",
    json={
        "text": [
            "List 3 countries and their capitals.",
            "List 3 countries and their capitals.",
        ],
        "sampling_params": {"max_new_tokens": 32, "temperature": 0},
        "lora_path": ["lora0", "lora1"],
    },
)
print(f"Output from lora0: {response.json()[0]['text']}")
print(f"Output from lora1: {response.json()[1]['text']}")

In [ ]:

response = requests.post(
    url + "/unload_lora_adapter",
    json={
        "lora_name": "lora0",
    },
)

In [ ]:

response = requests.post(
    url + "/load_lora_adapter",
    json={
        "lora_name": "lora2",
        "lora_path": "pbevan11/llama-3.1-8b-ocr-correction",
    },
)

if response.status_code == 200:
    print("LoRA adapter loaded successfully.", response.json())
else:
    print("Failed to load LoRA adapter.", response.json())

In [ ]:

response = requests.post(
    url + "/generate",
    json={
        "text": [
            "List 3 countries and their capitals.",
            "List 3 countries and their capitals.",
        ],
        "sampling_params": {"max_new_tokens": 32, "temperature": 0},
        "lora_path": ["lora1", "lora2"],
    },
)
print(f"Output from lora1: {response.json()[0]['text']}")
print(f"Output from lora2: {response.json()[1]['text']}")

In [ ]:

terminate_process(server_process)

Advanced: hosting adapters of different shapes

In some cases, you may want to load LoRA adapters with different ranks or target modules (e.g., q_proj, k_proj) simultaneously. To ensure the server can accommodate all expected LoRA shapes, it's recommended to explicitly specify --max-lora-rank and/or --lora-target-modules at startup.

For backward compatibility, SGLang will infer these values from --lora-paths if they are not explicitly provided. This means it's safe to omit them only if all dynamically loaded adapters share the same shape (rank and target modules) as those in the initial --lora-paths or are strictly "smaller".

In [ ]:

lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16"  # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj
lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora"  # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj


# The `--target-lora-modules` param below is technically not needed, as the server will infer it from lora0 which already has all the target modules specified.
# We are adding it here just to demonstrate usage.
server_process, port = launch_server_cmd(
    f"""
    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --lora-paths lora0={lora0} \
    --cuda-graph-max-bs 2 \
    --max-loras-per-batch 2 --lora-backend triton \
    --disable-radix-cache
    --max-lora-rank 64
    --lora-target-modules q_proj k_proj v_proj o_proj down_proj up_proj gate_proj
    """
)

url = f"http://127.0.0.1:{port}"
wait_for_server(url)

In [ ]:

response = requests.post(
    url + "/load_lora_adapter",
    json={
        "lora_name": "lora1",
        "lora_path": lora1,
    },
)

if response.status_code == 200:
    print("LoRA adapter loaded successfully.", response.json())
else:
    print("Failed to load LoRA adapter.", response.json())

In [ ]:

url = f"http://127.0.0.1:{port}"
json_data = {
    "text": [
        "List 3 countries and their capitals.",
        "AI is a field of computer science focused on",
    ],
    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
    # The first input uses lora0, and the second input uses lora1
    "lora_path": ["lora0", "lora1"],
}
response = requests.post(
    url + "/generate",
    json=json_data,
)
print(f"Output from lora0: {response.json()[0]['text']}")
print(f"Output from lora1: {response.json()[1]['text']}")

In [ ]:

terminate_process(server_process)

Future Works

The development roadmap for LoRA-related features can be found in this issue. Currently radix attention is incompatible with LoRA and must be manually disabled. Other features, including Unified Paging, Cutlass backend, and dynamic loading/unloadingm, are still under development.