extend sagemaker.Dockerfile serve script to allow all sglang serve flags (#13173)

docker/serve

@@ -1,31 +1,34 @@
 #!/bin/bash
-
 echo "Starting server"
 
-SERVER_ARGS="--host 0.0.0.0 --port 8080"
+PREFIX="SM_SGLANG_"
+ARG_PREFIX="--"
 
-if [ -n "$TENSOR_PARALLEL_DEGREE" ]; then
-    SERVER_ARGS="${SERVER_ARGS} --tp-size ${TENSOR_PARALLEL_DEGREE}"
+ARGS=()
+
+while IFS='=' read -r key value; do
+    arg_name=$(echo "${key#"${PREFIX}"}" | tr '[:upper:]' '[:lower:]' | tr '_' '-')
+
+    ARGS+=("${ARG_PREFIX}${arg_name}")
+    if [ -n "$value" ]; then
+        ARGS+=("$value")
+    fi
+done < <(env | grep "^${PREFIX}")
+
+# Add default port only if not already set
+if ! [[ " ${ARGS[@]} " =~ " --port " ]]; then
+    ARGS+=(--port "${SM_SGLANG_PORT:-8080}")
 fi
 
-if [ -n "$DATA_PARALLEL_DEGREE" ]; then
-    SERVER_ARGS="${SERVER_ARGS} --dp-size ${DATA_PARALLEL_DEGREE}"
+# Add default host only if not already set
+if ! [[ " ${ARGS[@]} " =~ " --host " ]]; then
+    ARGS+=(--host "${SM_SGLANG_HOST:-0.0.0.0}")
 fi
 
-if [ -n "$EXPERT_PARALLEL_DEGREE" ]; then
-    SERVER_ARGS="${SERVER_ARGS} --ep-size ${EXPERT_PARALLEL_DEGREE}"
+# Add default model-path only if not already set
+if ! [[ " ${ARGS[@]} " =~ " --model-path " ]]; then
+    ARGS+=(--model-path "${SM_SGLANG_MODEL_PATH:-/opt/ml/model}")
 fi
 
-if [ -n "$MEM_FRACTION_STATIC" ]; then
-    SERVER_ARGS="${SERVER_ARGS} --mem-fraction-static ${MEM_FRACTION_STATIC}"
-fi
-
-if [ -n "$QUANTIZATION" ]; then
-    SERVER_ARGS="${SERVER_ARGS} --quantization ${QUANTIZATION}"
-fi
-
-if [ -n "$CHUNKED_PREFILL_SIZE" ]; then
-    SERVER_ARGS="${SERVER_ARGS} --chunked-prefill-size ${CHUNKED_PREFILL_SIZE}"
-fi
-
-python3 -m sglang.launch_server --model-path /opt/ml/model $SERVER_ARGS
+echo "Running command: exec python3 -m sglang.launch_server ${ARGS[@]}"
+exec python3 -m sglang.launch_server "${ARGS[@]}"

docs/get_started/install.md

@@ -125,6 +125,58 @@ sky status --endpoint 30000 sglang
 ```
 
 3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve).
+
+</details>
+
+## Method 7: Run on AWS SageMaker
+
+<details>
+<summary>More</summary>
+
+To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy)
+
+To host a model with your own container, follow the following steps:
+
+1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script.
+2. Push your container onto AWS ECR.
+
+<details>
+<summary>Dockerfile Build Script: <code>build-and-push.sh</code></summary>
+
+```bash
+#!/bin/bash
+AWS_ACCOUNT="<YOUR_AWS_ACCOUNT>"
+AWS_REGION="<YOUR_AWS_REGION>"
+REPOSITORY_NAME="<YOUR_REPOSITORY_NAME>"
+IMAGE_TAG="<YOUR_IMAGE_TAG>"
+
+ECR_REGISTRY="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com"
+IMAGE_URI="${ECR_REGISTRY}/${REPOSITORY_NAME}:${IMAGE_TAG}"
+
+echo "Starting build and push process..."
+
+# Login to ECR
+echo "Logging into ECR..."
+aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REGISTRY}
+
+# Build the image
+echo "Building Docker image..."
+docker build -t ${IMAGE_URI} -f sagemaker.Dockerfile .
+
+echo "Pushing ${IMAGE_URI}"
+docker push ${IMAGE_URI}
+
+echo "Build and push completed successfully!"
+```
+
+</details>
+
+3. Deploy a model for serving on AWS Sagemaker, refer to [deploy_and_serve_endpoint.py](https://github.com/sgl-project/sglang/blob/main/examples/sagemaker/deploy_and_serve_endpoint.py). For more information, check out [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk).
+    1. By default, the model server on SageMaker will run with the following command: `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`. This is optimal for hosting your own model with SageMaker.
+    2. To modify your model serving parameters, the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script allows for all available options within `python3 -m sglang.launch_server --help` cli by specifying environment variables with prefix `SM_SGLANG_`.
+    3. The serve script will automatically convert all environment variables with prefix `SM_SGLANG_` from `SM_SGLANG_INPUT_ARGUMENT` into `--input-argument` to be parsed into `python3 -m sglang.launch_server` cli.
+    4. For example, to run [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) with reasoning parser, simply add additional environment variables `SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B` and `SM_SGLANG_REASONING_PARSER=qwen3`.
+
 </details>
 
 ## Common Notes

examples/sagemaker/deploy_and_serve_endpoint.py

@@ -0,0 +1,69 @@
+import json
+import boto3
+import sagemaker
+
+from sagemaker import serializers
+from sagemaker.model import Model
+from sagemaker.predictor import Predictor
+
+boto_session = boto3.session.Session()
+sm_client = boto_session.client("sagemaker")
+sm_role = boto_session.resource("iam").Role("SageMakerRole").arn
+
+endpoint_name="<YOUR_ENDPOINT_NAME>"
+image_uri="<YOUR_DOCKER_IMAGE_URI>"
+model_id="<YOUR_MODEL_ID>" # eg: Qwen/Qwen3-0.6B from https://huggingface.co/Qwen/Qwen3-0.6B
+hf_token="<YOUR_HUGGINGFACE_TOKEN>"
+prompt="<YOUR_ENDPOINT_PROMPT>"
+
+model = Model(
+  name=endpoint_name,
+  image_uri=image_uri,
+  role=sm_role,
+  env={
+      "SM_SGLANG_MODEL_PATH": model_id,
+      "HF_TOKEN": hf_token,
+  },
+)
+print("Model created successfully")
+print("Starting endpoint deployment (this may take 10-15 minutes)...")
+
+endpoint_config = model.deploy(
+    instance_type="ml.g5.12xlarge",
+    initial_instance_count=1,
+    endpoint_name=endpoint_name,
+    inference_ami_version="al2-ami-sagemaker-inference-gpu-3-1",
+    wait=True,
+)
+print("Endpoint deployment completed successfully")
+
+
+print(f"Creating predictor for endpoint: {endpoint_name}")
+predictor = Predictor(
+    endpoint_name=endpoint_name,
+    serializer=serializers.JSONSerializer(),
+)
+
+payload = {
+    "model": model_id,
+    "messages": [{"role": "user", "content": prompt}],
+    "max_tokens": 2400,
+    "temperature": 0.01,
+    "top_p": 0.9,
+    "top_k": 50,
+}
+print(f"Sending inference request with prompt: '{prompt[:50]}...'")
+response = predictor.predict(payload)
+print("Inference request completed successfully")
+
+if isinstance(response, bytes):
+    response = response.decode("utf-8")
+
+if isinstance(response, str):
+    try:
+        response = json.loads(response)
+    except json.JSONDecodeError:
+        print("Warning: Response is not valid JSON. Returning as string.")
+
+print(f"Received model response: '{response}'")
+