sglang/sgl-model-gateway/bindings/golang

SGLang Go gRPC SDK

A high-level Go SDK for interacting with SGLang gRPC API, designed with an OpenAI-style API for familiarity and ease of use.

Location: sgl-model-gateway/bindings/golang/

Table of Contents

• Features
• Installation
• Quick Start
  • Basic Usage
  • Streaming Usage
• Examples
• Configuration
• API Reference
• Testing
  • Unit Tests
  • Integration Tests
  • Benchmarks
• Documentation
• Development
• Troubleshooting
• License

Features

• OpenAI-style API: Familiar interface similar to OpenAI Go SDK
• Streaming Support: Real-time streaming chat completions
• Non-streaming Support: Simple request/response API
• Tool Calling: Support for function calling and tool use
• Type-safe: Full Go type definitions for requests and responses
• Comprehensive Testing: 18+ unit and integration tests
• Thread-safe: All public methods are safe for concurrent use
• Well-documented: Full API documentation with examples

Installation

go get github.com/sglang/sglang-go-grpc-sdk

Build Requirements

• Go 1.21 or later
• Rust toolchain (for building the FFI library)
• Python 3.x (for Python bindings in Rust FFI)
• Tokio runtime for async operations

Quick Start

Basic Usage (Non-streaming)

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/sglang/sglang-go-grpc-sdk"
)

func main() {
    // Create client
    client, err := sglang.NewClient(sglang.ClientConfig{
        Endpoint:      "grpc://localhost:20000",
        TokenizerPath: "/path/to/tokenizer",
    })
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    // Create completion
    resp, err := client.CreateChatCompletion(context.Background(), sglang.ChatCompletionRequest{
        Model: "default",
        Messages: []sglang.ChatMessage{
            {Role: "user", Content: "Hello!"},
        },
        Stream: false,
    })
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(resp.Choices[0].Message.Content)
    fmt.Printf("Usage: Prompt=%d, Completion=%d, Total=%d\n",
        resp.Usage.PromptTokens,
        resp.Usage.CompletionTokens,
        resp.Usage.TotalTokens)
}

Streaming Usage

package main

import (
    "context"
    "fmt"
    "io"
    "log"

    "github.com/sglang/sglang-go-grpc-sdk"
)

func main() {
    // Create client
    client, err := sglang.NewClient(sglang.ClientConfig{
        Endpoint:      "grpc://localhost:20000",
        TokenizerPath: "/path/to/tokenizer",
    })
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    // Create streaming completion
    ctx := context.Background()
    stream, err := client.CreateChatCompletionStream(ctx, sglang.ChatCompletionRequest{
        Model: "default",
        Messages: []sglang.ChatMessage{
            {Role: "user", Content: "Tell me a story"},
        },
        Stream:              true,
        MaxCompletionTokens: intPtr(500),
    })
    if err != nil {
        log.Fatal(err)
    }
    defer stream.Close()

    // Read streaming response
    for {
        chunk, err := stream.Recv()
        if err == io.EOF {
            break
        }
        if err != nil {
            log.Fatal(err)
        }

        for _, choice := range chunk.Choices {
            if choice.Delta.Content != "" {
                fmt.Print(choice.Delta.Content)
            }
        }
    }
    fmt.Println() // newline
}

// Helper functions for optional pointer fields
func intPtr(i int) *int {
    return &i
}

func float32Ptr(f float32) *float32 {
    return &f
}

Examples

The SDK includes several examples in the examples/ directory:

• simple: Basic non-streaming chat completion example
• streaming: Real-time streaming with performance metrics

Running Examples

# Run simple example
cd bindings/golang/examples/simple
bash run.sh

# Run streaming example
cd bindings/golang/examples/streaming
bash run.sh

# Or use Makefile from bindings/golang directory
cd bindings/golang
make run-simple
make run-streaming

Examples automatically detect the server endpoint and tokenizer path via environment variables or defaults.

Configuration

Environment Variables

• SGL_GRPC_ENDPOINT: gRPC server endpoint (default: grpc://localhost:20000)
• SGL_TOKENIZER_PATH: Path to tokenizer directory (required)
• CARGO_BUILD_DIR: Rust build output directory (auto-detected if not set)

ClientConfig

type ClientConfig struct {
    // Endpoint is the gRPC endpoint URL (e.g., "grpc://localhost:20000")
    // Required field. Must include the scheme (grpc://) and port number.
    Endpoint string

    // TokenizerPath is the path to the tokenizer directory containing
    // tokenizer configuration files (e.g., tokenizer.json, vocab.json)
    // Required field.
    TokenizerPath string
}

API Reference

Client Methods

type Client struct {
    // Thread-safe client for SGLang gRPC API
}

// Creates a new client with the given configuration
func NewClient(config ClientConfig) (*Client, error)

// Closes the client and releases all resources
func (c *Client) Close() error

// Creates a non-streaming chat completion
func (c *Client) CreateChatCompletion(ctx context.Context, req ChatCompletionRequest) (*ChatCompletionResponse, error)

// Creates a streaming chat completion
func (c *Client) CreateChatCompletionStream(ctx context.Context, req ChatCompletionRequest) (*ChatCompletionStream, error)

Request Types

• ChatCompletionRequest: Main request type for chat completions
  • Model, Messages, Stream, Temperature, TopP, MaxCompletionTokens, Tools, etc.
• ChatMessage: Individual message in a conversation
  • Role, Content
• Tool: Tool/function definition for function calling
  • Type, Function (name, description, parameters)

Response Types

• ChatCompletionResponse: Non-streaming response
  • ID, Model, Created, Choices, Usage
• ChatCompletionStreamResponse: Streaming response chunk
  • Same structure as above but for incremental updates
• Message: Complete message with content and tool calls
• ToolCall: Tool call information with function and arguments
• Usage: Token usage statistics
  • PromptTokens, CompletionTokens, TotalTokens

Testing

The SDK includes comprehensive testing infrastructure with both unit and integration tests.

Unit Tests

Unit tests are located in client_test.go and test individual components without requiring a server.

Running Unit Tests

# Run all unit tests
go test ./...

# Run with verbose output
go test -v ./...

# Run specific test
go test -run TestClientConfig

# Run tests with race detector (detects concurrency issues)
go test -race ./...

# Run with coverage analysis
go test -cover ./...

# Generate detailed coverage report
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html

Unit Test Coverage

• Configuration validation (TestClientConfig) - Validates ClientConfig requirements
• Type structures - Verifying all struct types work correctly
• Response handling - Testing response parsing and validation
• Concurrent operations (TestConcurrentClientOperations) - Thread-safety verification
• Benchmarks (BenchmarkChatCompletionRequest) - Performance measurement

Test Files:

• client_test.go - 10 unit tests covering core functionality
• Tests cover: config validation, message types, request validation, close operations, response types, streaming, tools, concurrency, and context cancellation

Integration Tests

Integration tests require a running SGLang server and test the full client-server interaction.

Prerequisites

1. Start an SGLang server:

# Using Python (requires sglang package installed)
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-hf

# Or using pre-built Docker image
docker run -p 20000:20000 lmsys/sglang:latest

# Or build your own
sglang launch_server --model-path <model_path>

2. Set required environment variables:

# Set the gRPC endpoint (default: grpc://localhost:20000)
export SGL_GRPC_ENDPOINT=grpc://localhost:20000

# Set the tokenizer path (required)
export SGL_TOKENIZER_PATH=/path/to/tokenizer

Running Integration Tests

# Run all integration tests
go test -tags=integration ./...

# Run specific integration test
go test -tags=integration -run TestIntegrationNonStreamingCompletion

# Run with verbose output
go test -tags=integration -v ./...

# Run with race detector
go test -tags=integration -race ./...

Integration Test Coverage

Test File: integration_test.go - 4 integration tests

• TestIntegrationNonStreamingCompletion - Basic non-streaming request/response
• TestIntegrationStreamingCompletion - Streaming response handling
• TestIntegrationConcurrentRequests - Multiple simultaneous requests
• TestIntegrationContextCancellation - Context timeout and cancellation

Benchmarks

Measure performance of SDK operations:

# Run all benchmarks
go test -bench=. -benchmem ./...

# Run specific benchmark
go test -bench=BenchmarkChatCompletionRequest -benchmem

# Run for longer duration
go test -bench=. -benchtime=10s ./...

Current benchmarks:

• BenchmarkChatCompletionRequest - Measures request creation performance

CI/CD Integration

Add to your GitHub Actions workflow:

- name: Run Go tests
  run: |
    go test -race -cover ./...

- name: Run integration tests (on main branch)
  if: github.ref == 'refs/heads/main'
  env:
    SGL_GRPC_ENDPOINT: grpc://localhost:20000
    SGL_TOKENIZER_PATH: /path/to/tokenizer
  run: go test -tags=integration ./...

Documentation

Code Documentation

All public types and functions include comprehensive documentation:

1. Package-level documentation in client.go with usage examples
2. Type documentation for all structs with field descriptions
3. Function documentation with:
   • Purpose and behavior description
   • Parameter documentation with types and constraints
   • Return value documentation
   • Error cases and handling
   • Safety notes (for FFI functions)
   • Usage examples

Key Documented Components

• Client - Main client with thread-safety notes
• ClientConfig - Configuration requirements and validation rules
• ChatCompletionRequest - Request structure with field descriptions
• ChatCompletionResponse - Response structure and usage
• ChatCompletionStreamResponse - Streaming response format
• Usage - Token usage information structure
• Tool, Function, ToolCall - Tool call structures

Viewing Documentation

Generate and view HTML documentation:

# Install godoc (if not already installed)
go install golang.org/x/tools/cmd/godoc@latest

# Generate and serve documentation
godoc -http=:6060

# Visit: http://localhost:6060/pkg/github.com/sglang/sglang-go-grpc-sdk/

Development

Building

cd bindings/golang

# Build the Go bindings (compiles Rust FFI library)
make build

# Clean build
make clean && make build

Code Quality

Ensure code quality before committing:

# Run Go vet (check for potential bugs)
go vet ./...

# Format code
go fmt ./...

# Run all tests with race detection
go test -race ./...

Project Structure

bindings/golang/
├── client.go                 # Main client implementation
├── client_test.go            # Unit tests
├── integration_test.go       # Integration tests
├── README.md                 # This file
├── Makefile                  # Build automation
├── Cargo.toml               # Rust FFI dependencies
├── examples/                # Example programs
│   ├── simple/             # Non-streaming example
│   └── streaming/          # Streaming example
├── src/                    # Rust FFI source
│   ├── client.rs          # Client FFI
│   ├── stream.rs          # Stream handling
│   ├── grpc_converter.rs  # Response conversion
│   └── ...
└── internal/               # Internal packages
    └── ffi/               # FFI bindings

Troubleshooting

Connection Errors

Error: connection refused or failed to dial

Solution:

1. Ensure SGLang server is running: python -m sglang.launch_server
2. Check endpoint: echo $SGL_GRPC_ENDPOINT
3. Verify port is not blocked: nc -zv localhost 20000

Tokenizer Not Found

Error: tokenizer path not found or tokenizer configuration missing

Solution:

1. Set SGL_TOKENIZER_PATH environment variable
2. Verify path contains required files: ls $SGL_TOKENIZER_PATH
3. Files should include: tokenizer.json, vocab.json, config.json

Build Failures

Error: library 'sglang_router_rs' not found

Solution:

1. Rebuild Rust library: cd sgl-model-gateway/bindings/golang && make build
2. Or manually with cargo: cd sgl-model-gateway/bindings/golang && cargo build --release
3. Set CARGO_BUILD_DIR if using non-standard build location
4. Ensure Rust toolchain is installed: rustup toolchain list

Tests Hanging

Error: Tests seem to hang indefinitely

Solution:

1. Use timeout for hanging tests: timeout 30s go test ./...
2. Run with verbose output to see which test hangs: go test -v ./...
3. Ensure server is responsive: grpcurl -plaintext localhost:20000 list

Memory Issues

Error: Out of memory during tests

Solution:

# Run with memory limit for long-running tests
GODEBUG=madvdontneed=1 go test -timeout 5m ./...

# Monitor memory during tests
watch -n1 'ps aux | grep test'

Contributing

When adding new features:

1. Add comprehensive documentation to public types/functions
2. Include usage examples for complex APIs
3. Add unit tests covering happy path and error cases
4. Add integration tests if server interaction required
5. Ensure code passes go vet and go test -race
6. Update this README if adding new features

License

See LICENSE file for details.

---

Need Help?

• Check examples in examples/ directory
• Run tests to see working code: go test -v ./...
• Review function documentation: godoc or inline comments
• Check troubleshooting section above