Files
sglang/python/sglang
laoyao0822 9c8e3e99cb Align OpenAI serving behavior with Para deployments
Absorb PR 11's final Para compatibility surface as an opt-in OpenAI serving layer rather than hard-coding business defaults into protocol models. The change adds server args for Para chat defaults, Kimi/GLM compatibility, tool-choice normalization, tool-role text flattening, and streaming first-chunk error preflight while preserving default upstream behavior unless explicitly enabled.

Reasoning token usage is also propagated through chat/completion usage paths, with GLM compatibility emitting completion_tokens_details.reasoning_tokens. Low-risk protocol fixes accept string image_url content parts and preserve GLM function-call argument value whitespace.

Constraint: Online Para-compatible deployments require request/response semantics that differ from default OpenAI serving behavior.

Constraint: Current CP/HiCache/bs>1 work must not be coupled to OpenAI serving compatibility changes.

Rejected: Merge PR 11 history directly | intermediate commits briefly hard-code chat max_tokens=32768 before later gating it by server args.

Rejected: Enable Para compatibility by default | would change non-Para OpenAI-compatible deployments.

Confidence: high

Scope-risk: moderate

Directive: Keep Para-specific serving policies behind explicit server args unless the business contract changes globally.

Tested: PYTHONPATH=python:. python -m unittest discover -s test/registered/unit/entrypoints/openai -p 'test_para_serving_protocol.py' -v (19 tests OK)

Tested: python -m py_compile modified OpenAI serving, tokenizer manager, server_args, function-call detector, and test files

Not-tested: Live router/prefill/decode OpenAI serving E2E after enabling Para flags.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 05:59:08 +08:00
..

Code Structure

  • eval: The evaluation utilities.
  • lang: The frontend language.
  • multimodal_gen: Inference framework for accelerated image/video generation.
  • srt: The backend engine for running local models. (SRT = SGLang Runtime).
  • test: The test utilities.
  • api.py: The public APIs.
  • bench_offline_throughput.py: Benchmark the performance in the offline mode.
  • bench_one_batch.py: Benchmark the latency of running a single static batch without a server.
  • bench_one_batch_server.py: Benchmark the latency of running a single batch with a server.
  • bench_serving.py: Benchmark online serving with dynamic requests.
  • check_env.py: Check the environment variables and dependencies.
  • global_config.py: The global configs and constants.
  • launch_server.py: The entry point for launching a local server.
  • profiler.py: The profiling entry point to send profile requests.
  • utils.py: Common utilities.
  • version.py: Version info.