sglang/gov/work/2026-06-07-rebase-runtime-env-to-torch-2-11-dev-cu13-image-deepgemm-sgl-deep-gemm-port-compat.toml

#:schema ../schema/work.schema.json

[govctl]
id = "WI-2026-06-07-001"
title = "Rebase runtime env to torch-2.11 dev-cu13 image (DeepGEMM sgl-deep-gemm port + compat)"
status = "active"
created = "2026-06-07"
started = "2026-06-07"

[content]
description = "Rebase the runtime environment to the torch-2.11 dev-cu13 image (built from upstream/main's current Dockerfile: torch 2.11.0, sgl-kernel 0.4.3, sgl-deep-gemm 0.1.2, flashinfer 0.6.12, transformers 5.8.1, mooncake 0.3.11.post1) WITHOUT a full code rebase. Phase 1 (done): port the DeepGEMM sgl-deep-gemm wheel migration (API compat + coupled correctness fixes) verified against upstream HEAD = wheel 0.1.2. Phase 2/3 (in image): rebuild tai-kernel against torch 2.11, import-smoke + fix any transformers-5.8/xgrammar-0.2/torch-2.11 breakages empirically, then harness coldchunk byte-equality + perf validate."

[[content.journal]]
date = "2026-06-07"
scope = "deepgemm-port"
content = "History review (user: don't rush, dig commit history of files to change) caught a CRITICAL bug in the initial port. Verified all 4 edited functions against upstream HEAD (= sgl-deep-gemm 0.1.2), NOT just PR#24268 (which targeted wheel 0.0.1). Findings: (1) compile_utils warmup hasattr-compile-mode-guard + positional m_indices = byte-identical to HEAD. OK. (2) fp8_utils transform_scale_ue8m0 DLPack stride fix = byte-identical to HEAD. OK. (3) entrypoint masked/contig deep_gemm signatures unchanged at HEAD (fp8_m_grouped_gemm_nt_masked enable_overlap/max_block_n; m_grouped_fp8_gemm_nt_contiguous positional) - our entrypoint already matches, no change. (4) CRITICAL: _to_2d_context_lens layout CHANGED between 0.0.1 and 0.1.2. PR#24268 used (batch_size, next_n); HEAD uses ALWAYS (N_total,1) with comment 'avoid deadlock at deep_gemm.fp8_paged_mqa_logits'. Passing (bs,next_n>=2) DEADLOCKS the kernel. Our EAGLE deploy = next_n=4 on SM90/H200 (DG-native broadcast path is SM100-only), so it takes the per-token (N_total,1) path. FIXED nsa_backend _to_2d_context_lens to HEAD's (N_total,1) form. nsa_indexer unsqueeze(-1) already yields (N_total,1) - consistent. Lesson: verify against HEAD/target-wheel, not the intro PR; wheel APIs drift between minor versions."

[[content.journal]]
date = "2026-06-07"
scope = "deepgemm-port"
content = "Complete-migration sweep (user: 要迁移就完整迁移). Verified EVERY deep_gemm symbol our code calls against HEAD/0.1.2. UNCHANGED+confirmed: fp8_mqa_logits (ks/ke cu_seqlens, not paged ctx - no 2D issue), fp8_gemm_nt, bf16_gemm_nt/nn, get_mk_alignment_for_contiguous_layout (no-arg), transform_sf_into_required_layout (same kwargs), get/set_num_sms, fp8_m_grouped_gemm_nt_masked (HEAD only ADDS optional recipe_a/b NVFP4 + _ensure_cuda, not required for FP8). PORTED beyond the 4 compat edits: (5) #26839 SBO masked-return unpack guard in moe_runner/deep_gemm.py (our line 388 had the unguarded unpack; our launch enables --enable-single-batch-overlap). (6) #23979 PDL-on-by-default in entrypoint.py (hasattr-guarded set_pdl(True), get_bool_env_var SGLANG_DEEPGEMM_PDL default true - perf default matching new wheel). (7) engine.py sglang-kernel assert 0.4.0->0.4.3 (migrated code requires sgl-deep-gemm-era wheel; (N_total,1) ctx would misbehave on old bundled DeepGEMM). SKIPPED with rationale (not on GLM-5.1-FP8/NSA/deepgemm/H200-SM90 path): #26025 fallback-unsupported-shapes (N/A - our tree has no _varlen_deep_gemm_silu_mul_quant/SGLANG_OPT_USE_JIT_EP_ACTIVATION), #25286 Gemma4 triton_scaled_mm scale layout (triton path not deepgemm), #22300 Minimax fp16/flashinfer-trtllm fallback, #26473/#17392 BF16 features, #26496/#24692 SM120/NVFP4, #26238/#25884 dsv4, AMD/ROCm. All 7 edited files py_compile clean. Runtime validation pending the torch-2.11 image (Phase 2/3)."

[[content.journal]]
date = "2026-06-07"
scope = "env-rebase"
content = "Phase-2 prep (tai-kernel rebuild characterization). tai-kernel has NO prebuilt .so / no setup.py — it JIT-compiles its CUDA (.cu/.cc, at::Tensor + TORCH_LIBRARY_FRAGMENT(tai_kernel_ops)) via torch.utils.cpp_extension.load at first import, caching to ~/.cache/tai-kernel/ops/<digest>. GOOD: auto-recompiles against the runtime torch (2.11) headers on first import — no manual wheel rebuild. GOTCHA: the cache digest = md5(source bytes + torch.version.cuda + ldflags) does NOT include torch.__version__; torch 2.9.1+cu130 -> 2.11.0+cu130 keeps cuda='13.0' so the digest is unchanged -> a persisted ~/.cache/tai-kernel would be WRONGLY REUSED (ABI-incompatible) against 2.11. Mitigation: fresh container has empty ~/.cache (auto-fresh compile); if cache is on a mounted/persistent volume, rm -rf ~/.cache/tai-kernel before first import. Phase-2 risk = whether the .cu/.cc compile cleanly against torch-2.11 ATen/c10 headers (empirical, verify in image). Phase 1 deep_gemm migration committed 32818784d3."

[[content.journal]]
date = "2026-06-07"
scope = "env-rebase"
content = "Phase 2/3 VALIDATED on B300 (g0036/g0037, 8x B300 SXM6 each, driver 590.48, image sglang:dev-cu13 = torch 2.11.0+cu130, sgl-kernel 0.4.3, sgl-deep-gemm 0.1.2, flashinfer 0.6.12, transformers 5.8.1, xgrammar 0.2.1, mooncake-transfer-engine-cuda13 0.3.11.post1, deep_ep 1.2.1, sglang-router 0.3.2 - EXACTLY the predicted target). Synced our branch (Phase-1 port) to shared beegfs /mnt/beegfs/syh/sglang-stable (036<->037 shared). In a privileged --gpus all container: pip install -e python + tai-kernel (--no-deps) OK; sglang 0.5.11 editable shadows image default. SMOKE: (a) deep_gemm 0.1.2 wheel has set_pdl=FALSE and get_compile_mode=FALSE -> BOTH my hasattr guards are load-bearing (would crash unguarded); get_paged_mqa/fp8_paged_mqa/contig/masked present. (b) all migrated modules import clean on torch 2.11 (entrypoint w/ PDL edit, nsa_backend w/ _to_2d, fp8_utils, moe_runner/deep_gemm). (c) DEEPGEMM_SCALE_UE8M0=True on Blackwell -> my transform_scale_ue8m0 DLPack stride fix is ACTIVE/load-bearing here (was dormant on H200 SM90). (d) tai-kernel nsa_prefill CUDA ext JIT-compiled vs torch-2.11 ABI in 46s (cleared ~/.cache/tai-kernel first per the digest-gotcha) - the big ABI risk PASSED. Remaining empirical unknowns: model load (transformers 5.8 + GLM-5.1 trust_remote_code), runtime kernels on SM100, mooncake P->D over RoCE roce_rail0..7 (host /sys GIDs read empty; multi-port roce_rail ports1-6 w/ port1 DOWN - mooncake device/port/GID TBD), deepep normal(P)/low_latency(D) on 1-node B300. Topology change: 1-node P (g0036, tp8/ep8) + 1-node D (g0037, dp8); decode drops nnodes2/dist-init. NICs: roce_rail0..7 (GPU rails), bond1 10.25.203.36/37 control plane."

[[content.journal]]
date = "2026-06-07"
scope = "b300-bringup"
content = "B300 networking diagnosed + RDMA verified. Newly-received cluster; inter-node RDMA appeared dead but was 2 self-inflicted red herrings: (1) per-rail VRFs (CX8_PLB-config.yaml tables 100-107) - main routing table sends 10.33.x out bond1, so naive ping=100% loss; VRF-bound ping (ip vrf exec vrf_vf_rail0 / ping -I eth_vf_rail0) = 0.15ms. (2) container has no iproute2 (ip absent) - irrelevant. Fabric = B300+CX8 dual-plane rail-optimized Spectrum-X RoCE; per-GPU rails roce_vf_rail0..7 RoCE v2 GID idx 3, 10.33.16-23.x/24, 400G, PXB 1:1 GPU (nvidia-smi topo NIC10-17). IB fabric mlx5_12-15 (Quantum-3) also present but rails are the KV path. VERIFIED RDMA 036<->037 in container: ib_write_bw -d roce_vf_rail0 -x 3, handshake to BOND 10.25.203.37, RDMA over rail -> peak 344.93 / AVG 4.53 Gb/s (sustained collapses -> PFC/ECN tuning open issue). Fixed draft scripts (roce_rail->roce_vf_rail0..7). Deliverables on /mnt/beegfs/syh/harness/: run_prefill_b300.sh, run_decode_b300.sh, run_router_b300.sh, b300_rdma_test.sh (host-runnable server/client), b300-bringup-networking.md. Doc also docs_internal/. Memory: reference_b300_cluster. Dispatched 2 opus bg agents: (A) verify prefill per-layer kernel sources on SM100, (B) survey upstream Blackwell/B300/GLM5 perf fixes to consider porting."

[[content.journal]]
date = "2026-06-07"
scope = "b300-perf"
content = "Two opus agents done. (A) prefill per-layer kernel-source audit on SM100: NO sm_90 lock-in anywhere; all kernels runtime-selected or JIT-built for live arch. Map: NSA attn=sgl_kernel flash_mla_sparse_fwd (heads 64->128 on Blackwell); indexer=deep_gemm fp8_mqa/paged_mqa/metadata (0.1.2); MoE=deep_gemm m_grouped_fp8_gemm_nt_contiguous + TAI three_step; CP shared-KV=tai_kernel CUDA; FP8 KV=Triton; dense block-fp8=deep_gemm fp8_gemm_nt + sgl_kernel ue8m0 quant. Validated the SECOND tai extension (quantization/three_step, the no-torch-fallback MoE hot path) JIT-compiles on cu130/SM100 in 35s (nsa_prefill was 46s). Notable non-correctness: --enable-single-batch-overlap is INERT on this config (Blackwell+deep_gemm+deepep-normal disables masked-down overlap; prefill uses contiguous); BS_GT1_DEBUG != DEBUG_CP_SHARED_KV (TAI fast paths stay active). (B) perf-fix survey -> user decisions: TOOK #25299 (cache MQA-logits budget, drop per-call mem_get_info host sync) + #22232 (data_ptr-guarded copy_ instead of rope clone; q_rope/k_rope are split views of query/key so in-place RoPE -> guard no-op). Verified provenance + correctness first-principles, import-validated on B300, committed 6bc23945e4. SKIPPED per user: #21332 (un-force MHA one-shot on Blackwell - risky), #23856 (torch.mm indexer GEMM). EVALUATE later: #27320 (kv-index parallelize), #26424 (EAGLE topk=1 skip cat/topk/sort), #25859 (MQA mem-ratio knob, pairs w/ #25299)."

[[content.journal]]
date = "2026-06-07"
scope = "indexcache"
content = "IndexCache (NSA indexer topk reuse across layers, user's 'every-4-layers' feature) = upstream #21405 (5a35316417 'Enable IndexCache for DeepSeek V3.2', 2026-04-05) + #27114 (393d0e169e gate bugfix + index_skip_topk_offset). NOT in our tree (no skip_topk/index_topk_freq plumbing). GLM-5.1-FP8 config.json has index_topk/index_n_heads/index_head_dim but NO index_topk_freq -> defaults to 1 (no sharing); user will set index_topk_freq=4 later (Zhipu confirmed accuracy OK, so no accuracy-validation needed on our side). Mechanism: thread prev_topk_indices through Model.forward loop -> DecoderLayer -> AttentionMLA forward/forward_prepare/forward_absorb_prepare (gate: 'if not skip_topk or (is_nextn and prev is None): indexer() else reuse') -> forward_absorb_core returns (output, topk_indices). Our forward_mla ALREADY threads topk_indices prepare->core via inner_state (350/363), so carrier exists. #24392 (capture/output infra, 18 files) is ORTHOGONAL - HEAD reuse path doesn't depend on it - SKIPPED. Dispatched opus agent in isolated WORKTREE to port (match HEAD form; find+fix ALL DecoderLayer.forward callers for the new 3-tuple return; py_compile). Will review diff + B300-validate before merging to main tree. Agent B perf-survey MISSED #21405 (in range but not surfaced) - survey not exhaustive."

[[content.journal]]
date = "2026-06-07"
scope = "indexcache"
content = "IndexCache ENABLED end-to-end. Reviewed the worktree-agent port line-by-line (caller coverage: Model loop, nextn, glm4_moe_lite/mistral inherit, TBO op-path via op_core; tuple-unpack placed after CP finally; freq=4 safe since our config has no TBO so all layers go monolithic path that propagates topk), applied to main tree, B300-import-validated, committed af2a41b8d3. Ported the #27114 TBO+index-sharing server_args guard (user: port guards for whole-repo correctness, don't 死扣配置) - committed with it. Edited GLM-5.1 config.json index_topk_freq=4 on BOTH g0036+g0037 (per-host /ssd, backups .bak_preindexcache). CRITICAL CATCH: GlmMoeDsaConfig.__init__ has NO **kwargs -> HF AutoConfig DROPS index_topk_freq (only named fields like index_topk survive) -> the edit was inert. Root cause = exactly #27114's 'Restore overridden HF config fields' piece. Ported it into our get_config (hf_transformers_utils.py, before the Phi4MM block): for GlmMoeDsaForCausalLM re-read raw config.json and setattr qk_rope_head_dim + index_topk_freq/pattern/offset + recompute qk_head_dim. Committed 92ff8b4747. VERIFIED on B300: sglang get_config('/ssd/models/GLM-5.1-FP8') now returns index_topk_freq=4. skip_topk pattern with freq=4: compute at layers 0,1,5,9,...; reuse for the 3 after each. LESSON saved: feedback_agents_read_only, feedback_whole_repo_correctness. Branch: 32818784d3(deepgemm) 6bc23945e4(perf) af2a41b8d3(indexcache) 92ff8b4747(config-restore), unpushed."

[[content.journal]]
date = "2026-06-07"
scope = "session-state"
content = "SESSION STATE for compaction (2026-06-08). BRANCH cjy-cp-refactor unpushed commits (user pushes): 32818784d3 deepgemm sgl-deep-gemm 0.1.2 migration; 6bc23945e4 perf(nsa) MQA-budget cache + rope-clone drop (#25299,#22232); af2a41b8d3 IndexCache port (#21405+#27114 gate + TBO guard); 92ff8b4747 GlmMoeDsa config-restore (index_topk_freq survives HF load). All B300-import-validated. (1) B300 BRING-UP [[reference_b300_cluster]]: code fully validated on torch-2.11 dev-cu13 image (deepgemm/tai-kernel JIT/IndexCache import-clean); networking SOLVED = per-GPU rails roce_vf_rail0..7 RoCE v2 GID idx 3 (per-rail VRF gotcha; RDMA ib_write_bw 036<->037 verified); GLM config index_topk_freq=4 set on BOTH g0036/g0037 (/ssd, .bak_preindexcache). BLOCKED: server won't start - NCCL hang(2.28.9)/segfault(2.29.7,2.30.4) at ncclCommInitRank on FIRST collective. Standalone torchrun micro-bench (/mnt/beegfs/syh/harness/nccl_microbench.py) ALSO hangs => SYSTEM P2P/NVLink, not sglang: NCCL fell back to NET/IB on the CX8 rails (single-node should use NVLink) => NVLink/fabric/peermem. User upgrading DOCA + will sweep NCCL. (2) flashmla: server_args.py:1502 trtllm-force UNCONDITIONALLY overrides --nsa-prefill-backend (only branch missing 'if not user_set_prefill'); flashmla_sparse DOES work on Blackwell (pads heads 64->128); 'needs DP attn' = FALSE ALARM for CP; FIX (parked, await go) = guard line 1502 -> then CP override picks flashmla_sparse. (3) bs>1 KV CONTAMINATION [[project_cp_bs_gt1_kv_contamination]]: 3 opus agents CONVERGED -> root cause = build_slot_page_inverse (cp_shared_kv_runtime.py:2973) value-keyed last-writer-wins page_inverse collapses duplicate logical pages across requests (non-atomic TAI scatter cp_shared_kv_materialize.cu:165). Known partial-fix bug (f75ffff8d9 masking only covers stale tails). 4th confirm+fix-validation agent (a9c8ebe4291826250) running. Fix dir: per-request page_inverse / key by (req_id,page) / use build_slot_page_remap per-slot dense_pages; disambig SGLANG_CP_SHARED_KV_PER_LAYER_TRANSFER=0. FEEDBACK saved: agents READ-ONLY [[feedback_agents_read_only]]; whole-repo correctness [[feedback_whole_repo_correctness]]; user wants confirm-state-before-any-operation. NEXT after compaction: process confirm-agent verdict -> fix bs>1 contamination (user sign-off first)."

[[content.journal]]
date = "2026-06-08"
content = "B300 decode DeepEP bring-up: bundled deep_ep 1.2.1 segfaults (nvshmem ibv_alloc_pd) on rails. Rebuilt deepseek-ai/DeepEP @29d31c0 vs nvshmem 3.6.5 (kept wheel), upgraded runtime nvidia-nvshmem-cu13 3.4.5->3.6.5 (fixed undefined-symbol import). Root fix for low_latency: NVSHMEM_HCA_LIST=roce_vf_rail0..7 (VF rails, defaults) -- nvshmem auto-enum hit zero-GUID devices. intranode test passes (~400 GB/s NVLink); decode relaunching with new wheel+env. See reference_b300_cluster memory."

[[content.acceptance_criteria]]
text = "Port deep_gemm sgl-deep-gemm 0.1.2 API compat (2D->N_total,1 ctx_lens, compile-mode hasattr guard, positional m_indices, DLPack stride fix, SBO masked unpack guard, PDL default, kernel version assert)"
status = "pending"
category = "added"

[[content.acceptance_criteria]]
text = "Rebuild tai-kernel against torch 2.11 and import-smoke + harness-validate in the new dev-cu13 image"
status = "pending"
category = "added"

[[content.acceptance_criteria]]
text = "govctl check passes"
status = "pending"
category = "chore"