Two upstream NSA-indexer perf ports (verified against our diverged tree), cutting
CPU launches / host syncs on the prefill critical path for GLM-5.1-FP8 on B300:
- #25299 (beaff00331): cache the MQA-logits chunking budget per device so prefill
stops calling torch.cuda.mem_get_info (a host sync) on every indexer pass. The
budget is computed once and capped by the mem_fraction_static serving headroom;
a static guard is used (uncached) during cuda-graph capture, and the first real
prefill caches the free-memory budget.
- #22232 (671fe73961): replace `dst = src.clone()` slice write-backs of the RoPE
output with a data_ptr-guarded `dst.copy_(src)`. q_rope/k_rope are torch.split
views of query/key, so when RoPE runs in place src/dst alias and the write-back
is a redundant no-op (guard skips it); otherwise one copy instead of clone+copy.
Saves an alloc+copy per q/k per indexer call. (Skipped the PR's AMD-only
@torch.compile cleanup.)
Both verified to import on the B300 / torch-2.11 image. Deliberately NOT taken:
#21332 (un-force MHA one-shot on Blackwell — risky, per user), #23856 (torch.mm
indexer GEMM — per user).
Refs: WI-2026-06-07-001
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 6bc23945e4ff5c68b0618173856c7daf4653fa10)
leavelet
4cf8885a74