The old min_new_tokens penalizer updated logits through boolean-mask indexing. That indexing is data-dependent and can force synchronization on the decode hot path.
Use an elementwise torch.where followed by inplace add so the operation stays tensorized and avoids the mask-index update path.
Constraint: Keep the numeric behavior for active rows equivalent without multiplying by zero, which would turn -inf penalties into NaN.
Rejected: Expanding the mask and assigning through logits[mask] | this is the synchronization pattern being removed.
Confidence: high
Scope-risk: narrow
Directive: Do not reintroduce boolean-mask writes in decode-step penalty paths without profiling the synchronization behavior.
Tested: RED/GREEN local pytest test/registered/unit/sampling/test_min_new_tokens_penalizer.py
Tested: RED/GREEN remote pytest in cjy-glm5-new for min_new_tokens penalizer together with related regression tests
Tested: git diff --check; py_compile for min_new_tokens.py
Not-tested: Full decode throughput benchmark
from_schedule_batch built temperature/top_p/top_k/min_p (+seed) with
4-5 separate list comprehensions and one synchronous H2D copy each,
plus 4 more passes for the is_all_greedy/need_* flags. Collect
everything in a single pass over reqs and upload the float params as
one pinned non-blocking H2D copy (disjoint device views of one
buffer; filter/merge only index and cat, producing fresh tensors, so
the shared buffer is safe), int32 top_k and optional int64 seeds as
their own pinned copies.
B300 (torch 2.11 cu130), scheduler-thread blocking time per call:
bs=8: 38.8 -> 18.2 us (2.1x); bs=32: 1.9x; bs=200: 1.2x
CPU-only construction at bs=200: 2890 -> 1387 us (2.1x).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
