[2/2] Support deterministic inference for temperature > 0 (#10678)

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com> Co-authored-by: hebiao064 <hebiaobuaa@gmail.com>
2025-09-21 19:36:08 -07:00
parent 86527a4799
commit e2ac7888b8
12 changed files with 117 additions and 11 deletions
@@ -270,9 +270,7 @@ class TpModelWorker:
                        logits_output, model_worker_batch
                    )
            else:
-                next_token_ids = self.model_runner.sample(
-                    logits_output, model_worker_batch
-                )
+                next_token_ids = self.model_runner.sample(logits_output, forward_batch)

            return logits_output, next_token_ids, can_run_cuda_graph
        else: