feat(disagg/mooncake): add non-blocking async transfer wrapper bindings

Add batch_transfer_async_submit() (non-blocking submit -> batch_id) and
wait_batch_transfers() (block-until-all, graceful) to MooncakeTransferEngine,
exposing the mooncake async API for the upcoming per-layer overlapped transfer.

Deliberately avoids batch_transfer_*_on_cuda: its CUDA host callback busy-waits
on the stream and calls _exit(1) on transfer failure (verified in the mooncake
source transfer_engine_py.cpp), which is unsafe for production (a decode-side
abort mid-transfer would crash the prefill). The async submit + wait pair lets
per-layer submits pipeline in the RDMA engine, then waits once on a background
thread with graceful failure.

Both bindings raise a clear upgrade error if the engine predates the API and
degrade to -1 on transient errors. Mock-engine unit tested (8 cases).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py

@@ -251,6 +251,54 @@ class MooncakeTransferEngine:
             )
         return ret
 
+    def batch_transfer_async_submit(
+        self,
+        session_id: str,
+        buffers: List[int],
+        peer_buffer_addresses: List[int],
+        lengths: List[int],
+    ) -> int:
+        """Submit a batch WRITE transfer WITHOUT blocking; returns a mooncake batch_id
+        handle (>= 0) or -1 on failure.
+
+        Unlike batch_transfer_sync (which submits and waits in one call), this returns
+        immediately after submitting, so multiple per-layer submits pipeline in the RDMA
+        engine before a single wait. Pair with wait_batch_transfers([batch_id, ...]).
+
+        Chosen over batch_transfer_*_on_cuda, whose host callback busy-waits on the CUDA
+        stream and calls _exit(1) on failure (verified in the mooncake source) — unsafe.
+        """
+        try:
+            return self.engine.batch_transfer_async_write(
+                session_id, buffers, peer_buffer_addresses, lengths
+            )
+        except Exception:
+            if not hasattr(self.engine, "batch_transfer_async_write"):
+                raise RuntimeError(
+                    "Mooncake's async transfer requires mooncake-transfer-engine "
+                    ">= 0.3.9. Please upgrade Mooncake."
+                )
+            logger.debug("Mooncake batch_transfer_async_write failed.")
+            return -1
+
+    def wait_batch_transfers(self, batch_ids: List[int]) -> int:
+        """Block until ALL submitted batch_ids complete; returns 0 on success, non-zero
+        on failure/timeout. Frees the batch_ids. Graceful (no process _exit). Call this
+        on a background thread, never the scheduler/compute thread.
+        """
+        if not batch_ids:
+            return 0
+        try:
+            return self.engine.get_batch_transfer_status(list(batch_ids))
+        except Exception:
+            if not hasattr(self.engine, "get_batch_transfer_status"):
+                raise RuntimeError(
+                    "Mooncake's async transfer status requires mooncake-transfer-engine "
+                    ">= 0.3.9. Please upgrade Mooncake."
+                )
+            logger.debug("Mooncake get_batch_transfer_status failed.")
+            return -1
+
     def get_session_id(self):
         return self.session_id