更早释放、更少暴露：双批次重叠下 NCCL-EP 的 MoE 分发卸载
Release Sooner, Expose Less: Offloading MoE Dispatch in NCCL-EP under Dual-Batch Overlap
Hsin-Chun Yin, Khaled Hamidouche, Ian Ziemba, Arjun Kashyap, Pak Markthub, Subhadeep Bhattacharya, Ke Wen, Assaf Ravid 等 10 位作者
AI 速读 基于摘要
作者提出将 NCCL-EP 中 MoE dispatch 的发送推进从 GPU 卸载到主机线程的 GDA-KI Free Flow 配置，以缩短 dispatch send 占用 GPU 的暴露时间，并在 GB300 NVL72 与分离式 vLLM 服务中报告了收益。
cs.DC
查看原文摘要
Dual-batch overlap (DBO) hides mixture-of-experts (MoE) communication behind another microbatch's computation, but only after dispatch send releases the GPU. The time dispatch send holds the GPU therefore stays exposed, while network progress after release can be hidden. We evaluate GPUDirect Async - Kernel Initiated (GDA-KI) Free Flow, an NCCL-EP research configuration that leaves WQE construction on the GPU but moves send progression to a host thread. On a GB300 NVL72 system, GDA-KI Free Flow reduces dispatch-send time relative to default GDA-KI at every operating point in the evaluated parameter sweep. An experiment that inserts a GPU delay between dispatch send and receive shows that offload increases the in-flight interval. Depending on the load, 32-96 microseconds of computation or other independent work between send and receive hides the deferred progress, leaving less exposed time than default GDA-KI. In disaggregated vLLM serving with DBO, GDA-KI Free Flow lowers median TPOT by 15-25% relative to NCCL-EP default GDA-KI and by 16-42% relative to DeepEP v1.