← 研踪 PaperTrailarXiv:2610.07593 · v2
TRANSIT: Transparent Scale-in for Multi-Node LLM Training
首次提交 2026-10-06 09:33
版本更新 2026-11-19 12:25
雷达观测 2026-11-20 10:50
摘要
The growing memory requirements of large language models (LLMs) increasingly force distributed training jobs to span multiple GPU nodes to accommodate their training state. This reduces per-GPU computational efficiency, increases network traffic, and exacerbates job queueing delays. To address these inefficiencies, we introduce TRANSIT, a system that transparently enables efficient distributed LLM training on fewer GPUs through memory oversubscription. TRANSIT optimizes CPU-GPU data movement during oversubscription by coordinating driver-managed page residency with execution-guided zero-copy access, exploiting recurring kernel execution patterns across training iterations. We evaluate TRANSIT on dense and mixture-of-experts (MoE) models across different parallelization strategies using up to 64 NVIDIA H100-80GB GPUs interconnected over RDMA over Converged Ethernet (RoCE). Compared with the minimum GPU count required to accommodate the training state in GPU memory, TRANSIT enables training on up to 50% fewer GPUs while retaining over 90% of the baseline per-GPU computational throughput. In some configurations, training on 17-33% fewer GPUs even improves throughput due to reduced distributed communication overhead. When training on fewer GPUs, TRANSIT achieves up to 243% and 87% higher throughput than conventional driver-runtime-managed memory oversubscription and framework-managed CPU offloading, respectively, while reducing CPU memory consumption by up to 79% compared with the latter. Training on fewer GPUs with TRANSIT also reduces per-node network traffic by up to 33% and improves throughput by up to 44% under constrained network bandwidth. Production-trace-driven simulations further show 15-49% lower inter-rack network traffic and up to 2-95% shorter job queue times for different cluster capacities.
作者备注:14 pages, 14 figures, 2 tables
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-19 12:25 · 最新观测版本
TRANSIT: Transparent Scale-in for Multi-Node LLM Training
The growing memory requirements of large language models (LLMs) increasingly force distributed training jobs to span multiple GPU nodes to accommodate their training state. This reduces per-GPU computational efficiency, increases network traffic, and exacerbates job queueing delays. To address these inefficiencies, we introduce TRANSIT, a system that transparently enables efficient distributed LLM training on fewer GPUs through memory oversubscription. TRANSIT optimizes CPU-GPU data movement during oversubscription by coordinating driver-managed page residency with execution-guided zero-copy access, exploiting recurring kernel execution patterns across training iterations. We evaluate TRANSIT on dense and mixture-of-experts (MoE) models across different parallelization strategies using up to 64 NVIDIA H100-80GB GPUs interconnected over RDMA over Converged Ethernet (RoCE). Compared with the minimum GPU count required to accommodate the training state in GPU memory, TRANSIT enables training on up to 50% fewer GPUs while retaining over 90% of the baseline per-GPU computational throughput. In some configurations, training on 17-33% fewer GPUs even improves throughput due to reduced distributed communication overhead. When training on fewer GPUs, TRANSIT achieves up to 243% and 87% higher throughput than conventional driver-runtime-managed memory oversubscription and framework-managed CPU offloading, respectively, while reducing CPU memory consumption by up to 79% compared with the latter. Training on fewer GPUs with TRANSIT also reduces per-node network traffic by up to 33% and improves throughput by up to 44% under constrained network bandwidth. Production-trace-driven simulations further show 15-49% lower inter-rack network traffic and up to 2-95% shorter job queue times for different cluster capacities.
查看此版本解读 →查看此版本原文 ↗