← 研踪 PaperTrailarXiv:2611.01859 · v1
Apto: Self-Adaptive CPU-GPU Hybrid MoE Inference on Shared PCs
首次提交 2026-11-01 20:50
版本更新 2026-11-01 20:50
雷达观测 2026-11-04 23:43
摘要
CPU/GPU hybrid inference enables Mixture-of-Experts (MoE) models to run on memory-constrained PCs by keeping all expert weights in host memory and executing activated experts across both processors. Existing hybrid systems, however, assume exclusive resource access and determine an upfront CPU/GPU partition, so CPU contention from colocated applications can severely imbalance the two execution paths. This paper presents Apto, a self-adaptive hybrid inference system that applies dynamic scheduling to expert execution. The CPU and GPU executors claim experts from opposite ends of a dual-ended queue as they progress, allowing the CPU/GPU work split to adapt to their execution progress without predicting contention. Apto further uses affinity-aware scheduling to expose each executor's preferred experts at its queue end and cancellable tail racing to mitigate final-expert stragglers. Compared with state-of-the-art GPU-only and hybrid baselines, Apto achieves average prefill and decode speedups of 2.96$\times$ and 1.52$\times$ under standalone execution, and 5.97$\times$ and 3.45$\times$ under concurrent CPU workloads. It also accelerates real-world coding-agent applications by 2.02$\times$ on average.
作者备注：15 pages, 13 figures
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-01 20:50 · 最新观测版本
Apto: Self-Adaptive CPU-GPU Hybrid MoE Inference on Shared PCs
CPU/GPU hybrid inference enables Mixture-of-Experts (MoE) models to run on memory-constrained PCs by keeping all expert weights in host memory and executing activated experts across both processors. Existing hybrid systems, however, assume exclusive resource access and determine an upfront CPU/GPU partition, so CPU contention from colocated applications can severely imbalance the two execution paths. This paper presents Apto, a self-adaptive hybrid inference system that applies dynamic scheduling to expert execution. The CPU and GPU executors claim experts from opposite ends of a dual-ended queue as they progress, allowing the CPU/GPU work split to adapt to their execution progress without predicting contention. Apto further uses affinity-aware scheduling to expose each executor's preferred experts at its queue end and cancellable tail racing to mitigate final-expert stragglers. Compared with state-of-the-art GPU-only and hybrid baselines, Apto achieves average prefill and decode speedups of 2.96$\times$ and 1.52$\times$ under standalone execution, and 5.97$\times$ and 3.45$\times$ under concurrent CPU workloads. It also accelerates real-world coding-agent applications by 2.02$\times$ on average.
查看此版本解读 →查看此版本原文 ↗