← 研踪 PaperTrailarXiv:2607.04164 · v2
BrownoutMoE: Learning Structure-Aware Expert Grouping for Adaptive Mixture-of-Experts Model Serving
首次提交 2026-07-05 16:08
版本更新 2026-11-15 09:44
雷达观测 2026-11-17 12:06
摘要
Mixture-of-Experts (MoE) large language models (LLMs) are increasingly served on multi-GPU clusters in cloud data centers, where inference must stay accurate and responsive under bursty demand. Expert access in MoE inference is highly imbalanced: a few hot experts process most tokens, whereas many cold experts fragment kernels and leave GPU parallelism underused. Existing serving systems optimize runtime execution but keep the original expert organization. We propose BrownoutMoE, a structure-aware serving framework inspired by the brownout paradigm in cloud computing. It learns layer-wise expert grouping offline with reinforcement learning, using the post-distillation error as the reward. Grouping-consistent distillation trains one united expert per group, and an online controller adjusts how often united experts replace cold experts under latency objectives. On eight A100 GPUs, BrownoutMoE reduces accuracy degradation by up to 71.4% relative to index-order grouping, improves unfused throughput by up to 2.24x over vLLM, and lowers saturated decode latency by 38%.
作者备注:25 pages, 11 figures
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-15 09:44 · 最新观测版本
BrownoutMoE: Learning Structure-Aware Expert Grouping for Adaptive Mixture-of-Experts Model Serving
Mixture-of-Experts (MoE) large language models (LLMs) are increasingly served on multi-GPU clusters in cloud data centers, where inference must stay accurate and responsive under bursty demand. Expert access in MoE inference is highly imbalanced: a few hot experts process most tokens, whereas many cold experts fragment kernels and leave GPU parallelism underused. Existing serving systems optimize runtime execution but keep the original expert organization. We propose BrownoutMoE, a structure-aware serving framework inspired by the brownout paradigm in cloud computing. It learns layer-wise expert grouping offline with reinforcement learning, using the post-distillation error as the reward. Grouping-consistent distillation trains one united expert per group, and an online controller adjusts how often united experts replace cold experts under latency objectives. On eight A100 GPUs, BrownoutMoE reduces accuracy degradation by up to 71.4% relative to index-order grouping, improves unfused throughput by up to 2.24x over vLLM, and lowers saturated decode latency by 38%.
查看此版本解读 →查看此版本原文 ↗