BrownoutMoE: Learning Structure-Aware Expert Grouping for Adaptive Mixture-of-Experts Model Serving
Yi Ding, Minxian Xu, Zhengxin Fang, Kejiang Ye, Chengzhong Xu
原文摘要
Mixture-of-Experts (MoE) large language models (LLMs) are increasingly served on multi-GPU clusters in cloud data centers, where inference must stay accurate and responsive under bursty demand. Expert access in MoE inference is highly imbalanced: a few hot experts process most tokens, whereas many cold experts fragment kernels and leave GPU parallelism underused. Existing serving systems optimize runtime execution but keep the original expert organization. We propose BrownoutMoE, a structure-aware serving framework inspired by the brownout paradigm in cloud computing. It learns layer-wise expert grouping offline with reinforcement learning, using the post-distillation error as the reward. Grouping-consistent distillation trains one united expert per group, and an online controller adjusts how often united experts replace cold experts under latency objectives. On eight A100 GPUs, BrownoutMoE reduces accuracy degradation by up to 71.4% relative to index-order grouping, improves unfused throughput by up to 2.24x over vLLM, and lowers saturated decode latency by 38%.
cs.DC