LionMuon: Alternating Spectral and Sign Descent for Efficient Training
Arman Bolatov, Artem Riabinin, Nikita Kornilov, Andrey Veprikov, Samuel Horváth, Martin Takáč, Aleksandr Beznosikov
Muon's spectral step gives a stronger direction than a sign step, but it is expensive. Every step runs Newton-Schulz iterations on the full matrix and, in distributed training, an extra all-reduce. Sign steps, as in Lion and Signum, are cheap and stay local to each device. We propose LionMuon, which takes one Muon step every P iterations and Lion steps in between, with a single dual-EMA momentum buffer shared by both. Muon's compute and communication are paid once per P steps, and the optimizer state is one buffer per matrix, half of AdamW's. A single-EMA variant, SignMuon, isolates the effect of alternation. We prove complexity bounds under heavy-tailed noise in which the period interpolates between the smoothness and noise constants of Muon and Lion, and which state, through measurable ratios of these constants, when LionMuon's complexity is below Muon's. On 124M and 355M models trained on FineWeb, LionMuon with P=2 and P=5 reaches a lower loss than Muon, AdamW, Lion and Signum at the same number of tokens. Under 4-GPU data-parallel training LionMuon reaches Muon's final loss with up to 30% less wall-clock on PCIe, and it beats the communication-efficient Muon variants Dion and MuonBP on loss at no more exposed communication, while keeping the exact gradient. Code: https://github.com/brain-lab-research/lion-muon
尚无 AI 解读，展示原文摘要