← 研踪 PaperTrailarXiv:2605.19811 · v4
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
首次提交 2026-05-19 21:07
版本更新 2026-10-29 08:53
雷达观测 2026-10-30 11:25
摘要
Muon's spectral step gives a stronger direction than a sign step, but it is expensive. Every step runs Newton-Schulz iterations on the full matrix and, in distributed training, an extra all-reduce. Sign steps, as in Lion and Signum, are cheap and stay local to each device. We propose LionMuon, which takes one Muon step every P iterations and Lion steps in between, with a single dual-EMA momentum buffer shared by both. Muon's compute and communication are paid once per P steps, and the optimizer state is one buffer per matrix, half of AdamW's. A single-EMA variant, SignMuon, isolates the effect of alternation. We prove complexity bounds under heavy-tailed noise in which the period interpolates between the smoothness and noise constants of Muon and Lion, and which state, through measurable ratios of these constants, when LionMuon's complexity is below Muon's. On 124M and 355M models trained on FineWeb, LionMuon with P=2 and P=5 reaches a lower loss than Muon, AdamW, Lion and Signum at the same number of tokens. Under 4-GPU data-parallel training LionMuon reaches Muon's final loss with up to 30% less wall-clock on PCIe, and it beats the communication-efficient Muon variants Dion and MuonBP on loss at no more exposed communication, while keeping the exact gradient. Code: https://github.com/brain-lab-research/lion-muon
作者备注：39 pages, 5 figures, 12 tables
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v4 · 2026-10-29 08:53 · 最新观测版本
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
Muon's spectral step gives a stronger direction than a sign step, but it is expensive. Every step runs Newton-Schulz iterations on the full matrix and, in distributed training, an extra all-reduce. Sign steps, as in Lion and Signum, are cheap and stay local to each device. We propose LionMuon, which takes one Muon step every P iterations and Lion steps in between, with a single dual-EMA momentum buffer shared by both. Muon's compute and communication are paid once per P steps, and the optimizer state is one buffer per matrix, half of AdamW's. A single-EMA variant, SignMuon, isolates the effect of alternation. We prove complexity bounds under heavy-tailed noise in which the period interpolates between the smoothness and noise constants of Muon and Lion, and which state, through measurable ratios of these constants, when LionMuon's complexity is below Muon's. On 124M and 355M models trained on FineWeb, LionMuon with P=2 and P=5 reaches a lower loss than Muon, AdamW, Lion and Signum at the same number of tokens. Under 4-GPU data-parallel training LionMuon reaches Muon's final loss with up to 30% less wall-clock on PCIe, and it beats the communication-efficient Muon variants Dion and MuonBP on loss at no more exposed communication, while keeping the exact gradient. Code: https://github.com/brain-lab-research/lion-muon
查看此版本解读 →查看此版本原文 ↗