← 研踪 PaperTrailarXiv:2605.09238 · v2
iMuon: Intrinsic Spectral Optimization on Riemannian Matrix Manifolds
首次提交 2026-05-10 08:39
版本更新 2026-11-17 07:41
雷达观测 2026-11-18 19:43
摘要
Muon and related norm-constrained optimizers, now central to large-scale learning, compute updates via a linear maximization oracle (LMO) over a matrix-norm ball in Euclidean space. However, they do not generalize cleanly to manifold-valued parameters such as low-rank factorizations, orthogonality constraints, or symmetric positive-definite (SPD) matrices. Naively restricting the Muon LMO to the tangent space (i) breaks quotient symmetries and (ii) couples the tangent-space constraint with an ambient norm bound, which obstructs closed-form solutions. We resolve both issues with a key observation: every Riemannian metric canonically lifts a unitarily invariant Euclidean norm to an intrinsic norm on each tangent space. The resulting intrinsic LMO is symmetry preserving and, under a structural condition on the scaled tangent space, reduces to a standard Euclidean LMO. Building on this, we introduce intrinsic Muon (iMuon), a unified framework with closed-form updates on the fixed-rank, SPD, Stiefel, and Grassmann manifolds for various unitarily invariant norms, including the spectral, Frobenius, and nuclear norms. We establish deterministic and stochastic convergence rates of iMuon with constants that depend only on the manifold dimension. Notably, on the fixed-rank manifold, the constant depends only on the rank, which makes the rate independent of factor conditioning and removes the runtime factor rescaling required by prior work. Experiments on LoRA fine-tuning of LLMs, image classification, and subspace learning illustrate the efficacy of the proposed approach.
作者备注:Code: https://github.com/1bang118/manifold-intrinsic-muon Accepted at NeurIPS 2026. Updated to match the camera-ready version
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-17 07:41 · 最新观测版本
iMuon: Intrinsic Spectral Optimization on Riemannian Matrix Manifolds
Muon and related norm-constrained optimizers, now central to large-scale learning, compute updates via a linear maximization oracle (LMO) over a matrix-norm ball in Euclidean space. However, they do not generalize cleanly to manifold-valued parameters such as low-rank factorizations, orthogonality constraints, or symmetric positive-definite (SPD) matrices. Naively restricting the Muon LMO to the tangent space (i) breaks quotient symmetries and (ii) couples the tangent-space constraint with an ambient norm bound, which obstructs closed-form solutions. We resolve both issues with a key observation: every Riemannian metric canonically lifts a unitarily invariant Euclidean norm to an intrinsic norm on each tangent space. The resulting intrinsic LMO is symmetry preserving and, under a structural condition on the scaled tangent space, reduces to a standard Euclidean LMO. Building on this, we introduce intrinsic Muon (iMuon), a unified framework with closed-form updates on the fixed-rank, SPD, Stiefel, and Grassmann manifolds for various unitarily invariant norms, including the spectral, Frobenius, and nuclear norms. We establish deterministic and stochastic convergence rates of iMuon with constants that depend only on the manifold dimension. Notably, on the fixed-rank manifold, the constant depends only on the rank, which makes the rate independent of factor conditioning and removes the runtime factor rescaling required by prior work. Experiments on LoRA fine-tuning of LLMs, image classification, and subspace learning illustrate the efficacy of the proposed approach.
查看此版本解读 →查看此版本原文 ↗