跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.26475 · v1

A new algorithmic approach for finding flat minima: critically damped Polyak stepsizes

寻找平坦极小值的新算法方法:临界阻尼Polyak步长

首次提交 2026-11-20 01:35
版本更新 2026-11-20 01:35
雷达观测 2026-11-20 12:52
arXiv ↗PDF ↗

摘要

We present a new adaptive stepsize rule that biases gradient descent toward flatter minima in both theory and practice, and establish the first uniform-in-time guarantees for both sharpness and loss reduction for a broad class of locally quadratic objectives. Our critically damped Polyak (CDP) stepsizes multiply the classical Polyak rule by 44 and a mild damping factor. The factor 44 is critical in that it produces equioscillations around the manifold of minimizers M\mathcal{M} to leading order. Damping provably drives convergence to M\mathcal{M}. CDP thus operates at the edge of stability by adapting its stepsizes to local curvature. Once the loss is small, its projected iterates follow gradient descent on log⁡λ1(∇2L)\log{λ_1(\nabla^2 \mathcal{L})} along M\mathcal{M} to leading order, paralleling the flatness bias of normalized gradient descent (Arora et al., 2022). Unlike normalized gradient descent, CDP step lengths shrink with the distance to M\mathcal{M}, enabling convergence to an interpolating solution that locally minimizes sharpness at finite learning rate. We establish our theory for a broad class of non-negative losses with locally quadratic Morse-Bott structure near M\mathcal{M}, capturing regular interpolation regimes in smooth overparameterized neural networks. Empirically, CDP stepsizes produce a flattening effect similar to that of Sharpness-Aware Minimization (SAM, Foret et al., 2021), reducing sharpness even in settings beyond our theory, including nonlinear teacher-student networks and fine-tuning of pre-trained LLMs.

作者备注:56 pages, 19 figures

AI 论文解读

基于摘要 · 非全文审阅

提出临界阻尼Polyak步长,在理论和实践中使梯度下降偏向更平坦的极小值,并首次给出局部二次目标下锐度与损失下降的一致时间保证。

正在读取账号与研究记录…

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-20 01:35 · 最新观测版本

A new algorithmic approach for finding flat minima: critically damped Polyak stepsizes

We present a new adaptive stepsize rule that biases gradient descent toward flatter minima in both theory and practice, and establish the first uniform-in-time guarantees for both sharpness and loss reduction for a broad class of locally quadratic objectives. Our critically damped Polyak (CDP) stepsizes multiply the classical Polyak rule by 44 and a mild damping factor. The factor 44 is critical in that it produces equioscillations around the manifold of minimizers M\mathcal{M} to leading order. Damping provably drives convergence to M\mathcal{M}. CDP thus operates at the edge of stability by adapting its stepsizes to local curvature. Once the loss is small, its projected iterates follow gradient descent on log⁡λ1(∇2L)\log{λ_1(\nabla^2 \mathcal{L})} along M\mathcal{M} to leading order, paralleling the flatness bias of normalized gradient descent (Arora et al., 2022). Unlike normalized gradient descent, CDP step lengths shrink with the distance to M\mathcal{M}, enabling convergence to an interpolating solution that locally minimizes sharpness at finite learning rate. We establish our theory for a broad class of non-negative losses with locally quadratic Morse-Bott structure near M\mathcal{M}, capturing regular interpolation regimes in smooth overparameterized neural networks. Empirically, CDP stepsizes produce a flattening effect similar to that of Sharpness-Aware Minimization (SAM, Foret et al., 2021), reducing sharpness even in settings beyond our theory, including nonlinear teacher-student networks and fine-tuning of pre-trained LLMs.

查看此版本解读 →查看此版本原文 ↗