← 研踪 PaperTrailarXiv:2605.06081 · v2
Fast Gauss-Newton for Multiclass Cross-Entropy
首次提交 2026-05-07 20:03
版本更新 2026-11-09 18:14
雷达观测 2026-11-10 15:22
摘要
In multiclass softmax cross-entropy, the full generalized Gauss-Newton (GGN) curvature couples all output logits through the softmax covariance, making curvature-vector products harder to scale as the number of classes grows. We show that the standard multiclass GGN can be decomposed exactly into a true-vs-rest term and a positive semidefinite within-competitor covariance term. Fast Gauss-Newton (FGN) retains the first term and drops the second, yielding a positive semidefinite under-approximation of the multiclass GGN that is exact for binary classification. The derivation uses an exact true-vs-rest scalar-margin representation of softmax cross-entropy: the loss and gradient are unchanged, and the approximation enters only at the curvature level. Exploiting the FGN curvature structure, the damped update can be written as an equivalent whitened row-space system with one row per mini-batch example. We solve this system matrix-free by conjugate gradient using Jacobian-vector and vector-Jacobian products of the scalar margin map. Targeted mechanism experiments show how the dropped covariance grows with competitor dispersion and how damping can reduce its effect on the resulting steps. In frozen-feature head optimization and end-to-end CIFAR-100 training, FGN achieves similar test accuracy to a full GGN baseline with faster updates.
作者备注：NeurIPS 2026. 28 pages (10 main text + 18 references and appendix)
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-09 18:14 · 最新观测版本
Fast Gauss-Newton for Multiclass Cross-Entropy
In multiclass softmax cross-entropy, the full generalized Gauss-Newton (GGN) curvature couples all output logits through the softmax covariance, making curvature-vector products harder to scale as the number of classes grows. We show that the standard multiclass GGN can be decomposed exactly into a true-vs-rest term and a positive semidefinite within-competitor covariance term. Fast Gauss-Newton (FGN) retains the first term and drops the second, yielding a positive semidefinite under-approximation of the multiclass GGN that is exact for binary classification. The derivation uses an exact true-vs-rest scalar-margin representation of softmax cross-entropy: the loss and gradient are unchanged, and the approximation enters only at the curvature level. Exploiting the FGN curvature structure, the damped update can be written as an equivalent whitened row-space system with one row per mini-batch example. We solve this system matrix-free by conjugate gradient using Jacobian-vector and vector-Jacobian products of the scalar margin map. Targeted mechanism experiments show how the dropped covariance grows with competitor dispersion and how damping can reduce its effect on the resulting steps. In frozen-feature head optimization and end-to-end CIFAR-100 training, FGN achieves similar test accuracy to a full GGN baseline with faster updates.
查看此版本解读 →查看此版本原文 ↗