跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.01335 · v2

Profile-aware diagnostics of discarded parameter-Hessian mass

Maxim BolshimAlexander Kugaevskikh
cs.LG
首次提交 2026-11-01 07:04
版本更新 2026-11-05 04:09
雷达观测 2026-11-06 11:06
arXiv ↗PDF ↗

摘要

Evaluating block-diagonal curvature models involves two questions: how much curvature they omit, and what a specific correction changes at a finite step. We measure the squared off-block fraction of the parameter Hessian against references for group sizes and the observed within-group curvature profile. In CIFAR-100 ResNet-18 and a skip-free counterpart, similar initial discarded fractions accompany different profiles. At saved trained ResNet-18 weights, changing from stage to layer groups moves estimated off-block mass from below to above the profile envelope; an approximate conditional screen resolves this comparison in all six seed/input cells, while some skip free comparisons remain unresolved. Separately, in saved ResNet-50/Stanford Cars states, we compare candidates formed from off-group Hessian action, the remaining model residual and their combination through the same Hessian-informed map. At unit scale off-group input alone raises native minibatch loss relative to the base preconditioner candidate in 17 of 18 state/geometry comparisons, yet the combined candidate has lower loss than the remainder-only candidate in nine of those cases; preferences vary with scale and correction-norm matching. The profiles help assess block models and alternative groupings, while same-state candidate comparisons test a particular correction without equating structural mass with training gain.

作者备注：36 pages, 6 figures, 17 tables, including appendices

查看 arXiv 原文 ↗阅读 PDF ↗

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存，登录后共享已有结果。

版本差异

仅对比实际观测的元数据，不代表 PDF 正文或实验变化。版本可能不连续；带删除线为移除，下划线为新增。

摘要变化

A discardedEvaluating parameterblock-Hessiandiagonal fractioncurvature dependsmodels oninvolves thetwo diagonalquestions: profilehow andmuch parametercurvature partition;they itomit, isand notwhat thea Fisherspecific informationcorrection discardedchanges byat K-FACa finite step. We comparemeasure off-blockthe squared Frobeniusoff-block massfraction withof group-sizethe andparameter measuredHessian diagonal-profileagainst references at specifiedfor weights,group inputssizes and coordinates.the Similarobserved initialwithin-group fractionscurvature inprofile. In CIFAR-100 ResNet-18 and itsa skip-free twincounterpart, similar initial discarded fractions accompany different profiles. On the sameAt saved trained ResNet-18 statesweights, changing from stage groups to layerslayer groups moves estimated off-block mass from below to above the profile envelope.; Anan approximate joint conditional Monte Carlo screen resolves this crossingcomparison in 6/6all six seed/input cells; the skip-free, networkwhile hassome 2skip resolvedfree andcomparisons 4remain unresolved comparisons. A separate comparison ofSeparately, ninein saved states under two geometryResNet-50/Stanford drawsCars splitsstates, thewe fullcompare modelcandidates residualformed intofrom off-group Hessian action, the remaining model residual and itstheir remainder:combination gradient,through within-groupthe actionsame andHessian-informed dampingmap. At unit correction scale, off-group input alone increasesraises native minibatch loss inrelative 17to ofthe 18base state-geometrypreconditioner comparisons;candidate in nine17 of those18 casesstate/geometry comparisons, yet the fullcombined candidate has lower loss than the remainder-only candidate. Afterin nine of those cases; preferences vary with scale and correction-norm matching, seven. ofThe theprofiles tenhelp comparisonsassess thatblock initiallymodels favouredand fullalternative favourgroupings, thewhile remainder.same-state Thesecandidate finite-losscomparisons contraststest dependa onparticular scalecorrection andwithout doequating notstructural establishmass awith restorationtraining benefitgain.

观测到的版本

相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-05 04:09 · 最新观测版本

Profile-aware diagnostics of discarded parameter-Hessian mass

Evaluating block-diagonal curvature models involves two questions: how much curvature they omit, and what a specific correction changes at a finite step. We measure the squared off-block fraction of the parameter Hessian against references for group sizes and the observed within-group curvature profile. In CIFAR-100 ResNet-18 and a skip-free counterpart, similar initial discarded fractions accompany different profiles. At saved trained ResNet-18 weights, changing from stage to layer groups moves estimated off-block mass from below to above the profile envelope; an approximate conditional screen resolves this comparison in all six seed/input cells, while some skip free comparisons remain unresolved. Separately, in saved ResNet-50/Stanford Cars states, we compare candidates formed from off-group Hessian action, the remaining model residual and their combination through the same Hessian-informed map. At unit scale off-group input alone raises native minibatch loss relative to the base preconditioner candidate in 17 of 18 state/geometry comparisons, yet the combined candidate has lower loss than the remainder-only candidate in nine of those cases; preferences vary with scale and correction-norm matching. The profiles help assess block models and alternative groupings, while same-state candidate comparisons test a particular correction without equating structural mass with training gain.

查看此版本解读 →查看此版本原文 ↗
v1 · 2026-11-01 07:04

Profile-aware diagnostics of discarded parameter-Hessian mass

A discarded parameter-Hessian fraction depends on the diagonal profile and parameter partition; it is not the Fisher information discarded by K-FAC. We compare off-block squared Frobenius mass with group-size and measured diagonal-profile references at specified weights, inputs and coordinates. Similar initial fractions in ResNet-18 and its skip-free twin accompany different profiles. On the same saved trained ResNet-18 states, changing from stage groups to layers moves estimated mass from below to above the profile envelope. An approximate joint conditional Monte Carlo screen resolves this crossing in 6/6 seed/input cells; the skip-free network has 2 resolved and 4 unresolved comparisons. A separate comparison of nine saved states under two geometry draws splits the full model residual into off-group Hessian action and its remainder: gradient, within-group action and damping. At unit correction scale, off-group input alone increases loss in 17 of 18 state-geometry comparisons; in nine of those cases, the full candidate has lower loss than the remainder-only candidate. After correction-norm matching, seven of the ten comparisons that initially favoured full favour the remainder. These finite-loss contrasts depend on scale and do not establish a restoration benefit.

查看此版本解读 →查看此版本原文 ↗