← 研踪 PaperTrailarXiv:2604.11639 · v2
Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures
首次提交 2026-04-13 23:48
版本更新 2026-11-06 05:49
雷达观测 2026-11-09 10:16
摘要
Automatic differentiation computes loss Hessians, while interpreting inter-layer interactions requires an explicit choice of graph-indexed blocks. This paper presents an analytical formalism that decomposes the full Hessian into blocks indexed by the DAG of an architecture. The standard decomposition $H = H^{GN} + H^T$ separates the Gauss-Newton component (positive semidefinite for a convex output loss) from the tensor component (residual curvature). If every node map is locally affine in its parent activations, the tensor component of the input Hessian vanishes on each smooth cell ($H^{T}_{v,w}=0$, $H^f_{v,w}=H^{GN}_{v,w}$); ReLU activations alone do not suffice. The full parametric Hessian contains residual terms that do not reduce to the GGN. Building on this decomposition, we use diagnostic metrics (inter-layer resonance $\mathcal{R}$, geometric coupling $\mathcal{C}$, stable rank $\mathcal{D}$, GN-Gap) with stochastic HVP-based estimates whose cost depends on graph evaluation and the number of probes. The theoretical analysis gives conditional bounds on curvature decay and residual-chain transport; empirical examples span fully connected MLPs (Exp. 1-5) and convolutional architectures (ResNet-18, ${\sim}11$M parameters, Exp. 6). When the architecture reduces to a single node, all definitions collapse to the standard Hessian $\nabla^2_θ\mathcal{L}(θ)\in\mathbb{R}^{p\times p}$. Archival correction. The original general novelty claims are no longer maintained: general DAG-Hessian calculus has prior precedents. This is a corrected archival version.
作者备注：45 pages, 9 figures, 17 tables. Submitted to Neural Networks (Elsevier). Code: https://github.com/comiam/dag-hesse
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-06 05:49 · 最新观测版本
Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures
Automatic differentiation computes loss Hessians, while interpreting inter-layer interactions requires an explicit choice of graph-indexed blocks. This paper presents an analytical formalism that decomposes the full Hessian into blocks indexed by the DAG of an architecture. The standard decomposition $H = H^{GN} + H^T$ separates the Gauss-Newton component (positive semidefinite for a convex output loss) from the tensor component (residual curvature). If every node map is locally affine in its parent activations, the tensor component of the input Hessian vanishes on each smooth cell ($H^{T}_{v,w}=0$, $H^f_{v,w}=H^{GN}_{v,w}$); ReLU activations alone do not suffice. The full parametric Hessian contains residual terms that do not reduce to the GGN. Building on this decomposition, we use diagnostic metrics (inter-layer resonance $\mathcal{R}$, geometric coupling $\mathcal{C}$, stable rank $\mathcal{D}$, GN-Gap) with stochastic HVP-based estimates whose cost depends on graph evaluation and the number of probes. The theoretical analysis gives conditional bounds on curvature decay and residual-chain transport; empirical examples span fully connected MLPs (Exp. 1-5) and convolutional architectures (ResNet-18, ${\sim}11$M parameters, Exp. 6). When the architecture reduces to a single node, all definitions collapse to the standard Hessian $\nabla^2_θ\mathcal{L}(θ)\in\mathbb{R}^{p\times p}$. Archival correction. The original general novelty claims are no longer maintained: general DAG-Hessian calculus has prior precedents. This is a corrected archival version.
查看此版本解读 →查看此版本原文 ↗