Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures
Maxim Bolshim, Alexander Kugaevskikh
原文摘要
Automatic differentiation computes loss Hessians, while interpreting inter-layer interactions requires an explicit choice of graph-indexed blocks. This paper presents an analytical formalism that decomposes the full Hessian into blocks indexed by the DAG of an architecture. The standard decomposition $H = H^{GN} + H^T$ separates the Gauss-Newton component (positive semidefinite for a convex output loss) from the tensor component (residual curvature). If every node map is locally affine in its parent activations, the tensor component of the input Hessian vanishes on each smooth cell ($H^{T}_{v,w}=0$, $H^f_{v,w}=H^{GN}_{v,w}$); ReLU activations alone do not suffice. The full parametric Hessian contains residual terms that do not reduce to the GGN. Building on this decomposition, we use diagnostic metrics (inter-layer resonance $\mathcal{R}$, geometric coupling $\mathcal{C}$, stable rank $\mathcal{D}$, GN-Gap) with stochastic HVP-based estimates whose cost depends on graph evaluation and the number of probes. The theoretical analysis gives conditional bounds on curvature decay and residual-chain transport; empirical examples span fully connected MLPs (Exp. 1-5) and convolutional architectures (ResNet-18, ${\sim}11$M parameters, Exp. 6). When the architecture reduces to a single node, all definitions collapse to the standard Hessian $\nabla^2_θ\mathcal{L}(θ)\in\mathbb{R}^{p\times p}$. Archival correction. The original general novelty claims are no longer maintained: general DAG-Hessian calculus has prior precedents. This is a corrected archival version.
cs.LG