The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning
James Hazelden, Laura Driscoll, Eli Shlizerman, Eric Shea-Brown
In training a neural network with gradient descent (GD), each iteration induces a linear operator that governs the first-order updates to a model's internal state variables. We call this operator the Global Empirical Neural Tangent Kernel (GeNTK). In finite-width networks, GeNTK is typically intractable to form, leading prior work to focus on restrictive settings, such as tracking only outputs or infinite-width limits. Here, we bridge this gap by studying the structure of GeNTK across many models. Formulating the model state as the solution to a global implicit constraint, we derive GeNTK as a product of two operators: K, accounting for immediate parameter-to-state interactions, and P, describing internal state-to-state dependencies. For a broad class of "weight-based models", such as RNNs and transformers, we prove a universal Kronecker-core theorem showing that K admits an exact, forward-pass-computable form given by the Gram matrix of weight-site variables. This core gives a measurable state-dependent restriction on Euclidean GD, termed "self-referential bias": current activity filters which adjoint directions can drive a first-order update. For recurrent models (GRUs and RNNs), we examine the spectrum of GeNTK and formalize notions of low-rank, biased learning in terms of reduced spatial or temporal ranks of the full operator. Practically, this explains why some RNN initializations yield better learning, while some plateau, by biasing the GeNTK and thus restricting learning to a small domain of learnable tasks. To demonstrate computational viability, we validate the main theorems in a self-attention transformer and a pretrained Pythia transformer language model with 410 million parameters. Overall, our results expose tractable local rank and mode-selective biases in the finite-width GD geometry and open the door to principled study of the GeNTK, a bridge between theory and practice.
尚无 AI 解读，展示原文摘要