arXiv:2610.21820 · v2

A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

首次提交 2026-10-15 21:06

版本更新 2026-11-11 22:48

雷达观测 2026-11-13 11:11

摘要

How the depth and width of neural networks shape learned representations is difficult to predict from the data and architecture. We use neural collapse (NC) as a probe of the representation geometry and express its first three properties, within-class variability collapse (NC1), simplex class means (NC2) and classifier alignment (NC3), through the learned feature kernel, extending a known identity for NC1 to NC2 and NC3. For deep linear networks with an exchangeable input kernel, the kernel equations yield a geometric law, exact within this theory, for the layerwise decrease of NC1. At large width $N$ the accumulated decrease becomes a function of the depth-to-width ratio $L/N$, with its rate determined self-consistently by the data and architecture, yielding a scaling law for representation geometry.

作者备注：40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: AXIOM Foundations of Efficient Deep Learning