Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks
Ying Chen, Aoxi Li, Jihun Kim, Javad Lavaei
原文摘要
Structured low-rank and sparse-plus-low-rank parameterizations, motivated by methods such as LoRA and low-rank factorization, impose strong structural constraints on the linear transformations within neural networks. We study the expressive consequences of these constrained structures from the perspective of universal approximation. We first characterize that purely low-rank neural networks, despite being able to exactly interpolate arbitrary finite scalar datasets, can fail to approximate continuous functions that vary along directions ignored by the low-rank hidden mapping. We show that augmenting the low-rank structure with a one-parameter diagonal component suffices to overcome this geometric bottleneck, yielding a Diagonal-plus-Low-Rank (DLoR) structure. We show that a full-rank linear transformation can be reconstructed using DLoR components either by expanding network width through additive decomposition or by extending network depth through multiplicative decomposition. We term this correspondence between additive width expansion and multiplicative depth extension the Structural Correspondence framework. By combining these constructions with local linearization of the activation function, we prove that DLoR neural networks restore universal approximation for general activation functions. These results characterize how architectural width and depth can compensate for severe rank constraints without requiring unrestricted dense hidden transformations. Diagonal terms are required for multiplicative construction, while additive expansion works with purely low-rank terms. We further determine its exact cost: both the minimal number of rank-$r$ DLoR factors representing $W$ and the minimal network depth for that layer equal $\lceilκ(W)/r\rceil$, where $κ(W)=\min_{γ\ne0}rank(W-γI)$, achieving a lower cost than its dense representation when $W$ is identity-centered.
cs.LG