通过谱神经网络与非线性格拉姆矩阵感知的隐式正则化
Implicit Regularization via Spectral Neural Networks and Non-linear Matrix Sensing
Hong T. M. Chu, Xiaoyu Dong, Subhro Ghosh, Chi Thanh Lam, Soumendu Sundar Mukherjee
AI 速览：作者提出谱神经网络（SNN），在矩阵感知中通过对输入矩阵奇异值施加非线性激活，从理论上证明梯度流训练具有指数快速损失衰减并隐式选择近似核范数最小解，并用实验展示其有效性。
The phenomenon of implicit bias has attracted interest in recent years as a fundamental aspect of the remarkable generalizing ability of neural networks. In a nutshell, it entails that gradient descent dynamics in many neural nets, even without any explicit regularizer in the loss function, converges to the solution of a regularized learning problem. In this work, we show that this phenomenon extends to non-linear spectral parameterizations in matrix sensing. More precisely, we propose a matrix network architecture called Spectral Neural Networks (abbrv. SNN) where each neuron applies nonlinear activation to the singular values of the input matrix. Conceptually, this entails coordinatizing the space of matrices by their singular values and singular vectors, as opposed to by their entries, a potentially fruitful perspective for matrix learning. We prove the exponentially fast decay of squared loss during the training dynamics of an SNN parameterized matrix sensing model under gradient flow (which is a continuous approximation of the discrete gradient descent), and also that the training implicitly selects an approximate solution that minimizes nuclear norm. We also demonstrate the effectiveness of the SNN architecture in the context of matrix sensing by empirical investigations. We believe that the SNN architecture has the potential to be of wide applicability in a broad class of matrix learning scenarios.