Langevin Monte-Carlo 可证明地学习任意规模与数据下的两层神经网络
Langevin Monte-Carlo Provably Learns Depth Two Neural Nets at Any Size and Data
Dibyakanti Kumar, Samyak Jha, Anirbit Mukherjee
AI 速读 基于摘要
作者证明 Langevin Monte Carlo 能在网络规模不限、训练数据任选的情况下最小化适当正则化两层网络的总体风险,并给出非渐近收敛速率。
cs.LG · math.FA · math.PR
查看原文摘要
In this note we prove that the Langevin Monte Carlo (LMC) algorithm can minimize the population risk of appropriately regularized depth-2 nets with no restrictions on the size of the net and for any choice of training data --- and we give non-asymptotic convergence rates for it. We achieve this via showing that in q-Renyi divergence, the iterates of LMC converge to the Gibbs distribution of certain Frobenius norm regularized losses for these nets --- in both classification and regression settings. The key technicality of the work is two-fold (a) to keep the amount of regularization needed be independent of the size of the net and (b) to obtain convergence rates that scale with just the Poincare constant of the corresponding Gibbs measure. This work synthesizes several recent observations about isoperimetry conditions under which LMC converges and that two-layer neural loss functions can always be regularized by a constant amount such that they satisfy the Villani conditions, and thus their Gibbs measures satisfy a Poincare inequality.