关于 ReLU 网络集合的(非)闭性
On the (non-)closedness of the set of ReLU networks
Eloi Martinet, Leon Bungert
AI 速读 基于摘要
作者证明单隐层 ReLU 网络集合在 Sobolev 空间中是弱闭的,而宽度有界的双隐层 ReLU 网络集合在 Sobolev 空间与连续函数空间中均不闭。
cs.LG · math.CA · math.FA · math.NA · math.OC
查看原文摘要
We prove that the set $\mathcal{N}_N$ of ReLU networks with one hidden layer and at most $N\in\mathbb N$ neurons is weakly closed in $W^{1,p}(Ω)$, for a bounded connected open set $Ω\subset\mathbb R^d$. Consequently, every coercive, weakly lower semicontinuous energy on this space attains its minimum over $\mathcal{N}_N$. This covers the Deep Ritz method with convex integrands, Sobolev training and Lipschitz learning. Moreover, as the weight decay parameter tends to zero, minimizers of the regularized problem converge, up to subsequences, to minimizers with minimal parameter norm. We also prove that the set of ReLU networks with two hidden layers of bounded widths is not closed in $W^{1,p}(Ω)$ or in $C(\overlineΩ)$. This disproves a conjecture of Petersen, Raslan and Voigtlaender.