Layer-wise Geometric Approximation Rates for Deep Networks
Shijun Zhang, Zuowei Shen, Yuesheng Xu
原文摘要
Depth is widely viewed as a central contributor to the success of deep neural networks, yet standard approximation results typically give error bounds only for the final output of a network. Inspired by classical multiscale expansions and multigrade deep learning, we give depth a scale-dependent interpretation by constructing a single network whose intermediate readouts approximate the target at successively finer scales. For any $f\in L^p([0,1]^d)$ with $1\le p <\infty$ and any integer $N\ge2$, we construct a single network of width $3dN+2d+4$ with the same activation pattern in every hidden layer, using sine in the first two neurons and $\mathtt{ReLU}$ in the rest. For every $\ell\ge0$, the $L^p$ approximation error of the readout $Φ_\ell$ at layer $\ell+4$ is bounded by $(2d+1)ω_{f,p}(N^{-\ell})$, where $ω_{f,p}$ denotes the $L^p$ translation modulus of $f$. Under the Hölder condition $ω_{f,p}(t)\le Ct^α$ with $C>0$ and $0< α\le 1$, this estimate yields the geometric bound $(2d+1)C N^{-α\ell}$. More generally, if $ω_{f,p}(t)$ has a known upper bound that tends to zero as $t\downarrow0$, we obtain a stopping rule for adaptive depth selection. We can stop at the first depth where the error bound meets the desired tolerance and later append layers to meet a tighter tolerance while keeping all earlier maps and readouts fixed.
cs.LG · math.NA · stat.ML