Deep Network With Approximation Error Being Reciprocal of Width to Power of Square Root of Depth

A new network with super-approximation power is introduced. This network is built with Floor (⁠$⌊x⌋$⁠) or ReLU (⁠$max{0,x}$⁠) activation function in each neuron; hence, we call such networks Floor-ReLU networks. For any hyperparameters $N∈N+$ and $L∈N+$⁠, we show that Floor-ReLU networks with width $max{d,5N+13}$ and depth $64dL+3$ can uniformly approximate a Hölder function $f$ on $[0,1]d$ with an approximation error $3λdα/2N-αL$⁠, where $α∈(0,1]$ and $λ$ are the Hölder order and constant, respectively. More generally for an arbitrary continuous function $f$ on $[0,1]d$ with a modulus of continuity $ωf(·)$⁠, the constructive approximation rate is $ωf(dN-L)+2ωf(d)N-L$⁠. As a consequence, this new class of networks overcomes the curse of dimensionality in approximation power when the variation of $ωf(r)$ as $r→0$ is moderate (e.g., $ωf(r)≲rα$ for Hölder continuous functions), since the major term to be considered in our approximation rate is essentially $d$ times a function of $N$ and $L$ independent of $d$ within the modulus of continuity.