A new network with super-approximation power is introduced. This network is built with Floor () or ReLU () activation function in each neuron; hence, we call such networks Floor-ReLU networks. For any hyperparameters and , we show that Floor-ReLU networks with width and depth can uniformly approximate a Hölder function on with an approximation error , where and are the Hölder order and constant, respectively. More generally for an arbitrary continuous function on with a modulus of continuity , the constructive approximation rate is . As a consequence, this new class of networks overcomes the curse of dimensionality in approximation power when the variation of as is moderate (e.g., for Hölder continuous functions), since the major term to be considered in our approximation rate is essentially times a function of and independent of within the modulus of continuity.
Deep Network With Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
Zuowei Shen, Haizhao Yang, Shijun Zhang; Deep Network With Approximation Error Being Reciprocal of Width to Power of Square Root of Depth. Neural Comput 2021; 33 (4): 1005–1036. doi: https://doi.org/10.1162/neco_a_01364
Download citation file: