On accuracy barriers and error-controlling parameters in neural networks

Many advances have improved our understanding of the depth, width, or number of parameters of a neural network (NN) that are needed to achieve a certain accuracy, and also our understanding of related topics such as overparameterization. Nevertheless, our theoretical understanding of these topics is incomplete, and perspectives with elementary principles can be valuable. A main aim here is to show, using simple concepts, that a relatively small number of parameters in the NN can be identified as the error-controlling parameters. Other parameters, which are more numerous, are identified as playing the role of defining the basis functions or fundamental features. Taken together, the error decays at multiple rates, where a rapid decay in error is governed by the error-controlling parameters, and subsequently a barrier or a slower decay in error can be seen. Theoretical results are proved for many common function classes (e.g., continuously differentiable functions, analytic functions, piecewise smooth functions), when approximated by a continuous piecewise linear function. Empirical results demonstrate these aspects in ReLU neural networks with standard training, including both synthetic examples and the MNIST dataset. Owing to a finite amount of training data, a second error barrier can arise, in addition to the representation error barrier mentioned above. We relate this data error barrier to a Monte Carlo numerical integration error, arising from the optimization problem for training.