acceptodds
Withdrawn from ICLR 2027

On accuracy barriers and error-controlling parameters in neural networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many advances have improved our understanding of the depth, width, or number of parameters of a neural network (NN) that are needed to achieve a certain accuracy, and also our understanding of related topics such as overparameterization. Nevertheless, our theoretical understanding of these topics is incomplete, and perspectives with elementary principles can be valuable. A main aim here is to show, using simple concepts, that a relatively small number of parameters in the NN can be identified as the error-controlling parameters. Other parameters, which are more numerous, are identified as playing the role of defining the basis functions or fundamental features. Taken together, the error decays at multiple rates, where a rapid decay in error is governed by the error-controlling parameters, and subsequently a barrier or a slower decay in error can be seen. Theoretical results are proved for many common function classes (e.g., continuously differentiable functions, analytic functions, piecewise smooth functions), when approximated by a continuous piecewise linear function. Empirical results demonstrate these aspects in ReLU neural networks with standard training, including both synthetic examples and the MNIST dataset. Owing to a finite amount of training data, a second error barrier can arise, in addition to the representation error barrier mentioned above. We relate this data error barrier to a Monte Carlo numerical integration error, arising from the optimization problem for training.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
neural networks, approximation theory, ReLU networks, piecewise linear approximation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SsubOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to show that a relatively small number of parameters in the NN can be identified as the error-controlling parameter, while other parameters are identified as playing the role of defining the basis functions or fundamental features.

Critical strengths

This paper attempts to build a new understanding into an interesting problem.

Critical weaknesses

This paper is poorly organized and hard to grasp the core message. It does not even give a clear problem formulation in the beginning. Also, I wonder why Figure 3 and 4 look the same as they should be the results of different experiments. As suggested by captions, Figure 3 is the approximation error of Fourier basis functions, and Figure 4 is the approximation error of neural networks. If it is something beyond a simple error, I would strongly suspect the validity of those empirical results.

Moreover, this work does not provide any new insights into the problem it attempts to discuss. All of its theoretical results presented in Section 2 and Appendix are well-known for decades. For example, Theorem 1 proves the approximation bound for 1D periodic functions, which can be found in every graduate-level functional analysis textbook. The theory has almost nothing to do with neural networks, and the closest connection is probably the approximation guarantee for piecewise smooth functions, which is again, a standard result in functional analysis.

I would recommend the authors to revise their work thoroughly before submitting it to the next venue.

Questions for authors

I have no question.

Reviewer_c8AFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an elementary perspective on neural network approximation error by identifying a small subset of parameters as "error-controlling parameters." The authors study global approximations built from piecewise-linear interpolants of spectral basis functions (Fourier, Legendre, Chebyshev) on a grid with NN elements, retaining KK modes. For analytic targets they prove an error bound of the form e−cK1/d+N−2/de^{-cK^{1/d}} + N^{-2/d}, revealing two regimes: a rapid initial decay controlled by KK, followed by a resolution-induced accuracy barrier controlled by NN. Extensions cover CmC^m/Sobolev and piecewise-smooth functions. Experiments with trained ReLU networks on synthetic functions, MNIST, and the NASA Airfoil dataset exhibit the same two-regime behavior. A second, data-induced error barrier is attributed to Monte Carlo numerical integration error, yielding a three-term scaling law e−cK1/d+N−2/d+K/Je^{-cK^{1/d}} + N^{-2/d} + \sqrt{K/J}

Critical strengths

Clear intuition and simple, accessible examples. The paper builds its message from elementary concepts (spectral truncation plus interpolation error) and illustrates each claim with simple 1D examples (e.g., f(x)=1/(2−cos⁡x)f(x) = 1/(2-\cos x)) before extending to higher dimensions and real datasets. The figures directly visualize the two-regime behavior, making the error-controlling-parameter viewpoint easy for readers to understand.

  • A unified and self-consistent story connecting representation, parameters, and data. The three-term decomposition in equation 8 cleanly attributes separate error contributions to the retained modes KK, the resolution NN, and the sample size JJ, and the MCNI interpretation of training (least squares with the Gram matrix replaced by its expectation) is a transparent and pedagogically appealing way to connect the finite-data barrier to a classical numerical-analysis quantity. The empirical results on trained networks are consistent with the theory across several settings.
Critical weaknesses
  • Limited novelty. The approximation results based on the Fourier basis appear to be extensions of classical results. The proof of Theorem 1 is a standard triangle-inequality combination of spectral truncation error (exponential coefficient decay for analytic functions) and classical piecewise-polynomial interpolation estimates.
  • The significance of the theoretical results remains implicit. The theory concerns a hand-constructed approximation family WK,NW_{K,N} (piecewise-linear interpolants of spectral modes), not trained neural networks. The link to actual NNs is only by analogy: the paper does not show that a trained ReLU network's parameters decompose into "error-controlling" and "basis-defining" subsets, nor that training dynamics realize anything like the proposed construction. Consequently, what the theory implies for practice (e.g., for pruning, architecture design, or scaling-law prediction) is never made concrete.
  • Curse of dimensionality and no rate improvement. The bound e−cK1/d+N−2/de^{-cK^{1/d}} + N^{-2/d} suffers the standard curse of dimensionality, and the rate matches what is already known for ReLU networks in approximation theory [1]. Given that the rates are not improved and the dimension dependence is standard, what is the new insight here? The paper would be considerably stronger if it identified a consequence of the two-regime viewpoint that cannot be read off from existing single-rate bounds, or demonstrated a setting where this perspective yields a prediction that classical theory misses.

[1] Shen Z, Yang H, Zhang S. Optimal approximation rate of ReLU networks in terms of width and depth[J]. Journal de Mathématiques Pures et Appliquées, 2022, 157: 101-135.

Questions for authors

See Weakness Section

Reviewer_Dm3bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper uses piecewise linear basis functions to approximate neural network outputs. Within this approximation framework, the authors decompose the error into three components: an insufficient number of features, finite representation resolution, and finite training data, and derive theoretical error bounds. They then fit the approximation model to the outputs of trained neural networks. In the experiments, the approximation error initially decreases as more basis functions are retained and then reaches a plateau. Based on these results, the authors propose a framework for interpreting accuracy barriers and the roles of parameters in neural networks.

Critical strengths
  1. The problem studied in this paper is very interesting. The idea of decomposing the error into different components is insightful.
  2. To the best of my understanding, the proofs of the error upper bounds within the approximation framework are solid.
Critical weaknesses
  1. The main theoretical proofs and experimental analyses are carried out within the approximation framework. This includes both the error upper bounds and the phenomena observed when fitting trained neural networks. Although these results are insightful, there seems to be no direct evidence that the error decomposition and observed phenomena extend to actual neural networks. This raises concerns about the theoretical value and practical relevance of the work.
Questions for authors
  1. Could the authors further explain how the theoretical analysis and experiments in this paper relate to actual neural networks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.