acceptodds
Under review as a conference paper at ICLR 2027

RELU NEURAL NETWORK APPROXIMATION TO SMOOTH FUNCTIONAL OPERATOR: DIMENSIONAL DECAY AND ERROR ANALYSIS

Abstract

We study the uniform approximation of smooth scalar-valued functionals on an infinite-dimensional separable Hilbert space by ReLU neural networks. A key feature in deep learning for functional data is the varying importance of different coordinates/dimensions. Representing the functional input in a basis expansion, we quantify the importance of each coordinate through both the magnitude of its corresponding basis scores and the directional sensitivity of the target functional. Our analysis combines coordinate truncation, anisotropic partitioning, local Taylor approximation, and ReLU network realization, while allowing unrestricted interactions among the retained coordinates. We establish a general nonasymptotic upper bound for the uniform approximation error and a complementary pseudo-dimension-based lower bound for the worst-case approximation error. Under generalized exponential coordinate decay , with , the upper and lower bounds match at the leading order, which is stretched-exponential in the logarithm of the network size budget, and thus yield the nearly optimal approximation rate. This work characterizes neural network approximation error for general smooth infinite-dimensional functional operator explicitly through the joint dimensional decay of coordinate magnitudes and directional sensitivities.

Primary area:
learning theory
Keywords:
Error Analysis, Functional Operator, ReLU Neural Network, Representation Learning.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MsifOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides nearly-optimal approximation bounds for ReLU networks approximating a smooth operator acting on L2[0,1]L_2[0, 1]. The network takes the scores from a truncated basis expansion and uses an anisotropic variant of the established Taylor-network approach to compute bounds. Operators with generalized exponential decay in coordinate importance are given special attention.

Critical strengths
  • The paper addresses a relevant and challenging topic in neural network approximation theory. The formulation in terms of basis coefficients is natural and powerful.
  • Extension of the standard proof technique to the new setting is nontrivial, especially the careful balancing of different sensitivities.
Critical weaknesses

None

Questions for authors
  1. I don't necessarily expect experiments in a theory paper, but as you already have some: can we confirm the predictions of your theorem more quantitatively in terms of convergence/approximation rates? In the Euclidean case it is fairly well known that practically trained networks rarely find configurations that can exploit higher-order smoothness, so I wouldn't be surprised if that would be the case here.
  2. I don't see where ξh,d0\xi_{h, d_0} is defined. I assume it's ⟨X−X0,νd⟩\langle X - X_0, \nu_d \rangle?
  3. The literature review of ReLU approximation theory effectively stops in 2021. A few more of the recent improvements over Lu2021 could be discussed.
Reviewer_GAqxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript studies the approximation of smooth functionals on an infinite-dimensional Hilbert space using ReLU neural networks. Each input function is represented using the expansion coefficients of an orthonormal basis, and the network takes only its first D coefficients. The authors introduce an effective dimension D0 based on coordinate sensitivity weights and coefficient decay, then partition the important coordinates using an anisotropic grid. They employ local Taylor expansions and ReLU subnetworks to approximate the Taylor coefficients and polynomial terms. The manuscript derives an approximation upper bound, a pseudo-dimension lower bound, and convergence rates under generalized exponential coordinate decay. Numerical experiments on synthetic Fourier-based functional data examine how truncation dimension, network width, and coordinate decay affect approximation errors.

Critical strengths

The idea of using a network-budget-dependent effective dimension D0 is very effective to describe how many input coordinates can be resolved given their relative importance. This approach provides an intuitive and interpretable connection between coordinate decay, network capacity and approximation difficulty.

Critical weaknesses
  1. The manuscript offers limited novelty relative to the existing literature on functional approximation using ReLU Neural Networks. Liu et al. [1] have shown how the decay of coordinate-dependent Fourier weights affects the tractability exponent and the resulting approximation rates of Fourier Functional Networks. Li et al. [2] have studies function spaces with fast coefficient decay and established approximation bounds involving the best s-term approximation error, with the decoder’s convergence rate depending on the sparsity level s rather than the dictionary size N. These results have already demonstrated that exploiting coordinate weights, coefficient decay, and sparse representations can improve neural-network approximation of nonlinear functionals. The authors should therefore compare their effective dimension D0D_0, based on wdsdw_d s_d against these existing results and explain what additional approximation advantages their method would provide, if any.

[1] Liu, Peilin, Yuqing Liu, Xiang Zhou, and Ding-Xuan Zhou. ``Approximation of functionals on Korobov spaces with Fourier Functional Networks.'' Neural Networks 182 (2025): 106922. Neural Networks, 182:106922.

[2] Li, Jianfei, Shuo Huang, Han Feng, Ding-Xuan Zhou, and Gitta Kutyniok. ``Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension.'' arXiv preprint arXiv:2604.06774 (2026).

2 There is a potentially critical gap in the proof of Theorem 1. On Page 17, the proof chooses L through the balancing condition (A.11) and, after setting ℓd=wd\ell_d = w_d at line 870, obtains H(D0)=D0/LH(D_0) = D_0/L. It then derives the approximation bound using this relation. However, the proof does not establish that this balancing condition is compatible with the definition of D0D_0 in Theorem 1.

  1. Minor issues:

Page 7: Theorem 1: For A=1\mathcal{A}=1, the defining set for D0D_0 in Theorem 1 is empty when wdsdw_ds_d is nonincreasing.

Page 14: Line 707: Here D0D_0 is different from D0D_0 defined in Theorem 1. Their equivalence is not shown.

Page 14: The derivative bound used in (A.3) requires the segment Xθ+τ(PDX−Xθ)X_\theta+\tau(P_DX-X_\theta) with τ∈[0,1]\tau\in[0,1] to lie in the domain where Assumption 2 holds; this is not established.

Page 14: In (A.4), it should be ∑d=D0+1Dwdsd≤H(D0)\sum_{d=D_0+1}^{D}w_ds_d\le H(D_0).

Page 15: In Step 1, it requires Wψ,d2Mψ2=LdW_{\psi,d}^{2}M_\psi^{2}=L_d, although a general integer LdL_d need not be a perfect square.

Page 15: In Step 2, ∏dLd=LD0∏dℓdsd\prod_d L_d=L^{D_0}\prod_d\ell_ds_d ignores the fact that Ld=⌊Lℓdsd⌋L_d=\lfloor L\ell_ds_d\rfloor.

Page 15: In (A.5)--(A.6), Lemma 2 is applied to approximate the coefficient maps Qd(s)\mathcal{Q}_d^{(s)} and Q(0)\mathcal{Q}^{(0)}. However, Lemma 2 requires target values in [0,1][0,1], whereas these maps may take negative values or values greater than one. The required normalization is not explained.

Page 15: In (A.7), Lemma 4 is stated on [0,1]n[0,1]^n, while the Taylor increments for d>D0d>D_0 can be negative.

Page 16: The derivation of (A.9) assumes ∑d≥1wdsd<∞\sum_{d\ge1}w_ds_d<\infty, but this condition is neither stated as an assumption nor established anywhere earlier in the paper.

Questions for authors

How can one resolve the gap in the proof of Theorem 1? If an additional decay assumption, such as H(D0)≲D0/LH(D_0)\lesssim D_0/L, is required, would this substantially restrict the class of admissible functionals and undermine the claimed generality of the approximation result?

How does the proposed effective dimension D0D_0 differ from the complexity measures in [1,2]? What improvement in approximation rates does it provide over these methods?

Reviewer_Tn6JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how well ReLU networks can approximate smooth scalar-valued functionals of infinite-dimensional inputs, relating approximation error to the decay of coefficient amplitudes and functional sensitivities.

Critical strengths

The distinction between input truncation and network approximation is useful: retaining more coefficients reduces information loss but gives the network a more complicated function to represent. Considering both coefficient amplitudes and functional sensitivities is sensible. The combination of upper and lower bounds is also valuable, and the experimental setup and model-selection procedures are clearly described.

Critical weaknesses
  1. I am concerned that Theorem 1 does not hold as stated. Take m=1m=1, sd=wd=1/[dlog⁡(d+1)]s_d=w_d=1/[\sqrt d\log(d+1)], and f0(X)=∑dwdξdf_0(X)=\sum_d w_d\xi_d on the full score domain. Choosing D=D0D=D_0 satisfies the theorem’s truncation condition, but any predictor using those coordinates has uniform error at least H(D0)≍1/log⁡D0H(D_0)\asymp1/\log D_0. The claimed bound is instead O(1/(log⁡D0)2)O(1/(\log D_0)^2). The proof appears to impose H(D0)=D0/LH(D_0)=D_0/L without establishing consistency with the definition of D0D_0. This concerns the general theorem; it does not itself disprove the exponential-decay result.

  2. The construction introduces logarithmic width and depth factors through Lemma 2, but these disappear from the stated complexity bounds. Their treatment needs clarification because the upper–lower comparison depends on network complexity.

Questions for authors
  1. What additional condition or revised bound resolves the counterexample, and do the exponential-decay and near-optimality conclusions survive after accounting for the logarithmic factors?

  2. How would sensitivity weights, including the higher-derivative bounds, be obtained for an unknown target? Assumption 2 also requires coefficient and sensitivity weights to decrease in the same ordering. When should this hold, and how could it be checked? Principal components alone do not establish this.

  3. Could the experiments report truncation error separately from fitting error? Since the reported maximum covers only sampled inputs, please also clarify what the experiments establish about the theoretical rate beyond the qualitative trends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.