The paper investigates the minimum number of hidden neurons required for arbitrary-accuracy approximation of multivariate Hölder-continuous functions. The authors construct a fixed activation function that achieves arbitrary uniform approximation using only d+1 hidden neurons and establish a matching lower bound for feedforward networks with locally integrable activations. They further provide constructions using an elementary floor–exponential activation and analyze the associated bit complexity, obtaining an upper bound that matches the metric-entropy lower bound up to a logarithmic factor.
The paper addresses an interesting theoretical problem and presents potentially significant mathematical results. In particular, the exact characterization of the minimum neuron count is a valuable contribution. However, I have several concerns.
- Incomplete discussion of closely related literature.
The authors should discuss the following two papers and include their relevant results in Table 1:
-
Maiti, Michelle, and Yang, "Optimal Neural Network Approximation for High-Dimensional Continuous Functions" (arXiv:2409.02363). This work establishes arbitrary-accuracy approximation using O(d) unique nonzero parameters and proves the optimality of linear scaling with the input dimension. Its construction uses an elementary universal activation function.
-
Li, Shen, Yang, and Zhang, "Optimal Tradeoffs Between Network Size and Parameter Magnitude in Neural Approximation and Minimax Regression" (arXiv:2609.25710). This work establishes sharp tradeoffs between network size and parameter magnitude, including fixed-size constructions with O(d) nonzero parameters and optimal-order parameter magnitude. It also establishes matching lower bounds through the information capacity of neural network classes.
These works are directly related to the manuscript's two central questions: optimal network size and the information required for arbitrary-accuracy approximation. In particular, the optimal O(d) dependence on input dimension has already been established, while information-theoretic limitations on network representations have also been investigated.
The authors should therefore clarify that their principal improvement in network size is the exact minimum of d+1 hidden neurons, rather than establishing the optimal order of dimensional scaling for the first time. The results should be compared carefully under their respective assumptions, particularly regarding approximation norms, activation regularity, neuron counts, nonzero parameters, parameter magnitude, and bit complexity.
- The comparison with Yarotsky (2021) needs clarification.
Yarotsky (2021) established superexpressiveness for several elementary activation families. In particular, Corollary 1 provides an O(d²) upper bound on the number of neurons and connections, while the proof of Theorem 3 also describes an O(d)-neuron construction for its first activation family.
However, Table 1 of the submitted manuscript reports an exact d+2 neuron count with three hidden layers and a skip connection for Yarotsky (2021). It is unclear how this exact count and architecture are derived from the original paper. The authors should provide a precise derivation or reference supporting this entry.
More generally, Table 1 should distinguish explicitly stated results from architectural bounds derived by interpreting previous constructions. This is particularly important because the claimed improvement in the exact neuron count is central to the manuscript.
- Interpretation of the bit-complexity contribution.
The authors establish a bit-complexity upper bound of O(ε^{-d/α} log(1/ε)) and compare it with the classical metric-entropy lower bound Ω(ε^{-d/α}). This provides a useful characterization of the storage requirements of their constructions.
However, information-theoretic lower bounds and optimal tradeoffs between network size and parameter magnitude have also been investigated in recent work, particularly Li, Shen, Yang, and Zhang (2026). Although parameter magnitude and binary encoding length are distinct complexity measures, they are closely related in understanding how fixed-size networks encode increasingly accurate approximations.
A more detailed comparison would help establish what is genuinely new in the present bit-complexity analysis. The authors should also discuss whether the remaining logarithmic gap is inherent to the proposed network architectures or a consequence of their particular encoding scheme.
- Limited computational practicality.
The proposed constructions rely on discontinuous or nonstandard activation functions and extremely large integer-valued parameters. Although the number of neurons is minimal, the required parameter precision increases rapidly as the approximation error decreases.
Moreover, the computational complexity of evaluating the specialized activation and performing the associated arbitrary-precision arithmetic is not fully characterized. Consequently, minimizing neuron count does not necessarily lead to computationally efficient approximation. The authors should better discuss the tradeoff between network size, parameter storage, and evaluation complexity.
- Limited numerical investigation.
The numerical examples primarily verify the theoretical constructions using exact arithmetic. While these experiments are appropriate for confirming correctness, they do not provide much additional insight into approximation efficiency.
It would be useful to report the computational costs associated with increasing approximation accuracy, particularly the time required for constructing the encoded parameters and evaluating the resulting networks.
Overall, the manuscript establishes an interesting exact neuron-count result and provides explicit approximation constructions. These are meaningful theoretical contributions. However, the comparison with prior work should be substantially strengthened, especially regarding previously established optimal O(d) scaling and information-theoretic complexity bounds. The precise novelty of the bit-complexity contribution and the computational implications of the proposed constructions also require clarification.
We thank the reviewer for the insightful suggestions. The sections, theorems, tables and figures mentioned below refer to the revised manuscript, in which all revisions are marked in blue.
Continuous functions and Hölder balls
Following your suggestion, all results on the neuron count are now stated for continuous functions. Universality is defined as density in C([0,1]d) (Definition 2.1). Theorems 3.1 and 4.1 now hold for every f∈C([0,1]d), with the grid size M chosen through the modulus of continuity of f in (5), and Theorem 3.4 shows that, for d≥2, no network with at most d hidden neurons is universal. The lower bound is not affected, because the counterexamples that we use to prove that networks with fewer than d+1 neurons are not universal lie in a Hölder ball and are therefore continuous. Your suggestion extends the scope of our results and makes the paper clearer. Thank you very much.
We keep the Hölder balls only for the bit complexity. A worst-case bit count needs a compact class: on the unit ball of C([0,1]d), no fixed architecture reaches accuracy ε<1/2 with a bounded number of bits. We choose the Hölder balls because they are compact and their ε-entropy is known [2], so that our bit complexity can be compared with the optimal one; in addition, our optimality in bit complexity is restricted to the Hölder balls. In detail:
We have added Remark 5.3 on this limitation of our optimality claim, and we list the case s>1 in Section 7 as an open problem.
Extension to vector-valued functions
Thank you for the question on output dimensions greater than 1, which inspired a new part of the paper (Remark 4.2 and Appendix F). We now prove that, for input dimension d≥2 and output dimension dout, the minimal number of hidden neurons among networks without skip connections is d+dout (Theorem F.4).
All four networks of the main text extend to dout outputs with the same activations (Propositions F.1 and F.2):
We also tested the four networks in exact arithmetic for d=dout=2 with f7(x)=(21sin(πx1)cos(πx2),21cos(πx1)sin(πx2)) and ε=2−1,…,2−6; all errors, measured in the Euclidean norm, are below ε (Table 2). The details are in Appendix F, and we have also updated the code in the supplementary material to include this example.
The practical aspects raised in the review, namely the discontinuous activations and training, are now discussed in the Limitations (Section 7).
We hope that these answers and revisions address your concerns, and we thank you again for the suggestions to improve the quality of our manuscript.
References
[1] G. G. Lorentz. Metric entropy and approximation. Bulletin of the American Mathematical Society, 1966.
[2] A. N. Kolmogorov and V. M. Tikhomirov. ε-entropy and ε-capacity of sets in function spaces. Uspekhi Matematicheskikh Nauk, 1959.