- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies when activation-normalized and preactivation-normalized networks can approximate a prescribed width-N one-hidden-layer target. Upper-bounds and lower-bounds of the extra width needed for the approximation is presented in specific cases.
Critical strengths
-
The theoretical analysis presented in this paper systematically covers a wide range of function classes, with discussions on AN/PN, two types of normalization layers, and different conditions on the activation. In certain cases, the core result establishes a sharp bound on the extra-width needed for the approximation.
-
The math derivation is rigorous. The exact RMS rate for -augmented architectures and for replicated architectures are supported by clear algebraic expansions. The authors also rigorously prove a finite-width obstruction in Proposition 5.2 demonstrating why fixed-width lower bounds cannot exist for general activations without affine parameters.
Critical weaknesses
- Overall this paper presents sound results on the analyzed domain, and thus I recommend acceptance. As stated in the weakness section, the analysis is restricted to shallow architectures and relies heavily on a large augmentation magnitude . Consequently, the theoretical guarantees might hold primarily in an asymptotic regime, offering limited insight into realistic, finite-parameter deep networks trained in practice.
Questions for authors
- In Theorem 6.1, is the LN-AN rate unavoidable for one-sided activations at width , given that the mean-zero pair appears to give when is two-sided?
Response to Reviewer pBZ5
Dear Reviewer pBZ5,
Thank you for your careful review, thoughtful questions, and support for our work. We appreciate your positive assessment of our theoretical results and the rigor of our proofs. We address your questions about the LN-AN approximation rate, deeper networks, and the role of large parameters below.
Q1: The LN-AN approximation rate for one-sided activations
Thank you for this very good question. We conjecture that the O(M−1) LN-AN rate in Theorem 6.1 is unavoidable in the general one-sided case at width N+2. This intuition comes from LN's additional centering operation, which introduces interactions between different neurons. In the construction of Theorem 6.1, centering couples the auxiliary coordinate M with the original activations, so the variance generally contains a cross term linear in M. Relative to the leading M2 term, this produces an O(M−1) perturbation in the normalization factor and leads to the stated O(M−1) approximation bound.
In contrast, Eqn. (21) in the proof of Theorem 4.1 shows why RMSNorm achieves an O(M−2) rate. When computing the RMS, the auxiliary coordinate M does not interact with the other coordinates. It contributes M2 directly to the squared norm, while the remaining term stays bounded on the compact target feature set. There is therefore no cross term linear in M, giving an O(M−2) perturbation and the corresponding approximation rate.
This explains why the RMSNorm argument does not directly give an O(M−2) LN-AN rate. Achieving such a rate would require cancelling the first-order centering effect. For general one-sided activations at width N+2, we therefore expect the O(M−1) rate to persist.
W1: Extension to deep networks and implications of large parameters for training
Thank you for pointing out these two concerns.
Our theorem can be extended to deep MLPs with normalization inserted in each hidden layer. Since an affine map can be split into a composition of two affine maps, a two-hidden-layer MLP of the form "Linear-Activation-Linear-Activation-Linear" can be written as a composition of two one-hidden-layer MLPs, each of the form "Linear-Activation-Linear". Repeating this argument, an L-hidden-layer MLP can be written as a composition of L one-hidden-layer MLPs. We can therefore approximate the network layer by layer and control the accumulated error. Under the corresponding activation and Lipschitz assumptions, the total error can be made smaller than any prescribed ε>0 for fixed finite L. We have provided a rigorous mathematical formulation of this extension in Proposition 7.4 (Section 7.2). Appendix E.4 provides its proof and an L-hidden-layer ReLU example with widths Nℓ+1 for RMSNorm and Nℓ+2 for LN.
Our proposed construction does require a large M. We agree with the reviewer that large parameters may cause difficulties in practical training. Here, however, we study approximation capability and focus on the existence of the constructed parameters.
There are also previous approximation results that use large parameters in their constructions. Examples include the tanh approximation results of De Ryck et al. [1] and the work of Zhang et al. [2] on approximating ReLU networks using networks with other activation functions. These works also use growing parameter scales to construct increasingly accurate approximations.
Although our proof uses a large M, this allows us to construct accurate approximants with less additional width. For example, Theorem 4.1 achieves vanishing approximation error by increasing parameter magnitudes while keeping the width fixed at N+1. In contrast, the construction in Theorem 4.2 improves accuracy by increasing the replication count r, and hence the width N+r. Thus, the large parameter M allows the first construction to improve approximation accuracy without further increasing the width.
If any questions remain unresolved, please feel free to raise them. We would be glad to address them.
References
[1] Tim De Ryck, Samuel Lanthaler, and Siddhartha Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 143:732-750, 2021.
[2] Shijun Zhang, Jianfeng Lu, and Hongkai Zhao. Deep Network Approximation: Beyond ReLU to Diverse Activation Functions. Journal of Machine Learning Research, 25(35):1-39, 2024.