acceptodds
Under review as a conference paper at ICLR 2027

Activation-Adjacent Layer Normalization: Effects on Approximation Power

Abstract

Layer normalization (LN) and root mean square normalization (RMSNorm) remain input-dependent and nonlinear at inference, so inserting either next to an activation need not preserve represented functions. We study when activation-normalized (AN) and preactivation-normalized (PN) shallow networks can approximate the function represented by a prescribed width- one-hidden-layer MLP. Viewing RMSNorm as input-dependent scaling, we use width augmentation to locally linearize normalization on compact feature sets. For AN networks, an unbounded activation range allows width , while every continuous nonconstant activation admits width with error . For PN networks, affine parameters or, without them, degree-one positive homogeneity recover width ; general continuous activations admit width with error tending to zero. The corresponding LN constructions use centered coordinate pairs, yielding widths and , respectively. We prove that for RMSNorm and for LN are also the worst-case bound for approximations, and show that applicable universal-approximation theorems for ordinary MLPs transfer to these normalized architectures. Numerical experiments recover the predicted approximation rates for both normalizations across all five settings and illustrate the shifted-ReLU finite-width obstruction.

Primary area:
learning theory
Keywords:
Deep Learning, layer normalization, RMSNorm, approximation theory, neural network expressivity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pBZ5OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies when activation-normalized and preactivation-normalized networks can approximate a prescribed width-N one-hidden-layer target. Upper-bounds and lower-bounds of the extra width needed for the approximation is presented in specific cases.

Critical strengths
  1. The theoretical analysis presented in this paper systematically covers a wide range of function classes, with discussions on AN/PN, two types of normalization layers, and different conditions on the activation. In certain cases, the core result establishes a sharp bound on the extra-width needed for the approximation.

  2. The math derivation is rigorous. The exact O(M−2)O(M^{-2}) RMS rate for MM-augmented architectures and O(r−1)O(r^{-1}) for replicated architectures are supported by clear algebraic expansions. The authors also rigorously prove a finite-width obstruction in Proposition 5.2 demonstrating why fixed-width lower bounds cannot exist for general activations without affine parameters.

Critical weaknesses
  1. Overall this paper presents sound results on the analyzed domain, and thus I recommend acceptance. As stated in the weakness section, the analysis is restricted to shallow architectures and relies heavily on a large augmentation magnitude MM. Consequently, the theoretical guarantees might hold primarily in an asymptotic regime, offering limited insight into realistic, finite-parameter deep networks trained in practice.
Questions for authors
  1. In Theorem 6.1, is the O(1/M)O(1/M) LN-AN rate unavoidable for one-sided activations at width N+2N+2, given that the mean-zero pair appears to give O(M−2)O(M^{-2}) when ϕ\phi is two-sided?
1 comment
AuthorsOpenReview ↗

Response to Reviewer pBZ5

Dear Reviewer pBZ5,

Thank you for your careful review, thoughtful questions, and support for our work. We appreciate your positive assessment of our theoretical results and the rigor of our proofs. We address your questions about the LN-AN approximation rate, deeper networks, and the role of large parameters below.

Q1: The LN-AN approximation rate for one-sided activations

Thank you for this very good question. We conjecture that the O(M−1)O(M^{-1}) LN-AN rate in Theorem 6.1 is unavoidable in the general one-sided case at width N+2N+2. This intuition comes from LN's additional centering operation, which introduces interactions between different neurons. In the construction of Theorem 6.1, centering couples the auxiliary coordinate MM with the original activations, so the variance generally contains a cross term linear in MM. Relative to the leading M2M^2 term, this produces an O(M−1)O(M^{-1}) perturbation in the normalization factor and leads to the stated O(M−1)O(M^{-1}) approximation bound.

In contrast, Eqn. (21) in the proof of Theorem 4.1 shows why RMSNorm achieves an O(M−2)O(M^{-2}) rate. When computing the RMS, the auxiliary coordinate MM does not interact with the other coordinates. It contributes M2M^2 directly to the squared norm, while the remaining term stays bounded on the compact target feature set. There is therefore no cross term linear in MM, giving an O(M−2)O(M^{-2}) perturbation and the corresponding approximation rate.

This explains why the RMSNorm argument does not directly give an O(M−2)O(M^{-2}) LN-AN rate. Achieving such a rate would require cancelling the first-order centering effect. For general one-sided activations at width N+2N+2, we therefore expect the O(M−1)O(M^{-1}) rate to persist.

W1: Extension to deep networks and implications of large parameters for training

Thank you for pointing out these two concerns.

  1. Our theorem can be extended to deep MLPs with normalization inserted in each hidden layer. Since an affine map can be split into a composition of two affine maps, a two-hidden-layer MLP of the form "Linear-Activation-Linear-Activation-Linear" can be written as a composition of two one-hidden-layer MLPs, each of the form "Linear-Activation-Linear". Repeating this argument, an LL-hidden-layer MLP can be written as a composition of LL one-hidden-layer MLPs. We can therefore approximate the network layer by layer and control the accumulated error. Under the corresponding activation and Lipschitz assumptions, the total error can be made smaller than any prescribed ε>0\varepsilon>0 for fixed finite LL. We have provided a rigorous mathematical formulation of this extension in Proposition 7.4 (Section 7.2). Appendix E.4 provides its proof and an LL-hidden-layer ReLU example with widths Nℓ+1N_\ell+1 for RMSNorm and Nℓ+2N_\ell+2 for LN.

  2. Our proposed construction does require a large MM. We agree with the reviewer that large parameters may cause difficulties in practical training. Here, however, we study approximation capability and focus on the existence of the constructed parameters.

    There are also previous approximation results that use large parameters in their constructions. Examples include the tanh approximation results of De Ryck et al. [1] and the work of Zhang et al. [2] on approximating ReLU networks using networks with other activation functions. These works also use growing parameter scales to construct increasingly accurate approximations.

    Although our proof uses a large MM, this allows us to construct accurate approximants with less additional width. For example, Theorem 4.1 achieves vanishing approximation error by increasing parameter magnitudes while keeping the width fixed at N+1N+1. In contrast, the construction in Theorem 4.2 improves accuracy by increasing the replication count rr, and hence the width N+rN+r. Thus, the large parameter MM allows the first construction to improve approximation accuracy without further increasing the width.

If any questions remain unresolved, please feel free to raise them. We would be glad to address them.


References

[1] Tim De Ryck, Samuel Lanthaler, and Siddhartha Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 143:732-750, 2021.

[2] Shijun Zhang, Jianfeng Lu, and Hongkai Zhao. Deep Network Approximation: Beyond ReLU to Diverse Activation Functions. Journal of Machine Learning Research, 25(35):1-39, 2024.

Reviewer_w69VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether inserting RMSNorm or LayerNorm immediately before or after the activation preserves the approximation power of a one-hidden-layer MLP. It constructs normalized networks that approximate a prescribed MLP on a compact set. The main idea is to add coordinates that make the input-dependent normalization scale nearly constant, with additional reparameterization to handle LayerNorm centering. The authors derive explicit width guarantees under different assumptions on the activation and affine parameters, and transfer applicable universal approximation results to the normalized architectures. They also establish joint worst-case sharpness of the fixed additional-width bounds and illustrate the predicted approximation rates in synthetic experiments.

Critical strengths

1. The width guarantees quantify the cost of preserving a target MLP.

The results connect the width of a given target MLP to the width of its normalized approximation. Under the corresponding assumptions, one additional coordinate is sufficient for RMSNorm, and two are sufficient for LayerNorm. This quantifies the additional width needed to reproduce a particular target network, providing information beyond universal approximation alone.

2. The constructions clarify the roles of activation placement and affine parameters.

The paper explains why AN and PN networks require different arguments, and how the activation range and positive homogeneity affect the guarantees. The finite-width obstruction is also informative: without affine parameters, no fixed normalized width can provide a guarantee covering all continuous activations in the PN setting.

Critical weaknesses

The importance of the relative approximation problem needs stronger justification.

The width guarantees provide useful quantitative preservation results. My main concern is whether approximating a prescribed shallow MLP with additional width is a sufficiently informative way to study the approximation power of normalized networks. The results establish conditions under which the target MLP’s function can be recovered after inserting normalization. However, I was looking for a more direct understanding of how normalization changes the representable function class. For example, a comparison of normalized and unnormalized architectures at a fixed width would help clarify this effect. I am not sufficiently convinced that the chosen recovery criterion provides insight into this broader question, which limits how significant I find the contribution.

Questions for authors

Why is recovering a prescribed shallow MLP with additional width an appropriate criterion for studying normalization’s approximation power? Beyond establishing this recovery guarantee, what does the formulation reveal about how normalization changes the representable function class? It would help to clarify the connection between these relative approximation results and a more direct comparison of normalized and unnormalized architectures.

6 comments
AuthorsOpenReview ↗

Response to Reviewer w69V (1/2)

Dear Reviewer w69V,

Thanks for your careful review and thoughtful comments on the motivation of our work. We appreciate your recognition of our width guarantees and the roles of activation placement and affine parameters. Below, we clarify the significance of the relative approximation problem and its connection to comparing function classes.

W1 / Q1: Function-class comparison and the significance of preserving a prescribed MLP

Thank you for this very good question. We address its two parts separately below.

1. Why is recovering a prescribed shallow MLP with additional width an appropriate criterion for studying normalization’s approximation power?

To address this question, we distinguish two perspectives on how normalization's nonlinearity affects expressive power. One concerns normalization's interaction with linear layers, using normalization itself as the source of nonlinearity. Ni et al. [1, 2] study this perspective through architectures combining LN or parallel LN with linear layers, as discussed in Section 2 (Related Work). The other concerns its interaction with an existing coordinatewise activation function, namely the focus of our paper. Mathematically, our work studies the approximation of prescribed MLPs by normalized networks with additional width. The broader significance of these results lies in clarifying how normalization affects the expressive power of networks with existing activation functions. We discuss this significance from the following two aspects.

First, inserting LN or RMSNorm may affect analyses of the original network's expressive power. For example, Montúfar et al. [3] study the number of linear regions of networks with piecewise-linear activations, including ReLU. These analyses rely on the network being affine within each region. Inserting LN introduces input-dependent nonlinear scaling [1], so the network may no longer be affine on the original regions.

Second, our results may help extend analyses that omit LN to architectures that include it. In [4], Yun et al. study the expressive power of Transformers, but omit LN from their theoretical analysis. The result therefore does not directly answer whether a Transformer with LN retains universal approximation capability. Depending on where LN is placed, the corresponding LayerNorm approximation results in Section 6 may help extend the analysis to architectures that include LN. This is one way in which our results can contribute to a more complete understanding of normalization in deep neural networks.

References are listed at the end of Part 2.

AuthorsOpenReview ↗

Response to Reviewer w69V (2/2)

W1 / Q1: Function-class comparison and the significance of preserving a prescribed MLP (continued)

2. Beyond establishing this recovery guarantee, what does the formulation reveal about how normalization changes the representable function class?

Most of our paper studies which functions remain approximable after inserting LN or RMSNorm, with additional width where needed. We also investigate which functions cannot be approximated arbitrarily well at a fixed normalized width.

In the original submission, Proposition 5.2 and its LN counterpart show that, for each fixed width dd, the shifted-ReLU activation ψd(t)=max⁡{0,t−d}\psi_d(t)=\max\{0,t-\sqrt d\} makes every no-affine PN network at that width constant, although an ordinary network with the same activation can represent nonconstant functions. Proposition 7.1 and its LN counterpart establish width lower bounds using the identity activation.

In response to Reviewer dY57's question about practical nonlinear activations, we have further investigated this issue for fixed activations, including ReLU and tanh. We have added these results as Propositions 7.2 and 7.3 in the revised manuscript, with proofs in Appendices E.2 and E.3. Using ReLU MLPs with RMSNorm inserted after the activation as an example, we next explain in detail how normalization affects the expressive power of the original MLP.

Illustrative example. Fix both the input and output dimensions at NN and consider ordinary width-NN ReLU MLPs on the unit ball. Proposition 7.2 shows that some such targets cannot be approximated arbitrarily well by width-NN RMS-AN networks. Theorem 4.1 shows that width N+1N+1 suffices to approximate every such target to arbitrary accuracy. Thus, normalization can prevent recovery at the original width, while one additional coordinate suffices to restore approximation capability in this setting.

If any questions remain unresolved, please feel free to raise them. We would be glad to address them.


References

[1] Yunhao Ni, Yuxin Guo, Junlong Jia, and Lei Huang. On the Nonlinearity of Layer Normalization. In Proceedings of the 41st International Conference on Machine Learning, PMLR 235:37957-37998, 2024.

[2] Yunhao Ni, Yuxin Guo, Yuhe Liu, Wenxin Sun, Jie Luo, Wenjun Wu, and Lei Huang. Parallel Layer Normalization for Universal Approximation. arXiv preprint arXiv:2505.13142, 2025.

[3] Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the Number of Linear Regions of Deep Neural Networks. In Advances in Neural Information Processing Systems, volume 27, 2014.

[4] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Are Transformers universal approximators of sequence-to-sequence functions?. In International Conference on Learning Representations, 2020.

Thank you for the clarification and the additional lower bounds. These address an important part of my original concern by providing a direct comparison and strengthening the preservation guarantees through worst-case necessity.

My remaining concern is how informative this preservation formulation is for understanding normalization’s approximation power. At fixed width, ordinary and normalized networks can have advantages for different target functions. The paper quantifies the additional width needed to recover ordinary MLP functions, but provides a more limited account of the expressive capabilities introduced by normalization itself. The added lower bounds sharpen the answer to the preservation question, while my concern about the breadth and significance of the resulting insight remains.

I understand that preservation is the paper’s intended focus, and I recognize the theoretical value of the results. Nevertheless, I remain unconvinced that the contribution is sufficiently significant to support acceptance. I therefore keep my score of 2.

Reviewer_dY57OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how LayerNorm and RMSNorm affects the expressive power of shallow neural networks. The central idea is to analyze how width-augmentation can preserve the approximation power of a neural net f after adding layer normalization before or after the activation function. The authors establish error bounds to quantify the corresponding errors and prove convergence of the augmented neural net to the original neural net. Additionally, the authors provide lower bounds on the required width-augmentation. Furthermore, geometric intuition about the approach is provided. The theoretical results are further supported by numerical experiments.

Critical strengths
  • The central ideas are elegant and well presented.
  • The authors give a thorough analysis of the approximation errors, providing both quantitative upper bounds on approximation errors and lower bounds on the necessary network width.
  • The proofs are constructive and relatively easy to follow
Critical weaknesses
  • The analysis is limited to networks with only one-hidden layer and a single normalization operation. It is unclear to what extent these results provide insight into normalization in practical scenarios, where multiple normalization layers interact, and whether the same linearization ideas would apply in those settings.

  • The lower bounds in Section 7 establish worst-case sharpness using the identity activation only, they do not cover practical nonlinear activations such as ReLU or tanh

  • The numerical experiments use synthetic one-dimensional functions and explicitly constructed networks, without training. While they confirm the analytical convergence rates, the experiments provide little additional insight into the relevance of the results for practical neural networks.

Minor presentation issues. The Introduction and Related Work sections contain substantial repetition, particularly in their discussion of Ni et al. (2024, 2025). These sections should be revised

Questions for authors
  • Can the authors provide more insight into whether the width bounds are optimal for fixed, commonly used activations such as ReLU?

  • To what extent could the proposed constructions extend to networks with multiple normalization layers?

  • Some approximation results, like Th. 4.1 require M-> infty in order for the error to vanish, but at the same time the magnitude of parameters after augumentation is diverging as O(M). Would the authors say this intrinsic to the approximation problem or specific to the proposed contruction?

4 comments
AuthorsOpenReview ↗

Response to Reviewer dY57 (1/2)

Dear Reviewer dY57,

Thanks for your careful review and constructive feedback. We appreciate your positive assessment of our main idea, approximation analysis, and constructive proofs. We address your questions and comments below.

Q1 / W2: Width lower bounds for fixed nonlinear activations

We consider this an excellent question. Our Proposition 7.1 aims to answer the following question: if we want to preserve the original expressive power for any coordinatewise activation, how much width is necessary? Taking RMSNorm as an example, the identity activation ϕ(x)=x\phi(x)=x shows that N+1N+1 is necessary in the worst case. For a specific coordinatewise activation, the lower bound needs to be examined separately. We discuss ReLU and tanh below.

  1. For the AN mode, where normalization is added after the activation, we can still establish the necessary widths N+1N+1 for RMSNorm and N+2N+2 for LN. In fact, this holds for every fixed continuous nonconstant activation. Consider a width-NN target MLP with input and output dimensions NN. We can choose its first-layer weights and biases so that the target output covers an NN-dimensional region. RMSNorm constrains the hidden representation to a sphere, leaving d−1d-1 dimensions at width dd. LN adds a zero-mean constraint, leaving d−2d-2 dimensions. This gives the geometric intuition for the two width thresholds. In response to your question, we have added Proposition 7.2 to the revised manuscript, with its proof in Appendix E.2. The proof uses the geometry of the affine readout to establish a positive uniform approximation error below these width thresholds.

  2. For the PN mode, where normalization is added before the activation, normalization can still reduce the dimension of the hidden representation, causing information loss that the subsequent activation cannot recover. We establish the same necessary widths N+1N+1 for RMSNorm and N+2N+2 for LN for continuous activations that are strictly monotone on some nondegenerate interval, including ReLU and tanh. In response to your question, we have also added Proposition 7.3 to the revised manuscript, with its proof and explicit ReLU and tanh examples in Appendix E.3.

Together with the existing upper bounds, these results establish worst-case sharpness for fixed ReLU in both AN and PN modes, and for fixed tanh in PN mode with affine normalization parameters. Both the AN and PN results and their proofs are now included in the revised manuscript. We have also clarified the activation assumptions for the lower bounds in Table 1 and the summary statements.

Q2 / W1: Extension to deep networks with multiple normalization layers

Yes, our theorem can be extended to deep MLPs with normalization inserted in each hidden layer. Since an affine map can be split into a composition of two affine maps, a two-hidden-layer MLP of the form "Linear-Activation-Linear-Activation-Linear" can be written as a composition of two one-hidden-layer MLPs, each of the form "Linear-Activation-Linear". Repeating this argument, an LL-hidden-layer MLP can be written as a composition of LL one-hidden-layer MLPs. We can therefore approximate the network layer by layer and control the accumulated error. Under the corresponding activation and Lipschitz assumptions, the total error can be made smaller than any prescribed ε>0\varepsilon>0 for fixed finite LL. We have added this extension as Proposition 7.4 in Section 7.2. Appendix E.4 provides its proof and an LL-hidden-layer ReLU example with widths Nℓ+1N_\ell+1 for RMSNorm and Nℓ+2N_\ell+2 for LN.

AuthorsOpenReview ↗

Response to Reviewer dY57 (2/2)

Q3 / W3: Large parameters and their implications for training

Our proposed construction does require a large MM. We agree with the reviewer that large parameters may cause difficulties in practical training. Here, however, we study approximation capability and focus on the existence of the constructed parameters.

There are also previous approximation results that use large parameters in their constructions. Examples include the tanh approximation results of De Ryck et al. [1] and the work of Zhang et al. [2] on approximating ReLU networks using networks with other activation functions. These works also use growing parameter scales to construct increasingly accurate approximations.

Although our proof uses a large MM, this allows us to construct accurate approximants with less additional width. For example, Theorem 4.1 achieves vanishing approximation error by increasing parameter magnitudes while keeping the width fixed at N+1N+1. In contrast, the construction in Theorem 4.2 improves accuracy by increasing the replication count rr, and hence the width N+rN+r. Thus, the large parameter MM allows the first construction to improve approximation accuracy without further increasing the width.

W4: Presentation issues

Thank you for pointing this out. We have reorganized the Introduction and Related Work to reduce repetition. We have also polished other parts of the manuscript to improve clarity and readability.

If any questions remain unresolved, please feel free to raise them. We would be happy to address them.


References

[1] Tim De Ryck, Samuel Lanthaler, and Siddhartha Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 143:732-750, 2021.

[2] Shijun Zhang, Jianfeng Lu, and Hongkai Zhao. Deep Network Approximation: Beyond ReLU to Diverse Activation Functions. Journal of Machine Learning Research, 25(35):1-39, 2024.

The authors successufully addressed my main concerns, in particular they managed to extend their results to deep networks, non-linear activations, and clarified the role of parameters with large magnitude by establishing connections with the existing literature. The presentation of the paper has also improved. I thank the authors for their efforts and for the discussion. I will increase my rating.

Dear Reviewer dY57,

Thank you for your careful review and constructive feedback. We are glad that our revisions and responses have addressed your main concerns. Your suggestions have helped us strengthen both the theoretical results and the presentation. We sincerely appreciate your reconsideration and your decision to raise the rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.