acceptodds
Under review as a conference paper at ICLR 2027

Parallel Layer Normalization for Universal Approximation

Abstract

Layer Normalization (LN) is not only an optimization component: between affine layers it also supplies nonlinearity, and grouping changes how this nonlinearity scales with width. We study this mechanism through universal approximation. In Parallel Layer Normalization networks (PLN-Nets), fixed-dimensional LN units are applied independently to groups between two affine layers. We prove that a shallow LN-Net with one LN unit is non-universal regardless of width, whereas PLN-Nets become universal as the number of parallel groups grows. The mechanism extends to deep networks, Root Mean Square Normalization (RMSNorm) variants, position-wise feed-forward networks, and convolutional networks under suitable grouping structures. Controlled MLP and physics-informed neural network (PINN) experiments support PLN's practical viability, while CNN and ViT experiments show that group count shapes the finite-sample fitting envelope but does not act alone: the grouped axes, replacement site, and label structure lead to distinct fitting and generalization behavior.

Primary area:
learning theory
Keywords:
Layer Normalization, RMSNorm, Group Normalization, Universal Approximation, Neural Network Expressivity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xcK7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes parallel layer normalization (PLN), which, in simple terms, applies layer norm (LN) to groups of features to serve as a nonlinear activation. The paper mainly proves that a shallow PLN network achieves universal approximation as the number of parallel groups increases, whereas a shallow network with a single LN unit (normalizing the entire hidden layer) cannot. Experiments investigate: 1) function approximation and 2) random-label fitting and clean-label classification using PLN-based networks.

Critical strengths
  1. The paper is overall interesting, as it provides an interesting perspective on using LN as an activation function. However, the credit is partly shared with the cited Ni et al. (2024), which has already investigated LN's nonlinearity. At the implementation level, PLN is perhaps not fundamentally different from existing group norm, although the normalization domains is different. I think the main contribution of this paper is more on the theoretical side, particularly how PLN can represent certain activation functions and achieve universal approximation, whereas a single LN unit cannot.

  2. The methodology writing seems to have been paid attention to. Symbols are generally properly defined, although given the number of symbols involved, it would make sense to provide a notation table. The lemma and theorem parts are more difficult to go through, but I find the overall arguments make sense. I should declare that I have not gone through every detail of the proofs, so I cannot fully vouch for their mathematical correctness.

Critical weaknesses
  1. While the overall message is interesting: a shallow PLN can achieve universal approximation, I find its practical significance insufficiently demonstrated. In particular, I would expect a ReLU-to-PLN replacement experiment on CIFAR-10 to complement Theorem 4.4, which says that a PLN-Net with depth 2L and width 3n_sN can approximate a ReLU network. This would help demonstrate whether the theoretical construction has practical value beyond synthetic function approximation. Instead, the CIFAR-10 experiment shifts to a ViT with GELU activation, for which no specific replacement guarantee is provided in the methodology. Also, using a customised ViT with 224x224 inputs for the 32x32 CIFAR-10 images seems unnecessarily complicated, especially when the test accuracy is only 70+%. A standard ResNet-18 with ReLU would likely achieve 90+% test accuracy with a relatively straightforward training setup. If ViT needs to be used, there are larger image-sized classification datasets (don't necessarily need to go for ImageNet, though). The authors can perhaps comment on the reasons for the experiment design.

  2. Regarding the random-label fitting experiments, while I understand that they are intended to evaluate the fitting capacity of the network, again I find it difficult to appreciate their practical significance, as it is unclear how it demonstrates generalization ability - which is perhaps what matters most.

  3. What would be the computational overhead and model parameter increases associated with the configurations reported in Table 3? I assume that LN-to-PLN/Mix-Norm replacement does not increase the number of learnable parameters, but the increased number of independent normalization groups may introduce additional computational overhead. For GELU-to-PLN/Mix-Norm replacement, is the number of units kept unchanged, and how does the parameter count compare with the original ViT? It would be helpful to report train/val times for these comparisons.

Questions for authors

Please check the critical weaknesses. Overall, the weak reject rating is slightly stronger than what I have in mind, but I feel the paper is not quite there for a weak accept yet.

Additional question:

  1. Overall, the theoretical argument makes sense to me. However, since the methodology largely relies on constructing particular cases, namely the sign function and ϕ\phi, and then connecting them to established universal-approximation results, I wonder how relevant these constructions are to practically trained PLN networks. In particular, there is no guarantee that gradient-based training would lead PLN units to learn functions resembling these theoretically constructed activations. Have the authors considered investigating (or thought about whether it is possible?) what nonlinear functions the PLN units actually approximate after training?

Minor:

  1. Define FFN.
  2. Fig. 3 sign or ϕ\phi?
  3. Perhaps try to have a contribution statement at the end of the intro so it is clear what the claims are.
10 comments
AuthorsOpenReview ↗

Response to Reviewer xcK7 (1/2)

Dear Reviewer xcK7,

Thank you for your careful review and helpful suggestions. We address the connection between our theory and experiments, the random-label results, computational cost, and the presentation issues below.

To make the contributions of this work clearer, we have added a summary of our contributions in the final paragraph of the Introduction.

W1: ReLU-to-PLN replacement and the ViT experimental setup

Theorem 4.4 gives a constructive function approximation guarantee. For a given depth-LL, width-NN ReLU network on the unit cube and any positive uniform error tolerance, there exists a PLN network of depth 2L2L and width 3nsN3n_sN, with ns≥3n_s\geq3, that meets that tolerance. The proof combines an existing ReLU-to-ϕ\phi approximation with the exact layer-wise PLN representation of the resulting ϕ\phi network. The GELU replacement in our ViT experiment is a separate empirical comparison.

Our original submission already includes CIFAR-10 activation comparisons in VGG-16 (Appendix D.2.2, Figure A3), together with the small-CNN normalization comparison in Table 2. To illustrate the constructive mechanism behind Theorem 4.4 more directly, we additionally converted a trained CIFAR-10 ResNet-18 by replacing all 17 ReLU calls with analytically constructed PLN[4] blocks, without further training or fine-tuning.

The original network was trained for 100 epochs with seed 40. We retain its trained convolutional and linear weights, frozen BatchNorm statistics, and residual connections. Each ReLU is replaced by two PLN stages, with two and three LN[4] groups per original scalar, respectively. The finite-difference construction uses double precision internally. Per-module scales RR are calibrated on 1,024 training images without using their labels.

Here kk controls the sharpness of the smooth step vk(y)=(1+ϕ(ky))/2v_k(y)=(1+\phi(ky))/2, where y=x/Ry=x/R and ϕ(t)=t/1+t2\phi(t)=t/\sqrt{1+t^2}. Larger kk gives a sharper step, while the network width and group count remain fixed. We report all three predeclared values k∈{100,1000,10000}k\in\{100,1000,10000\} on all 10,000 CIFAR-10 test images under the same evaluation settings:

ModelTest accuracy (%)Prediction agreement with ReLU (%)Mean per-image logit ℓ∞\ell_\infty errorMaximum absolute logit error
ReLU baseline94.90100.0000
Constructed PLN[4], k=100k=10094.1097.542.570599.33432
Constructed PLN[4], k=1,000k=1{,}00094.9199.890.08806760.517718
Constructed PLN[4], k=10,000k=10{,}00094.9199.990.00154120.00932789

At the finest construction strength, the converted model achieves 99.99% prediction agreement with the original network, with maximum absolute logit difference below 0.00933. This illustrates preservation of a learned function through the expanded affine–LN construction. The 0.01-percentage-point accuracy difference is not evidence of improved generalization.

Regarding the ViT input size, we retained the 224×224224\times224 input resolution of the adopted ViT configuration and therefore resized CIFAR-10 images accordingly. All compared variants used the same input resolution and preprocessing, allowing us to study normalization domains under a fixed experimental setup.

W2: What the random-label experiments tell us

Our experiments explore PLN's properties, including how its fitting behavior changes with the label structure. We find the contrast between clean and random labels interesting. Standard supervised learning experiments usually use the original labels. Looking only at that setting, one might conclude that PLN offers little advantage over LN. In Table 3, replacing LN with PLN raises the reported mean clean-test accuracy from 71.6% to 72.3%, whereas the reported mean fitting accuracy on 20,000 randomly labeled examples rises from 48.2% to 75.3%. These configuration-grid summaries reveal a substantial difference in empirical fitting capacity that is less apparent on the standard classification task.

This does not imply a comparable improvement in generalization. PLN's empirical performance may depend on the data distribution and vary across tasks. The grouping principle also has practical relevance: the grouped RMS normalization cores in RetNet [1] and Mamba-2 [2] correspond to position-wise PLS, as clarified in Remark 5.1 of the revised manuscript.

References are listed at the end of Part 2.

AuthorsOpenReview ↗

Response to Reviewer xcK7 (2/2)

W3: Parameter counts and computational cost

We have added computational-cost tables for CNNs (Appendix D.2.1, Table III, p. 30) and ViTs (Appendix D.3, Table IV, p. 34), using clean-label CIFAR-10 with 50,000 training images. The ViT FFN hidden width remains 768 in all settings, with output width 192. Replacing LN preserves the baseline's 1,967,434 parameters. Replacing GELU adds 6,144 learned scale and bias parameters, an increase of approximately 0.312%. All CNN configurations have 528,266 parameters.

We measured three training epochs per configuration on one RTX 4090 in FP32, using the original batch sizes and one fixed seed. Training times exclude initialization, warm-up, validation, and checkpoint I/O. Inference latency averages 30 forward passes after 10 warm-up passes, with inputs already on the GPU. The ViT results for Clean 50k are:

SettingParametersTraining time for 3 epochs (s)Inference time (ms per batch of 256)
LN–GELU baseline1,967,43480.217.5
LN → PLN1,967,43482.2–86.920.7–23.8
LN → Mix-Norm1,967,43480.5–88.820.7–24.6
GELU → PLN1,973,57898.4–101.329.7–31.4
GELU → Mix-Norm1,973,57898.1–116.329.5–33.4

The ranges cover all grouping configurations in each family, without accuracy-based selection. They are not uncertainty intervals across repeated runs. Detailed training and inference times for each grouping configuration are provided in Appendix D.4 (Tables VI–VIII, pp. 36–38). These measurements characterize the current implementations.

Additional question: Do trained PLN units recover the proof construction?

Trained PLN units need not reproduce the sign or ϕ\phi responses used in our proofs. These constructions establish approximation guarantees and complexity bounds. They do not predict the internal representations learned by gradient-based training. The same distinction applies to constructive approximation results for Transformers [3], which use quantization and contextual mappings, and ReLU networks [4], which construct approximate squaring and multiplication modules.

In our experiments, training minimizes the task loss without requiring intermediate units to match these auxiliary functions. A trained PLN network may therefore approximate the target through different internal representations. We have not systematically characterized the responses of individual trained PLN units, and do not claim that training recovers the proof construction. In the ResNet conversion above, the PLN parameters are set analytically after training the ReLU network.

Minor comments: FFN, Figure 3, and the contribution statement

  1. FFN abbreviation. The Introduction now introduces “position-wise feed-forward networks (FFNs)” before using the abbreviation.

  2. Figure 3. Figure 3 illustrates the construction of ϕ(x)=x/1+x2\phi(x)=x/\sqrt{1+x^2} in Lemma 4.2. The output label “sign” was a typographical error. We have corrected it to αϕ(w1x+b1)+β\alpha\phi(w_1x+b_1)+\beta and clarified the caption.

  3. Contribution statement. We have added a contribution statement at the end of the Introduction, organized into the construction method, theoretical results, and exploratory empirical study, with references to the corresponding theoretical statements.

If any questions remain unresolved, please feel free to raise them. We would be glad to address them.


References

[1] Yutao Sun et al. Retentive Network: A Successor to Transformer for Large Language Models. arXiv:2307.08621, 2023.

[2] Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML, 2024.

[3] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Are Transformers Universal Approximators of Sequence-to-Sequence Functions?. ICLR, 2020.

[4] Dmitry Yarotsky. Error Bounds for Approximations with Deep ReLU Networks. Neural Networks, 94:103–114, 2017.

I thank the authors for their response.

W1. No further questions.

W2. What is the label structure contained in a randomly assigned label? Essentially, it is fitting random noise, which is possible as long as a model has sufficient learning capacity, but I still do not find that a higher random-noise-fitting power is of much practical meaning. If random noise must be examined, perhaps it is more sensible to test a partially noise-seeded label setting to determine whether the proposal makes any difference.

Can the authors please use a colored diff to show what's changed in the updated manuscript?

Reviewer_joNAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies normalization as the sole nonlinearity between affine layers. It proves that a shallow network containing one full-width LayerNorm unit is not universally approximating, regardless of its width, whereas parallel fixed-size LayerNorm blocks become universal as their number increases. The positive construction exactly realizes a smooth scalar activation and transfers existing approximation results to PLN networks. The paper also establishes a function-class equivalence with grouped RMS normalization and extends the construction to deep networks, position-wise FFNs, and convolutional networks. Experiments on synthetic regression, a Helmholtz PINN, and CIFAR-10 CNNs and ViTs investigate trainability, grouping choices, and fitting versus generalization.

Critical strengths
  1. The separation between widening a single LN unit and adding parallel LN units is clear and informative. Theorem 4.1 identifies an explicit structural obstruction, while the positive results show how parallelization removes it. This provides a concrete explanation of why width has different consequences under different normalization structures.

  2. The representation construction is explicit and reusable. The PLN–PLS equivalence, the stabilized-LN treatment in Appendix A.8, and the convolutional construction make the result applicable beyond an idealized scalar example. The shallow regression experiments also provide useful evidence that the resulting networks can be trained successfully at finite width.

Critical weaknesses
  1. The incremental theoretical contribution is relatively narrow. Ni et al. (2024) already studied LN-only networks, grouped LN, and position-wise grouped normalization. The new shallow non-universality/universality separation is useful, but once Lemma 4.2 embeds a standard smooth activation, the approximation rates and most architectural extensions largely follow by transferring existing results. The paper offers limited quantitative insight into the additional expressive or approximation efficiency of PLN at a fixed resource budget.

  2. The experiments do not yet establish a sufficiently robust practical benefit to strengthen this theoretical contribution. CNN/ViT random-label fitting under fixed optimization settings conflates representation with trainability. Table 3 explicitly reports independently selected maxima across configuration grids, rather than a validation-selected comparison with matched tuning budgets. The PINN tables report best errors despite using five seeds, leaving typical performance unclear; moreover, ReLU is a weak comparator for the stated second-derivative residual because its second derivative vanishes almost everywhere. The full grids and acknowledged limitations are helpful, but these issues restrict the conclusions supported by the empirical results.

Questions for authors
  1. Clarification of novelty: Under matched grouping domains and affine settings, are PLN and CPLN the same operators as LN-G and LN-G-Position in Ni et al. (2024)? Please identify the precise additional theoretical conclusions, distinguishing the shallow separation from consequences of the scalar-activation embedding.

  2. Experimental clarification: For Table 3, was a separate validation set used to select configurations for the clean-test comparison? Please report the selected configurations and their mean and standard deviation, and clarify whether the LN–GELU baseline received a comparable learning-rate tuning budget. This would affect my assessment of the reported improvements.

  3. Clarification and aggregation of existing results: What exactly does “best” aggregate over in Tables 1 and II—seeds, optimizers, checkpoints, or other choices? Can the five-seed mean and standard deviation be reported for fixed optimizer and loss-weight settings, particularly against Tanh and SAT? This would determine whether the PINN advantage is consistent across runs.

5 comments
AuthorsOpenReview ↗

Response to Reviewer joNA (1/2)

Dear Reviewer joNA,

Thank you for your careful review and detailed questions. Below, we clarify the operator equivalences, identify the new theoretical results, and provide the selection rules and seed statistics underlying our experiments.

W1 / Q1: Operator equivalence and new theoretical results

We have added a paragraph at the end of the Introduction summarizing our contributions. We clarify the operator equivalences and theoretical contributions below.

  1. Operator equivalence. Yes. At a fixed width, PLN is the same operator as LN-G, and CPLN is the same operator as LN-G-Position, when the normalization domains, stabilization constant, and affine settings match.

  2. New theoretical results. To the best of our knowledge, this is the first work to investigate the universal approximation capabilities of neural networks whose nonlinearities are provided solely by normalization layers. Ni et al. [1] show that sufficiently deep LN-Nets can fit arbitrary labels on a finite set of distinct inputs and use a Hessian-based measure to study grouping. Our results concern uniform approximation of continuous functions. Theorem 4.1 shows that a shallow network with one full-width LN unit fails to be universal regardless of width. Theorem 4.2 shows that, for any LL-Lipschitz function on [0,1][0,1], ⌊L/(2ϵ)⌋+1\lfloor L/(2\epsilon)\rfloor+1 fixed-size LN groups suffice for uniform error below ϵ\epsilon. We also establish the equivalence to the RMS-based PLS construction (Lemma 3.2) and extend the representation construction to position-wise FFNs and CNNs (Theorems 5.1 and 5.2).

  3. Role of the scalar construction. Lemma 4.2 shows that a fixed-size LN block between affine maps exactly realizes ϕ(x)=x/1+x2\phi(x)=x/\sqrt{1+x^2}. Theorems 4.3 and 4.4 use this construction to transfer existing approximation results to PLN-Nets. The contribution is the connection that allows normalization to supply all nonlinearities, together with the structural separation and explicit bounds above.

W2 / Q2: Table 3 selection rules, tuning budget, and seed statistics

The experimental settings were provided in Appendix D.3 of our original submission. All ViT configurations in Table 3, including the LN–GELU baseline, used AdamW with the same fixed learning rate of 10−410^{-4} and a 100-epoch training schedule. Given the large number of configurations and our computational budget, we did not perform additional learning-rate tuning for any method.

We did not use a separate validation split to select the configurations in Table 3. For the clean-label grid, we trained on all 50,000 CIFAR-10 training examples and evaluated on the standard 10,000-example test set. The clean-test column reports the largest three-seed mean final-epoch test accuracy and the largest single-run final-epoch test accuracy, selected separately within each family. Clean training accuracy is also selected independently. Thus, different entries in the same row need not come from the same configuration.

The baseline is the per-token LN–GELU endpoint, (GT,GC)=(197,1)(G_T,G_C)=(197,1), whereas each grouped family includes multiple grouping configurations. The learning-rate protocol is shared, but the configuration-search budgets are different. We therefore interpret the maxima as descriptive summaries of the explored grids, rather than estimates of a generalization advantage after independent validation-based model selection.

We have added Table V in Appendix D.3 (p. 35), listing the configurations selected for each mean and maximum, all three seed results at each mean-selected configuration, the mean and sample standard deviation, and the seed attaining each independently selected maximum. The clean-test summary is reproduced below. Here GTG_T and GCG_C denote token and channel group counts, respectively. The seeds are 40, 41, and 42.

SettingConfiguration selected by test mean (GT,GC)(G_T,G_C)Final test accuracy (%), mean ± sample SDIndependent single-run maximum (%), configuration
LN–GELU baseline(197, 1)71.65 ± 0.6972.31 at (197, 1)
LN → PLN(197, 4)72.30 ± 0.6472.87 at (197, 4)
LN → Mix-Norm(4, 1)73.55 ± 0.8174.50 at (128, 12)
GELU → PLN(197, 2)69.71 ± 0.4871.02 at (197, 48)
GELU → Mix-Norm(8, 192)70.20 ± 0.2670.65 at (16, 2)

References are listed at the end of Part 2.

AuthorsOpenReview ↗

Response to Reviewer joNA (2/2)

W2 / Q2: Table 3 selection rules, tuning budget, and seed statistics (continued)

For Random 20k, each run contributes its highest fitting accuracy during training on the fixed random-label training set. We then select the configuration with the highest three-seed mean. The corresponding statistics are:

Random 20k settingConfiguration selected by fitting mean (GT,GC)(G_T,G_C)Best fitting accuracy (%), mean ± sample SD
LN–GELU baseline(197, 1)48.19 ± 29.89
LN → PLN(197, 48)75.34 ± 21.18
LN → Mix-Norm(32, 192)89.49 ± 1.60
GELU → PLN(197, 192)73.73 ± 26.68
GELU → Mix-Norm(64, 192)91.66 ± 3.58

These experiments measure empirical fitting under a specified training procedure, which depends on both representation and optimization. The representation guarantees come from our theorems. Random-label results explore how the models behave at finite width under training. Fitting 20,000 random labels is challenging for these finite ViTs, with substantial variation across seeds in several settings. Given this difficulty and variability, we did not extend the ViT experiments to the full CIFAR-10 training set with randomized labels.

W2 / Q3: PINN aggregation and five-seed results

Our PINN experiments aim to investigate fitting capacity. Accordingly, the original manuscript reports the best results achieved under the tested settings rather than averages across random seeds. For each fixed width, nonlinearity, and loss-weight pair, Table II reports the lowest final relative L2\mathcal L_2 error among completed 6,000-step runs across five seeds and the two optimizer settings. Table 1 additionally minimizes over the three loss-weight pairs. These are final-run minima, not the best intermediate checkpoints.

The table reports final relative L2\mathcal L_2 error across five seeds at fixed Adam learning rate 3×10−43\times10^{-4} and fixed loss weights. Entries are mean ± sample standard deviation. Lower is better.

Width(λPDE,λBC)(\lambda_{\mathrm{PDE}},\lambda_{\mathrm{BC}})PLN[4] (×10−3\times10^{-3})Tanh (×10−3\times10^{-3})SAT (×10−3\times10^{-3})
128(1, 10)1.89 ± 3.461.70 ± 1.326.99 ± 10.81
128(1, 1)151.89 ± 222.759.10 ± 3.706.31 ± 4.30
128(3, 1)273.79 ± 270.4813.66 ± 7.7223.29 ± 13.03
256(1, 10)2.61 ± 3.105.20 ± 6.695.55 ± 5.34
256(1, 1)79.45 ± 105.3616.57 ± 13.7220.65 ± 17.09
256(3, 1)170.50 ± 206.6919.83 ± 3.6237.57 ± 15.73

PLN[4]'s low best-run errors do not imply a consistent advantage over Tanh and SAT in mean error or variability across seeds. For example, at loss weights (1,10)(1,10), Tanh has a slightly lower mean at width 128, whereas PLN[4] has the lowest mean at width 256. At the other two loss-weight settings, both smooth baselines have lower means than PLN[4]. We focus this comparison on Tanh and SAT because the PDE residual involves a second derivative, which vanishes almost everywhere for a ReLU MLP.

If any questions remain unresolved, please feel free to raise them. We would be glad to address them.


References

[1] Yunhao Ni, Yuxin Guo, Junlong Jia, and Lei Huang. On the Nonlinearity of Layer Normalization. ICML, PMLR 235:37957–37998, 2024.

Thank you for the clarifications and seed statistics, which address my reporting questions. However, I still find the theoretical advance limited, and the PINN averages do not show a consistent advantage over the smooth baselines. I therefore maintain my rating of 2.

Reviewer_ziGxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper demonstrates that a layer of multiple parallel LN blocks is a universal approximator. It extends to RMSNorm, deeper networks and various types of networks. Experiments demonstrate the constructed models can fit random labels.

Critical strengths

The problem this paper studies is a clear one - the original LN is not a universal approximator, and making it parallel LN blocks makes it a universal approximator.

The range of types of normalization and the types of networks covered in the discussion is broad.

Critical weaknesses
  1. I understand this is a more theory-leaning paper. However, the experiment part is too small-scale, and it is hard to see the practical value of the proposed parallel LN layer. It stops at CIFAR-10 without ImageNet, LM, or other larger-scale baselines.

  2. The main part of the proof is to approximate a x/sqrt(1+x^2) function, and the rest of the conclusions (universal approximation) follow classical results. As such, the theory contribution of the paper is limited as well.

  3. Ni et al. 2024 ICML paper already established the non-linearity of LN as one of its important properties, and thus the additional understanding that a variant is a universal approximator is not too far to construct.

  4. The experiments also show limited benefit: PLN mainly helps fit random labels, while clean-label ViT test accuracy barely improves and even drops when replacing GELU.

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ziGx (1/2)

Dear Reviewer ziGx,

Thank you for your careful review and constructive feedback. We address your concerns about our theoretical contributions, the relationship to prior work, and the purpose of our experiments below.

W1: Experimental scope and practical relevance

The main contribution of this work is theoretical: we establish universal approximation guarantees for networks that use PLN as their sole nonlinearity. Our experiments are exploratory and examine how grouping affects fitting and generalization at finite width.

As for practical relevance, the same normalization structure is already used in sequence models. RetNet [1] normalizes each retention head independently at each token, and Mamba-2 [2] supports grouped RMS normalization in its gated state-space block. The RMS normalization core in both cases corresponds to position-wise PLS, the RMS-based counterpart of PLN, with matching groups and numerical stabilization. We have added this discussion in Remark 5.1 (Section 5.1, p. 7) of the revised manuscript.

W2: Use of existing approximation results and theoretical novelty

We have added a paragraph at the end of the Introduction summarizing our contributions. We would like to clarify three points.

  1. Use of existing approximation results. Only Theorems 4.3 and 4.4 directly invoke existing approximation theorems. Together, they occupy about half a page in the main text.

  2. What we establish about LN. To our knowledge, this is the first work to establish a separation in universal approximation between shallow networks with a single full-width LN unit and those with parallel LN units. A single LN unit fails to give universal approximation at any width (Theorem 4.1), while increasing the number of fixed-size LN blocks enables universal approximation. We also extend the PLN construction to position-wise FFNs and CNNs (Section 5).

  3. Why the scalar construction matters. Lemma 4.2 shows that a fixed-size LN block between affine maps exactly realizes ϕ(x)=x/1+x2\phi(x)=x/\sqrt{1+x^2}. This gives a concrete link between normalization and classical universal approximation theory. Each scalar activation in a network using this function can be replaced by an LN block, transferring the approximation guarantee to networks in which normalization is the sole nonlinearity.

W3: Relationship to Ni et al. (2024)

Ni et al. [3] established the nonlinearity of LN. Our work develops its approximation theory in the following directions.

  1. From finite-sample classification to universal approximation. Ni et al. [3] showed that sufficiently deep LN-Nets can classify any labeling of a finite set of distinct inputs. We study uniform approximation of continuous functions and show that shallow PLN-Nets achieve universal approximation as the number of parallel LN groups grows.

  2. A quantitative approximation guarantee for grouping. Ni et al. [3, Section 5.1] use a Hessian-based measure to compare the nonlinearity of LN and LN-G. Theorem 4.2 quantifies the expressive power of PLN through a uniform approximation bound: for any LL-Lipschitz function on [0,1][0,1], ⌊L/(2ϵ)⌋+1\lfloor L/(2\epsilon)\rfloor+1 groups suffice to achieve uniform error below ϵ\epsilon.

  3. Limits of a single LN unit and extensions to structured models. As discussed in our response to W2, Theorem 4.1 gives a width-independent lower bound on the approximation error of a shallow network with one full-width LN unit. Section 5 extends the PLN representation construction to position-wise FFNs and CNNs.

References are listed at the end of Part 2.

AuthorsOpenReview ↗

Response to Reviewer ziGx (2/2)

W4: Random-label fitting and clean-label generalization

We agree that PLN behaves quite differently under clean and random labels. We find this difference interesting because it changes what we learn about PLN from the experiment.

In standard supervised learning experiments, models are usually evaluated with the original labels. If we evaluated PLN only under this setting, we might conclude that replacing LN with PLN makes little difference: the reported mean test accuracy rises from 71.6% to 72.3%. On 20,000 randomly labeled examples, the same replacement raises the reported mean fitting accuracy from 48.2% to 75.3%, revealing a difference in finite-sample fitting that is easy to miss on the standard classification task.

This greater random-label fitting capacity does not imply a comparable generalization gain. In particular, replacing GELU with PLN reduces the reported clean-test accuracy to 69.7%, below the 71.6% LN–GELU baseline. We acknowledge that PLN's empirical performance may depend on the data distribution and vary across tasks. As discussed in W1, the same grouping principle already appears in sequence models such as RetNet [1] and Mamba-2 [2], where the grouped RMS normalization core corresponds to position-wise PLS.

If any questions remain unresolved, please feel free to raise them. We would be glad to address them.


References

[1] Yutao Sun et al. Retentive Network: A Successor to Transformer for Large Language Models. arXiv:2307.08621, 2023.

[2] Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML, 2024.

[3] Yunhao Ni, Yuxin Guo, Junlong Jia, and Lei Huang. On the Nonlinearity of Layer Normalization. ICML, PMLR 235:37957–37998, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.