- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes parallel layer normalization (PLN), which, in simple terms, applies layer norm (LN) to groups of features to serve as a nonlinear activation. The paper mainly proves that a shallow PLN network achieves universal approximation as the number of parallel groups increases, whereas a shallow network with a single LN unit (normalizing the entire hidden layer) cannot. Experiments investigate: 1) function approximation and 2) random-label fitting and clean-label classification using PLN-based networks.
Critical strengths
-
The paper is overall interesting, as it provides an interesting perspective on using LN as an activation function. However, the credit is partly shared with the cited Ni et al. (2024), which has already investigated LN's nonlinearity. At the implementation level, PLN is perhaps not fundamentally different from existing group norm, although the normalization domains is different. I think the main contribution of this paper is more on the theoretical side, particularly how PLN can represent certain activation functions and achieve universal approximation, whereas a single LN unit cannot.
-
The methodology writing seems to have been paid attention to. Symbols are generally properly defined, although given the number of symbols involved, it would make sense to provide a notation table. The lemma and theorem parts are more difficult to go through, but I find the overall arguments make sense. I should declare that I have not gone through every detail of the proofs, so I cannot fully vouch for their mathematical correctness.
Critical weaknesses
-
While the overall message is interesting: a shallow PLN can achieve universal approximation, I find its practical significance insufficiently demonstrated. In particular, I would expect a ReLU-to-PLN replacement experiment on CIFAR-10 to complement Theorem 4.4, which says that a PLN-Net with depth 2L and width 3n_sN can approximate a ReLU network. This would help demonstrate whether the theoretical construction has practical value beyond synthetic function approximation. Instead, the CIFAR-10 experiment shifts to a ViT with GELU activation, for which no specific replacement guarantee is provided in the methodology. Also, using a customised ViT with 224x224 inputs for the 32x32 CIFAR-10 images seems unnecessarily complicated, especially when the test accuracy is only 70+%. A standard ResNet-18 with ReLU would likely achieve 90+% test accuracy with a relatively straightforward training setup. If ViT needs to be used, there are larger image-sized classification datasets (don't necessarily need to go for ImageNet, though). The authors can perhaps comment on the reasons for the experiment design.
-
Regarding the random-label fitting experiments, while I understand that they are intended to evaluate the fitting capacity of the network, again I find it difficult to appreciate their practical significance, as it is unclear how it demonstrates generalization ability - which is perhaps what matters most.
-
What would be the computational overhead and model parameter increases associated with the configurations reported in Table 3? I assume that LN-to-PLN/Mix-Norm replacement does not increase the number of learnable parameters, but the increased number of independent normalization groups may introduce additional computational overhead. For GELU-to-PLN/Mix-Norm replacement, is the number of units kept unchanged, and how does the parameter count compare with the original ViT? It would be helpful to report train/val times for these comparisons.
Questions for authors
Please check the critical weaknesses. Overall, the weak reject rating is slightly stronger than what I have in mind, but I feel the paper is not quite there for a weak accept yet.
Additional question:
- Overall, the theoretical argument makes sense to me. However, since the methodology largely relies on constructing particular cases, namely the sign function and , and then connecting them to established universal-approximation results, I wonder how relevant these constructions are to practically trained PLN networks. In particular, there is no guarantee that gradient-based training would lead PLN units to learn functions resembling these theoretically constructed activations. Have the authors considered investigating (or thought about whether it is possible?) what nonlinear functions the PLN units actually approximate after training?
Minor:
- Define FFN.
- Fig. 3 sign or ?
- Perhaps try to have a contribution statement at the end of the intro so it is clear what the claims are.
10 comments
Response to Reviewer xcK7 (2/2)
W3: Parameter counts and computational cost
We have added computational-cost tables for CNNs (Appendix D.2.1, Table III, p. 30) and ViTs (Appendix D.3, Table IV, p. 34), using clean-label CIFAR-10 with 50,000 training images. The ViT FFN hidden width remains 768 in all settings, with output width 192. Replacing LN preserves the baseline's 1,967,434 parameters. Replacing GELU adds 6,144 learned scale and bias parameters, an increase of approximately 0.312%. All CNN configurations have 528,266 parameters.
We measured three training epochs per configuration on one RTX 4090 in FP32, using the original batch sizes and one fixed seed. Training times exclude initialization, warm-up, validation, and checkpoint I/O. Inference latency averages 30 forward passes after 10 warm-up passes, with inputs already on the GPU. The ViT results for Clean 50k are:
| Setting | Parameters | Training time for 3 epochs (s) | Inference time (ms per batch of 256) |
|---|---|---|---|
| LN–GELU baseline | 1,967,434 | 80.2 | 17.5 |
| LN → PLN | 1,967,434 | 82.2–86.9 | 20.7–23.8 |
| LN → Mix-Norm | 1,967,434 | 80.5–88.8 | 20.7–24.6 |
| GELU → PLN | 1,973,578 | 98.4–101.3 | 29.7–31.4 |
| GELU → Mix-Norm | 1,973,578 | 98.1–116.3 | 29.5–33.4 |
The ranges cover all grouping configurations in each family, without accuracy-based selection. They are not uncertainty intervals across repeated runs. Detailed training and inference times for each grouping configuration are provided in Appendix D.4 (Tables VI–VIII, pp. 36–38). These measurements characterize the current implementations.
Additional question: Do trained PLN units recover the proof construction?
Trained PLN units need not reproduce the sign or responses used in our proofs. These constructions establish approximation guarantees and complexity bounds. They do not predict the internal representations learned by gradient-based training. The same distinction applies to constructive approximation results for Transformers [3], which use quantization and contextual mappings, and ReLU networks [4], which construct approximate squaring and multiplication modules.
In our experiments, training minimizes the task loss without requiring intermediate units to match these auxiliary functions. A trained PLN network may therefore approximate the target through different internal representations. We have not systematically characterized the responses of individual trained PLN units, and do not claim that training recovers the proof construction. In the ResNet conversion above, the PLN parameters are set analytically after training the ReLU network.
Minor comments: FFN, Figure 3, and the contribution statement
-
FFN abbreviation. The Introduction now introduces “position-wise feed-forward networks (FFNs)” before using the abbreviation.
-
Figure 3. Figure 3 illustrates the construction of in Lemma 4.2. The output label “sign” was a typographical error. We have corrected it to and clarified the caption.
-
Contribution statement. We have added a contribution statement at the end of the Introduction, organized into the construction method, theoretical results, and exploratory empirical study, with references to the corresponding theoretical statements.
If any questions remain unresolved, please feel free to raise them. We would be glad to address them.
References
[1] Yutao Sun et al. Retentive Network: A Successor to Transformer for Large Language Models. arXiv:2307.08621, 2023.
[2] Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML, 2024.
[3] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Are Transformers Universal Approximators of Sequence-to-Sequence Functions?. ICLR, 2020.
[4] Dmitry Yarotsky. Error Bounds for Approximations with Deep ReLU Networks. Neural Networks, 94:103–114, 2017.
I thank the authors for their response.
W1. No further questions.
W2. What is the label structure contained in a randomly assigned label? Essentially, it is fitting random noise, which is possible as long as a model has sufficient learning capacity, but I still do not find that a higher random-noise-fitting power is of much practical meaning. If random noise must be examined, perhaps it is more sensible to test a partially noise-seeded label setting to determine whether the proposal makes any difference.
Can the authors please use a colored diff to show what's changed in the updated manuscript?
Response to Reviewer xcK7 (1/2)
Dear Reviewer xcK7,
Thank you for your careful review and helpful suggestions. We address the connection between our theory and experiments, the random-label results, computational cost, and the presentation issues below.
To make the contributions of this work clearer, we have added a summary of our contributions in the final paragraph of the Introduction.
W1: ReLU-to-PLN replacement and the ViT experimental setup
Theorem 4.4 gives a constructive function approximation guarantee. For a given depth-L, width-N ReLU network on the unit cube and any positive uniform error tolerance, there exists a PLN network of depth 2L and width 3nsN, with ns≥3, that meets that tolerance. The proof combines an existing ReLU-to-ϕ approximation with the exact layer-wise PLN representation of the resulting ϕ network. The GELU replacement in our ViT experiment is a separate empirical comparison.
Our original submission already includes CIFAR-10 activation comparisons in VGG-16 (Appendix D.2.2, Figure A3), together with the small-CNN normalization comparison in Table 2. To illustrate the constructive mechanism behind Theorem 4.4 more directly, we additionally converted a trained CIFAR-10 ResNet-18 by replacing all 17 ReLU calls with analytically constructed PLN[4] blocks, without further training or fine-tuning.
The original network was trained for 100 epochs with seed 40. We retain its trained convolutional and linear weights, frozen BatchNorm statistics, and residual connections. Each ReLU is replaced by two PLN stages, with two and three LN[4] groups per original scalar, respectively. The finite-difference construction uses double precision internally. Per-module scales R are calibrated on 1,024 training images without using their labels.
Here k controls the sharpness of the smooth step vk(y)=(1+ϕ(ky))/2, where y=x/R and ϕ(t)=t/1+t2. Larger k gives a sharper step, while the network width and group count remain fixed. We report all three predeclared values k∈{100,1000,10000} on all 10,000 CIFAR-10 test images under the same evaluation settings:
At the finest construction strength, the converted model achieves 99.99% prediction agreement with the original network, with maximum absolute logit difference below 0.00933. This illustrates preservation of a learned function through the expanded affine–LN construction. The 0.01-percentage-point accuracy difference is not evidence of improved generalization.
Regarding the ViT input size, we retained the 224×224 input resolution of the adopted ViT configuration and therefore resized CIFAR-10 images accordingly. All compared variants used the same input resolution and preprocessing, allowing us to study normalization domains under a fixed experimental setup.
W2: What the random-label experiments tell us
Our experiments explore PLN's properties, including how its fitting behavior changes with the label structure. We find the contrast between clean and random labels interesting. Standard supervised learning experiments usually use the original labels. Looking only at that setting, one might conclude that PLN offers little advantage over LN. In Table 3, replacing LN with PLN raises the reported mean clean-test accuracy from 71.6% to 72.3%, whereas the reported mean fitting accuracy on 20,000 randomly labeled examples rises from 48.2% to 75.3%. These configuration-grid summaries reveal a substantial difference in empirical fitting capacity that is less apparent on the standard classification task.
This does not imply a comparable improvement in generalization. PLN's empirical performance may depend on the data distribution and vary across tasks. The grouping principle also has practical relevance: the grouped RMS normalization cores in RetNet [1] and Mamba-2 [2] correspond to position-wise PLS, as clarified in Remark 5.1 of the revised manuscript.
References are listed at the end of Part 2.