- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a non-normalized residual block, named noNorm, which replaces normalization with explicit magnitude control through element-wise clipping of activations and weights. The paper notes that normalization layers in models can lead to costly reduction operations, introduce outliers, and hinder low-precision quantization. It argues that a bounded numerical range helps stabilize training. noNorm replaces both activation and weight normalization. The experiments in the paper show that this method enables controllable and consistent learning rates across each layer, resulting in activation and weight distributions free of outliers. Evaluations on the FineWeb-EDU language model and LibriSpeech speech recognition tasks showed that models without normalization matched or exceeded the performance of strong pre-norm baselines while exhibiting more controlled internal dynamics. Among all compared models, noNorm demonstrated the strongest quantization robustness.
Critical strengths
- It is a very good topic as during my research experience, I also recognize the Normalization layer could play a very important role in training.
-
- The results obtained after training with noNorm demonstrate strong quantization robustness. While other baselines largely fail or degrade in performance, noNorm essentially maintains its original performance with 4-bit weights. 3 . Experiments show that, after removing normalization, the method described in this paper achieves performance comparable to that of the pre-norm baseline, allowing the model to be scaled to deeper layers without modifying hyperparameters.
Critical weaknesses
- This paper states that normalization can lead to issues such as computational overhead, gradient vanishing, and excessively large logits; however, no experiments or references are cited to support these claims. As such, it is unclear whether these challenges actually have a real impact.
- The core contribution of this paper is the use of KKT clipping to constrain outliers. How does this differ from standard clipping? The paper does not discuss direct clipping and fails to fully demonstrate the benefits of constrained optimization.
- The paper lacks comparisons with other normlization-free methods.
Questions for authors
The content discussed in this paper is very interesting, particularly the explanation that removing normalization (noNorm) allows the model to maintain its original performance even after per-tensor quantization. However, I have some questions regarding the paper’s premise and experimental aspects that I would like to have answered.
- In the Introduction, the authors mention that normalization incurs computational overhead and causes gradients to decay in deep layers. Could you provide additional experiments or references to address this issue? Does noNorm indeed resolve these problems?
- The paper discusses constrained optimization, which leads to KKT clipping. However, the main text does not clarify the difference between KKT clipping and standard clipping. I hope you could include experiments to compare whether the differences are significant.
- Regarding the experimental setup, the authors used Transformer-class models. Would similar issues arise with Vit-class models, and would the authors’ method remain effective in those cases?
- In the “Related Work” section, the authors discuss other non-normalization methods, such as Fixup and ReZero; however, the main table does not compare these baselines—it only compares methods like pre-norm. I would appreciate the inclusion of some comparative experiments.
In summary, the research topic addressed in this paper is very interesting. The paper would be even more complete if the experiments and reasoning were further elaborated.
Thank you for the authors' responses. My concerns have been adequately addressed, and I will maintain my original score of 3 (Weak Acceptance).
2 comments
Thank you again for your feedback, we have now uploaded a second revision and we think that the article benefits from the added data and discussion. We hope the new content addresses all your concerns, in particular:
- We added an explicit comparison with Fixup, NF-ResNet, BHyT, DyT and Derf to our results section
- We added references on the effects of normalization on gradients and training, as well as to other normalization-free methods mentioned by reviewers
- We clarified the difference between standard clipping (projection of values to the feasible region) and additional gradient projection in the main text. We reworked the KKT framing and now consistently call our modified clipping variant ‘GP-clip’ (gradient-projection clip), with unified naming in the appendix.
- We added the requested baseline of GPT++ with RMSNorm plus weight and residual-output clipping. From this, it becomes clear that the quantization benefits are in part attributable to the clipping, and in part to the noNorm model design.
- In Tables 3 and 11, additional per-channel RTN and GPTQ results are now shown for all the key configurations and ablations. While results for all models improve, noNorm remains significantly better.
- We have now included the full performance measurements in the latest revision (see Appendix L) and note the current gain in the main text.
Thank you for your review and feedback.
[W1, Q1] Issues of normalization: The issues of normalization have been studied and analyzed extensively in prior work, e.g.
We will add appropriate references.
[W2, Q2] Framing of KKT clipping: The interesting question for us was to find the right perspective on clipping in order to reason about its impact and best way of application. The KKT problem primarily formalizes clipping in the context of the optimization problem. We show results for applying the clipping operator to weights in different ways in Figure 8. Practically, modifying gradients versus just forcing values results in slightly smoother histograms, while performance is comparable. Theoretically, value forcing is a slightly cruder approximation of solving the constrained optimization problem.
[W3, Q4] Comparison to other normalization-free methods: We found that none of the prior work was competitive. Please also see our comment above in the collective response. We will add the comparison to other normalization-free methods to the revised paper.
[Q3] Application to Vit-class models: We agree that applying weight clipping to ViTs would be an interesting application. We focus on LLMs in this work, but since the architectures are very similar, we expect extending the same methods to ViT to be promising future work.