acceptodds
Under review as a conference paper at ICLR 2027

Normalization-Free Residual Networks by Bounding Weights and Residuals

Abstract

Normalization layers are widely used for stabilizing deep residual networks, yet they incur computational overhead and obscure training dynamics. We propose a stable, normalization-free residual block rooted in constrained nonlinear optimization that replaces traditional activation normalization with an inexpensive, element-wise clipping operation on the output of each residual branch. We extend this framework to weights, combining it with a parameterization that enables a single global constant to bound the magnitudes of both activations and weights. This parameterization additionally results in controllable and uniform effective learning rates across all layers. Compared to networks that do not use clipping, our approach yields activation and weight distributions that are remarkably smooth and free of outliers, enabling direct post-training quantization of weights to INT4 with only a minor loss degradation. Our experiments demonstrate that this approach successfully eliminates the need for both activation and weight normalization, offering a simple and principled foundation for training deep residual networks.

Primary area:
generative models
Keywords:
architecture research, constrained optimization, language modeling, transformers, quantization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6YzAOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a non-normalized residual block, named noNorm, which replaces normalization with explicit magnitude control through element-wise clipping of activations and weights. The paper notes that normalization layers in models can lead to costly reduction operations, introduce outliers, and hinder low-precision quantization. It argues that a bounded numerical range helps stabilize training. noNorm replaces both activation and weight normalization. The experiments in the paper show that this method enables controllable and consistent learning rates across each layer, resulting in activation and weight distributions free of outliers. Evaluations on the FineWeb-EDU language model and LibriSpeech speech recognition tasks showed that models without normalization matched or exceeded the performance of strong pre-norm baselines while exhibiting more controlled internal dynamics. Among all compared models, noNorm demonstrated the strongest quantization robustness.

Critical strengths
  1. It is a very good topic as during my research experience, I also recognize the Normalization layer could play a very important role in training.
    1. The results obtained after training with noNorm demonstrate strong quantization robustness. While other baselines largely fail or degrade in performance, noNorm essentially maintains its original performance with 4-bit weights. 3 . Experiments show that, after removing normalization, the method described in this paper achieves performance comparable to that of the pre-norm baseline, allowing the model to be scaled to deeper layers without modifying hyperparameters.
Critical weaknesses
  1. This paper states that normalization can lead to issues such as computational overhead, gradient vanishing, and excessively large logits; however, no experiments or references are cited to support these claims. As such, it is unclear whether these challenges actually have a real impact.
  2. The core contribution of this paper is the use of KKT clipping to constrain outliers. How does this differ from standard clipping? The paper does not discuss direct clipping and fails to fully demonstrate the benefits of constrained optimization.
  3. The paper lacks comparisons with other normlization-free methods.
Questions for authors

The content discussed in this paper is very interesting, particularly the explanation that removing normalization (noNorm) allows the model to maintain its original performance even after per-tensor quantization. However, I have some questions regarding the paper’s premise and experimental aspects that I would like to have answered.

  1. In the Introduction, the authors mention that normalization incurs computational overhead and causes gradients to decay in deep layers. Could you provide additional experiments or references to address this issue? Does noNorm indeed resolve these problems?
  2. The paper discusses constrained optimization, which leads to KKT clipping. However, the main text does not clarify the difference between KKT clipping and standard clipping. I hope you could include experiments to compare whether the differences are significant.
  3. Regarding the experimental setup, the authors used Transformer-class models. Would similar issues arise with Vit-class models, and would the authors’ method remain effective in those cases?
  4. In the “Related Work” section, the authors discuss other non-normalization methods, such as Fixup and ReZero; however, the main table does not compare these baselines—it only compares methods like pre-norm. I would appreciate the inclusion of some comparative experiments.

In summary, the research topic addressed in this paper is very interesting. The paper would be even more complete if the experiments and reasoning were further elaborated.

Thank you for the authors' responses. My concerns have been adequately addressed, and I will maintain my original score of 3 (Weak Acceptance).

2 comments

Thank you for your review and feedback.

[W1, Q1] Issues of normalization: The issues of normalization have been studied and analyzed extensively in prior work, e.g.

We will add appropriate references.

[W2, Q2] Framing of KKT clipping: The interesting question for us was to find the right perspective on clipping in order to reason about its impact and best way of application. The KKT problem primarily formalizes clipping in the context of the optimization problem. We show results for applying the clipping operator to weights in different ways in Figure 8. Practically, modifying gradients versus just forcing values results in slightly smoother histograms, while performance is comparable. Theoretically, value forcing is a slightly cruder approximation of solving the constrained optimization problem.

[W3, Q4] Comparison to other normalization-free methods: We found that none of the prior work was competitive. Please also see our comment above in the collective response. We will add the comparison to other normalization-free methods to the revised paper.

[Q3] Application to Vit-class models: We agree that applying weight clipping to ViTs would be an interesting application. We focus on LLMs in this work, but since the architectures are very similar, we expect extending the same methods to ViT to be promising future work.

Thank you again for your feedback, we have now uploaded a second revision and we think that the article benefits from the added data and discussion. We hope the new content addresses all your concerns, in particular:

  • We added an explicit comparison with Fixup, NF-ResNet, BHyT, DyT and Derf to our results section
  • We added references on the effects of normalization on gradients and training, as well as to other normalization-free methods mentioned by reviewers
  • We clarified the difference between standard clipping (projection of values to the feasible region) and additional gradient projection in the main text. We reworked the KKT framing and now consistently call our modified clipping variant ‘GP-clip’ (gradient-projection clip), with unified naming in the appendix.
  • We added the requested baseline of GPT++ with RMSNorm plus weight and residual-output clipping. From this, it becomes clear that the quantization benefits are in part attributable to the clipping, and in part to the noNorm model design.
  • In Tables 3 and 11, additional per-channel RTN and GPTQ results are now shown for all the key configurations and ablations. While results for all models improve, noNorm remains significantly better.
  • We have now included the full performance measurements in the latest revision (see Appendix L) and note the current gain in the main text.
Reviewer_cFDQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a normalization free residual block based on constrained nonlinear optimization where they replace the normalization with element-wise clipping both for activations and weights, using a parametrization that enables a single global constant to bound the magnitudes of activations and weights. This way of clipping results in activations and weight distribution that are smooth, has less outliers and therefore direct post training quantization do not result in major loss degradation which is interesting.

Critical strengths
  1. Element wise clipping with a single global constant replacing the normalization layer is easy to implement.
  2. The quantization results in Table 3 look promising.
  3. The RMS skip path activation grows severely over depth for Pre-LN GPT++ whereas it grows slowly for noNorm resulting in less outliers. This intuitively should help quantization.
  4. It covers autoregressive LM, block diffusion, and ASR, and compares against both nGPT and a forced-weight-normalization variant.
  5. It admits that full-precision loss is not much better than GPT++.
Critical weaknesses
  1. The reviewer thinks that the full-precision gains are overstated. “Matches or surpasses” is not supported. For instance in Fig. 2, noLN-FWN (2.430) beats noNorm (2.439), and the 600M runs are single-seed with no noise estimate at that scale. The reviewer believes that “matches” would be the accurate claim.
  2. The clipping-vs-FWN comparison is confounded by weight decay. In the paper noNorm uses weight decay (0.002) while noLN-FWN uses none, so the comparison does not cleanly isolate clipping from normalization. The effect of decay within FWN, and its impact on quantizability is unexplored.
  3. Missing baseline Zhu et al. (2025) and incomplete tanh study. “General case” is undefined andZhu et al. (2025) is not run as a baseline. The tanh ablations test tanh only as a standalone clamp, not combined with the inward-pointing gradient rule, so tanh’s weakness on weights cannot be attributed to the function versus its gradient treatment.
  4. Efficiency is motivated but never measured. Removing normalization overhead is a stated motivation, yet no wall-clock, memory, or throughput numbers are given, at training or inference, despite the method adding per-step clipping, a custom backward, and an effective-weight computation.
Questions for authors
  1. In Fig. 2, among the full precision versions, since noLN-FWN actually beats noNorm in terms of loss, and the runs are single seed, it is better to use “performance matches” instead of “performance matches or surpasses”.
  2. I think the difference in performance is the difference in how the optimization process progresses on the loss landscape. A lower weight decay is good for noNorm since it deals with small weight values compared to GPT++ where a higher weight decay is used. For a fair comparison, authors have considered a model identical to noNorm but with forced normalization instead of weight clipping. However, FWN does not use weight decay whereas weight clipping does. Without weight decay, FWN may have many weights closer to the boundary close to ±1\pm 1. So the reviewer wants to know, how does noLN-FWN behave with weight decay applied after renormalization? Does it improve quantizability, and does it compromise the effective-learning-rate uniformity you claim for the normalized variant?
  3. You state that the approach of Zhu et al. (2025), clamping activations with tanh at the residual input, “was not sufficient in the general case,” but “general case” is not defined. Could you specify what fails (which tasks, depths, or scales), and include DyT as a direct baseline rather than only citing it? Relatedly, your tanh ablations (Tables 9 and 11) use tanh as a standalone clamp. tanh already saturates smoothly, so its gradient decays near the bound. Have you tried combining tanh with the inward-pointing gradient rule of Eq. 5, and does the KKT-style correction change the picture, particularly the weight-clamping gap where tanh underperforms (+0.0502)? More broadly, is the inward-gradient idea even well-defined for a smooth saturating function, or specific to hard clipping?
  4. What is the actual wall-clock, memory, and throughput difference vs pre-LN? Removing normalization is a stated motivation, but the method adds per-step clipping, a custom backward, and an effective-weight computation. Is it net faster at training and at inference?
4 comments

Thank you very much for your careful review.

[W1, Q1] Positioning: The statement that the noNorm model matches or surpasses the baseline pre-LN model remains valid: noLN-FWN is a variant of noNorm that achieves a slight performance improvement at the cost of additional effort during training.

[W2, Q2] Combination of FWN and weight decay: Weight decay does not have a lasting effect when combined with Forced Weight Normalization, because the normalization step immediately undoes the scaling introduced by weight decay. The corresponding results are therefore expected to match.

[W3, Q3] Missing baseline: Both our own experiments and prior work indicate that DyT does not achieve competitive performance. Please also see our comment above in the collective response. If permitted, we will add the corresponding results to the appendix. Combination of tanh and inward pointing gradient rule: Tanh cannot be combined with the inward-pointing gradient rule, because tanh is a smooth bounded transformation rather than a clamping operation. Moreover, the gradient of tanh becomes small in the saturated regions: its derivative approaches zero for large absolute input values, i.e., when the activation is close to ±1. This is a fundamental issue of saturation.

[W4, Q4] Performance numbers: Both training throughput and inference latency depend on the execution environment. The paper does not make performance-gain claims; it only notes that the method can enable such gains, since it avoids expensive synchronization operations and relies instead on low-cost element-wise operations. We therefore refrain from reporting absolute performance numbers, as these would provide limited insight, see the common introductory response.

[Q1] Influence of different random seeds: The influence of different random seeds is evaluated in Table 7.

Thanks to the authors for the responses. Below are my comments.

  1. noLN-FWN is forced weight normalization proposed by Karras et al., 2024 which differs from noNorm as they use FWN instead of clipping which is novelty of this work. If by using FWN, the model acheives the similar/better performance, then the reviewer is worried about the novelty of the paper.
  2. The reviewer believes the answer addresses the standard ordering (decay, then renormalization), where the renormalization indeed cancels the shrinkage. The question concerned the reverse order: renormalization first, then weight decay applied as the final step. In that case nothing undoes the shrinkage, so the row settles below the FWN target and its magnitude becomes governed by λ\lambda rather than pinned by the normalization.
  3. Thank you for comparing with DyT. I understand that gradient pointing inward will not be applicable to tanh due to gradient saturation problem for tanh.
  4. I appreciate that absolute numbers are environment-dependent. However, the abstract states that normalization layers “incur computational overhead” and the introduction refers to “costly reduction operations,” which reads as a performance motivation. A single fixed-hardware comparison (same GPUs, same batch size, pre-LN vs noNorm) would support that motivation. Could you report training throughput and inference latency for one such setup, noting the method’s own added costs (per-step clipping, custom backward, effective-weight computation)?

Thank you for your reply.

  1. We see the novelty in our work in our normalization-free architecture that results from the combination of weight and activation clipping and thus proves itself particularly suitable for quantization due to its well-controlled weight distributions. Note that noLN-FWN is itself a novel architecture: it is noNorm with FWN instead of weight clipping (with residual clipping, query scale, layer-input scaling etc). EDM2 was a convolutional image generation model. However, noLN-FWN does not exhibit the same beneficial properties for quantization as our more distinct noNorm with clipping. We include the comparison to show that weight clipping behaves similarly to weight normalization when it comes to training dynamics, while it is cheaper and simpler, as can be seen in the weight and activation trajectories over training, and in the shape of the loss curve.
  2. With FWN (see the Karras EDM2 paper), normalization of weights actually happens in both the forward pass and post-optimization steps, so weight scale and thus weight decay are order-independent. Note that even when removing this double normalization, the weight decay is multiplied by the LR schedule and thus approaches zero over time, which would also push weight vectors back to unit norm.
  3. Indeed, we hope that the added comparison in our latest revision addresses the concerns.
  4. The reviewer is correct that this wording might mislead readers that the method is targeting improved speed; we will reword the introduction to avoid suggesting this. In our top-level reply we now added a measured throughput number: noNorm models are about 1% faster than GPT++ during training and inference at the 500M scale. While we do not consider these numbers representative of production settings, we can add them for completeness. In our setup, measurements show the per-step clipping and custom backward cost around 0.5% training speed. For inference, 1/sqrt(fan-in scale), query scale, and the output scale can be folded into weights for simplicity, and there are no weight clipping or custom backward operations. For training, effective-weight computation can be fused with linear layers into a single operation.

Thank you again for your feedback, we have now uploaded a second revision and we think that the article benefits from the added data and discussion. We hope the new content addresses all your concerns:

  • We added the requested baseline of GPT++ with RMSNorm plus weight and residual-output clipping. We now also show additional per-channel scaling and GPTQ results for all ablations in Table 6. From this, it becomes clear that the quantization benefits are in part attributable to the clipping, and in part to the noNorm model design.
  • In Tables 3, 6 and 11, additional per-channel RTN and GPTQ results are now shown for all the key configurations and ablations. While results for all models improve, noNorm remains significantly better.
  • We have now included the full performance measurements in the latest revision (see Appendix L) and note the current gain in the main text.

We hope that these explanations clarify our work and that you will consider re-evaluating your score. We remain available for any further discussion.

Reviewer_pZwcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to replace the normalization layers of a Transformer by applying element-wise clipping [−2.1,2.1][-2.1, 2.1] to the scaled output of the residual branch, as well as scaled weights. The paper also uses a depth-dependent input scaling (similar form as in Brock et al. (2021)) and a learnable per-head attention temperature (as in Henry et al., 2020, and Ryan, 2024). These design choices are supported by large scale (in terms of research) experiments, which show that the resulting model matches normalized baselines in full precision and is more robust when applied to low-bit weight quantization.

Critical strengths

(1) The method is simple. A single constant bounds both the residual-branch updates and the weights, so the scale of every quantity in the network is bounded by construction instead of being rescaled at run time. Unlike DyT, which places a bounded function where the normalization layer was, the paper clips the scaled output of each residual branch before it is added to the skip path. With this design the model removes all normalization layers and still matches normalized baselines in full precision.

(2) The paper shows that bounding each weight entry makes a normalization-free Transformer robust to 4-bit post-training rounding.

(3) Experiments and the ablations are extensive. For example, the core quantization result is clear and large. Table 10 shows that disabling weight clipping raises the W4 increase from +0.025+0.025 to +1.874+1.874.

Critical weaknesses

(1) Positioning. Bounded replacements of normalization (DyT, Derf, BHyT, EDM2, NF-ResNet) and outlier-free pretraining for quantization (OSP, Park et al. 2025; Bondarenko et al. 2023; the OP block of He et al. 2024) are closely related; the last three are not cited, and DyT is compared only qualitatively. The weight-side KKT clipping is projected gradient on a box constraint, and the parameterization follows EDM2/u-μ\muP. The novelty claim of the work is a bit vague and could be made more clear.

(2) Attribution. It is not shown whether the quantization gain comes from removing normalization or from the bounds.

(3) Quantization protocol. Only per-tensor RTN is reported, which is the setting least favorable to GPT++. Per-channel RTN and GPTQ results, and downstream accuracy after quantization, are needed to judge the practical gain.

(4) The efficiency claims in the abstract and conclusion are not supported by any throughput or latency measurement.

(5) Internal inconsistencies: the 150M comparison is described three ways (Sec. 4.4, App. F.3, Table 15); the skip RMS growth is 400$\times$ in Sec. 4.2 but about 13$\times$ in the Figure 19 caption; App. F.7 says adding pre-LN does not help while Table 14 shows −0.0144-0.0144; the depth scaling is 1/l+11/\sqrt{l+1} in the text and 1/2l+11/\sqrt{2l+1} in Algorithm 1; Eq. (7) gives s=dins=\sqrt{d_{\text{in}}} while App. E uses 50; the same clipping operator has six names.

(6) The paper presents its training procedure as "rooted in constrained nonlinear optimization" (abstract) and derives Eq. (5) from KKT theory (Sec. 3.1), but the derivation does not go beyond introducing the term "KKT-constrained optimization". For the weights, Eqs. (5) and (8) amount to standard projected gradient descent on a box, and under SGD Eq. (5) is already implied by Eq. (8). For the activations, Eq. (5) is a one-sided straight-through estimator. KKT conditions characterize optimal points but do not by themselves yield either rule, so the framing should be presented as motivation rather than derivation.

Questions for authors

(1) What is the W4 increase of GPT++ with RMSNorm kept plus weight and residual-output clipping?

(2) How do noNorm and GPT++ compare under per-channel RTN and GPTQ?

(3) Can you add numerical comparisons with DyT and the OP block, including kurtosis and max/median activation ratios?

(4) What fraction of weights sits on the boundary at the end of training for one-sided vs hard clipping?

(5) Please report training throughput and inference latency.

5 comments

Thank you for your careful review and valuable feedback.

[W1] Positioning: Please see summary common introductory response. If allowed, we will add a paragraph with the numbers we evaluated, noting that these methods did not prove competitive under our experimental setup.

[W1] Novelty of clipping/KKT: EDM2 still has normalized weight matrices and uses forced weight normalization. We further simplify to component-wise clipping, for which we discover the nice property that resulting weight distributions are also more uniform and easier to quantize. This is a new contribution to the field.

[W2, Q1] Attribution of the quantization gains: Table 10 compares validation loss under quantization across different model variants, including configurations with and without weight and activation clipping. Without activation clipping, training diverges. Disabling weight clipping leads to a significant increase in validation loss under quantization. Improving a model’s quantization robustness by adding weight clipping is beyond the scope of our paper, which focuses on evaluating the necessity of normalization rather than the quantizability of arbitrary models.

[W3, Q2] Quantization protocol: Our evaluation of quantization behavior focuses on the worst-case quantization setting, namely round-to-nearest quantization without fine-tuning or additional scaling factors. In practical applications, the degradation of a pre-LN model can be reduced with more advanced techniques. Crucially, however, our norm-free model does not require them to achieve competitive results.

[W4] Efficiency claims: Synchronization and barrier operations incurred by normalization carry a certain overhead. We refrain from quantifying this overhead, or from providing a direct comparison with a normalization-based method, because the measured difference depends heavily on the execution environment and would provide limited insight. Neither the abstract nor the conclusion makes efficiency claims beyond this point; the element-wise operations used by our method are among the cheapest possible operations. See our comment above in the collective response.

[W5] Internal inconsistencies:

  • Different comparisons: Section 4.4 presents the main ablation study. It focuses on the most relevant aspects, which were selected based on the additional ablations in Appendix F3. Table 15 addresses a more specific question, namely whether the proposed approach is robust for deeper networks. These comparisons are complementary rather than inconsistent, as they evaluate different aspects of the method. We note that the results for the 150M-parameter model are not directly comparable to those for the 600M-parameter model, as they correspond to different model scales.
  • Mistake in caption: The skip RMS values in the caption of Figure 19 are incorrectly stated as relative to the embedding. In fact they are absolute values and we will correct this mistake.
  • pre-LN results in Table 14: Table 14 contains an ablation study of different model hyperparameters. The comparison showing an advantage for pre-LN corresponds to a suboptimal configuration for noNorm that does not have additional functionality for compensating attention query scales, and is included for completeness. Therefore, when comparing the noNorm model with the suggested hyperparameters against the best pre-LN configuration, our statement that “adding pre-LN does not help” remains correct.
  • Depth Scaling: This is an ambiguity in the common nomenclature of transformer architectures. Depending on the context, transformer layers denote function block groups (attn + MLP) or individual function blocks. We will clarify by using different letters: ‘B’ for (attn + mlp) and ‘L’ for individual blocks.
  • Derivation of factor 50: The explanation is given in line 1084: For the baseline initialized with 0.02 standard deviation, a factor of 50 is required when we reparameterize to a standard deviation of 1. The confusion may have arisen from the fact that the baseline does not use fan-in based initialization. This is in appendix C, but we will add an explicit note to appendix D as well for clarity.
  • Six names for clipping: what ‘six names’ is the reviewer referring to? If it is the different clipping operations, such as hard clipping, one-sided clipping, tanh clipping, …, these are different variants of the clipping operation and are compared for evaluating their performance. We would appreciate any clarification regarding this concern.

(continued)

[W6] Framing of KKT formulation: Our main concern in 3.1 is to point out that training with bounded weights is a constrained optimization problem that comes with many mathematical tools. The KKT formulation makes the objective both concise and explicit. Our goal is not to derive new mathematical theory with this paper, but rather to follow the curiosity of which gradients can be informative even in the face of hard constraints, since gradients play such a central role in deep learning. This is particularly interesting because modern optimizers are not typically simple SGD, therefore we expect the differential perspective to remain key in developing the strongest possible optimization algorithms going forward.

[Q3] Comparison to DyT and OP block: Please see summary common introductory response. If allowed, we will add a paragraph with the numbers we evaluated, noting that these methods did not prove competitive under our experimental setup.

[Q4] Weight histograms: The fraction of weights that sits directly on the boundary is usually small compared to the overall sample mass of the histograms (see Figure 7). For reasonably finely resolved histograms, the outer bins are typically no larger than 2-3x the maximal bin. We can add the exact respective numbers to the figure.

[Q5] Performance numbers: Both training throughput and inference latency depend on the execution environment. The paper does not make performance-gain claims; it only notes that the method can enable such gains, since it avoids expensive synchronization operations and relies instead on low-cost element-wise operations. We therefore refrain from reporting absolute performance numbers, as these would provide limited insight, see the common introductory response

I appreciate the clarification that the KKT formulation serves as motivation rather than derivation. However, I cannot find this change in the revision.

Two of my questions remain open. First, the attribution question: the W4 increase of GPT++ with RMSNorm kept plus weight and residual-output clipping would show whether the quantization gain comes from removing normalization or from the bounds; this is central to the paper's message rather than out of scope. Second, per-channel RTN or GPTQ results are still needed to judge the practical gain, since per-tensor RTN is the setting least favourable to GPT++. I also share Reviewer cFDQ's concern that noLN-FWN matches or exceeds noNorm in full precision.

If a revision addressing these points is provided, I will keep my score.

Comments on the paper

3 comments

Dear Reviewers,

We sincerely thank all reviewers for their thoughtful engagement with our manuscript and for providing detailed, constructive feedback.

We are very encouraged by the positive reception of our key contributions, particularly regarding:

  • Quantization Robustness: Enhanced resilience to quantization errors (pZwc, cFDQ, 6YzA).
  • Implementation Simplicity: Straightforward and efficient architectural implementation (pZwc, cFDQ).
  • Clearer Dynamics: More controlled and predictable network dynamics without normalization layers (cFDQ, 6YzA).
  • Empirical Rigor: Insightful experimental validation and comprehensive ablations (pZwc, cFDQ).

Below, we summarize our plan to address the key points and recommendations raised in the reviews, which will be incorporated into the final revision. We trust these explanations clarify our work and hope you will consider re-evaluating your scores. We remain available for any further discussion.

Positioning with respect to related work and choice of baseline (pZwc, cFDQ, 6YzA)

The particular benefits that we describe for our normalization-free architecture result from the combination of weight and activation clipping. In preliminary evaluations of the related normalization-free approaches that partially realized such architectures before (as raised by the reviewers), we found that previous work that just replaces normalization layers by alternative functions (DyT, Derf, BHyT), different initialization (Fixup) or additional scale factors (ReZero, NF-ResNet) without controlled weight ranges consistently degraded performance. Aspects of these previous works (DyT in Table 9, Fixup in Table 14, ReZero in Table 13) are already reflected in our ablations, and we will clarify the respective correspondences to previous work. We suggest adding a comparison with prior work on normalization-free networks in their original form in the Appendix.

Model150M / 10B600M / 100B
Fixup3.1122.545
NF-ResNet (ReLU)2.954training
BHyT2.9452.465
DyT2.9362.479
Derf2.9212.486
noNorm2.8702.439
GPT++2.8762.450

Additional Table: Validation loss of different normalization-free models (best / 2nd best)

Clarify efficiency claims (pZwc, cFDQ, 6YzA)

We will clarify that our algorithm is more efficient because of simpler data dependencies: it removes the synchronization and barrier operations that normalization requires and relies only on element-wise operations, which are among the cheapest available. All scaling factors can be folded into weight matrices for inference. Building production implementations is beyond the scope of our work. Hence, the article does not claim measured speedups. In our training runs with standard pytorch, noNorm is about 1% faster than GPT++. Using optimized implementations this can likely be improved. We note that the main benefit of noNorm is not training speed but better suitability for quantization.

Quantization comparisons (pZwc)

As the reviewers note, the quantization results are very promising, but the main contribution of our research is a controlled, normalization-free architecture. We deliberately evaluate the worst-case setting: round-to-nearest (RTN) without fine-tuning or extra scaling factors, which results in the most efficient inference kernels. RTN is the most transparent way to show how much error reduction comes from the architecture itself. More sophisticated quantization methods reduce the error for all models, including pre-LN, but the same trends hold. The ablations in Figure 5 and Table 10 attribute the gains: without activation clipping, training diverges, and disabling weight clipping substantially increases validation loss under quantization.

Clarify impact of normalization on gradients (6YzA)

The issues of explicit normalization have been extensively studied (e.g., https://proceedings.mlr.press/v119/huang20f.html, arXiv:2502.05795, arXiv:2601.09719). We will more explicitly note this in the related work.

Impact of KKT framing (pZwc, 6YzA)

We will clarify the difference between “standard clipping” (projection of values to the feasible region) and additional gradient projection in the main text. To reason about optimization under clipping, we need a concise formalization, and the KKT view makes this objective explicit in the form of constrained optimization. We see our main contribution in the resulting simple architecture based on clipping, which has useful side effects such as improved quantizability. In practice (Figure 8), modifying gradients via the inward-pointing rule and simply forcing values to the feasible region give comparable performance, with the gradient-based approach producing slightly smoother weight histograms. Thus, value forcing still works as a cruder approximation of the constrained optimization for deep learning in practice, with less informed gradients.

(continued)

Architectural ablations and variants (pZwc, 6YzA)

The reviewer acknowledged our extensive ablations, including clipping variants (Appendix F3), depth scaling (Table 15), and seed variance (Table 7). The attribution of the quantization benefits is currently most apparent in the ablations in the appendix (Table 10), following the suggestions of the reviewers we will also add it explicitly to the main text. Hybrids of our proposed architecture and a pre-LN architecture are interesting future work. Table 14 shows that simply adding normalization layers to our proposed noNorm-architecture does not improve the validation loss. Extending the approach to vision transformers is promising future work.

AuthorsOpenReview ↗

Summary of revision changes

Dear Reviewers,

thank you for your feedback on our proposed changes. We have just uploaded a revision. In the following, we provide a summary of the applied changes as proposed in our initial answer, as well as additional changes that address some of today’s discussion. We also note the changes we still intend to apply. The latest revision now includes all of the following changes:

Changes as proposed in the rebuttal

  • Added an explicit comparison with Fixup, NF-ResNet, BHyT, DyT and Derf to our results section
  • Added references on the effects of normalization on gradients and training, as well as to other normalization-free methods mentioned by reviewers
  • Clarified the difference between standard clipping (projection of values to the feasible region) and additional gradient projection in the main text. We reworked the KKT framing and now consistently call our modified clipping variant ‘GP-clip’ (gradient-projection clip), with unified naming in the appendix
  • Moved the table for quantization attribution to the main text as Table 6
  • Added discussion of the impact of basic clipping vs. gradient projection on the final histograms to the results section in the main text. Also added the exact boundary-weight fractions to the weight histograms in Figure 8
  • Fixed the Figure 19 caption: the skip RMS values are absolute, not relative to the embedding.
  • Renamed the depth-scaling indices: B for (attn + MLP) blocks and L for individual subblocks.
  • Add an explicit note on the factor of 50 (the baseline's 0.02 initialization versus unit-variance reparametrization) to appendix D.
  • Clarified the claim that adding pre-LN does not improve performance in Appendix F.7
  • Clarified that noLN-FWN has lower full-precision loss, but is much more difficult to quantize. The limitations/future work explicitly address this.

Additional changes

  • The main quantization table now includes per-channel RTN and GPTQ results. This is a new experiment that addresses pZwc's Q2.
  • Swapped the scaling and clipping ablation studies for logical ordering

Items that still need to be addressed are now addressed in our second revision

  • In accordance with the latest reply of reviewer pZwc, further clarify attribution of W4 improvements with additional numbers for GPT++ with RMSNorm plus weight and residual-output clipping
  • Further clarified ablations and attribution of benefits in the results section
  • Added more GPTQ numbers to ablations (see Appendix)
  • Clarify practical efficiency gains in test setup as noted in the updated initial official comment to address reviewer cFDQ’s concerns

We hope that these changes address the reviewer’s concerns. Each reviewer’s questions are addressed in detail in our individual responses. If you find that we have addressed your concerns, we would kindly ask you to consider revising your score, and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.