acceptodds
Under review as a conference paper at ICLR 2027

NORMALISATION CHOICES IN DECODER-ONLY TRANSFORMERS ARE REGIME-DEPENDENT

Abstract

Three claims about normalisation in Transformers circulate as folk theorems: Pre-LN is stable and Post-LN is not; RMSNorm is a free drop-in for LayerNorm; and QK-Norm stabilises attention. They are rarely tested against each other in one codebase, on one corpus, with shared seeds. We do so with 175 runs of a GPT-2-style decoder (21.3M non-embedding parameters at depth 12) on FineWeb-Edu, comparing six variants arranged as two single-axis chains rooted at Pre-LN and Post-LN, five of them across four depths, a learning-rate sweep, reruns at the selected rates, a gradient-clip stress test and per-layer gradient diagnostics. The apparent answer to “which variant is better?” changes materially with the optimisation regime. At a shared learning rate the Post-LN penalty is +0.0495 nats; at each variant’s sweep-selected learning rate it is +0.1442 nats, a 2.9-fold change from a single hyperparameter. A single-architecture chain attributes the Post-LN rescue cleanly: swapping LayerNorm for RMSNorm moves the loss by +0.0001 nats, while adding QK-Norm moves it by −0.0504 nats and lifts the Post-LN learning rate ceiling threefold. That headroom is depth-limited: at depth 48 five variants converge to within 0.021 nats, but at 10−3 both Post-LN variants—QK-Norm included—collapse whether gradient clipping is relaxed 100× or left at its default, so of the two variables crossed the learning rate determines the outcome and the clip does not. The failures are not blow-ups: collapsed runs settle +0.020 nats from the corpus unigram entropy, which we measure directly. We release all 175 per-run artefacts with a pipeline that regenerates every number here.

Primary area:
optimization
Keywords:
layer normalization, Pre-LN, Post-LN, RMSNorm, QK-Norm, transformer training stability, learning rate sensitivity, gradient clipping, controlled ablation, language model pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3RY6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Using 175 training runs, this paper compares normalization placement, LayerNorm versus RMSNorm, and QK normalization in decoder-only transformers. Results show that performance rankings and stability depend on learning rate, depth, and training regime.

Critical strengths
  • Single-axis comparisons help distinguish the contributions of RMSNorm and QK normalization within each residual layout.
  • Seed pairing, closure calibration, and detailed run records improve confidence in the reported comparisons.
Critical weaknesses
  • Novelty is modest. Most findings refine established observations about normalization and learning-rate sensitivity. Evidence from one width and one corpus leaves the broader significance of these refinements unclear.
  • Learning rates are selected using one seed at 5k steps, then used to compare architectures at 20k steps. Some selections lie at the grid boundary or are barely distinguishable from neighboring rates. These experiments establish rankings under the chosen rates, but not rankings after adequate tuning.
  • Loss close to unigram entropy does not demonstrate context-independent predictions. Reduced gradient norms also do not establish that a model cannot escape the plateau. Calling this a warm-up trap requires direct measurements of prediction distributions and interventions on warm-up or recovery.
  • Cross-block comparisons remain difficult to interpret. Identical configurations differ by up to 0.018 nats for reasons that cannot be reconstructed, and depth-48 parity has not been checked under the canonical final-norm convention.
  • Readability suffers from frequent block identifiers, implementation-history details, and qualifications. Readers must repeatedly consult the appendices to determine which comparisons are controlled and which conclusions remain provisional.
Questions for authors

Typos

Appendix G states that every collapse is complete by step 250, despite the unsettled run documented in Appendix B. Add the exception.

Reviewer_mFGQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical investigation of how normalization choices interact with optimization regimes in decoder-only Transformers. The authors examine three architectural dimensions: normalization placement (Pre-LN versus Post-LN), normalization function (LayerNorm versus RMSNorm), and query-key normalization (QK-Norm).

Using a GPT-2-style Transformer trained on FineWeb-Edu, the study conducts 175 runs covering learning-rate sweeps, different network depths, gradient-clipping thresholds, and gradient diagnostics.

The main findings are:

The performance gap between Pre-LN and Post-LN depends substantially on learning-rate selection, increasing from 0.0495 nats under a shared learning rate to 0.1442 nats under individually selected learning rates.

QK-Norm, rather than replacing LayerNorm with RMSNorm, accounts for most of the observed performance improvement in the Post-LN configuration at depth 12.

The learning-rate tolerance gained through QK-Norm does not persist at depth 48, where Post-LN configurations exhibit training collapse at higher learning rates regardless of the two tested clipping thresholds.

Collapsed models converge to a finite-loss plateau close to the empirical unigram entropy of the validation corpus.

The paper also provides training artifacts, per-run configurations, and an analysis pipeline intended to reproduce the reported results.

Critical strengths
  1. Carefully controlled ablations provide credible evidence for the contributions of individual normalization components.

A notable strength is the use of single-axis comparison chains to separate normalization placement, normalization function, and QK-Norm. This design addresses an important limitation of comparisons in which multiple architectural modifications are introduced simultaneously. In particular, Table 3 provides a relatively clean decomposition of the Post-LN improvement: replacing LayerNorm with RMSNorm produces a negligible change (+0.0001 nats), whereas subsequently introducing QK-Norm reduces validation loss by approximately 0.0504 nats.

The use of paired random seeds, shared data ordering within experimental blocks, and explicit calibration of the final-normalization implementation strengthens the credibility of these comparisons. This is consequential because the isolated contribution of QK-Norm within a Post-LN architecture is one of the paper's most distinctive empirical results.

  1. Detailed failure-mode characterization and reproducibility efforts add value beyond conventional performance comparisons.

The paper goes beyond reporting final validation losses by examining learning-rate sensitivity, gradient norms, collapse trajectories, and the relationship between the observed loss plateau and corpus unigram entropy.

The observation that unstable Post-LN runs can converge to a finite, nearly stationary high-loss regime rather than producing NaNs is practically relevant for diagnosing unsuccessful training runs.

Furthermore, the release of artifacts for 175 runs, together with the authors' explicit discussion of cross-block differences and implementation changes, facilitates verification and further investigation.

These efforts increase confidence in the reported observations, even though the underlying collapse mechanism is not fully established.

Critical weaknesses
  1. Limited novelty and experimental scope substantially weaken the significance and generalizability of the findings.

The central observations are closely related to conclusions already established in prior work. Learning-rate sensitivity and the stabilizing effects of QK normalization have been investigated by Wortsman et al. (2024) and Rybakov et al. (2024). Differences in the optimization behavior of normalization placements have been examined at substantially larger scales by Kim et al. (2025). Moreover, the relationship between Post-Norm collapse, vanishing gradients, and token-frequency predictions has been investigated by Wang et al. (2026). The principal additional contribution appears to be the controlled isolation of RMSNorm and QK-Norm within a Post-LN configuration. Although useful, this contribution is experimentally demonstrated within a narrow setting.

All experiments use a fixed model width of 384, a GPT-2-style architecture, a context length of 512, and a single training corpus. Increasing network depth does not substitute for investigating changes in model width, parameterization, and data distribution. In addition, the absence of experiments using modern large-scale decoder architectures limits the practical relevance of the conclusions.

This limitation is particularly consequential because the paper's central claim is that normalization behavior is regime-dependent. Without validation across substantially different architectural or training regimes, it remains unclear whether the observed interactions constitute generalizable principles or properties of this specific experimental setup.

Given the extensive existing literature and the absence of a new normalization method or theoretical result, I find the current empirical evidence may be insufficient to establish a sufficiently significant advance.

  1. Important optimization-regime conclusions are stronger than the experimental controls and mechanistic evidence currently justify.

First, the learning-rate sweep uses only four candidate values and one seed per configuration, with learning rates selected from 5,000-step experiments and subsequently evaluated over 20,000 steps. These experiments also employ different warm-up durations. Consequently, the selected learning rates are not necessarily representative of the optimal configurations at the final training horizon. The reported 2.9-fold change in the Pre-LN/Post-LN performance gap is a valid observation for the tested settings, but its magnitude should not be interpreted as a robust characterization of optimally tuned architectures.

Second, the depth-48 clipping analysis in Table 4 combines runs with substantially different training durations (1,000, 12,000, and 20,000 steps). Although the early trajectories provide evidence that learning rate is the dominant factor in the observed failures, the design does not constitute a fully matched-horizon factorial comparison. More importantly, Appendix B reports a canonical-closure run that remains above the collapse threshold at 1,000 steps but is still improving substantially. This makes the classification of that run as an irreversible collapse premature.

Third, the proposed explanation of collapse remains primarily observational. The proximity between the final cross-entropy loss and corpus unigram entropy is consistent with convergence toward a context-independent predictor, but it does not by itself establish that the model's conditional predictions have become independent of context. Similarly, reduced gradient norms do not establish the precise mechanism responsible for the inability to recover.

These limitations matter because the learning-rate ceiling, depth-dependent failure of QK-Norm, and characterization of the collapse mechanism are central to the paper's claimed contributions. More carefully matched optimization controls and direct mechanistic diagnostics are needed to substantiate the strongest interpretations.

Questions for authors
  1. Generalization beyond the current architecture (Critical Weakness 1).

Can the authors provide evidence that the main findings, especially the isolated QK-Norm improvement in Post-LN and its depth-dependent learning-rate tolerance, persist at a substantially different model width or on a second data distribution?

A targeted replication at a larger model width would materially strengthen the significance of the study. I am particularly interested in whether the relative ordering of QKNorm-PreLN and QKNorm-PostLN remains consistent.

  1. Sensitivity to learning-rate selection and warm-up (Critical Weakness 2).

How sensitive are the reported learning-rate ceilings and selected-learning-rate rankings to the sweep resolution, training horizon, and warm-up duration?

In particular, could the authors evaluate additional learning rates around the observed stability boundaries and examine whether extending the Post-LN warm-up changes the depth-48 collapse behavior?

Since the failures occur during warm-up, this distinction is important for determining whether the instability is intrinsic to the normalization configuration or depends primarily on the chosen optimization schedule.

  1. Verification of the depth-48 collapse mechanism (Critical Weakness 2).

Can the authors provide a matched-duration comparison between clipping thresholds 1 and 100 at depth 48, particularly for the canonical-closure configuration, and clarify whether the still-improving run reported in Appendix B eventually converges or collapses?

Additionally, can the authors directly quantify the discrepancy between the model's predicted token distributions and the empirical unigram distribution, ideally across different input contexts?

Such evidence would help establish whether the observed plateau genuinely represents a context-independent prediction regime rather than simply a high-loss stationary point.

Reviewer_yDFJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper does a controlled ablation of 3 choices with regards to normalization in transformers: 1)Pre-LN vs Post-LN, 2) Layernorm vs RMSNorm and 3) attention-logit normalization (QK-Norm vs No QK-Norm). The paper then runs ablations rooted in Pre-LN and Post-LN to generate different possible combinations. The authors control for depth, learning rate , model size etc and train on fineWeb-Edu. The authors claim that i) Post-LN relativve to Pre-LN penalty grows from +0.05 to +0.14 nats when accounting for lr since Pre-LN tolerates at least 10x higher rate ii) QK-Norm is responsible for the improvements from the Post-LN sweeps and RMSNorm does not contribute anything meaningful. iii) At larger depths e.g 48, all variants are on par with each other and smaller lr but Post-LN collapses at 10-3.

Critical strengths
  • The authors conduct a thorough ablation on LLM normalization choices, with a good experimental hygiene; controlling for the lr etc. This can be helpful to understand and helps separate commonly bundles architectural changes. Breaking down the performance gains from each part/normalization choice is interesting and can be useful to the community.

  • The paper is well written and clear to follow. The authors are also transparent about experimental limitations.

Critical weaknesses
  • The scope of this paper is very limited and I am not sure this is the scope for an ICLR paper. It very much reads like an ablation sweep that can completed in a single day. The contribution is useful but is ts scope is limited by the width, depth, corpus etc, the experiment has been run on.

  • The model sizes used in the ablation are also very small its very hard to draw any conclusions at this scale 21.3M params or in general <100m params. Alot of the findings here might easily get washed out and not matter in the way they are presented at larger scales.

Questions for authors
  • How do these observations scale beyon 100m param models ?

  • Does the QK-norm benefit remain if you do not have RMSNorm in the residual blocks ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.