acceptodds
Under review as a conference paper at ICLR 2027

From Front-Loading to Gradual Refinement: Simplex Gating in Transformer Decoders

Abstract

We observe pronounced _front-loading_ of residual-norm growth in transformer decoders: the first block sharply amplifies the residual stream, while subsequent blocks produce much smaller relative increases. Our analysis identifies a structural asymmetry underlying this pattern: pre-normalization standardizes sublayer inputs but leaves residual-update magnitudes unconstrained, giving a write of a given size greater relative influence near the input, where the stream is smallest. Motivated by this asymmetry, we introduce _simplex gating_ to regulate residual writes in both attention and MLP sublayers. An input-dependent scalar gate scales the attention output, while normalized group gates reweight MLP activations before the down-projection. The MLP gates are nonnegative and sum to one for each token, enabling adaptive allocation of a fixed total gate weight across activation groups. Under matched pre-training conditions, simplex gating substantially alleviates front-loading, reducing the first-block residual-norm growth ratio from 9.10× to 1.84×. It also improves pre-training convergence at two model scales, achieving consistently lower training loss. Ablations show that the simplex constraint mitigates compensatory activation growth that can weaken independent sigmoid gates. These findings demonstrate the practical benefits of controlling residual writes and suggest a path from front-loaded residual growth toward more gradual refinement across depth.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer architecture, normalized gating, layer-wise probing, depth utilization, language modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rnYqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a simplex gating approach to handle the front-loading of residual-norm growth in transformer decoders.

Critical strengths
  1. The idea sounds simple but effective.
  2. The result presentation is clear.
Critical weaknesses
  1. No comparison against any method other than the ungated baseline. Related works mentioned a few methods.
  2. Novelty is limited
Questions for authors

check weakness.

Reviewer_ouXpOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper measures the mean norm of the residual stream after each block of pre-norm decoders and reports that the first block multiplies it by a large factor (9.10×9.10\times for a 152M LLaMA-style model trained by the authors, 19.34×19.34\times for Qwen2.5-3B-Instruct), while later blocks grow it by 1.11.1 to 1.3×1.3\times on average. The authors call this front-loading and attribute it to pre-normalization: a sublayer reads a normalized input, so the size of what it writes does not depend on the size of the stream, and a write counts most where the stream is smallest. They propose simplex gating: a scalar gate 2σ(Wgattnh)2\sigma(W_g^{attn} h) on the attention output, and, in the MLP, HH groups of hidden units weighted by sigmoid scores normalized to sum to one. On two models (152M and 589M) trained on OpenWebText, the first-block ratio falls from 9.10×9.10\times to 1.84×1.84\times and the training loss is lower throughout training. Three analyses of the 152M models (removal of the token embedding after block 0, effective rank, INT4 weight quantization) are given as explanations of the gain.

Critical strengths
  • They provide some evidence that a plain gate cannot control the first write, which motivates normalising the gates. The paper shows that the first block writes a vector much larger than the token embedding it is added to (r_0 = 9.10 at 152M, 19.34 in Qwen2.5-3B-Instruct). More informatively, Appendix A.1 shows that independent sigmoid gates do not reduce this write: the first-layer gates fall to a mean of 0.082, but the activations they multiply grow to 14.84 against 3.64 in the baseline, so the layer undoes the gate. It is the clearest support for the paper's main design choice, the normalized group gate. Its weight is limited because it rests on a single run with no loss reported (see Q2).
Critical weaknesses
  • The training gain is not established:
    • The lower train loss is the paper's main evidence that support the method but it rests on one run per model with no measure of variance. Fig. 4 states that the two runs "share the data, optimizer and learning rate schedule", but does not report seeds, data and batch order or more than one run. The gated model has additional parameters, so unless this was handled explicitly the two models do not even start from the same initialization of the shared weights.
    • Read from Fig. 4, the difference goes from about 0.070.07 at step 10000 to about 0.03 at step 40000 at 152M, and from about 0.08 to about 0.06 at 589M. Without a variance estimate, we do not know how much of the final gap is beyond run-to-run noise.
    • Furthermore, both models use a peak learning rate of 10−310^{-3}. Qiu et al. (2025), whose gated attention is half of the method, report that output gating tolerates larger learning rates, so a single shared learning rate may favour the gated model.
    • Finally, at initialization every MLP gate equals 1/H, so the method starts as the baseline with every MLP write divided by H (H = 12 at 152M). The paper compares with neither this fixed scaling, nor a learnable scalar (ReZero, LayerScale, which it cites), nor the attention gate alone (close to Qiu et al., 2025), nor the MLP gate alone. The gate-placement ablation is announced in the reproducibility statement as part of Appendix A, but only one sentence without numbers exists (end of Section 4.2), and Appendix A.1 gives no loss for the soft variant.
  • The paper doesn't shows that its method does something different from baselines or that front-loading is the reason for the gain:
    • Controlling the write already exists, contrary to what Section 2.1 states (no prior remedy "imposes a hard budget on the write itself, the increment a sublayer emits") NormFormer (Shleifer et al., 2021), which the paper cites, computes x + LN(MHA(LN(x))), i.e. it normalizes the attention output before adding it to the stream. Peri-LN (Kim et al., ICML 2025) normalizes the output of every sublayer, x + Norm(Module(Norm(x))), and is the placement used in the Gemma and OLMo families. nGPT (Loshchilov et al., 2024) keeps hidden states at unit norm and adds each update with a learned step size. None of these is compared, and Peri-LN and nGPT are not cited. Input-dependent sigmoid gates on attention and MLP outputs also exist (Highway networks, cited; GateSkip, Laitenberger et al., 2025, not cited). What remains specific to the paper is the normalized group gate inside the MLP.
    • Front-loading is largely set by the embedding scale, and is not controlled. The first-block ratio is r0=xˉ1/xˉ0r_0 = \bar x_1 / \bar x_0, where xˉl\bar x_l is the mean residual norm after block ll and xˉ0\bar x_0 the embedding norm. The embedding norm is 0.94, close to its value at initialization. Since each pre-norm sublayer only sees the direction of its input (Section 3.2), multiplying the embedding by a constant cc leaves the write of block 0 unchanged, so r0r_0 drops by about cc as long as c xˉ0c\,\bar x_0 stays small against that write; the only other change is the weight of the embedding in the stream seen by later blocks. The original Transformer multiplies embeddings by d\sqrt{d}, with dd the model width (the embeddings here are tied, so this would be a multiplier on the input side only). Without this control, the claim that "any remedy must be architectural" is untested, and we do not know whether a small r0r_0, rather than the gates, is what helps.
    • The embedding-removal test really measures the size of the removed vector instead. Section 5 removes the embedding x0x_0 from x1=x0+block0(x0)x_1 = x_0 + \mathrm{block}_0(x_0) and reports a loss increase of +0.0083 for the baseline and +5.22 for the gated model. The removed vector is about 11% of the norm of x1x_1 in the baseline (0.94 / 8.59) and about 54% in the gated model (1/1.84), so the second perturbation is much larger by construction. An increase of 5.22 from a loss near 2.74 is what one expects from deleting half of the stream. Moreover, block0(x0)\mathrm{block}_0(x_0) is itself computed from the token, so a small difference in loss does not show that "downstream layers barely use the original token identity". McDougall et al. (2023, Appendix I) ran the same test on GPT-2 Small (loss 3.047 against 3.044) and read it as the first MLP producing an effective embedding. The two other analyses do not locate the effect at the first block either: the effective rank of the two models is the same after blocks 0 and 1 (about 600, Fig. 5) and diverges at block 2, and the quantization advantage shrinks from 0.080 (per-channel INT4) to 0.016 (NF4) (Table 2).
Questions for authors

Clarifications:

  1. Do the baseline and gated runs share the seed and the data order? What are the held-out losses of both 589M models, and of the 152M models during training?
  2. In Appendix A.1, the soft gates settle at a mean of 0.082, almost the mean of the simplex gates (1/12≈0.0831/12 \approx 0.083), yet the activation norm is 14.84 against 3.15. What explains the difference, and what are the loss and r0r_0 of the soft variant?
  3. What are the numbers of the gate-placement ablation, and how was HH chosen?

Additional results that would change my assessment:

  1. Held-out loss over at least three seeds at both scales, with a small learning-rate sweep for each model.
  2. The loss and r0r_0 of: the baseline with the embedding multiplied by a constant (e.g. d\sqrt{d}); the baseline with MLP writes multiplied by a fixed 1/H and by a learnable scalar; the attention gate alone; the MLP gate alone; and at least one write-normalization method (NormFormer or Peri-LN).
  3. Difference in loss when a random vector of the same norm as x0x_0 is removed instead, for both models.
Reviewer_KfSKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper aimed at addressing the asymmetry in residual updates in Pre-norm decoder-only LLMs, where magnitude of residual writes are unconstrained, especially in the early layers. Authors have introduced simplex gating mechanism which constraint the magnitude of residual write in both, attention and MLP, sub-blocks. This has shown to reduce the magnitude of residual norm in early layers and improve the convergence speed across various scale of model (1B+).

Critical strengths

Authors have very well articulated the problem statements, the source of asymmetry in residual update, and the proposed approach to mitigate the asymmetry. The systematic explanation for explaining the under-laying mechanism of simplex gating, and their difference with respect to gating mechanism in MLP is easy to understand and follow.

Critical weaknesses

I see two major weaknesses which needs to be addressed during the rebuttal period.

First, the residual norm growth in pre-LN architecture is a known phenomenon, and their exact mitigation is proposed in MixLN paper [1]. In particular, residual norm can be controlled by having post-LN in first few layers (e.g., 25% of layers), and rest as pre-LN. Authors must have discussed and further compare the pros and cons of simplex gating with MixLN approach.

Second, to test whether the residual update asymmetry is a fundamental property of Pre-Ln architecture or it emerges due to AdamW optimizer, authors needs to pre-train, at least one smaller LLaMA model (152M), with Muon/NorMuon optimizer. In fact, Dion has nice property to constrain the update rank (1/2, 1/4, 1/8, 1/16) and allow a trade-off between Muon and AdamW like optimization behavior.

This is particular important because earlier observations, prior to the advent of Muon optimizer, such as attention-induced rank collapse, were understood as fundamental property of transformer architecture. However, later studies, such as [2] showed it as specific to AdamW.

[1] Li et al., Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN, ICLR 2025

[2] Jha et al., NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks, ICLR 2026

Questions for authors

See the weaknesses.

Additionally, could authors use a different metric, other than Shannon entropy of normalized Eigen values, for effective dimensionality of learned representation, such as participation ratio, which is sensitive to the concentration of Eigen distribution, and see if the observation made in the paper is consistent or not.

Shannon entropy hides the distribution of Eigen-values. That is, it does not become clear whether the eigenvalue distribution has long tail or just a few dominant values.

Comments on the paper

1 comment
Area_Chair_9LpjOpenReview ↗

Reminder: Author–Reviewer Discussion Deadline Approaching

Dear Reviewers and Authors,

Thank you to the reviewers for your time, effort, and dedication in reviewing this paper. This AC also appreciates the authors' detailed rebuttal, which addresses the reviewers' comments and includes additional clarifications and experimental results.

To the reviewers: As the author–reviewer discussion deadline approaches, please carefully review the authors' responses and assess whether they adequately address your concerns. If any concerns have not been fully addressed, please let the authors know what further clarification or evidence would be helpful.

Thank you again for your valuable contributions and constructive discussion.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.