acceptodds
Under review as a conference paper at ICLR 2027

Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining

Abstract

Causal-decoder Transformers form predictions through a hierarchy in which lower layers construct the residual features consumed by upper-layer attention. This paper identifies a GPT pretraining failure mode: upper layers can specialize their Q/K attention patterns too early, before lower-layer representations have stabilized. Temporarily slowing only upper-layer Q/K projections during early training improves final perplexity and downstream accuracy while leaving the rest of the model unchanged. The intervention prevents upper attention from collapsing onto an immature residual basis. In LLaMA-style blocks, the same intervention is largely unnecessary. Through ablations, we trace this difference to multiplicative gated feed-forward networks, which suppress the upstream residual writes that drive the failure. A pathwise analysis connects the two mechanisms: the learning-rate intervention reduces a step-size factor, while gated FFNs reduce a residual-energy factor along the same growth pathway. The results identify upper-layer Q/K timing as a concrete interaction point between decoder architecture and optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model pretraining, Transformer optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_M9YMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a simple but effective approach by temporarily reducing the learning rate of upper-layer query and key projections to handle the premature upper-layer attention specialization problem in language model pretraining.

Critical strengths
  1. The idea is simple but sounds effective at different scales.
  2. Experiments are relatively comprehensive.
Critical weaknesses
  1. The learning-rate-control scheme itself is not novel, and the paper's novelty claim rests on the diagnosis — which is only established through the intervention
  2. The "best scheme" ranking is not established. Only the magnitude and timing of one intervention are explored; the target is fixed by assumption.
Questions for authors

check the weakness

Reviewer_zkGYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether premature upper-layer attention specialisation harms language model pretraining. The authors hypothesise that upper-layer attention becomes sharp and heavily relied upon before lower-layer representations have matured. To address this, they propose temporarily reducing the learning rate of upper-layer Q/K projections while keeping other parameters on their normal schedules. In the experiments, they show improvements in validation loss and token efficiency across different sizes of models. The authors provide additional analyses to show that the intervention delays upper-layer attention specialisation and reduces early dependence on upper-layer Q/K.

Critical strengths
  • The paper shows that reducing the early learning rate of upper-layer Q/K improves validation loss across different model sizes. The method is easy to understand and implement.
  • Sections 5.3–5.4 show that the intervention delays attention sharpening and reduces early dependence on upper-layer Q/K, providing evidence that early optimisation choices can affect the model's later internal computations.
Critical weaknesses
  • The motivation in Section 1 is reasonable, but the proposed "premature upper-layer attention specialisation" relies on several unverified assumptions. First, the lower-copy score used in Section 3.2 does not necessarily indicate that lower-layer representations have matured, and the paper provides little evidence connecting this threshold to representation stability or downstream behaviour. Second, although the authors show that upper-layer attention becomes sharp early, they do not demonstrate, either qualitatively or quantitatively, that it learns incorrect or unstable matching patterns.
  • Section 5.5 explores different learning-rate multipliers, and Section 5.7 compares against global learning-rate reduction. However, the paper does not compare slowing upper-layer Q/K against simple baselines, such as slowing lower-layer Q/K, or individual attention heads. It is unclear whether the improvement comes specifically from delaying upper-layer attention specialisation without these comparisons.
Questions for authors
  • The hypothesis raised in the second paragraph of Section 1 is interesting. Could the authors provide additional analysis of checkpoints trained with the standard pretraining recipe to show that upper-layer attention learns incorrect or unstable matching patterns before lower-layer representations mature?
  • What evidence supports using the lower-copy score threshold of 0.005 as an indicator of mature lower-layer representations (Section 3.2)?
Reviewer_PtGoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies premature upper-layer attention specialization during GPT-style language-model pretraining. The authors hypothesize that upper-layer attention becomes sharp and strongly relied upon while lower-layer routing features are still developing, leading to a worse subsequent optimization trajectory. They propose temporarily reducing the learning rate of query and key projections in the upper half of the network, followed by a return to the standard schedule. Experiments on 270M, 700M, and 1B models report improvements in validation loss and token efficiency, with additional downstream evaluation at 270M. Training-time measurements and ablations connect the intervention to delayed attention sharpening and reduced dependence on upper-layer Q/K.

Critical strengths
  1. The paper offers a useful temporal perspective on attention specialization, showing that a brief early intervention can have lasting effects on training.
  2. The proposed method is simple to implement and adds no model parameters or inference-time cost.
Critical weaknesses
  1. Insufficiently controlled hyperparameter comparisons. The proposed method is evaluated across several learning-rate multipliers and release schedules, while entropy regularization uses only one threshold and strength. The resulting early entropy also differs substantially (0.799 versus 0.966), so the performance gap may reflect weaker entropy preservation rather than an advantage of learning-rate scheduling.

  2. Limited validation of key design choices. The paper does not compare intervening on the upper half with other layer subsets or all-layer Q/K interventions. Schedule ablations use one seed, and the threshold/patience robustness analysis only replays existing logs without retraining, leaving final-performance robustness unclear.

  3. Limited novelty beyond existing attention-control methods. Zhai et al. (ICML 2023), Stabilizing Transformer Training by Preventing Attention Entropy Collapse, already connect low attention entropy with training instability, while Anson and Aitchison (2025), Controlling changes to attention logits, directly control logits through Q/K-specific learning rates, including fixed-multiplier experiments. The potentially novel contribution is the claim that early upper-layer specialization on immature lower-layer features causes lasting harm. However, the current experiments do not sufficiently distinguish this timing-mismatch mechanism from general benefits of controlling Q/K updates. Direct comparisons with these methods and stronger controls over intervention location and timing would clarify the contribution.

Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.