- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the origin of the neural scaling law of training, extending prior work on a single softmax head learning peaked distributions. Using gate-head (GH) toy models, the authors show that any softmax learning peaked distributions, regardless of its position, becomes a training bottleneck, with its logit magnitudes growing as and its loss contribution decaying as . A loss universality theorem shows this scaling is not specific to loss definitions. In Pythia and OLMo-2, losses follow the 1/3 time scaling, and attention heads rather than the LM head show logit growth, suggesting that attention trying to concentrate drives the scaling law.
Critical strengths
- Extends the softmax-based scaling theory to multiple softmax functions at arbitrary depth, with a loss universality theorem showing the 1/3 exponent holds for any reasonable loss.
- Validates the theory with targeted toy-model experiments across gate non-linearities and optimizers.
- Verifies the theory’s assumptions and predictions in Pythia and OLMo-2, and shows via MSE training that the LM head is not the sole bottleneck.
Critical weaknesses
- Lack of causal evidence that attention drives the 1/3 scaling. The central claim rests on the correlational observation that attention logits grow as . An intervention that caps or fixes attention temperature, even at small scale, would show whether attention is the cause or merely a co-occurring phenomenon.
- Unexplained inconsistency regarding the LM head. The paper places the LM head in the low-temperature regime, yet its logit magnitude saturates rather than grows, which conflicts with the theory that any low-temperature softmax is a bottleneck. One possibility is that the LM head has already reached its target coldness.
- Limited representativeness of the evidence. The key evidence (Figures 4d and 5a) relies on a few selected heads, and the MSE experiment uses a single model size, a single head, and a saturation point chosen by eye. A quantitative summary across all heads, such as the distribution of late-training growth exponents, would establish generality.
Questions for authors
- Effect of the learning rate schedule. Prior work (Tissue et al., 2024; Luo et al., 2025) shows that learning-rate annealing reduces loss beyond what the cumulative learning rate predicts. Does the scaling hold under cosine decay and other schedules such as WSD?
- Relation to attention sinks. Do the low-temperature heads whose logits grow as coincide with attention-sink heads? This would clarify whether the bottleneck is a distinct phenomenon or the same one seen from a training-dynamics perspective.
4 comments
First reply to reviewer qKCN continued
Question 1: Learning rate schedule
This is a very interesting question that we are actively working on. In short, holds under any schedule; the schedule changes only the loss coefficient. Under cosine decay, holds (Pythia, Figure 4c, is trained with cosine decay). Under WSD, it holds for the final checkpoints across different runs, but not locally during the fast decay phase. The details follow.
We first need to consider gradient noise, Here, is the density of the energy gap at zero, is a constant depending on the loss, and is the variance of the weights in the transverse directions.
For a perfectly aligned student, , and the learning rate schedule has no effect.
For an approximately aligned student, . In general, is a dynamic variable that depends on the trajectory; Eq. (11) of https://arxiv.org/pdf/2507.02119 has the same flavor. Assuming the transverse motion is much faster than the norm growth, we can solve for from its quasi-equilibrium where is the width, is the learning rate at , is the batch size, and is a nonlinear function without an analytic form.
In general, holds robustly, and the learning rate schedule affects the coefficient through . If is already very small or changes slowly, holds, which is the case for cosine decay. If changes fast, as in the final decay phase of WSD, the coefficient drops dramatically and we cannot see locally. However, if we fix the shape of the WSD schedule and the peak learning rate and vary the total number of steps, the final losses of these runs, plotted against , will follow , as they have similar .
We will report the results in future work, and we plan to emphasize the learning rate schedule question as a limitation. Thank you for the references, which we will include in discussions.
Question 2: Relation to attention sinks
This is an interesting question. The heads with growth partly coincide with attention-sink heads. In Pythia models, roughly, the later 2/3 of the layers have attention sinks, and their heads often try to concentrate. But the first few layers also have heads with growth.
The heads want to concentrate as required by the data, and the vanishing gradient caused by the nonlinearity and gradient-based optimization makes the concentration slow, i.e., .
The data determine why a head concentrates. Inputs that need no information from the context make the head concentrate on BOS, which produces an attention sink. Inputs that need specific information make it concentrate on particular tokens, which does not. Both lead to growth, which is why the two sets of heads partly coincide.
Less than one day left in discussion period
Dear Reviewer qKCN,
Thank you very much for your time and effort in reviewing.
With the discussion closing in less than 24 hours (Nov. 18 AOE), we want to follow up on our rebuttal. Please let us know whether any of your concerns or questions have been addressed and whether you have follow-up questions.
We’d be grateful if you would revisit your assessment in light of the rebuttal. Deeply appreciated!
Best,
The authors
Thank you for the detailed response. The LM head explanation and new head-level statistics address several of my questions. While I understand the difficulty of clean interventions in LLMs, the evidence for attention’s role remains correlational, and even a small-scale intervention would strengthen the central claim. I will keep my score.
First reply to reviewer qKCN
We thank the reviewer for their careful reading and positive view of our contribution. The reviewer's concern is about the strength of the empirical evidence in LLMs.
Weakness 1: Lack of causal evidence
We want to clarify that we are not claiming a rigorous "causal relation" in this paper. Our title "Emergent One-Third Scaling Law as Attention Tries to Concentrate" uses "as", which is closer to co-occurring. In the natural sciences such as physics, if event A (attention logits growing as τ1/3) always co-occurs with event B (loss decaying as τ−1/3), and A is "hidden" while B is superficial, we tend to say "A causes B". But this is not a rigorous logical causal relation. Mathematically, we have L=Θ(β−1), which is an equation (co-occurrence), not a causal relation. We will revise our paper to be more precise.
The intervention the reviewer suggested is ideal, but reality is more complicated. First, to prove A→B, one should show ¬B→¬A. However, the intervention at most shows ¬A→¬B. Second, capping or fixing the attention temperature may not affect the attention logits cleanly. Because LLMs are so complicated, the intervention may affect many other things including aligning or rotation, making it hard to claim even ¬A→¬B rigorously.
That said, we believe the evidence is a strength of this work. Besides LLM data, we have a solvable toy model in which the causal direction is explicit, and the resulting theory makes falsifiable causal predictions that we verify in Pythia and OLMo-2 (several model sizes in different model families). This is not a proof, but it supports that the theory applies to LLMs, and then the theory together with the LLM data point to attention as the bottleneck. Few theories of neural scaling laws are tested against internal quantities of LLMs, and few empirical studies of LLM scaling come with a falsifiable mechanism. Relative to what is currently attainable in this field, we see the quality of our evidence as a strength rather than a weakness.
Weakness 2: Unexplained inconsistency
We are sorry for the confusion. The reviewer's intuition is correct: the LM head has already reached its target coldness. There is no inconsistency. We omitted the full picture to save space in the main text. We will add the details to the Appendix and briefly explain in the main text why the LM head can saturate, which we believe will improve the clarity. Thank you very much!
In our theory, we focused on β⋆≫β≫2lnd, where L∼β−1 and ∇βL∼β−2. But if β⋆≈β≫2lnd, we have ∇βL∼β−2−β⋆−2. Near β⋆, the dynamics of β become an exponential convergence to β⋆, as β⋆ is the fixed point.
When β⋆ is in the low-temperature regime, β passes through three regimes: (i) high temperature, β≲2lnd; (ii) 1/3 scaling, β⋆≫β≫2lnd; and (iii) exponential convergence, β≈β⋆. For the LM head, β⋆ is only moderately large, so regime (ii) is short and β reaches β⋆ early in training.
Weakness 3: Representativeness
We thank the reviewer for bringing this up. Our theory does not require all heads to have 1/3 growth, only one. In our view, the representativeness most relevant to our theory is that there exists at least one cold head with 1/3 growth across model sizes and families. We do have data showing such generality in the Appendix (original Figures 11-14 and 16 in first version). It is therefore representative to show one head in the main text, given the page limit. We ran a single MSE experiment due to limited compute, which is a limitation of our work. The saturation point in this experiment is determined from the increment of the LM-head coldness.
Whether 1/3 growth holds across heads within one model is less central to our theory, but it is worth reporting. Please see Figures 12 and 13. We have now also fitted the growth exponents of all attention heads in Pythia-12B and will report the statistics in the main text. From layer 21 to 32, the 480 heads that try to concentrate have growth exponents close to 1/3, with a mean exponent of 0.32 (please see Figure in this link).
Please see responses to Questions below