acceptodds
Under review as a conference paper at ICLR 2027

Emergent One-Third Scaling Law as Attention Tries to Concentrate

Abstract

The neural scaling law relating longer training to better performance through a power law is central to today's large language models (LLMs), yet its origin remains debated. One recent proposal is that power laws can emerge from the strong non-linearity of a single softmax head learning peaked distributions. What happens with multiple softmax functions, as in LLMs, is unclear. Here, we show through toy models that any softmax learning peaked distributions, regardless of its position in the model, can develop logit magnitudes that grow in a power law with exponent , becoming a training bottleneck whose loss contribution decays as a power law with the same exponent . The overall loss therefore obeys scaling whenever at least one softmax learns peaked distributions. We confirm that many softmax functions in LLMs learn peaked distributions and that LLM loss scaling matches this prediction. Moreover, logit growth dynamics reveal that attention heads, rather than the language modeling head, are the bottleneck likely driving the loss scaling in LLMs. Attention trying to concentrate on specific information, which is the heart of Transformers, may therefore also be the heart of the neural scaling law of training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, attention, neural scaling laws, universality

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qKCNOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the origin of the neural scaling law of training, extending prior work on a single softmax head learning peaked distributions. Using gate-head (GH) toy models, the authors show that any softmax learning peaked distributions, regardless of its position, becomes a training bottleneck, with its logit magnitudes growing as τ1/3\tau^{1/3} and its loss contribution decaying as τ−1/3\tau^{-1/3}. A loss universality theorem shows this scaling is not specific to loss definitions. In Pythia and OLMo-2, losses follow the 1/3 time scaling, and attention heads rather than the LM head show τ1/3\tau^{1/3} logit growth, suggesting that attention trying to concentrate drives the scaling law.

Critical strengths
  • Extends the softmax-based scaling theory to multiple softmax functions at arbitrary depth, with a loss universality theorem showing the 1/3 exponent holds for any reasonable loss.
  • Validates the theory with targeted toy-model experiments across gate non-linearities and optimizers.
  • Verifies the theory’s assumptions and predictions in Pythia and OLMo-2, and shows via MSE training that the LM head is not the sole bottleneck.
Critical weaknesses
  • Lack of causal evidence that attention drives the 1/3 scaling. The central claim rests on the correlational observation that attention logits grow as τ1/3\tau^{1/3}. An intervention that caps or fixes attention temperature, even at small scale, would show whether attention is the cause or merely a co-occurring phenomenon.
  • Unexplained inconsistency regarding the LM head. The paper places the LM head in the low-temperature regime, yet its logit magnitude saturates rather than grows, which conflicts with the theory that any low-temperature softmax is a bottleneck. One possibility is that the LM head has already reached its target coldness.
  • Limited representativeness of the evidence. The key evidence (Figures 4d and 5a) relies on a few selected heads, and the MSE experiment uses a single model size, a single head, and a saturation point chosen by eye. A quantitative summary across all heads, such as the distribution of late-training growth exponents, would establish generality.
Questions for authors
  • Effect of the learning rate schedule. Prior work (Tissue et al., 2024; Luo et al., 2025) shows that learning-rate annealing reduces loss beyond what the cumulative learning rate predicts. Does the τ−1/3\tau^{-1/3} scaling hold under cosine decay and other schedules such as WSD?
  • Relation to attention sinks. Do the low-temperature heads whose logits grow as τ1/3\tau^{1/3} coincide with attention-sink heads? This would clarify whether the bottleneck is a distinct phenomenon or the same one seen from a training-dynamics perspective.
4 comments
AuthorsOpenReview ↗

First reply to reviewer qKCN

We thank the reviewer for their careful reading and positive view of our contribution. The reviewer's concern is about the strength of the empirical evidence in LLMs.

Weakness 1: Lack of causal evidence

We want to clarify that we are not claiming a rigorous "causal relation" in this paper. Our title "Emergent One-Third Scaling Law as Attention Tries to Concentrate" uses "as", which is closer to co-occurring. In the natural sciences such as physics, if event A (attention logits growing as τ1/3\tau^{1/3}) always co-occurs with event B (loss decaying as τ−1/3\tau^{-1/3}), and A is "hidden" while B is superficial, we tend to say "A causes B". But this is not a rigorous logical causal relation. Mathematically, we have L=Θ(β−1)L = \Theta(\beta^{-1}), which is an equation (co-occurrence), not a causal relation. We will revise our paper to be more precise.

The intervention the reviewer suggested is ideal, but reality is more complicated. First, to prove A→BA \rightarrow B, one should show ¬B→¬A\neg B \rightarrow \neg A. However, the intervention at most shows ¬A→¬B\neg A \rightarrow \neg B. Second, capping or fixing the attention temperature may not affect the attention logits cleanly. Because LLMs are so complicated, the intervention may affect many other things including aligning or rotation, making it hard to claim even ¬A→¬B\neg A \rightarrow \neg B rigorously.

That said, we believe the evidence is a strength of this work. Besides LLM data, we have a solvable toy model in which the causal direction is explicit, and the resulting theory makes falsifiable causal predictions that we verify in Pythia and OLMo-2 (several model sizes in different model families). This is not a proof, but it supports that the theory applies to LLMs, and then the theory together with the LLM data point to attention as the bottleneck. Few theories of neural scaling laws are tested against internal quantities of LLMs, and few empirical studies of LLM scaling come with a falsifiable mechanism. Relative to what is currently attainable in this field, we see the quality of our evidence as a strength rather than a weakness.

Weakness 2: Unexplained inconsistency

We are sorry for the confusion. The reviewer's intuition is correct: the LM head has already reached its target coldness. There is no inconsistency. We omitted the full picture to save space in the main text. We will add the details to the Appendix and briefly explain in the main text why the LM head can saturate, which we believe will improve the clarity. Thank you very much!

In our theory, we focused on β⋆≫β≫2ln⁡d\beta^\star \gg \beta \gg \sqrt{2 \ln d}, where L∼β−1L \sim \beta^{-1} and ∇βL∼β−2\nabla_{\beta} L \sim \beta^{-2}. But if β⋆≈β≫2ln⁡d\beta^\star \approx \beta \gg \sqrt{2 \ln d}, we have ∇βL∼β−2−β⋆−2\nabla_{\beta} L \sim \beta^{-2} - \beta^{\star-2}. Near β⋆\beta^\star, the dynamics of β\beta become an exponential convergence to β⋆\beta^\star, as β⋆\beta^\star is the fixed point.

When β⋆\beta^\star is in the low-temperature regime, β\beta passes through three regimes: (i) high temperature, β≲2ln⁡d\beta \lesssim \sqrt{2 \ln d}; (ii) 1/31/3 scaling, β⋆≫β≫2ln⁡d\beta^\star \gg \beta \gg \sqrt{2 \ln d}; and (iii) exponential convergence, β≈β⋆\beta \approx \beta^\star. For the LM head, β⋆\beta^\star is only moderately large, so regime (ii) is short and β\beta reaches β⋆\beta^\star early in training.

Weakness 3: Representativeness

We thank the reviewer for bringing this up. Our theory does not require all heads to have 1/31/3 growth, only one. In our view, the representativeness most relevant to our theory is that there exists at least one cold head with 1/31/3 growth across model sizes and families. We do have data showing such generality in the Appendix (original Figures 11-14 and 16 in first version). It is therefore representative to show one head in the main text, given the page limit. We ran a single MSE experiment due to limited compute, which is a limitation of our work. The saturation point in this experiment is determined from the increment of the LM-head coldness.

Whether 1/31/3 growth holds across heads within one model is less central to our theory, but it is worth reporting. Please see Figures 12 and 13. We have now also fitted the growth exponents of all attention heads in Pythia-12B and will report the statistics in the main text. From layer 21 to 32, the 480 heads that try to concentrate have growth exponents close to 1/31/3, with a mean exponent of 0.320.32 (please see Figure in this link).

Please see responses to Questions below

AuthorsOpenReview ↗

First reply to reviewer qKCN continued

Question 1: Learning rate schedule

This is a very interesting question that we are actively working on. In short, β∼τ1/3\beta \sim \tau^{1/3} holds under any schedule; the schedule changes only the loss coefficient. Under cosine decay, L∼τ−1/3L \sim \tau^{-1/3} holds (Pythia, Figure 4c, is trained with cosine decay). Under WSD, it holds for the final checkpoints across different runs, but not locally during the fast decay phase. The details follow.

We first need to consider gradient noise, L≈ρ(0)β(AL+Γ/2).L \approx \frac{\rho(0)}{\beta}(A_L + \Gamma / 2). Here, ρ(0)\rho(0) is the density of the energy gap at zero, ALA_L is a constant depending on the loss, and Γ\Gamma is the variance of the weights in the transverse directions.

For a perfectly aligned student, Γ=0\Gamma = 0, and the learning rate schedule has no effect.

For an approximately aligned student, Γ≠0\Gamma \neq 0. In general, Γ\Gamma is a dynamic variable that depends on the trajectory; Eq. (11) of https://arxiv.org/pdf/2507.02119 has the same flavor. Assuming the transverse motion is much faster than the norm growth, we can solve for Γ\Gamma from its quasi-equilibrium Γ=mητBΨ(Γ),\Gamma = \frac{m\eta_\tau}{B}\Psi(\Gamma), where mm is the width, ητ\eta_\tau is the learning rate at τ\tau, BB is the batch size, and Ψ\Psi is a nonlinear function without an analytic form.

In general, β∼τ1/3\beta \sim \tau^{1/3} holds robustly, and the learning rate schedule affects the coefficient through Γ\Gamma. If Γ\Gamma is already very small or ητ\eta_\tau changes slowly, L∼τ−1/3L\sim \tau^{-1/3} holds, which is the case for cosine decay. If ητ\eta_\tau changes fast, as in the final decay phase of WSD, the coefficient drops dramatically and we cannot see L∼τ−1/3L\sim \tau^{-1/3} locally. However, if we fix the shape of the WSD schedule and the peak learning rate and vary the total number of steps, the final losses of these runs, plotted against τ\tau, will follow τ−1/3\tau^{-1/3}, as they have similar Γ\Gamma.

We will report the results in future work, and we plan to emphasize the learning rate schedule question as a limitation. Thank you for the references, which we will include in discussions.

Question 2: Relation to attention sinks

This is an interesting question. The heads with τ1/3\tau^{1/3} growth partly coincide with attention-sink heads. In Pythia models, roughly, the later 2/3 of the layers have attention sinks, and their heads often try to concentrate. But the first few layers also have heads with τ1/3\tau^{1/3} growth.

The heads want to concentrate as required by the data, and the vanishing gradient caused by the nonlinearity and gradient-based optimization makes the concentration slow, i.e., τ1/3\tau^{1/3}.

The data determine why a head concentrates. Inputs that need no information from the context make the head concentrate on BOS, which produces an attention sink. Inputs that need specific information make it concentrate on particular tokens, which does not. Both lead to τ1/3\tau^{1/3} growth, which is why the two sets of heads partly coincide.

AuthorsOpenReview ↗

Less than one day left in discussion period

Dear Reviewer qKCN,

Thank you very much for your time and effort in reviewing.

With the discussion closing in less than 24 hours (Nov. 18 AOE), we want to follow up on our rebuttal. Please let us know whether any of your concerns or questions have been addressed and whether you have follow-up questions.

We’d be grateful if you would revisit your assessment in light of the rebuttal. Deeply appreciated!

Best,

The authors

Thank you for the detailed response. The LM head explanation and new head-level statistics address several of my questions. While I understand the difficulty of clean interventions in LLMs, the evidence for attention’s role remains correlational, and even a small-scale intervention would strengthen the central claim. I will keep my score.

Reviewer_ADm6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how learning peaked distributions with softmax induces the power-law loss decay during training, regardless of the softmax position in the model. Through theoretical analysis of toy models in teacher-student setup, it is shown that the logits magnitudes of the student grow as a power law with exponent 1/3, and consequently the loss decays with the same exponent. To support the relevance of this result in practice, the authors empirically demonstrate that attention heads tend to learn peaked distributions, and that losses follow power-law in dynamic training time with all exponents close to 1/3, for different Pythia and OLMo models.

Critical strengths

This paper explores an interesting perspective on loss scaling : modeling choices, such as softmax attention, can become the training bottleneck, instead of the hidden structure in the data. The power-law gate coldness growth illustrates this point well.

The authors clearly demonstrate how their toy model assumptions can be relevant to LLMs as well, by checking the entropy of the softmax outputs in different layers of Pythia and OLMo models and checking that the density of zero energy gap is non-zero.

Critical weaknesses

My main concern is that this work does not introduce sufficient contribution compared to (Liu et al., 2026). The main additional result is Result 2 (Loss universality), but it does not introduce a novel proof technique or relax the Aligned Student assumption. In the introduction, the main question is "How do multiple softmaxes affect the training dynamics?", but the result derivation is based purely on the reduction to the case of single softmax very similar to (Liu et al., 2026). This motivate my final rating, weak reject.

Questions for authors

i) Have the authors studied the interaction between the proposed peaked softmax mechanism and some complementary scaling mechanisms ? E.g. if the alignment between the teacher and the student is slow, how would it affect the later dynamics of β\beta ?

ii) Do the authors expect the power-law scaling to hold in the large learning rate SGD regime, where the dynamics can not be simply reduced to gradient flow ? I would expect that stochasticity pushes the norms of the parameters up ; but it is not clear to me wether this can speed up the logits magnitude growth significantly enough to break the power-law.

4 comments
AuthorsOpenReview ↗

First reply to reviewer ADm6

We thank the reviewer for their positive views on clarity, informativeness, and soundness. The reviewer's main concern is the novelty and significance of our work relative to previous work.

Weakness

The reviewer is concerned that our work "does not introduce sufficient contribution compared to (Liu et al., 2026)", that the derivation is "based purely on the reduction to the case of single softmax", and that Result 2 "does not introduce a novel proof technique". These concerns are understandable, but we respectfully disagree, for the reasons below.

  • The question we focus on is orthogonal to previous work. We study how multiple softmaxes may affect the training dynamics and scaling, a question previous work cannot answer. The answer is not obvious: multiple softmaxes may further slow down the scaling, or they may provide more flexible pathways and speed up the scaling. Our finding that they lead to the same exponent 1/31/3 when one softmax is cold is therefore non-trivial. The conclusion looks similar to that of previous work, but it answers a different question and is necessary to bridge the gap between a single softmax head and real LLMs.
  • The fact that "reduction" worked is a finding and is also our contribution. Through experiments and theoretical analysis, we found that the multi-softmax GH model may be reduced to a single softmax under some conditions. This is a non-trivial finding and was not guaranteed to be true before our study. It is part of the explanation of why 1/31/3 scaling persists as long as one softmax is cold. In hindsight, the reduction may look natural. Before our study, however, it was only one of several possibilities. Reducing a new, unsolved problem to a simpler one is a standard and valuable strategy, and showing that it works here is itself a contribution.
  • The single softmax case we reduce to is different from previous work. Previous work (Liu et al., 2026) explicitly claimed that the 1/31/3 scaling comes from softmax and CE loss. Our single softmax model does not rely on the CE loss.
  • We disagree that loss universality is not a sufficient contribution. The reviewer evaluated the significance of a mathematical result by whether its proof technique is novel. We take a different view: the novelty of a mathematical theorem lies in the statement itself, the rich phenomena it can explain, and the inspirations and implications it provides. We were quite surprised to observe that ⟨∥p−q∥2⟩\langle \|p-q\|^2 \rangle and ⟨∥p−q∥4⟩\langle \|p-q\|^4 \rangle have the same scaling exponent 1/31/3. Loss universality explains why an internal softmax contributes to the 1/31/3 scaling of the overall loss. It also inspires our future study on the scaling of reinforcement learning, which may also show 1/31/3 scaling when plotting "loss" rather than reward. We therefore consider loss universality a significant contribution, worth reporting as a standalone mathematical result.
  • On the Aligned Student assumption. Its strict version can be relaxed, but this is not essential and does not change our central claims.
  • Our most significant contribution is about LLMs, and it says something totally different from previous work. Our most important claim is that in LLMs, attention heads, not the LM head, are the bottleneck behind power-law loss scaling. This differs fundamentally from the previous claim (Liu et al., 2026) that the LM head is the origin. If this nonlinear mechanism dominates neural scaling laws, our new results on LLMs could have a broad impact on the field, pointing out the correct bottleneck on which future improvements should focus. As this result is the most relevant to the ICLR community, we hope the reviewer will take it into account in the overall assessment.

To summarize, our contributions are sufficiently significant and distinct from previous work. In particular, our new understanding of the origin of neural scaling laws in LLMs is the most important to the ML community, and it corrects the LM head leads to 1/31/3 loss scaling claim in previous work.

Please see responses to Questions below

AuthorsOpenReview ↗

First reply to reviewer ADm6 continued

Question 1: Complementary scaling mechanisms

A slow alignment delays the onset of the cold regime and may change the coefficient of the scaling, but it does not change the later dynamics β∼τ1/3\beta \sim \tau^{1/3}: once the student is aligned and grows its norm to become more confident, the 1/31/3 scaling holds regardless.

The reason is that there is no "interaction" between the two mechanisms, only a "handoff": the complementary mechanism works in the linear (warm, alignment) phase, while ours works in the nonlinear (cold) regime. We will emphasize this argument in the Related Work section to show the full picture. Thank you for bringing this up.

We have not studied the warm regime, or the transition to the cold regime, theoretically, which will be stated as a limitation. The intuition for the warm regime is that the student needs to see all the directions and align with them. Inevitably, more frequent directions in the data covariance have larger signals and are learned faster, which is why a power-law covariance in the input data can lead to a power-law loss (the complementary mechanism). When the alignment process ends depends on the smallest eigenvalue of the covariance. How the coefficient depends on the data covariance is under active study.

Question 2: Large learning rate

We expect the 1/31/3 scaling to hold as long as training remains stable: stochasticity does not speed up the logit growth, but it increases the coefficient of the scaling.

When the learning rate increases, the effect of gradient noise cannot be ignored. Starting from an SDE rather than simple gradient flow, the 1/31/3 scaling still holds (Kuhn et al., 2026). When the learning rate is increased further, the SDE approximation may break down, and the edge of stability may be reached. We did observe divergent loss at large learning rates, where the model cannot enter the cold regime at all. However, we do not have a theory for the breakdown of the SDE approximation, which can be discussed in the limitations and future work.

We do not think stochasticity will make the norm grow faster. It is the larger learning rate that helps the norm grow, given the same number of steps. But a larger learning rate introduces larger noise in the transverse directions, leading to higher loss. The first effect changes nothing if we plot the loss against τ\tau (which already accounts for the learning rate), but the second effect increases the coefficient of the 1/31/3 scaling.

If you are interested in the learning rate schedule, please also see our response to reviewer qKCN's Question 1.

I thank the authors for addressing my concern. Indeed, the reduction is not obvious and it leads to identifying attention heads as the cause of the power-law scaling, which is a significant result. Additionally, the answers to the questions clarified the overall picture on the learning dynamics for me. Therefore, I update my score to accept.

AuthorsOpenReview ↗

Second reply to reviewer ADm6

We sincerely thank the reviewer for the kind follow-up and for raising the score. We are glad that the answers helped clarify the overall picture of the learning dynamics.

If you have any further thoughts on what would make the paper stronger, we would be happy to hear them, including ones that may go beyond the scope of the current paper. We will try our best to address them in the revision, or else state them clearly as limitations and future work. Your feedback has already improved the paper.

We would also like to share some further thoughts on loss universality, which we could not fit into our first reply. It is the theoretical result we are most excited about.

  • From its generality, it can connect settings that look unrelated. One example is the classic online learning of a perceptron (Statistical Mechanics of Learning, Engel & Van den Broeck, 2001, Ch. 9). That setup has almost nothing in common with ours or with LLMs. The task is binary classification with sign outputs, not softmax. The student model is trained with the perceptron learning rule, which updates only on misclassified examples, instead of gradient descent. Performance is measured by the 0–1 generalization error. Yet this error is known to decay as α−1/3\alpha^{-1/3}, where α\alpha is the number of examples per input dimension (proportional to training time). This classic exponent can also be understood through loss universality. Without loss universality and our sequence of toy models of increasing complexity, it would be hard to imagine a possible link between this decades-old result and the late training of LLMs.
  • The conditions are as important as the proof. Identifying the conditions under which different losses inherit the same exponent took considerable exploration with toy models. The conditions we present are sufficient. Whether they are necessary, or minimal, remains open.
  • It points to a more fundamental and general theory. As the GH-II model shows (Appendix A.4), softmax is not the only nonlinearity that leads to 1/31/3 scaling. We conjecture that the 1/31/3 law holds whenever the nonlinearity and the loss jointly satisfy certain conditions. Specifying these conditions would clarify where emergent power laws come from. It would also raise the question of whether such nonlinearities are necessary for language modeling. In this sense, we see loss universality as opening a door to more fundamental principles of learning and language modeling.

We plan to add a short discussion of these points to the paper, and would welcome your view on whether it belongs in the main text or the Appendix. More broadly, we would love to hear any thoughts you have on these directions, as they would help make the discussion more inspiring to the mathematical community.

Finally, as promised in our first reply, the revision will describe in Related Work how data-driven scaling mechanisms hand off to our nonlinear mechanism. It will also expand the limitations to cover the warm-to-cold transition and very large learning rates. Any comments on these planned revisions are very welcome.

Reviewer_EZ5cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether attention concentration can explain one-third scaling in networks with internal softmax layers. It combines gate–head toy models, theoretical analysis, and experiments on Pythia and OLMo-2. The exponent is mathematically derived under specific assumptions, and the LLM experiments examine the relevance of the proposed mechanism.

Critical strengths
  1. The paper investigates deeper mechanisms behind scaling laws. Its hypothesis about attention concentration offers a fresh perspective on what limits learning. The toy models isolate the roles of different components and provide controlled tests. I consider this combination of mechanistic reasoning and careful experimental design a valuable contribution.
  2. The experiment using logit MSE is particularly interesting. It shows that similar scaling can persist without an output softmax loss, providing useful motivation for investigating internal attention layers. The progression from toy models to LLM experiments is clear, and the proposed diagnostics may help training.
Critical weaknesses
  1. The explanation could be clearer. The one-third exponent follows from specific theoretical assumptions, but it is less clear which of these conditions hold in the tested language models. The paper could distinguish the theoretical guarantees from the empirical observations and explain where the proposed mechanism is expected to apply.
  2. The evidence that attention is the dominant bottleneck remains indirect. The logit-MSE experiment shows that an output softmax loss is not necessary for the observed scaling, but it does not by itself establish attention as the cause. Attention-logit growth is also mainly assessed visually against reference slopes. Quantifying these trends would strengthen the connection between the proposed mechanism and the language-model results.
Questions for authors

I would appreciate it if the authors could provide some additional clarification on the following points:

  1. Could the authors comment on whether the fitted loss exponents remain close to one-third when different starting points of the fitting interval are considered?

  2. It would be helpful if the authors could report the fitted growth exponents for the attention heads highlighted in the main figures, to provide more quantitative support for the observed trends.

  3. Could the authors provide a brief explanation or derivation of the relative-error estimate in Eq. (41) in the appendix?

3 comments
AuthorsOpenReview ↗

First reply to reviewer EZ5c

We thank the reviewer for their positive views and rating. The reviewer's concerns are about the connection between theory and LLMs and the empirical evidence in LLMs. We address them below with proposed revisions.

Weakness 1: Theoretical assumptions and their validity in LLMs

This is a critical point and we thank the reviewer for their suggestion. Below we list the assumptions needed for a rigorous theory, and explain which have been verified in LLMs and which have not (but are expected to hold approximately based on observations).

  • Complex and diverse data. The theory requires complex and diverse data such that the hidden states are distributed all over the hidden space, i.e., the "energy gap" has non-zero density at zero. This is verified in attention heads of LLMs (Figure 4).
  • Softmax trying to concentrate. The theory requires the target distribution to be peaked, i.e., the softmax in the trainable model is trying to concentrate. This is verified in LLMs (Figure 4).
  • Reasonable loss. Theorem 1 or Result 2 (loss universality) requires a "reasonable loss" defined on probabilities, whose conditions cannot be verified rigorously for internal attention heads in LLMs, but are expected to hold approximately; otherwise, the heads would have trainability issues.
  • Aligned student. Our theory requires the student weights to be perfectly aligned with the teacher. This is not exactly true even in the toy model, but it holds approximately at small learning rates (see our reply to ADm6's Question 2), as our toy experiments confirm. We then extrapolate this to LLMs. In the toy model, the matrix need to rorate or align first and then grow in norm. Otherwise, the model produces confidently wrong predictions, which leads to unacceptably high loss. In LLMs, since the logits are already growing large, we expect (approximated) alignment to hold.
  • Gradient flow. In the derivation, we assumed that β\beta follows gradient flow. In LLMs this is violated by gradient noise, different optimizers, etc., and cannot be checked directly. We therefore tested it in GH models with different learning rates, optimizers, and architectures (mainly in the Appendix A): the 1/3 scaling of both the loss and β\beta persists, so we expect gradient flow to be a good approximation in LLMs as well.

Revision. We will add a detailed discussion to the Appendix and summarize it in the main text, either at the beginning of Section 4 or (our preference, space permitting) in the limitations paragraph of Section 6. We would welcome the reviewer's opinion.

Weakness 2: Evidence is indirect

In our view, the evidence based on attention logit growth in Figure 4 is direct evidence. If a quantitative prediction agrees with experiments, this is often considered direct evidence for the theory (but not a proof). That said, we agree that we should report the fitted growth exponent, 0.36±0.020.36\pm 0.02 (from the last 7 points). We have now also fitted the growth exponents of all attention heads in Pythia-12B. From layers 21 to 32, the 480 heads that try to concentrate have growth exponents close to 1/31/3, with a mean exponent of 0.320.32 (please see Figure in this link). We will add the fitted growth exponents to the main text (Figure 4) and the Appendix (after Figure 13).

The logit-MSE experiment by itself is not direct evidence. Rather, the results in Figure 5 should be read as auxiliary to those in Figure 4, providing additional support for the proposed mechanism.

Please see responses to Questions below

AuthorsOpenReview ↗

First reply to reviewer EZ5c continued

Question 1: Fitting window

We thank the reviewer for this question. In the field of scaling laws, choosing the fitting window is important in practice (Broken Neural Scaling Laws, 2023).

Based on our theory and toy model experiments, we expect the 1/31/3 scaling to hold in the late stage of training. The initial stage may not follow any power law, or may follow a different one (see all toy model experiments and Figure 4c). We therefore need to choose a starting point for fitting that excludes the initial stage while retaining enough points for fitting.

In practice, we fitted the loss with Eq. (13) while varying the starting point, obtaining the fitted exponent and its standard error. If we include the initial stage, the standard error is large, as those points do not follow the same power law. If we exclude too many points, the standard error also increases. We therefore choose the starting point that minimizes the standard error of the fit; such a minimum indeed exists (please see Figure in this link). The exponents reported in Figure 4c use this starting point. We will add the starting-point sweep results to the Appendix.

Question 2: Fitted growth exponents

Thank you for the suggestion. Please see our response to Weakness 2.

Question 3: Eq. (41)

The estimate was motivated by comparing the probability mass omitted by the two-state approximation with its distance from the one-hot teacher. Writing a=e−βΔϵa=e^{-\beta\Delta\epsilon} and R=∑i=3de−βΔϵiR=\sum_{i=3}^{d}e^{-\beta\Delta\epsilon_i}, we obtain

= \frac{R(1+a)}{a(1+a+R)} \leq (d-2)e^{-\beta(\epsilon_3^\uparrow-\epsilon_2^\uparrow)}.$$ This explains the exponential factor in Eq. (41). We will add more details to the proof. There is also a less heuristic and more direct proof. We can show $L=\Theta(\beta^{-1})$ by establishing upper and lower bounds. Lipschitz continuity gives $D(q^\uparrow,e_1)\leq C e^{-\beta\Delta\epsilon}$, whose average is $O(\beta^{-1})$. Conversely, inputs with $\Delta\epsilon\leq1/\beta$ have probability $\Theta(\beta^{-1})$, and their student distributions remain a fixed distance from the one-hot teacher. Continuity and positivity of the loss then give a matching $\Omega(\beta^{-1})$ lower bound. This establishes the loss scaling without requiring the relative-error estimate as an extra step.
AuthorsOpenReview ↗

Less than one day left in discussion period

Dear Reviewer EZ5c,

Thank you very much for your time and effort in reviewing.

With the discussion closing in less than 24 hours (Nov. 18 AOE), we want to follow up on our rebuttal. Please see if any of your concerns or questions have been addressed and if you have further questions.

We’d be grateful if you would revisit your assessment in light of the rebuttal. Deeply appreciated.

Best,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal

Overview

We thank all reviewers for their constructive feedback. The reviewers (ADm6, EZ5c, qKCN) were generally positive about the clarity, soundness, and informativeness of the paper. Their concerns centered on three aspects: the novelty of our contribution relative to previous single-softmax work (ADm6), the strength and directness of the empirical evidence in LLMs (EZ5c, qKCN), and how well the theoretical assumptions carry over to LLMs (EZ5c, qKCN). Their questions mainly concerned the robustness of the 1/31/3 scaling beyond the idealized setting, including fitting choices (EZ5c), learning rates and schedules (ADm6, qKCN), and interplay with other scaling mechanisms (ADm6), as well as its relation to known phenomena such as attention sinks (qKCN).

Summary of reply

  • On novelty (ADm6), we clarified that our central question, how multiple softmaxes shape training dynamics, cannot be answered by previous work. The finding that a multi-softmax model reduces to a single cold softmax is itself a contribution, and loss universality explains why an internal softmax dictates the scaling of the overall loss. Most importantly, our LLM results show that attention heads, not the LM head, are the bottleneck behind the 1/31/3 scaling, which corrects the claim in previous work.
  • On evidence (EZ5c, qKCN), we provided new quantitative results. The highlighted attention head grows with exponent 0.36±0.020.36\pm0.02, and the 480 concentrating heads in layers 21–32 of Pythia-12B have a mean growth exponent of 0.320.32. A sweep over the start of the fitting window supports the reported loss exponents (EZ5c).
  • On assumptions (EZ5c, qKCN), we listed which assumptions are directly verified in LLMs (peaked targets, non-zero gap density) and which hold approximately (reasonable loss, alignment, gradient flow), as supported by toy experiments across optimizers, learning rates, and architectures (EZ5c). We also explained that the saturation of the LM head is consistent with our theory: its target coldness is only moderately large and is reached early in training (qKCN). Finally, we answered the questions on large learning rates and slow alignment (ADm6), learning rate schedules and attention sinks (qKCN), and gave a more detailed derivation of the step in the proof of Theorem 1 that was questioned (EZ5c).

Summary of revision

We have made the following revisions. (1) Add a discussion of the theoretical assumptions and their status in LLMs to the Appendix, summarized in the main text (EZ5c). (2) Report fitted attention-logit growth exponents in Figure 4, and per-head exponent statistics for Pythia-12B in the main text and Appendix (EZ5c, qKCN). (3) Add the fitting-window sweep to the Appendix (EZ5c). (4) Explain the three regimes of coldness dynamics and why the LM head saturates, briefly in the main text and in detail in the Appendix (qKCN). (5) Expand the proof of Theorem 1 (EZ5c). (6) Phrase the link between attention concentration and loss scaling more cautiously, avoiding causal claims (qKCN). (7) Describe in Related Work how data-driven scaling mechanisms hand off to our nonlinear mechanism (ADm6). (8) Expand the limitations to cover the warm-to-cold transition and very large learning rates (ADm6), as well as learning rate schedules and the single logit-MSE experiment (qKCN).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.