acceptodds
Under review as a conference paper at ICLR 2027

Weight-Scale Growth Tracks Training Effort, Not Learning Quality

Abstract

Does weight growth mean learning? Previous work showed that data predictability forecasts growth in the Weibull weight-scale parameter λ during transformer training. We test what this growth records, why it can diverge from generalization, and how it accumulates. In a controlled factorial grid crossing token shuffling with fixed-budget repetition, fully shuffled, highly repeated data produces the largest Δλ² but the worst held-out loss. Under fixed AdamW, Δλ² closely tracks accumulated radial-alignment work. A component- and time-resolved follow-up further shows that data conditions change both when this effort accumulates and whether it is routed toward Transmission or query/key Selection weights. Within the main grid, pairing Δλ² with the train–held generalization gap separates learning, memorization, and nothing-to-learn without seed-level label flips. The same frozen thresholds preserve this separation under matched corner interventions on C4 and Python code. These experiments trace weight-scale growth from controlled data construction through optimizer work and component-resolved weight evolution to behavior. Weight-scale growth therefore describes training activity, but alone does not determine whether the fitted structure generalizes.

Primary area:
optimization
Keywords:
training dynamics, AdamW, optimization effort, weight-space diagnostics, memorization and generalization, data repetition, Weibull weight scale, language model pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wDozOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors explore the underlying meaning behind the LLMs’ weight growth during the training as was discovered by the previous works: “learning or merely optimization effort.” In doing so, they build a testing framework based on the WikiText by incorporating a shuffling-repetition scheme. The authors conduct experiments on the grid of the described parameters without altering the optimization process and report the results. The findings reveal that the high weights’ magnitude does not necessarily mean better generalization. In fact, the largest magnitude is achieved by the experiments with the fully shuffled tokens and high repetition. By exploring other parameter combinations, the authors describe three regimes: “LEARNED, MEMORIZED, and NOTHING-TO-LEARN.”

Critical strengths
  • The experiments setup with the idea of the two parameters (shuffling-repetition) is a clean way to modify the data’s information capacity to suit the authors’ needs.
Critical weaknesses
  • The main problem is the lack of novelty. I do not see why this is in any significant way different from the classic ML overfitting theory. The same results can be achieved with the simple logistic regression: when there is no underlying learning signal (such as fully shuffled data) and the model is forced to minimize cross-entropy (high repetition), the optimizer simply memorizes the noise by constructing an arbitrary separating hyperplane. To maximize the margin on this arbitrary boundary, the model pushes its weight magnitudes to infinity. Therefore, the observation that fully shuffled, highly repeated data produces the largest weight-scale growth is a predictable consequence of margin maximization, not a novel discovery about transformer training. The paper in its current state simply adds experimental complexity to a phenomenon that is already well-understood in classical ML theory.
  • The authors note that "transformer weights grow systematically during training (Merrill et al., 2021)" and later cite Kosson et al., 2024. The latter is the more recent work that shows that unconstrained growth is actually not correct. Weight decay balances the weight increase, eventually reaching the "rotational equilibrium". The authors claim that the Δλ2\Delta \lambda^2 tracks "accumulated radial-alignment work." However, the summary admits that in high-repetition setting, weight growth shrinks due to WD (as the rotational equilibrium shows). Then, Δλ2\Delta \lambda^2 is not a measure of "training effort". Instead, the metric is heavily affected by the AdamW hyperparameters. Using a metric that is actively affected by the optimizer's regularization to measure "how much the optimizer is working" is a flawed premise.
Questions for authors
  • 8000 steps is a small budget for the training, where early-training dynamics effects can be still influential. Could you perhaps repeat your experiments while scaling down your model size? The current number of tokens (98M) seems too low to make any conclusions (way too far Chinchilla optimality).
  • How important is the learning rate decay to the obtained results? Optimization dynamics say that under a constant learning rate, the opposing forces of gradient updates and weight decay should eventually stabilize at a steady state (rotational equilibrium). When the learning rate decays, this equilibrium boundary artificially drops, allowing weight decay to dominate and pull the weight magnitudes down. What happens to these weight scale trajectories if you run the exact same experiments but keep the learning rate strictly constant?
Reviewer_tvnTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper attempts to sort out whether weight growth is an indicator of generalization or simply learning effort. They do this through a series of controlled experiments to determine if models with large weights always generalize or if this growth can be attributed to something else.

Critical strengths

[S1] Good controlled experiments showing that growth is not a function of generalizability.

Critical weaknesses

[W1] The novelty of this work is limited. The title claim that weight growth does not imply generalization has been established in prior work. E.g. Jiang*, Y., Neyshabur*, B., Mobahi, H., Krishnan, D., & Bengio, S. (2020). Fantastic Generalization Measures and Where to Find Them. International Conference on Learning Representations.

[W2] I found this paper extremely hard to read. The writing is difficult to parse and depends heavily on a lot of terms/jargon and notation.

Questions for authors

[Q1] In figure 2, for f = 0% why is the weight scale largest for p=16, while for all the other fs p=64 has the largest magnitude?

Reviewer_2ebhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether growth in transformer weights indicates better learning. In the main experiment, shuffled and heavily repeated text produces the largest weight growth, but the model memorizes training examples and performs poorly on unseen text.

Critical strengths
  • The experimental setup appears sound. The authors are careful to keep the amount of processed tokens fixed, and to separate the effects of shuffling and repetition from simply taking more training steps.
  • Investigating the significance of weight growth is worthwhile. These experiments show that greater weight growth does not necessarily imply better generalization performance.
Critical weaknesses
  • The paper is difficult to follow without already knowing the narrow line of work it builds on. This is a significant clarity issue: several terms are discussed before being explained. Examples include "central-body shape", "radial-alignment work" and "matched fitted-scale readout", all on the first page. For other terms, it is not clear if they are technical "data–optimization–weights–behavior chain" and "causal chain".

  • I think that the term "training effort" can be misleading. The authors point out that their measure of effort captures one contribution to weight growth and misses changes in weight direction. Moreover there are also several results in the loss of plasticity literature showing that weight changes can become small with time, as weight norms increase. In that regime, training for a long time will result in little further increase in the weight norm, but substantial "training effort". The title suggests a broader measure of training, and would benefit from focusing more on how/when such growth occurs vs the directional changes.

  • The paper's motivation does not clearly explain how it adds beyond existing understanding of weights and thier growth. Weight decay can improve generalization, some linear learning settings favor minimum-norm solutions, and work on loss of plasticity correlate increasing weight magnitudes with a reduction in the ability to learn. There are many reasons for not interpreting larger weights as better learning. Although the paper discusses some related work, it should explain more clearly its motivations and expectations in relation to existing work.

Questions for authors
  • Could the main text include a short example showing how the training text is shuffled and repeated, and how the separate test text is prepared? Furthermore, could these choices be ablated against to see whether the conclusions are specific to the setup?
  • What is learned from the fitted curve, which would otherwise be missed by measuring weight size with a Frobenius norm?
  • Can training and test losses alone distinguish the three groups (e.g., "learned", "memorized" and "nothing to learn")? What does measuring weight growth improve beyond those losses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.