- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper looks at activation covariance spectra and per-sample gradient spectra as ways to understand what is happening during language-model training. Using modded-NanoGPT, the authors study how these spectra change with batch size and as they add architecture and optimizer changes one after another.
The main finding is that the activation-spectrum tail exponent, measured early in training, can rank batch-size settings by how many tokens they need to reach a target validation loss. The authors report a mean within-family Spearman correlation of 0.970, compared with 0.212 for early validation loss, and test this relationship on models with up to 3.57B parameters. They also look at activation and gradient spectra together to see what changes inside the model when token efficiency, throughput, or both improve.
To help explain this, the authors use an exactly solvable kernel model for modular-arithmetic next-token prediction. It illustrates why early spectral tails may pick up differences in how quickly different modes learn that early loss does not capture. They also run toy Transformer experiments to explore how spectral changes relate to the Fourier representations the models learn.
Critical strengths
-
The early-prediction result is interesting. In the tested batch-size sweeps, the activation-spectrum tail exponent gives a much stronger ranking of eventual token cost than early validation loss. This suggests that internal representations contain useful information about training efficiency before the loss curves clearly separate.
-
Separating token efficiency from throughput makes the analysis useful. It helps distinguish modifications that reduce the tokens needed to reach a target loss from those that mainly improve execution speed. The activation and gradient spectra provide complementary views of the internal changes accompanying these gains.
Critical weaknesses
-
The comparison with early validation loss does not fully control the measurement budget. Appendix B specifies approximately 33.6 million validation tokens for the activation spectrum, whereas the released training scripts default to approximately 10.5 million tokens for validation loss. Both use FineWeb validation data, but the token counts and sequence segmentation differ. This leaves open whether the reported correlation gap partly reflects differences in measurement noise, weakening the evidence for a central claim of the paper. A stronger comparison would evaluate both metrics on the same inputs with matched token budgets and repeat the measurements across validation subsets to report ranking uncertainty.
-
The large-model experiments cover a very short training horizon. From Figure 1(b), the 2.00B and 3.57B models reach the target loss after approximately 1.2–2.5% and 0.27–0.62% of the common 20-token-per-parameter 1xChinchilla reference, respectively. These results therefore concern a very early loss threshold. It is unclear whether the batch-size rankings remain stable later in training. The reported large-model results do not convince me that the diagnostic reliably predicts training efficiency over standard horizons.
-
The practical benefit remains untested. The spectra are computed offline from saved checkpoints, without an experiment using them to select or adjust configurations during training. It is unclear whether the method saves compute compared with loss-based selection once diagnostic costs are included.
-
Some spectral probes use a different numerical path from training. The default FP8Head analysis script replaces the FP8 head with an ordinary
nn.Linearand omits the training autocast context. This mismatch may affect the measured gradient spectra.
Questions for authors
-
How many validation tokens were actually used for the early-loss baseline and the activation-spectrum probe? Appendix B specifies approximately 33.6M tokens for the spectrum, whereas the released training scripts default to approximately 10.5M tokens for validation loss. Does the correlation advantage persist when both metrics use the same inputs and token budget?
-
Wen et al. [1] evaluate optimizers at 1–8× Chinchilla budgets and show that early validation-loss rankings can reverse during learning-rate decay (§4.2, Figure 6). Your 2.00B and 3.57B experiments stop far below the common 20-token-per-parameter reference [2]. What evidence supports extending the observed spectral rankings beyond this early loss threshold? Do you have results at later target losses or longer training budgets showing that the rankings remain stable?
-
Have you conducted a prospective experiment using the spectral ranking to select training configurations? If so, how much compute does it save compared with loss-based selection after including diagnostic costs? The offline correlations alone do not establish this practical benefit.
-
Which FP8Head analysis script produced the reported results? The default
16_sa_fp8lmhead_1gpu.pyuses an ordinarynn.Linearhead and omits the training autocast context. Have you checked whether the gradient spectra and associated conclusions remain consistent when the probe reproduces the training computation? -
Please clarify the notation and labels in Figure 1(b). I could not tell what each point and the numbers next to it represent; these should be explained in the caption. Figure 1(c) and its text are also too small to read comfortably. I had to zoom in substantially to make them out. Please enlarge the panel and its labels.
-
Figure 1(b) labels the 3.57B spectral measurement at 0.25B training tokens, while the bs2 configuration appears to reach the target loss at approximately 0.19B tokens. Was its spectrum measured after it had already reached the target? Since training is described as stopping at the target loss, which checkpoint was actually used? Could you clarify the actual probe timing for each configuration and show whether the ranking holds at a common checkpoint before any configuration reaches the target?
[1] Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fantastic Pretraining Optimizers and Where to Find Them. ICLR 2026. arXiv:2509.02046.
[2] Jordan Hoffmann et al. Training Compute-Optimal Large Language Models. 2022. arXiv:2203.15556.
I regret that my questions went unanswered. Without clarification on the diagnostic’s advantage over batch size alone or its net compute savings, I am lowering my rating from 3 to 2.