acceptodds
Under review as a conference paper at ICLR 2027

Understanding Optimization from Activation and Gradient Spectra: A Modded-NanoGPT Study

Abstract

We study the activation covariance and per-sample gradient SVD spectra throughout training in a modded-NanoGPT study. We show that the tail exponents of these spectra reveal differences in internal geometry that the training loss does not. We propose using these spectra as practical predictors of training optimization in two ways. First, we show that the tail exponent of the activation covariance spectrum at an early checkpoint predicts the terminal token efficiency. In controlled experiments in which the runs reach the same terminal loss, it achieves a Spearman correlation of , compared with for the early test loss. Second, we show that along a cumulative chain of modded-NanoGPT interventions, the activation and gradient spectra together distinguish the changes in internal geometry that accompany gains in token efficiency, in throughput, or in both. For example, the MuonUntied transition improves token efficiency by while throughput rises by only , a shift marked primarily by a change in the activation covariance tail. Both signals hold across architectures and for model sizes up to 3.57B parameters. We explain this behavior with an exactly solvable kernel model of next-token prediction on modular arithmetic.

Primary area:
optimization
Keywords:
LLM Optimization, Training Dynamics, Activation Spectra, Representation Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dkebOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper looks at activation covariance spectra and per-sample gradient spectra as ways to understand what is happening during language-model training. Using modded-NanoGPT, the authors study how these spectra change with batch size and as they add architecture and optimizer changes one after another.

The main finding is that the activation-spectrum tail exponent, measured early in training, can rank batch-size settings by how many tokens they need to reach a target validation loss. The authors report a mean within-family Spearman correlation of 0.970, compared with 0.212 for early validation loss, and test this relationship on models with up to 3.57B parameters. They also look at activation and gradient spectra together to see what changes inside the model when token efficiency, throughput, or both improve.

To help explain this, the authors use an exactly solvable kernel model for modular-arithmetic next-token prediction. It illustrates why early spectral tails may pick up differences in how quickly different modes learn that early loss does not capture. They also run toy Transformer experiments to explore how spectral changes relate to the Fourier representations the models learn.

Critical strengths
  1. The early-prediction result is interesting. In the tested batch-size sweeps, the activation-spectrum tail exponent gives a much stronger ranking of eventual token cost than early validation loss. This suggests that internal representations contain useful information about training efficiency before the loss curves clearly separate.

  2. Separating token efficiency from throughput makes the analysis useful. It helps distinguish modifications that reduce the tokens needed to reach a target loss from those that mainly improve execution speed. The activation and gradient spectra provide complementary views of the internal changes accompanying these gains.

Critical weaknesses
  1. The comparison with early validation loss does not fully control the measurement budget. Appendix B specifies approximately 33.6 million validation tokens for the activation spectrum, whereas the released training scripts default to approximately 10.5 million tokens for validation loss. Both use FineWeb validation data, but the token counts and sequence segmentation differ. This leaves open whether the reported correlation gap partly reflects differences in measurement noise, weakening the evidence for a central claim of the paper. A stronger comparison would evaluate both metrics on the same inputs with matched token budgets and repeat the measurements across validation subsets to report ranking uncertainty.

  2. The large-model experiments cover a very short training horizon. From Figure 1(b), the 2.00B and 3.57B models reach the target loss after approximately 1.2–2.5% and 0.27–0.62% of the common 20-token-per-parameter 1xChinchilla reference, respectively. These results therefore concern a very early loss threshold. It is unclear whether the batch-size rankings remain stable later in training. The reported large-model results do not convince me that the diagnostic reliably predicts training efficiency over standard horizons.

  3. The practical benefit remains untested. The spectra are computed offline from saved checkpoints, without an experiment using them to select or adjust configurations during training. It is unclear whether the method saves compute compared with loss-based selection once diagnostic costs are included.

  4. Some spectral probes use a different numerical path from training. The default FP8Head analysis script replaces the FP8 head with an ordinary nn.Linear and omits the training autocast context. This mismatch may affect the measured gradient spectra.

Questions for authors
  1. How many validation tokens were actually used for the early-loss baseline and the activation-spectrum probe? Appendix B specifies approximately 33.6M tokens for the spectrum, whereas the released training scripts default to approximately 10.5M tokens for validation loss. Does the correlation advantage persist when both metrics use the same inputs and token budget?

  2. Wen et al. [1] evaluate optimizers at 1–8× Chinchilla budgets and show that early validation-loss rankings can reverse during learning-rate decay (§4.2, Figure 6). Your 2.00B and 3.57B experiments stop far below the common 20-token-per-parameter reference [2]. What evidence supports extending the observed spectral rankings beyond this early loss threshold? Do you have results at later target losses or longer training budgets showing that the rankings remain stable?

  3. Have you conducted a prospective experiment using the spectral ranking to select training configurations? If so, how much compute does it save compared with loss-based selection after including diagnostic costs? The offline correlations alone do not establish this practical benefit.

  4. Which FP8Head analysis script produced the reported results? The default 16_sa_fp8lmhead_1gpu.py uses an ordinary nn.Linear head and omits the training autocast context. Have you checked whether the gradient spectra and associated conclusions remain consistent when the probe reproduces the training computation?

  5. Please clarify the notation and labels in Figure 1(b). I could not tell what each point and the numbers next to it represent; these should be explained in the caption. Figure 1(c) and its text are also too small to read comfortably. I had to zoom in substantially to make them out. Please enlarge the panel and its labels.

  6. Figure 1(b) labels the 3.57B spectral measurement at 0.25B training tokens, while the bs2 configuration appears to reach the target loss at approximately 0.19B tokens. Was its spectrum measured after it had already reached the target? Since training is described as stopping at the target loss, which checkpoint was actually used? Could you clarify the actual probe timing for each configuration and show whether the ranking holds at a common checkpoint before any configuration reaches the target?

[1] Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fantastic Pretraining Optimizers and Where to Find Them. ICLR 2026. arXiv:2509.02046.

[2] Jordan Hoffmann et al. Training Compute-Optimal Large Language Models. 2022. arXiv:2203.15556.

Reviewer_R1W9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the internal optimization dynamics of large language models (LLMs) using trace-normalized activation covariance spectra and per-sample gradient singular value decomposition (SVD) spectra in the modded-NanoGPT framework.

Critical strengths

The authors make two primary empirical claims: 1. The tail exponent of the final-layer activation covariance spectrum (αtail\alpha_{tail}) measured early in training (e.g., at 20% progress) strongly ranks batch tiers by their later token efficiency (Spearman ρS=0.970\rho_S = 0.970), while early validation loss is a poor predictor (ρS=0.212\rho_S = 0.212). 2. Combining activation and gradient spectra allows classification of architectural and optimizer interventions into activation-dominant, gradient-dominant, execution-dominant, or mixed categories. The empirical findings are supported by a solvable kernel model of next-token prediction on modular arithmetic (Zc\mathbb{Z}_c).

Critical weaknesses
  1. The 16 architectural and optimizer interventions (from Baseline →\rightarrow RoPE →\rightarrow Muon →\rightarrow Untied →⋯→\rightarrow \dots \rightarrow AttnScale) are applied cumulatively. Each variant inherits all preceding modifications. Consequently, the observed spectral shift and performance gain attributed to a specific step (e.g., the Muon →\rightarrow Untied transition or FlexWin →\rightarrow VTE) are path-dependent and confounded by interactions with all prior modifications. It is impossible to isolate whether the spectral signature is characteristic of the named intervention or an artifact of the cumulative accumulation. Without isolating individual interventions against a common baseline, the "taxonomy of tricks" lack causal validity.

  2. The paper trains earlier and less efficient variants on the 100B-token split of FineWeb and switches to the 10B-token split from ValueMix onward. Although the authors argue that random token windows from both splits show a tiny Jensen-Shannon divergence (4.31×10−44.31 \times 10^{-4}), training across vastly different data volumes changes epoch repetition rates, data ordering effects, and representation geometry. This introduces an unquantified confounding variable right in the middle of the intervention chain. A rigorous study should maintain a uniform dataset split across all compared variants.

  3. The core claim is that activation spectra reveal differences between batch sizes that the loss does not. However, every batch tier undergoes an independent hyperparameter sweep (using ASHA) to select its optimal learning rate multiplier. Therefore, the differences in activation tails (αtail\alpha_{tail}) across batch tiers may simply reflect differences in the effective learning rate schedules and edge-of-stability dynamics instead of an intrinsic geometric property of batch size alone. The authors must demonstrate that batch tiers trained under identical (or non-tuned) learning rates show the same spectral divergence, or explicitly isolate learning-rate effects from batch-size effects.

  4. Theory-Practice Gap: To explain the empirical findings, the authors rely on an exactly solvable kernel model of next-token prediction on modular arithmetic (Zc\mathbb{Z}_c). Modular arithmetic relies heavily on periodic Fourier features and cyclic group structures, which show radically different spectral properties and learning dynamics compared to autoregressive transformers trained on web text (FineWeb). Claiming that a linear kernel model on Zc\mathbb{Z}_c explains multi-billion parameter LLM pretraining dynamics is a massive conceptual leap. The toy model proves what happens in that specific kernel regime, but it does not mathematically justify why αtail\alpha_{tail} behaves identically in deep non-linear transformers. The authors must tone down claims that the toy model "explains" the LLM results, and reframe it strictly as a qualitative pedagogical illustration.

Questions for authors

see Weaknesses.

Reviewer_i6pYOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies activation covariance and per-sample gradient spectra during language-model training. It finds that an early activation-tail exponent predicts the tokens needed to reach a target loss better than early loss-based diagnostics. It also uses spectral changes to characterize architectural and optimizer interventions. The experiments cover several modded-NanoGPT variants and models up to 3.57B parameters, supported by a solvable kernel model.

Critical strengths
  1. The early prediction result is strong: the activation-tail exponent achieves a within-family Spearman correlation of 0.970, compared with 0.212 for early test loss. The held-out-family analysis and larger-model results provide meaningful support beyond a single configuration.

  2. Studying activation and gradient spectra together provides a useful view of training dynamics. The intervention analysis documents internal changes that loss curves alone do not reveal, while the toy model illustrates how early loss and eventual token efficiency can favor different configurations.

Critical weaknesses
  1. The practical value of the ranking remains incompletely established. The comparisons do not show how much information the spectrum adds beyond batch size itself. Each batch tier also requires a separate LR search, so the reported correlations do not yet establish the total compute savings of using this diagnostic.

  2. The mechanistic interpretation is less convincing than the empirical prediction result. The cumulative intervention chain does not isolate individual effects, and the spectral taxonomy is largely descriptive. The kernel model explains a possible mechanism under specific assumptions, but those assumptions are not established for the language-model experiments.

Questions for authors
  1. Using the existing runs, does the activation-tail exponent improve batch-tier selection over a predictor based on batch size alone? How close is the selected tier to the token-optimal tier?

  2. Including LR tuning and spectral probing, what compute savings would selection at 0.5B tokens provide on the existing sweeps, compared with selection based on early loss?

  3. Which language-model measurements support the proposed mechanism, particularly the common rate-budget assumption and the interpretation of the fitted spectral band as unresolved modes?

1 comment

I regret that my questions went unanswered. Without clarification on the diagnostic’s advantage over batch size alone or its net compute savings, I am lowering my rating from 3 to 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.