- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper asks whether spectral scaling laws for FFN representations depend on the optimizer. For each layer it computes the covariance spectrum of FFN post-activations and tracks two effective ranks: a soft rank (exponential of the Shannon entropy) and a hard rank (the participation ratio). It then fits power laws as the FFN width grows. The width sweep covers with for a 160M modded-nanoGPT model and for a 350M model, trained on FineWeb-Edu with AdamW, Muon, NorMuon and Dion at update-rank fractions from 1/2 to 1/16. Representations are also split by token frequency into HEAD, MID and TAIL buckets that each cover a third of occurrences. The main findings are:
- Hard-rank exponents depend strongly on the optimizer (aggregate 0.29 for AdamW vs. 0.82 for Muon; on TAIL tokens 0.44 vs. 1.02), while soft-rank exponents differ less.
- The gap between soft- and hard-rank exponents is largest for AdamW.
- AdamW trained twice as long matches Dion (1/16) in perplexity, but its hard-rank scaling disappears.
- Lowering Dion's update rank steadily lowers hard-rank scaling.
- The qualitative ordering holds at 350M.
The appendix adds a Rényi-order sweep, layer-wise fits, a learning-rate sweep, and architectural comparisons: fewer attention heads, removing RoPE, and partial Post-LN placement. From these the authors argue that optimizer effects on spectral scaling often exceed, and interact with, architectural effects.
Critical strengths
1. The Dion rank sweep is a clean, controlled result. Varying Dion's update-rank fraction keeps the orthonormalized update, the learning-rate scaling and the scalar optimizer fixed, so it isolates update rank better than any comparison involving AdamW. The effect is monotonic and well fit. TAIL hard-rank exponents fall from 0.88 to 0.67, 0.48 and 0.40 as goes from 1/2 to 1/16, all with . Soft-rank exponents fall more gently, from 0.95 to 0.72, and validation perplexity follows the same order. This is the most convincing evidence in the paper that properties of the optimizer update shape how added FFN width shows up in representation geometry, and it is the main point in the paper's favor.
2. The question is relevant, and the authors attempt several controls. Asking whether optimizers that reach similar loss also produce similar representation geometry is a useful complement to loss-level optimizer scaling studies. The paper includes:
- an extended-training control and a small learning-rate sweep;
- layer-wise fits showing the ordering is not produced by a few layers;
- a 350M replication;
- a Rényi-order sweep showing the separation already appears before the nonlinearity (pre-activation of 28.0 for AdamW vs. 164.8 for Muon at 350M).
Critical weaknesses
1. The AdamW comparison is confounded with the training recipe, in a way that directly affects width-scaling exponents. The matrix-aware optimizers use per-parameter spectral-norm learning-rate scaling (Yang et al., 2023) and Lion for scalar parameters. AdamW uses a single global learning rate at every width, with no warmup and no gradient clipping. Since is fit across FFN widths, an optimizer whose effective step size is not transferred across widths will produce distorted exponents for reasons unrelated to update geometry. There are signs that AdamW training is not well behaved across the sweep. Its perplexity is about 6 points worse than Muon's and is not monotonic in width (34.12 at 3d vs. 34.34 at 4d; 31.68 at 6d vs. 32.43 at 8d). At a learning rate of its fits collapse ( for TAIL hard rank). The learning-rate sweep has only three values and does not touch warmup, clipping or width-dependent scaling. The headline result, a 2.3× larger TAIL hard-rank exponent for Muon, is an AdamW-vs.-Muon comparison. So this confound is the main reason I lean toward rejection: the gap may reflect parameterization and tuning rather than optimizer geometry.
2. The exponents are statistically fragile, and the capacity interpretation is not validated. Each (optimizer, width) point is a single seed, the error bars are inter-layer standard deviations, and fits use eight points at 160M and four at 350M. Several fits behind the main claims are weak:
- AdamW aggregate has , and its MID fit has .
- The headline TAIL value is .
- Muon's aggregate exponent at 350M is 1.21, i.e. hard rank growing faster than width, which points to noise.
Questions for authors
Clarifications and additional results that could change my assessment
-
AdamW parameterization. Can you rerun the 160M width sweep with AdamW using width-aware learning-rate scaling (μP or the same spectral-norm prescription), warmup and gradient clipping, and the same treatment of scalar parameters as the matrix optimizers? Does the gap in hard-rank exponents persist? Please also explain why AdamW perplexity is non-monotonic in width (e.g. 31.68 at 6d vs. 32.43 at 8d).
-
Seed variation. What is the seed-to-seed variation of and , for at least AdamW and Muon at 160M? How are the "±" uncertainties in Table 19 computed (standard error, confidence interval, bootstrap over layers)?
-
Link to model quality. Can you report validation loss separately for HEAD, MID and TAIL tokens as a function of width for each optimizer? Does a larger hard-rank exponent correspond to better rare-token loss scaling? How do you reconcile AdamW 12K reaching lower perplexity with a hard-rank exponent near zero?
-
Frequency buckets. The buckets differ greatly in the number of token types (30, 1,215, 49,059). Do the cross-regime conclusions hold if you match the number of types or tokens per bucket, or remove per-type means before computing the covariance?
-
Effect-size comparisons. Can you compare a fixed optimizer pair (e.g. AdamW→Muon) against the architectural shifts at the same model scale, with uncertainties, instead of the best-of-four optimizer gain? Several entries in Tables 20–25 are smaller than the fit uncertainties reported in Table 19.
3 comments
(c) Downstream evaluation: All four evaluated widths favor Dion in LAMBADA likelihood
| Width | Dion PPL change | Mean Δ log-likelihood, Dion−AdamW | Paired 95% interval | Accuracy change |
|---|---|---|---|---|
| 1d | −3.27% | +.03325 | [−.00405, +.07004] | −.47 pp |
| 2d | −3.72% | +.03787 | [+.00046, +.07516] | .00 pp |
| 4d | −4.92% | +.05044 | [+.01543, +.08592] | +.47 pp |
| 5d | −3.54% | +.03608 | [−.00032, +.07183] | +1.09 pp |
(d) SFT on SCAN datasets For the matched-loss comparison, we apply a common SFT procedure on the official SCAN length split [1]
| Width | Pretraining configuration | ID correct at SFT-300 | ID correct at SFT-400 | Length-OOD correct at SFT-400 | ID validity at SFT-400 |
|---|---|---|---|---|---|
| 1d | AdamW-12K | 1/128 | 3/128 | 0/128 | 61.72% |
| 1d | Dion-1/16-6K | 13/128 | 23/128 | 4/128 | 100% |
| 2d | AdamW-12K | 0/128 | 4/128 | 0/128 | 99.22% |
| 2d | Dion-1/16-6K | 15/128 | 33/128 | 4/128 | 100% |
All four start at zero pre-SFT exact accuracy. For SFT, we used AdamW, LR 1e-5, batch 16, and 400 updates. Greedy evaluation has a shared 320-token limit. Each model has one pretraining seed (1234) and one SFT trajectory.
(e) SFT on Vietnamese dataset on the same matched-loss configurations Separate Vietnamese CulturaX continuation corroborates the target-learning ordering. Below is the result.
| Width | Pretraining configuration | Initial target CE | Final target CE | Final FineWeb CE increase |
|---|---|---|---|---|
| 1d | AdamW-12K | 2.7318 | 2.1335 | +.0409 |
| 1d | Dion-1/16-6K | 2.7276 | 2.0201 | +.1007 |
| 2d | AdamW-12K | 2.6754 | 2.0885 | +.0473 |
| 2d | Dion-1/16-6K | 2.6987 | 1.9603 | +.1534 |
Dion has a larger target-loss reduction and lower final target CE at both widths, with greater source forgetting. The settings are common within this task and differ from SCAN's protocol. Together, these SFT experiments demonstrate functional differences at nearly matched pretraining loss.
4. Frequency buckets: unequal type counts and centering The available controls support within-bucket comparisons (see Table 8, and Appendix A.4)
5. Fixed-pair architectural effect sizes Table 22 allows a fixed AdamW–Muon comparison at the 160M reference scale:
| Optimizer | TAIL hard β, 12 heads | TAIL hard β, 6 heads | Absolute head-count change |
|---|---|---|---|
| AdamW | .438 | .751 | .313 |
| Muon | 1.021 | 1.388 | .367 |
The 12-head Muon vs AdamW contrast is .583, while the architectural changes .313/.367. This avoids selecting the best optimizer from four candidates.
[1] Lake et al., Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks, ICML 2018
[2] Hernandez-Garcia et al., Can Scale Save Us From Plasticity Loss in Large Language Models? arXiv 2026
Thank you for the response and the additional experiments. The second seed is helpful: the AdamW–Muon ordering of TAIL hard-rank exponents is reproduced (0.39/0.31 vs. 1.02/1.18), although the AdamW fits remain weak (R² 0.48–0.59) and Muon's exponent is again at or above 1. My main concern, however, is not addressed. The weight-decay variation does not change the fact that AdamW uses a single global learning rate across widths, without warmup, clipping or width-aware scaling, while the matrix optimizers use spectral-norm scaling. Since β is fit across widths, this difference in parameterization can affect the exponents directly, and the requested width-aware AdamW rerun was not performed. The Dion rank sweep remains the clean result, but the headline claims rest on the AdamW–Muon comparison. The new functional evidence is interesting but does not answer my question: at matched loss, the frequency-bucket CE residuals are at most about 0.015 despite 1.2–1.6× differences in rank, which suggests that hard rank does not translate into rare-token loss. The LAMBADA, SCAN and Vietnamese results are single-seed, partly within their confidence intervals, and in the last case show a learning–forgetting trade-off rather than a clear advantage. The bucket-size question was not addressed with new results, and the fixed-pair contrast (0.58 vs. 0.31–0.37) is still reported without uncertainties. I therefore keep my score of 2.
Authors' Rebuttal
We thank the reviewer D4RP for identifying the controlled Dion sweep as the strongest evidence. Below, we answer all the five questions, with the additional results from the experiments performed during the rebuttal.
1. AdamW parameterization, tuning, and non-monotonic perplexity
First, the cleanest attribution is within Dion rank sweep experiments. Having a fixed model architecture and training recipe, changing the update-rank fraction 1/2 to 1/4 to 1/8 to 1/16 gives TAIL hard exponents 0.88, 0.67, 0.48, and 0.40, respectively, with R² ≥ .98. This identifies dependence on the rank setting within the tested optimizer family.
Second, Appendix F provides a scalar-learning-rate sweep. Third, we performed the weight-decay variation during the rebuttal period, below is the results.
The higher weight decay improves global and bucket perplexity across all the width configurations, while TAIL hard growth remains sublinear. That is, across the learning-rate and weight decay variation our observations on the qualitative trends for Soft and Hard Spectral rank scaling remains same, even when pre-training performance improved.
Finally, we do not tune the AdamW hyper-parameters across the width settings in order to isolate the effect of training recipe with optimizer family. Note that, the purpose of the spectral rank scaling study is not to find the best configurations for performance, rather showing how the optimizer selection and update rank itself change the representation geometry.
2. Seed variation, error bars, and fragile exponent estimates: We distinguish three sources of variation and provide the independent-seed results.
(a) The reported uncertainties have different meanings. The Layer variation and power-law slope uncertainty are different. Individual ranks are depth-averaged, and we've observed the substantial variation across the layers (see Fig. 7 & 8, Appendix D). However, layer heterogeneity does not resolve a poor aggregate fit, and we did the seed variation study.
(b) Two completed seeds support qualitative trend. We trained the models with FFN width variations on a different seed, and below we showed that how the qualitative trends remain same for AdamW vs Muon.
(c) AdamW's hard rank fits remain weaker, and we found that GLOBAL exponents are more seed-sensitive. Moreover, the quality of fit itself is token-regime sensitive (see Table 1), where HEAD exhibits poorer fit while TAIL R^2 values are closer to 1. We therefore restrict the claim and do not treat low-R² fits as reliable numerical laws.
3. Functional relevance and frequency-conditioned model quality: The additional comparisons show different geometry, intervention sensitivity, and learning outcomes at nearly matched pretraining loss.
(a) Matched loss, spectral ranks, and frequency-bucket losses Within each width, compare AdamW-12K with Dion-1/16-6K. The 160M label denotes the 4d reference model; 2d has 133.9M parameters and 4d has 162.2M.
Rank ratios are Dion/AdamW.
(b) Direct intervention on dominant activation directions We remove top post-activation principle-components (PCs) at a pre-specified layer, layer 5. We define C as the CE increase from top-PC removal minus the mean increase from three deterministic dimension-matched random removals.