acceptodds
Under review as a conference paper at ICLR 2027

Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws

Abstract

Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat the optimizer as a fixed training detail. We show that this assumption misses a fundamental axis of representation scaling: how effectively the optimizer converts added FFN width into utilized spectral capacity. Using eigenspectra of feed-forward network representations, measured through soft and hard spectral-ranks, we find that the same Transformer architecture realizes markedly different spectral scaling laws when trained with different optimizers. Holding architecture and width schedule fixed, AdamW exhibits weak hard-rank scaling (=0.44) on rare-token (TAIL) representations where learning is known to be hardest, whereas Muon achieves linear scaling (=\bf 1.02) in the same regimes, a increase in the scaling exponent. This difference is not reducible to validation loss: AdamW configurations can match low-rank Dion variants in perplexity, under extended training, while exhibiting sharply different spectral geometry, demonstrating that matched loss does not imply matched representation structure. Hard–soft rank asymmetry further reveals that optimizers differ not only in how much capacity is realized, but also in how that capacity is structured across eigenmodes. To disentangle optimizer effects from architectural ones, we compare against architectural interventions (e.g., attention rank and positional encoding), and find that optimizer-induced spectral shifts often exceed the architectural effects. These results suggest optimization as a first-class axis of representation scaling, motivating optimizer–architecture co-design.

Primary area:
optimization
Keywords:
representation scaling laws, optimization, spectral geometry, effective rank, feed-forward networks, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_D4RPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether spectral scaling laws for FFN representations depend on the optimizer. For each layer it computes the covariance spectrum of FFN post-activations and tracks two effective ranks: a soft rank (exponential of the Shannon entropy) and a hard rank (the participation ratio). It then fits power laws R(D)∝DβR(D) \propto D^{\beta} as the FFN width DD grows. The width sweep covers D=m⋅dmodelD = m \cdot d_{\text{model}} with m=1,…,8m = 1, \dots, 8 for a 160M modded-nanoGPT model and m=1,…,4m = 1, \dots, 4 for a 350M model, trained on FineWeb-Edu with AdamW, Muon, NorMuon and Dion at update-rank fractions from 1/2 to 1/16. Representations are also split by token frequency into HEAD, MID and TAIL buckets that each cover a third of occurrences. The main findings are:

  • Hard-rank exponents depend strongly on the optimizer (aggregate βhard\beta_{\text{hard}} 0.29 for AdamW vs. 0.82 for Muon; on TAIL tokens 0.44 vs. 1.02), while soft-rank exponents differ less.
  • The gap between soft- and hard-rank exponents is largest for AdamW.
  • AdamW trained twice as long matches Dion (1/16) in perplexity, but its hard-rank scaling disappears.
  • Lowering Dion's update rank steadily lowers hard-rank scaling.
  • The qualitative ordering holds at 350M.

The appendix adds a Rényi-order sweep, layer-wise fits, a learning-rate sweep, and architectural comparisons: fewer attention heads, removing RoPE, and partial Post-LN placement. From these the authors argue that optimizer effects on spectral scaling often exceed, and interact with, architectural effects.

Critical strengths

1. The Dion rank sweep is a clean, controlled result. Varying Dion's update-rank fraction keeps the orthonormalized update, the learning-rate scaling and the scalar optimizer fixed, so it isolates update rank better than any comparison involving AdamW. The effect is monotonic and well fit. TAIL hard-rank exponents fall from 0.88 to 0.67, 0.48 and 0.40 as rr goes from 1/2 to 1/16, all with R2≥0.98R^2 \ge 0.98. Soft-rank exponents fall more gently, from 0.95 to 0.72, and validation perplexity follows the same order. This is the most convincing evidence in the paper that properties of the optimizer update shape how added FFN width shows up in representation geometry, and it is the main point in the paper's favor.

2. The question is relevant, and the authors attempt several controls. Asking whether optimizers that reach similar loss also produce similar representation geometry is a useful complement to loss-level optimizer scaling studies. The paper includes:

  • an extended-training control and a small learning-rate sweep;
  • layer-wise fits showing the ordering is not produced by a few layers;
  • a 350M replication;
  • a Rényi-order sweep showing the separation already appears before the nonlinearity (pre-activation R2R_2 of 28.0 for AdamW vs. 164.8 for Muon at 350M).
Critical weaknesses

1. The AdamW comparison is confounded with the training recipe, in a way that directly affects width-scaling exponents. The matrix-aware optimizers use per-parameter spectral-norm learning-rate scaling (Yang et al., 2023) and Lion for scalar parameters. AdamW uses a single global learning rate at every width, with no warmup and no gradient clipping. Since β\beta is fit across FFN widths, an optimizer whose effective step size is not transferred across widths will produce distorted exponents for reasons unrelated to update geometry. There are signs that AdamW training is not well behaved across the sweep. Its perplexity is about 6 points worse than Muon's and is not monotonic in width (34.12 at 3d vs. 34.34 at 4d; 31.68 at 6d vs. 32.43 at 8d). At a learning rate of 6×10−36\times10^{-3} its fits collapse (R2=0.04R^2 = 0.04 for TAIL hard rank). The learning-rate sweep has only three values and does not touch warmup, clipping or width-dependent scaling. The headline result, a 2.3× larger TAIL hard-rank exponent for Muon, is an AdamW-vs.-Muon comparison. So this confound is the main reason I lean toward rejection: the gap may reflect parameterization and tuning rather than optimizer geometry.

2. The exponents are statistically fragile, and the capacity interpretation is not validated. Each (optimizer, width) point is a single seed, the error bars are inter-layer standard deviations, and fits use eight points at 160M and four at 350M. Several fits behind the main claims are weak:

  • AdamW aggregate βhard\beta_{\text{hard}} has R2=0.34R^2 = 0.34, and its MID fit 0.24±0.320.24 \pm 0.32 has R2=0.36R^2 = 0.36.
  • The headline TAIL value is 0.44±0.310.44 \pm 0.31.
  • Muon's aggregate exponent at 350M is 1.21, i.e. hard rank growing faster than width, which points to noise.
Questions for authors

Clarifications and additional results that could change my assessment

  1. AdamW parameterization. Can you rerun the 160M width sweep with AdamW using width-aware learning-rate scaling (μP or the same spectral-norm prescription), warmup and gradient clipping, and the same treatment of scalar parameters as the matrix optimizers? Does the gap in hard-rank exponents persist? Please also explain why AdamW perplexity is non-monotonic in width (e.g. 31.68 at 6d vs. 32.43 at 8d).

  2. Seed variation. What is the seed-to-seed variation of βhard\beta_{\text{hard}} and βsoft\beta_{\text{soft}}, for at least AdamW and Muon at 160M? How are the "±" uncertainties in Table 19 computed (standard error, confidence interval, bootstrap over layers)?

  3. Link to model quality. Can you report validation loss separately for HEAD, MID and TAIL tokens as a function of width for each optimizer? Does a larger hard-rank exponent correspond to better rare-token loss scaling? How do you reconcile AdamW 12K reaching lower perplexity with a hard-rank exponent near zero?

  4. Frequency buckets. The buckets differ greatly in the number of token types (30, 1,215, 49,059). Do the cross-regime conclusions hold if you match the number of types or tokens per bucket, or remove per-type means before computing the covariance?

  5. Effect-size comparisons. Can you compare a fixed optimizer pair (e.g. AdamW→Muon) against the architectural shifts at the same model scale, with uncertainties, instead of the best-of-four optimizer gain? Several entries in Tables 20–25 are smaller than the fit uncertainties reported in Table 19.

3 comments
AuthorsOpenReview ↗

Authors' Rebuttal

We thank the reviewer D4RP for identifying the controlled Dion sweep as the strongest evidence. Below, we answer all the five questions, with the additional results from the experiments performed during the rebuttal.

1. AdamW parameterization, tuning, and non-monotonic perplexity

First, the cleanest attribution is within Dion rank sweep experiments. Having a fixed model architecture and training recipe, changing the update-rank fraction 1/2 to 1/4 to 1/8 to 1/16 gives TAIL hard exponents 0.88, 0.67, 0.48, and 0.40, respectively, with R² ≥ .98. This identifies dependence on the rank setting within the tested optimizer family.

Second, Appendix F provides a scalar-learning-rate sweep. Third, we performed the weight-decay variation during the rebuttal period, below is the results.

AdamW weight decayTAIL soft βSoft-fit R²TAIL hard βHard-fit R²
.01.669.964.360.569
.10.614.968.425.830

The higher weight decay improves global and bucket perplexity across all the width configurations, while TAIL hard growth remains sublinear. That is, across the learning-rate and weight decay variation our observations on the qualitative trends for Soft and Hard Spectral rank scaling remains same, even when pre-training performance improved.

Finally, we do not tune the AdamW hyper-parameters across the width settings in order to isolate the effect of training recipe with optimizer family. Note that, the purpose of the spectral rank scaling study is not to find the best configurations for performance, rather showing how the optimizer selection and update rank itself change the representation geometry.

2. Seed variation, error bars, and fragile exponent estimates: We distinguish three sources of variation and provide the independent-seed results.

(a) The reported uncertainties have different meanings. The Layer variation and power-law slope uncertainty are different. Individual ranks are depth-averaged, and we've observed the substantial variation across the layers (see Fig. 7 & 8, Appendix D). However, layer heterogeneity does not resolve a poor aggregate fit, and we did the seed variation study.

QuantityWhat it measuresWhat it does not measure
Figure error bars (Fig. 1, 2, 4 & 5)Inter-layer standard deviation of rank measurementsIndependent training-seed variability
Appendix D layer fits (Fig. 7 & 8)Variation in separately fitted layer exponentsReplication across independently trained models

(b) Two completed seeds support qualitative trend. We trained the models with FFN width variations on a different seed, and below we showed that how the qualitative trends remain same for AdamW vs Muon.

OptimizerSeedTAIL soft βTAIL hard βHard-fit R²
AdamW1337.646.385.587
AdamW1234.629.311.480
Muon13371.0261.021.808
Muon12341.0731.182.891

(c) AdamW's hard rank fits remain weaker, and we found that GLOBAL exponents are more seed-sensitive. Moreover, the quality of fit itself is token-regime sensitive (see Table 1), where HEAD exhibits poorer fit while TAIL R^2 values are closer to 1. We therefore restrict the claim and do not treat low-R² fits as reliable numerical laws.

3. Functional relevance and frequency-conditioned model quality: The additional comparisons show different geometry, intervention sensitivity, and learning outcomes at nearly matched pretraining loss.

(a) Matched loss, spectral ranks, and frequency-bucket losses Within each width, compare AdamW-12K with Dion-1/16-6K. The 160M label denotes the 4d reference model; 2d has 133.9M parameters and 4d has 162.2M.

WidthFineWeb CE gap (Dion - AdamW)Soft-rank ratioHard-rank ratioMaximum bucket CE residual
1d+.008461.177×1.255×.00135
2d+.004061.191×1.287×.00056
4d+.023361.478×1.638×.01288
5d+.024271.261×1.590×.01495

Rank ratios are Dion/AdamW.

(b) Direct intervention on dominant activation directions We remove top post-activation principle-components (PCs) at a pre-specified layer, layer 5. We define C as the CE increase from top-PC removal minus the mean increase from three deterministic dimension-matched random removals.

WidthkVariance removed, AdamW / DionExcess CE C, AdamW / DionAdamW C / Dion C
2d12.62% / 1.93%.00372 / .001382.69×
2d56.41% / 6.44%.01232 / .003713.32×
2d2013.22% / 15.72%.02349 / .011142.11×
2d5021.46% / 25.49%.03598 / .017822.02×
4d12.30% / 1.13%.00209 / .000962.18×
4d57.72% / 4.87%.00671 / .003511.91×
4d2014.27% / 12.82%.01678 / .008791.91×
4d5021.38% / 21.69%.02818 / .014191.99×

(c) Downstream evaluation: All four evaluated widths favor Dion in LAMBADA likelihood

WidthDion PPL changeMean Δ log-likelihood, Dion−AdamWPaired 95% intervalAccuracy change
1d−3.27%+.03325[−.00405, +.07004]−.47 pp
2d−3.72%+.03787[+.00046, +.07516].00 pp
4d−4.92%+.05044[+.01543, +.08592]+.47 pp
5d−3.54%+.03608[−.00032, +.07183]+1.09 pp

(d) SFT on SCAN datasets For the matched-loss comparison, we apply a common SFT procedure on the official SCAN length split [1]

WidthPretraining configurationID correct at SFT-300ID correct at SFT-400Length-OOD correct at SFT-400ID validity at SFT-400
1dAdamW-12K1/1283/1280/12861.72%
1dDion-1/16-6K13/12823/1284/128100%
2dAdamW-12K0/1284/1280/12899.22%
2dDion-1/16-6K15/12833/1284/128100%

All four start at zero pre-SFT exact accuracy. For SFT, we used AdamW, LR 1e-5, batch 16, and 400 updates. Greedy evaluation has a shared 320-token limit. Each model has one pretraining seed (1234) and one SFT trajectory.

(e) SFT on Vietnamese dataset on the same matched-loss configurations Separate Vietnamese CulturaX continuation corroborates the target-learning ordering. Below is the result.

WidthPretraining configurationInitial target CEFinal target CEFinal FineWeb CE increase
1dAdamW-12K2.73182.1335+.0409
1dDion-1/16-6K2.72762.0201+.1007
2dAdamW-12K2.67542.0885+.0473
2dDion-1/16-6K2.69871.9603+.1534

Dion has a larger target-loss reduction and lower final target CE at both widths, with greater source forgetting. The settings are common within this task and differ from SCAN's protocol. Together, these SFT experiments demonstrate functional differences at nearly matched pretraining loss.

4. Frequency buckets: unequal type counts and centering The available controls support within-bucket comparisons (see Table 8, and Appendix A.4)

5. Fixed-pair architectural effect sizes Table 22 allows a fixed AdamW–Muon comparison at the 160M reference scale:

OptimizerTAIL hard β, 12 headsTAIL hard β, 6 headsAbsolute head-count change
AdamW.438.751.313
Muon1.0211.388.367

The 12-head Muon vs AdamW contrast is .583, while the architectural changes .313/.367. This avoids selecting the best optimizer from four candidates.

[1] Lake et al., Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks, ICML 2018

[2] Hernandez-Garcia et al., Can Scale Save Us From Plasticity Loss in Large Language Models? arXiv 2026

Thank you for the response and the additional experiments. The second seed is helpful: the AdamW–Muon ordering of TAIL hard-rank exponents is reproduced (0.39/0.31 vs. 1.02/1.18), although the AdamW fits remain weak (R² 0.48–0.59) and Muon's exponent is again at or above 1. My main concern, however, is not addressed. The weight-decay variation does not change the fact that AdamW uses a single global learning rate across widths, without warmup, clipping or width-aware scaling, while the matrix optimizers use spectral-norm scaling. Since β is fit across widths, this difference in parameterization can affect the exponents directly, and the requested width-aware AdamW rerun was not performed. The Dion rank sweep remains the clean result, but the headline claims rest on the AdamW–Muon comparison. The new functional evidence is interesting but does not answer my question: at matched loss, the frequency-bucket CE residuals are at most about 0.015 despite 1.2–1.6× differences in rank, which suggests that hard rank does not translate into rare-token loss. The LAMBADA, SCAN and Vietnamese results are single-seed, partly within their confidence intervals, and in the last case show a learning–forgetting trade-off rather than a clear advantage. The bucket-size question was not addressed with new results, and the fixed-pair contrast (0.58 vs. 0.31–0.37) is still reported without uncertainties. I therefore keep my score of 2.

Reviewer_cNFROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper says that optimizers should be considered as a part of how we scale representations not just as something fixed during training. When looking at the eigenspectra of FFN representations they show that the same GPT-2 architecture follows different spectral scaling laws when different optimizers are used. AdamW leads to hard-rank scaling for rare-token representations while Muon and NorMuon lead to almost linear scaling. Dion is in the middle, with its scaling following the update rank, which makes it easy to separate orthonormalisation from rank. The main claims are: the hard–soft gap is optimizer-dependent; matched loss does not mean matched geometry; optimizer-induced spectral shifts often beat architectural tweaks; and optimizer choice even widens the trainable normalisation space.

Critical strengths

Clean, well-motivated question. Treating the optimizer as a scaling axis is genuinely underexplored, and the FFN probe is a reasonable operationalization

I like the rank sweep design. Varying update rank while keeping orthonormalisation fixed lets the paper pin down which bit of Muon-style optimizers actually drives hard-capacity scaling. It's a properly controlled intervention, not just another optimizer bake-off.

Critical weaknesses
  • Only one random seed per configuration. The paper explicitly states: "All scaling-law fits use single seeds per (optimizer, FFN width) configuration; error bars in figures correspond to the inter-layer standard deviation of per-layer measurements." This is a fundamental statistical flaw. The paper's core claims are precise exponent differences, but each point rests on a single seed. The error bars in the figures are inter-layer standard deviations, which is a completely different thing from seed-to-seed variance.

  • Scale is too small; "scaling law" is overclaimed; 160M and 350M parameters, 3.15B / 4.19B tokens, 6000 / 8000 training steps. That's very small by modern LLM standards. The paper acknowledges this ("testing at 1B+ scale would further strengthen"), but the title and abstract wording ("scaling laws") implies more generality than the evidence supports. A more accurate framing would be "spectral scaling trends in small-scale Transformers".

  • Training is far from converged: 6000 steps and 3.15B tokens is also very short training for a 160M model. Validation perplexity sits at 28–38 (Table 9), well above what these models reach when properly trained. This raises a key question: is the observed "scaling law" just transient early-training behaviour? The paper's AdamW 12K extended-training experiment (Section 4.2) actually makes this worry worse: with longer training, AdamW's hard-rank scaling drops from 0.29 to 0.03, meaning the "scaling law" itself collapses over the course of training. If AdamW's exponent isn't even stable over 12K steps, does Muon's β=1.02 hold up under longer training?

Questions for authors
  • how much could different random seeds shift the fitted exponents? The paper's core claims rest on precise contrasts. however, with only one single seed, it's hard to know whether those differences are real. Would the authors consider at least 3–5 seeds with reported uncertainty on the exponents?

  • "Scaling law" usually implies power-law relationships validated across orders of magnitude. Would the authors consider softening the framing to "spectral scaling trends in small-scale Transformers", or do they have preliminary evidence the exponents hold at 1B+?

  • 6000–8000 steps is quite short, and validation perplexity (28–38) suggests the models are far from converged. Does Muon's β=1.02 survive extended training, and if not, what does that mean for the central claim?

  • the paper is purely empirical. Why do orthonormalized updates produce more concentrated spectra, and why does AdamW's extra width become diffuse spectral mass rather than dominant-mode capacity? Even a simplified linear-network analysis connecting Newton–Schulz orthogonalization to covariance spectra would substantially strengthen the contribution.

Minor: the tables are very hard to read.. font too small; Also, table 25 in appendix seems have some format issue as well

3 comments
AuthorsOpenReview ↗

Authors' Rebuttal

Thanks for the thorough reading and a detailed review, and also for recognizing the controlled Dion sweep. We tried our best to address each concern below.

Independent seeds and uncertainty

We have performed the additional seed experiments during the rebuttal time, and evidence supports qualitative the Soft-Hard spectral scaling trend in TAIL token regime. Particularly, we would like to discuss the source of uncertainty, and their implications for the qualitative trends observed in spectral scaling laws.

(1) Layer variation and power-law slope uncertainty are different. Individual ranks are depth-averaged, and we've observed the substantial variation across the layers (see Fig. 7 & 8, Appendix D). However, layer heterogeneity does not resolve a poor aggregate fit, and we did the seed variation study.

QuantityWhat it measuresWhat it does not measure
Figure error bars (Fig. 1, 2, 4 & 5)Inter-layer standard deviation of rank measurementsIndependent training-seed variability
Appendix D layer fits (Fig. 7 & 8)Variation in separately fitted layer exponentsReplication across independently trained models

At each width, we first took the median over last 5 rank values, from 5K to 6K steps, at 200-step intervals within each layer. Then, the median across 12 layers, followed by log-log OLS.

(2) Variation across training seeds We trained all the models with FFN width variations on a different seed, and below we showed that how the qualitative trends remain same for AdamW vs Muon, for 160M models.

OptimizerSeedTAIL soft βTAIL hard βHard-fit R²
AdamW1337.646.385.587
AdamW1234.629.311.480
Muon13371.0261.021.808
Muon12341.0731.182.891

(3) AdamW's hard rank fits remain weaker, and we found that GLOBAL exponents are more seed-sensitive. Moreover, the quality of fit itself is token-regime sensitive (see Table 1), where HEAD exhibits poorer fit while TAIL R^2 values are closer to 1. We therefore restrict the claim and do not treat low-R² fits as reliable numerical laws.

Model scale and generality of the claim

We agree with the requested scope correction, and we have mentioned this in limitation. We further like to clarify the score and limitation claims: (1) The 160M/350M labels identify the 4d base-model reference configurations, parameter counts change with FFN width; (2) our experiments cover finite width ranges (1d to 8d).

Our strongest evidence for optimizer-induced spectral rank scaling remain the Dion rank seep (Figure 4), where the only variable is Dion's update rank, and everything else -- model and training recipe -- remain same.

Dion update-rank fraction1/21/41/81/16
TAIL hard-rank exponent.88.67.48.40

Note that all four hard-rank fits have R² ≥ .98. Nonetheless, we will clarify the score in the draft as empirical FFN width-scaling trends in the tested small-scale Transformer families.

Training duration and persistence of Muon's exponent

We extend the AdamW to 12K only to match the pre-training performance with Muon-family, following the observations from [1]. Hence, persistence of β=1.02 under longer training is not established. The existing evidence supports two narrower conclusions:

(1) AdamW's width dependence changes with training. Appendix E reports these TAIL hard ranks:

FFN widthTraining stepTAIL hard rank
8d4K394
8d12K246
4d12K365

Validation loss improves while the 8d hard rank decreases and falls below 4d. The aggregate 12K training why validation loss keeps improving but the width utilization plateau (see the MID and TAIL trends for 8d width in Figure 9)

(2) Matching loss establishes neither convergence nor stable exponents. The observation supports separation between loss and representation geometry at the evaluated checkpoints. Our claim is that optimizer recipes and update-rank settings affect spectral utilization at specified budgets, and we do not claim that the fitted exponents are independent of training horizon.

Table readability and numerical corrections

We acknowledge this, and will improve the readability as suggested.

[1] Wen et al., Fantastic Pretraining Optimizers and Where to Find Them, ICLR 2026

Theoretical proof and mechanistic understanding for why should polar normalization affect R2R_2 more strongly than R1R_1?

Under active-subspace isotropy, polar normalization equalizes the singular amplitudes contributing to the one-step activation-covariance update and therefore maximizes R2R_2 (R1R_1 denotes soft rank, and R2R_2 hard rank). Moreover, compared to R1R_1, R2R_2 is more sensitive to deviations from a uniform spectrum.

For a (optimizer's) update rank-rr

ΔW=−ηUΣV⊤,ΔWpolar=−ηUV⊤,defineK(ΔW)=ΔWCxΔW⊤.\Delta W = -\eta U\Sigma V^\top, \qquad \Delta W_{\mathrm{polar}} = -\eta UV^\top, \qquad \text{define}\quad K(\Delta W) = \Delta W C_x \Delta W^\top.

If V⊤CxV=cIrV^\top C_x V = cI_r, the nonzero eigenvalues of K(ΔW)K(\Delta W) are proportional to σi2\sigma_i^2, whereas those of K(ΔWpolar)K(\Delta W_{\mathrm{polar}}) are equal. Hence

R2(K(ΔW))=(∑iσi2)2∑iσi4≤r=R2(K(ΔWpolar)),R_2(K(\Delta W)) = \frac{\left(\sum_i \sigma_i^2\right)^2}{\sum_i \sigma_i^4} \le r = R_2(K(\Delta W_{\mathrm{polar}})),

with equality only when the nonzero singular amplitudes are equal.

For normalized eigenvalues pi=1/r+δip_i = 1/r + \delta_i near uniformity,

log⁡R1=log⁡r−r2∑iδi2+O(∣δ∣3),andlog⁡R2=log⁡r−r∑iδi2+O(∣δ∣4).\log R_1 = \log r - \frac{r}{2}\sum_i \delta_i^2 + O(|\delta|^3), \qquad \text{and}\qquad \log R_2 = \log r - r\sum_i \delta_i^2 + O(|\delta|^4).

Thus, R2R_2 has twice the leading-order log sensitivity to spectral concentration. The spectral/operator-norm steepest-descent geometry motivating the exact-polar idealization follows [2], while [3] relates finite-step Newton--Schulz Muon to this polar map.

Note that, this is a one-step covariance result for the exact-polar idealization, not a theory of complete multi-step AdamW--Muon dynamics or the final post-GELU exponent. Finite-step Newton--Schulz Muon only approximates this map.

[2] Bernstein et al. Old Optimizer, New Norm: An Anthology. OPT Workshop, NeurIPS 2024

[3] Kim et al., Convergence of Muon with Newton-Schulz, ICLR 2026

Thank you for the detailed response and additional experiments. The second-seed results strengthen the evidence for the qualitative AdamW–Muon separation. I also appreciate the proposed scope correction and the simplified mechanistic analysis, which addresses my request for an initial theoretical explanation.

However, two concerns remain. First, although the additional seed is useful, two seeds without reported uncertainty across independent training runs still leave the precision of the fitted exponents insufficiently characterized. The results support the qualitative ordering more convincingly than the specific numerical exponent claims.

Second, and more importantly, the persistence of Muon’s stronger width dependence under extended training remains untested, as explicitly acknowledged in the response. Given the substantial training dependence observed for AdamW, the current evidence does not establish whether Muon’s reported behavior persists beyond the evaluated budget. An extended Muon width sweep would directly address this concern; the exponent need not remain exactly 1.02, but the stability of the qualitative separation should be established.

I welcome the narrower framing as empirical width-scaling trends in the tested small Transformer families at specified budgets. This qualification should be reflected consistently in the title, abstract, and conclusions. The additions improve the paper, but uncertainty in the exponents and robustness across training horizons remain unresolved.

Reviewer_Ps23OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how optimizer choice affects the scaling of FFN effective rank with hidden width. Across 160M and 350M base models, Muon-style optimizers show stronger hard-rank scaling than AdamW, particularly for mid- and low-frequency tokens. Extended-training comparisons also show that similar validation perplexity can coexist with different spectral geometry.

Critical strengths

A systematic empirical comparison across optimizers, FFN widths, and two model scales.

Token-frequency stratification and Dion update-rank ablations provide useful detail beyond aggregate spectral measurements.

Learning-rate and extended-training controls strengthen the observation that similar perplexity can coexist with different spectral geometry.

Critical weaknesses
  1. i have issues to detail the novelty aspect. last years "NerVE: Nonlinear eigenspectrum dynamics" (ICLR) showed already the Muon/Adam relation, with similar interpretation. The main novelty shrinks down to how the rank scales with width, it seems. Can the authors please comment on novelty? W.r.t. Nerve and the already cited "spectral scaling laws"?

  2. many hard rank fits have low R^2 value and only one seed which limits the explanatory statements

Questions for authors

in general clean and strong paper imho. before submission i'd like to see more seeds/ stronger discrimination about true derivable correlations.

my main concern resides with the novelty aspect.

1 comment
AuthorsOpenReview ↗

Authors' Rebutal

We thank the Reviewer Ps23 for appreciating our systematic study on controlled Dion rank sweep, which indeed is most convincing empirical results on optimizer-induced spectral scaling laws. Below, we first address the Novelty concerns and then reliability of hard-rank scaling exponents.

Novelty: NerVE (ICLR 2026) showed the representational effect of Muon/Dion, and Spectral Scaling Laws (EMNLP 2025) found the gap between Soft and Hard Spectral Rank Scaling. This work's novelty is three fold:

  1. First we showed that gap between Soft and Hard spectral rank scaling is not an artifact of transformer architecture -- gap is highest when trained with AdamW, and lowest when trained with Muon variants. And we have showed the systematic gap variation with Dion rank sweep, where update ran constraint appears to be a dial along the soft and hard rank scaling.

  2. Second is frequency-dependent optimizer-architecture effects. The Spectral rank scaling substantially differ across frequent-to-rare token regimes. In fact, we found the optimizer-depend scaling effect most prominent for rare-tokens. When contrasting the representational effect of architectural variation vs optimizer variation, we found that architectural effects dominates only for frequent token ( Figure 10 in Appendix H.1).

We believe that, this frequency-dependent optimizer-architecture study may benefit the application domains where more discriminative representation is required for rare-facts in the training data -- such as Medical and health-care applications -- where focus should be more on optimizer to enhance the representation of rare-data.

  1. Third is the matched-pretrianing-loss case study, where we showed that even when extended AdamW training achieve similar pre-training performance, their spectral rank scaling varies substantially, which has implications for downstream applications and future learning such as SFT.

Note that similar observation is found in [1] where they showed the matched-loss transfer learning difference, and reason them through flatness of loss curves and transferable features.

Low-R² fits, layer variability, and independent seeds: We agree that weak fits cannot support precise exponent claims. We have discussed the source of this variability in Appendix D, and during the rebuttal period, we ran the independent seed experiments to further demonstrate the consistency.

  1. What does Appendix D establish? Its independent layer fits (Figs. 7 & 8, and Tables 15 & 16) give median TAIL hard exponents .397 for AdamW, .746 for Muon, and .751 for Dion 1/2. This describes depth heterogeneity, as the source of variability in rank and exponent estimation.

  2. Replicate across training seeds? Below is the seed comparison across two seed:

OptimizerSeedTAIL soft βTAIL hard βHard-fit R²
AdamW1337.646.385.587
AdamW1234.629.311.480
Muon13371.0261.021.808
Muon12341.0731.182.891

Both seeds retain qualitative TAIL separation. AdamW's hard fits remain weaker, and GLOBAL exponents are more seed-sensitive. We agree that weak fits should be described cautiously as limited or non-monotonic width dependence.

  1. Liu et al., Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models, ICML 2023

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.