acceptodds
Withdrawn from ICLR 2027

Optimizer Lineage: The Pretraining Optimizer Leaves a Causal Spectral Fingerprint on Base-Model Weights

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A base model is usually treated as a black box that downstream users prune, quantize, LoRA-adapt, or merge—as if these operations depend only on its architecture and final weights. We show that its weight spectrum also carries a durable, causal imprint of a choice fixed long before any user touches the checkpoint: the optimizer it was pretrained with. Adaptive optimizers such as Adam apply per-coordinate, effectively low-rank updates, whereas spectral optimizers such as Muon orthogonalize each update via a matrix-sign (msign) step, spreading energy across the full singular spectrum. Our single load-bearing result is a causal spectral fingerprint (H1). Under a clean-room control that pretrains 124M GPT pairs varying only the optimizer—architecture, data, initialization seed, and step count all held fixed— Muon raises the per-layer stable rank of hidden matrices by 6–14× (from  17 to  121) and lightens their spectral tail relative to Adam. The effect is unusually robust: it holds consistently across every layer, replicates across two independent seeds, and reproduces at two scales (124M and 350M). It is causal rather than a loss artifact—under an iso-loss control that trains Adam to matched (or lower) loss the gap persists and grows, at both scales (a 350M pair keeps a  12–25× stable-rank gap, p=7e-15)—so it tracks update geometry, not training progress. The fingerprint has a concrete deployment consequence: it sets a checkpoint’s compression budget. Muon-lineage weights are 5–7× less low-rank-truncatable and  29× less amenable to 4-bit quantization than Adam-lineage weights (again robust to the iso-loss control), so the pretraining optimizer—not just the architecture and final weights—determines how cheaply a base model can be compressed. Optimizer lineage is thus a measurable, causal, and deployment-relevant property that deserves to travel with a checkpoint as first-class metadata.

Primary area:
optimization
Keywords:
optimizer, Muon, Adam, weight spectrum, singular values, stable rank, model compression, quantization, low-rank, science of deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_EpwTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the effect optimizer has on the weight distribution of the model, and how it affects downstream compression techniques.

The authors gather statistics of the singular value spectra of weight matrices: spectral entropy, effective rank and power-law of the distribution tail. Then they compare the compressibility of the trained models and match it with the weight statistics.

Experimental results include GPT models trained with AdamW and Muon, while architecture and data were kept fixed. The model sizes are 124M and 350M while training data is 150M tokens, which is unusually small with token-to-parameters ratio ~ 1 . Compute-optimal data-to-parameters ratio is 20 ([1] Hoffman et al.) and often models are trained to ratios 100-1000 [2]. Loss values of ~4-5 additionally show the models are far from convergence.

Overall, the contribution of the paper is small. Experimental validation is very limited and the results intersect closely with previous work [2,3,4].

[1] https://arxiv.org/abs/2203.15556

[2] https://arxiv.org/abs/2502.16982

[3] https://proceedings.iclr.cc/paper_files/paper/2026/file/99ac2e91bacb6102a524464dae107bb3-Paper-Conference.pdf

[4] https://arxiv.org/pdf/2605.10468

Critical strengths
  1. The paper studies an important question of why model weights produced by different optimizers are easier / harder to compress. I believe it is a promising research direction.
Critical weaknesses
  1. Experimental validation is weak. The model sizes are 124M and 350M. Training data is 150M tokens, which is unusually small and means results are not applicable to common regime of token-to-parameters ratio of at least 20 [1].

  2. The results in the paper are not novel. Weights statistics such as spectral entropy has been compared between Muon and AdamW in [2]. Performance of optimizers under quantization have been analyzed in [3]. Muon having larger effective rank has been demonstrated, for example, in [4].

Questions for authors

I have no questions to the authors.

Reviewer_a8GGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how pretraining with Muon or AdamW affects the singular-value spectra of model weights and their compressibility. The results show that weights optimized with Muon have substantially higher stable rank than weights optimized with AdamW. Further experiments show that models optimized with Muon suffer larger performance drops after low-rank truncation and 4-bit quantization.

Critical strengths

The paper connects spectral differences to model compressibility. The results suggest that the choice of pretraining optimizer can affect downstream compression performance, which is a useful practical observation.

Critical weaknesses
  • The observation that Muon and AdamW produce different weight spectra is not entirely new, as similar findings have been reported in previous work (e.g., arXiv: 2502.16982). The connection between higher stable rank and reduced low-rank compressibility is also relatively straightforward, limiting the novelty of the paper.
  • The proposed lineage-matched PEFT method has not been empirically evaluated. It remains unclear whether matching the fine-tuning optimizer to the pretraining optimizer provides any benefit over existing approaches.
Questions for authors

Could the authors evaluate the proposed lineage-matched PEFT method? In particular, how does Muon-based fine-tuning compare with AdamW-based fine-tuning on both Muon- and AdamW-pretrained models? This would help determine whether matching the optimizer lineage offers any benefit beyond the choice of fine-tuning optimizer itself.

Reviewer_zSFPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how pretraining with MUON or AdamW affects weight spectra, model compression, and parameter-efficient fine-tuning. The authors propose three hypotheses: H1 concerns optimizer-dependent weight spectra, H2 examines their downstream effects, and H3 proposes a lineage-matched PEFT method. Experiments on 124M and 350M models show differences in stable rank, low-rank compressibility, and quantization. However, H1 overlaps with prior work, H2 lacks convincing evidence of downstream implications, and H3 is not evaluated.

Critical strengths

The paper studies a practical question: whether the choice of pretraining optimizer affects downstream compression and adaptation.

The authors report negative results on pruning, LoRA rank, model connectivity, and fine-tuning dynamics, helping clarify the limitations of the proposed spectral fingerprint.

Critical weaknesses

H1 and the quantization part of H2 largely overlap with NuMuon and Beyond Outliers, without establishing a new qualitative finding or a causal link between weight spectra and quantization.

Low-rank truncation largely follows from the spectral measurements, the LoRA experiment saturates at r=1r=1, and H3 is not evaluated, leaving both H2 and H3 insufficiently supported.

Questions for authors

What new qualitative finding does this paper provide beyond NuMuon and Beyond Outliers, and how is the quantization result reconciled with Metis?

Can the authors evaluate LoRA on a non-saturated task and provide experimental results for lineage-matched PEFT?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.