- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the effect optimizer has on the weight distribution of the model, and how it affects downstream compression techniques.
The authors gather statistics of the singular value spectra of weight matrices: spectral entropy, effective rank and power-law of the distribution tail. Then they compare the compressibility of the trained models and match it with the weight statistics.
Experimental results include GPT models trained with AdamW and Muon, while architecture and data were kept fixed. The model sizes are 124M and 350M while training data is 150M tokens, which is unusually small with token-to-parameters ratio ~ 1 . Compute-optimal data-to-parameters ratio is 20 ([1] Hoffman et al.) and often models are trained to ratios 100-1000 [2]. Loss values of ~4-5 additionally show the models are far from convergence.
Overall, the contribution of the paper is small. Experimental validation is very limited and the results intersect closely with previous work [2,3,4].
[1] https://arxiv.org/abs/2203.15556
Critical strengths
- The paper studies an important question of why model weights produced by different optimizers are easier / harder to compress. I believe it is a promising research direction.
Critical weaknesses
-
Experimental validation is weak. The model sizes are 124M and 350M. Training data is 150M tokens, which is unusually small and means results are not applicable to common regime of token-to-parameters ratio of at least 20 [1].
-
The results in the paper are not novel. Weights statistics such as spectral entropy has been compared between Muon and AdamW in [2]. Performance of optimizers under quantization have been analyzed in [3]. Muon having larger effective rank has been demonstrated, for example, in [4].
Questions for authors
I have no questions to the authors.