- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper develops a unified analysis of Kronecker approximation that unifies the three existing divergences (Frobenius、von Neumann、KL) into a trace-power Bregman family and introduces a fourth one, which yields a new method SQ-Shampoo. They show how the family exponent determines which parts of the spectrum are prioritized. They also show that empirical gradient second moments severely underestimate tail eigenvalues on GPT-2.
Critical strengths
This paper studies Shampoo, an important optimizer in LLM training. They develops unified analysis of Kronecker approximation that unifies the three existing divergences (Frobenius、von Neumann、KL) into a trace-power Bregman family and introduces a fourth one, which yields a new method SQ-Shampoo.
Critical weaknesses
- Core training results lack replication, making it difficult to assess robustness.
The GPT-2 training explicitly states that each configuration was run only once, with no multiple random seeds, error bars, or significance analysis. The differences between SQ and KL across multiple widths are very small—for instance, nearly identical at width 768, and only approximately 0.006 loss difference at width 1536. It is currently impossible to determine whether these differences exceed training stochasticity. The authors also note that no further per-method learning rate search was conducted beyond width 768. Given that training performance is one of the paper's main conclusions, this insufficient evidence is my primary reason for leaning toward rejection.
- Significant gap between finite-sample theory and practical problems.
Proposition 3 assumes gradients come from an exact Kronecker Gaussian model, and when estimating one factor, the other factor is known and fixed. While this isolates the effect of divergence, it differs substantially from real training scenarios involving non-Kronecker distributions, non-stationarity, joint estimation of both factors, and the use of EMA. The authors verify the empirical ordering on a synthetic non-Kronecker distribution, but this remains insufficient to support generalizing "KL is most stable, F is least stable." I recommend more clearly distinguishing between theoretical guarantees, synthetic experimental observations, and empirical phenomena in real training.
Questions for authors
No
3 comments
Follow-up on Our Response
Dear Reviewer Sw37,
With the discussion period ending soon, we wanted to briefly check whether our response has addressed your concerns, especially the new 3-seed per-method LR sweeps across all widths and the clarified scope of Proposition 3.
Please let us know if you have any remaining questions before the discussion closes. Thank you again for your time and feedback!
Best regards,
The Authors
Thanks for the response.
We thank the reviewer for recognizing the importance of studying Shampoo and our unified trace-power Bregman analysis. To directly address your main concern on replication and learning-rate tuning across widths, we reran Section 6 across all five widths with independent per-method LR sweeps and 3 random seeds (280 runs in total; see A1 and our General Response). We now address each point below.
W1: Training results lack replication and per-method tuning across all widths
A1: Thanks for raising this point. We have rerun Section 6 across all five widths using a two-stage per-method LR sweep over 3 matched random seeds (280 runs in total; please see our General Response above for the setup, error bars, and full grid losses). With per-method tuning and 3-seed error bars, the method ranking at every width and our conclusions in Section 6 remain unchanged:
W2: Gap between finite-sample theory and practical problems
A2: Thanks for the insightful comment. Proposition 3 uses a controlled model to isolate the effect of the exponent qD and derive an exact, unified formula for the factor error. We now explicitly distinguish our theoretical guarantees, synthetic observations, and real-gradient results below:
Thus, Proposition 3 provides the exact guarantee under the idealized model, Figure 4 (right) and Table 3 provide empirical evidence on both synthetic and real neural network gradients, and Section 6 reports end-to-end training where stability and gain calibration act jointly. We have revised Section 3.2 to make this distinction clearer and removed broad statements such as "KL is most stable, F is least stable"; we also updated Table 1 and its caption.
Since the experiments in Section 6 were the main reason for your initial score, we hope these new results resolve your concerns and respectfully ask you to re-evaluate the paper.