acceptodds
Under review as a conference paper at ICLR 2027

How Bregman Divergences Shape Shampoo

Abstract

Understanding the principles behind Shampoo has recently guided the development of more effective neural network optimizers. These methods learn a preconditioner by optimizing the Frobenius or Kullback–Leibler (KL) divergence against the gradient second moment. In this work, we investigate how the choice of divergence shapes preconditioning, which remains unclear and blocks further improvements. To do so, we develop a unified Bregman divergence framework that connects all popular divergences, allowing us to study them jointly. Through spectral analysis and empirical investigation on the gradient second moment, we examine how divergence choice influences approximation behavior and preconditioner estimation. We find that some divergences can compensate for the bias introduced by approximating the gradient second moment with a finite number of gradients, and that this helps explain the behavior of their Shampoo variant in GPT-2 pretraining and synthetic experiments. By connecting divergence choice to practical training behavior, we believe our framework provides principled guidance for understanding the foundations of, and further improving, Shampoo.

Primary area:
optimization
Keywords:
Stochastic optimization, Shampoo, Kronecker preconditioning, Bregman divergences

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Sw37OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper develops a unified analysis of Kronecker approximation that unifies the three existing divergences (Frobenius、von Neumann、KL) into a trace-power Bregman family and introduces a fourth one, which yields a new method SQ-Shampoo. They show how the family exponent determines which parts of the spectrum are prioritized. They also show that empirical gradient second moments severely underestimate tail eigenvalues on GPT-2.

Critical strengths

This paper studies Shampoo, an important optimizer in LLM training. They develops unified analysis of Kronecker approximation that unifies the three existing divergences (Frobenius、von Neumann、KL) into a trace-power Bregman family and introduces a fourth one, which yields a new method SQ-Shampoo.

Critical weaknesses
  1. Core training results lack replication, making it difficult to assess robustness.

The GPT-2 training explicitly states that each configuration was run only once, with no multiple random seeds, error bars, or significance analysis. The differences between SQ and KL across multiple widths are very small—for instance, nearly identical at width 768, and only approximately 0.006 loss difference at width 1536. It is currently impossible to determine whether these differences exceed training stochasticity. The authors also note that no further per-method learning rate search was conducted beyond width 768. Given that training performance is one of the paper's main conclusions, this insufficient evidence is my primary reason for leaning toward rejection.

  1. Significant gap between finite-sample theory and practical problems.

Proposition 3 assumes gradients come from an exact Kronecker Gaussian model, and when estimating one factor, the other factor is known and fixed. While this isolates the effect of divergence, it differs substantially from real training scenarios involving non-Kronecker distributions, non-stationarity, joint estimation of both factors, and the use of EMA. The authors verify the empirical ordering on a synthetic non-Kronecker distribution, but this remains insufficient to support generalizing "KL is most stable, F is least stable." I recommend more clearly distinguishing between theoretical guarantees, synthetic experimental observations, and empirical phenomena in real training.

Questions for authors

No

3 comments

We thank the reviewer for recognizing the importance of studying Shampoo and our unified trace-power Bregman analysis. To directly address your main concern on replication and learning-rate tuning across widths, we reran Section 6 across all five widths with independent per-method LR sweeps and 33 random seeds (280280 runs in total; see A1 and our General Response). We now address each point below.

W1: Training results lack replication and per-method tuning across all widths

A1: Thanks for raising this point. We have rerun Section 6 across all five widths using a two-stage per-method LR sweep over 33 matched random seeds (280280 runs in total; please see our General Response above for the setup, error bars, and full grid losses). With per-method tuning and 33-seed error bars, the method ranking at every width and our conclusions in Section 6 remain unchanged:

  • Gaps over VN and F: SQ and KL achieve lower mean losses than VN and F at all five widths, and the gap grows at every width: from 0.0030.003–0.0060.006 over VN (0.040.04 over F) at width 256256 to 0.0270.027–0.0290.029 over VN (more than 0.100.10 over F) at width 15361536.
  • SQ vs. KL: Consistent with your observation at width 768768, SQ and KL have very close mean losses at widths 256256–768768. At widths 10241024 and 15361536, seed SDs are at most 0.00100.0010, and KL achieves lower loss than SQ on all three matched seeds, with mean gaps of 0.00270.0027 and 0.00280.0028.

W2: Gap between finite-sample theory and practical problems

A2: Thanks for the insightful comment. Proposition 3 uses a controlled model to isolate the effect of the exponent qDq_D and derive an exact, unified formula for the factor error. We now explicitly distinguish our theoretical guarantees, synthetic observations, and real-gradient results below:

LevelRef.SettingResult
Theoretical guaranteeProp. 3Exact Kronecker Gaussian model, opposite factor fixedExpected relative factor error: KL ≤ SQ ≤ VN ≤ F
Synthetic observationFig. 4 (right)Gaussian gradients with a non-Kronecker second moment and long-tailed spectrum; both factors jointly fittedMean squared relative Frobenius error against each method’s large-sample fit follows the same ordering at every tested sample size
Real-gradient observationTable 3Joint 40-sample fits and EMA on GPT-2 gradients, without Gaussian or Kronecker assumptionsMean recovery error of large-sample inverse-root gains follows the same ordering across nine checkpoint–projection pairs per training optimizer, for both AdamW and Shampoo checkpoints

Thus, Proposition 3 provides the exact guarantee under the idealized model, Figure 4 (right) and Table 3 provide empirical evidence on both synthetic and real neural network gradients, and Section 6 reports end-to-end training where stability and gain calibration act jointly. We have revised Section 3.2 to make this distinction clearer and removed broad statements such as "KL is most stable, F is least stable"; we also updated Table 1 and its caption.

Since the experiments in Section 6 were the main reason for your initial score, we hope these new results resolve your concerns and respectfully ask you to re-evaluate the paper.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer Sw37,

With the discussion period ending soon, we wanted to briefly check whether our response has addressed your concerns, especially the new 3-seed per-method LR sweeps across all widths and the clarified scope of Proposition 3.

Please let us know if you have any remaining questions before the discussion closes. Thank you again for your time and feedback!

Best regards,

The Authors

Thanks for the response.

Reviewer_nTmPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper unifies a number of recent variations on Shampoo that make different choices for the objective function used to approximate the kronecker factorization of the empirical Fisher Information matrix. This unification can help to explain differences between the variants and their performance, and guide further development of similar optimizers. The paper then proposes a new member of the Shampoo family and shows its connection to an independently developed Kronecker-based optimizer.

Critical strengths
  • The paper provides a nice framework to understand how an entire family of Shampoo variants relate to each other. The framework is insightful with apparent explanatory power, and the accompanying theory is illuminating. The empirical analysis of the interaction between sampling and fitting bias is particularly interesting. It gives algorithm developers and practitioners a concrete diagnostic tool and a potential design parameter to consider.
Critical weaknesses
  • The conclusion that the paper draws is that lower qq is better, with Frobenius at q=2q=2 and KL sitting at q→0q\rightarrow 0. This raises the question of why the proposed variant falls squarely in the middle of the range of explored values at q=1/2q=1/2 rather than the interesting regime q<0q<0. The omission undermines the framing of this parameter as having predictive design power.
  • The link between covariance fit and training loss is only demonstrated on a quadratic function. Testing the covariance estimation bias correction (as inspired by Ledoit & Wolf, 2012) more thoroughly would strengthen the narrative that the analysis results in a useful design intervention rather than merely functioning as a diagnostic.
Questions for authors
  • Why not prove the predictive power by demonstrating the better optimizer that is being suggested by the theory and empirical evidence? Why choose the apparently suboptimal operating point q=1/2q=1/2 instead for the proposal, and what about the gain error correction?
4 comments

We thank the reviewer for the constructive feedback and address each point below.

W1 & Q (Part 1): The choice of q=12q=\frac{1}{2} and the q<0q<0 regime

A1: Thanks for the insightful comment. We address both parts of your question below.

1. Why we focus on q=12q=\frac{1}{2} and q∈[0,2]q \in [0, 2]:

Our primary goal is to unify and explain existing optimizers instead of finding a new optimal exponent qq. We study q=12q = \frac{1}{2} (SQ-Shampoo) because its objective is equivalent to PSGD-Kron's gradient-whitening objective (Proposition 1), which was developed independently of Shampoo and has shown strong empirical results [1]. Implementing q=12q = \frac{1}{2} lets us compare PSGD-Kron's objective under the same Shampoo update scheme with F (q=2q = 2), VN (q→1q \to 1), and KL (q→0q \to 0), which naturally places our analysis in q∈[0,2]q \in [0, 2]. Within this range, decreasing qq toward 00 reduces the weight on large eigenvalues (reaching scale independence as q→0q \to 0) and penalizes underestimation more strongly, avoiding over-amplification under SD−12S_D^{-\frac{1}{2}} and improving few-sample stability. In most of our experiments, q=12q = \frac{1}{2} falls between VN and KL, which directly supports the predictive power of qq and helps explain, from an objective perspective, why PSGD and KL-Shampoo work well.

2. The q<0q < 0 regime:

The trace-power Bregman family remains well-defined for negative qq. Following your comment, we evaluated q=−12q = -\frac{1}{2} on our metrics in Section 5 across the AdamW checkpoints (held-out gain error in Table 2 and recovery error in Table 3); it yields results very close to KL (q→0q \to 0), even with some small improvements (all within one SD):

MethodTable 2: Large-sampleTable 2: 40-sampleTable 2: EMATable 3: Few-sampleTable 3: EMA
KL (q→0q\to 0)0.1024±0.01820.1024\pm0.01820.1561±0.02970.1561\pm0.02970.1071±0.02030.1071\pm0.02030.066±0.0200.066\pm0.0200.044±0.0110.044\pm0.011
q=−12q=-\frac{1}{2}0.0997±0.01840.0997\pm0.01840.1553±0.02800.1553\pm0.02800.1067±0.02350.1067\pm0.02350.063±0.0180.063\pm0.0180.045±0.0090.045\pm0.009

However, we also found that q=−12q = -\frac{1}{2} is less numerically stable: the factor updates (Equation 3) need R−32R^{-\frac{3}{2}} and L−32L^{-\frac{3}{2}}, which are more sensitive to small eigenvalues. Thoroughly studying the q<0q < 0 regime is an interesting direction for future work, and we have noted this in Section 7.

W2 & Q (Part 2): Gain interventions and bias correction

A2: Thanks for the insightful comment. As with A1, our goal is to explain existing optimizers rather than propose a new bias-corrected optimizer, so we did not test an explicit online bias correction in GPT-2 training. We clarify this through three parts below:

  1. Roles of the stochastic quadratic and GPT-2. Noisy quadratic models are widely used to study neural network optimization and preconditioning dynamics [2–4]. Knowing the exact population second moment here lets us test how changing directional gains affects optimization: replacing KL's top or tail gains with F's substantially raises the loss, whereas replacing them with SQ's keeps the loss close to KL's. To verify whether the predicted ordering holds in practice, we evaluate all four divergences in end-to-end GPT-2 pretraining across widths, learning rates, and seeds (see Section 6 and General Response).

  2. Role of held-out scales. Accordingly, the held-out scales in Section 4 serve strictly as an offline reference to evaluate preconditioner gains in Section 5; directly repeating this full-spectrum held-out procedure at every training step would be too costly due to the extra backward passes and memory overhead.

  3. Divergences and bias correction. In our experiments, KL and SQ's properties yield larger fitted tail scales, partially compensating for tail underestimation without extra held-out data. Turning this insight into a low-cost online intervention is a promising direction for designing new optimizers, which we have added to future work.

We have also updated Section 4 to state more explicitly that held-out scales serve only as an evaluation reference, and expanded Appendix D.3 with additional motivation and references [2–4].

References:

[1] Frans et al., "A Stable Whitening Optimizer for Efficient Neural Network Training," NeurIPS 2025.

[2] Zhang et al., "Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model," NeurIPS 2019.

[3] Kunstner et al., "Limitations of the Empirical Fisher Approximation for Natural Gradient Descent," NeurIPS 2019.

[4] Meterez et al., "A Defense of the Quadratic Model," arXiv:2607.21716, 2026.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer nTmP,

With the discussion period ending soon, we wanted to briefly check whether our response has addressed your questions on the q<0q < 0 regime, SQ-Shampoo (q=12q = \frac{1}{2}), and the offline held-out reference.

Please let us know if you have any follow-up questions before the discussion closes. Thank you again for your time and feedback!

Best regards,

The Authors

Thanks to the authors for the thorough general and individualized responses and corresponding paper edits.

I appreciate the experiments at the new operating point q=−1/2q=-1/2. It seems that the results there suggest q<0q<0 is in fact not likely to be an interesting direction for future work, or that the usefulness of the analysis in this paper diminishes in that regime without additional explanatory frameworks or computational techniques/considerations (e.g. numerics)?

Thanks for the interesting follow-up question! We would like to clarify two points below:

  • Our analysis still holds here: Our paper focuses on q∈[0,2]q \in [0, 2] to unify and explain existing optimizers, and by q→0q \to 0 (KL), most of the preconditioning gain error is removed (0.10240.1024 vs. 0.97590.9759 for F). Therefore, there is not much room to improve on these metrics; even so, q=−12q =-\frac{1}{2} penalizes underestimation more strongly and still lowers the gain error slightly (with comparable recovery errors), which is in line with our analysis. Since we only tested one point (q=−12q = -\frac{1}{2}) offline and have not yet tested training performance across q<0q < 0, there may still be improvement space.

  • Computational considerations: In practice, as you noted, exploring q<0q < 0 does require extra computational considerations and techniques (such as damping for the factors or higher precision), because the factor updates use Rq−1R^{q-1} (q−1<−1q - 1 < -1) and can amplify tiny eigenvalues. Since these techniques are coupled with the divergence properties in this regime, jointly considering them in training is why we still think q<0q < 0 is an interesting direction for future work.

Reviewer_wCewOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a unified view, through Bregman divergences, of popular Kronecker based algorithms in literature (Shampoo variants). Following, the authors show the interplay of the different divergences with the estimation error coming from using noisy estimates of the ggTgg^T matrix, and the estimation error coming from learning the Kronecker factorization of the second moment matrix.

Critical strengths

Presentation-wise, the paper is very clearly written, and the flow from section to section is very coherent, great job to the authors for this.

From a technical point of view:

  • the unifying view through the Bregman divergence is interesting and useful. The comparison across divergences in Figures 3 and 4 clearly shows that Frobenius-Shampoo is good mostly in the head, with KL-Shampoo spreading weight evenly across the spectrum
  • back to the initial plus on the presentation, it makes sense to split the error into noisy estimation error (on the second moment matrix) which the authors carefully ablate in Section 4, and the kronecker factor estimation error in Section 5 (and its interaction with the noisy second moment matrix)
Critical weaknesses
  • not sure if I missed it, but for proposition 2, I didnt see Δq\Delta_q definition anywhere (except for the appendix)? dqd_q is parameterized below (line 211)
  • for FF and VNVN, both in table 2 and table 3 there seem to be a large error band around the mean, basically on the same order of magnitude as the men but much smaller for the other divergences. Why is this showing up?

However my biggest concern currently is regarding tuning, and the main reason behind my current score. You mentioned on line 1733 that you use a shared learning rate across optimizers, however this seems suspicious beause the different optimizers have different scales. Could you provide the grid sweep and the losses at each learning rate? Secondly, since you do not resweep across widths, are your models parameterized under muP, and if so could you provide the learning rate transfer plots? Otherwise this would be an unfair comparison.

Questions for authors

Asked in weaknesses section.

3 comments

We thank the reviewer for the positive feedback on our paper's clarity and analysis. To directly address your main concern on learning-rate tuning, we have rerun Section 6 with per-width, per-method LR sweeps over 33 seeds (280280 runs in total; see A3 below and our General Response). We now address each point below.

W1: Missing definition of Δq\Delta_q

A1: Thanks for the comment. Previously, the explicit formula for Δf\Delta_f was only in Appendix B.3; we have now added it (along with the explicit formula for dfd_f) directly in Proposition 2: Δf(Λ,Ω;P):=−∑i,j=1dλif′(ωj)(Pij−δij), Δq:=Δfq,\Delta_f(\Lambda,\Omega;P):=-\sum_{i,j=1}^d\lambda_i f'(\omega_j)(P_{ij}-\delta_{ij}),\ \Delta_q:=\Delta_{f_q}, where δij\delta_{ij} is the Kronecker delta (δij=1\delta_{ij}=1 if i=ji=j, and 00 otherwise).

W2: Large SDs for F and VN in Tables 2 and 3

A2: Thanks for this insightful question. A key reason for the large SDs is that F and VN are highly sensitive to the spectral shape, which varies greatly across the nine matrices we evaluate (Figure 13). We explain this and provide additional experiments below.

1. What the error band represents: In Tables 2 and 3, the reported mean ±\pm SD is computed across nine distinct matrices (3 steps ×\times 3 projections: Q,V,Q, V, and OO) with different spectral shapes per training optimizer, not repeated fits of a single matrix. Thus, a large SD reflects sensitivity to the spectral shape.

2. Why F and VN have larger SDs: Across the nine matrices, spectra vary from relatively flat to highly concentrated in the leading eigenvalues. Because KL (q→0q\to 0) treats large and small eigenvalues equally and SQ (q=12q=\frac{1}{2}) gives large ones only mild extra weight, both stay relatively consistent across spectral shapes and keep small SDs. In contrast, F (q=2q=2) and VN (q→1q\to 1) focus heavily on leading eigenvalues over the tail, so their errors become large on concentrated spectra and drive up the SD:

  • Table 2 (Held-out Gain Error): Since F and VN also favor underestimation, they severely underestimate the tail on concentrated spectra; taking the inverse square root SD−12S_D^{-\frac{1}{2}} turns these underestimated tail eigenvalues into huge gains and causes much larger gain errors than on flatter spectra.
  • Table 3 (Few-Sample Recovery Error): Here, the target is each method's own large-sample gain. On concentrated spectra, F and VN rely on just a few dominant directions, so their few-sample fits are more sensitive to sampling noise and yield larger recovery errors than on flatter spectra.

The real-gradient spectrum results at AdamW step 4500 illustrate this directly. Because VV has a much more concentrated spectrum than QQ (Figure 13), F and VN errors jump sharply from QQ to VV while SQ and KL remain stable:

MetricProjectionFVNSQKL
Table 2: Large-sample held-out gain errorQQ0.24000.24000.13200.13200.10330.10330.09480.0948
VV1.40771.40770.33830.33830.12370.12370.10440.1044
Table 3: 40-sample recovery errorQQ0.78250.78250.12130.12130.05080.05080.04190.0419
VV1.95391.95390.27730.27730.07630.07630.06190.0619

3. Additional results on Block 9:

Applying the same analysis to Block 9 of GPT-2 (99 additional checkpoint–projection pairs per training optimizer) reproduces this phenomenon:

OptimizerMethodTable 2: Large-sampleTable 2: 40-sampleTable 3: 40-sample
AdamWF1.1228±0.32981.1228\pm0.32982.9377±0.66972.9377\pm0.66971.9462±0.54751.9462\pm0.5475
VN0.4267±0.21610.4267\pm0.21610.7072±0.36430.7072\pm0.36430.3274±0.19250.3274\pm0.1925
SQ0.1558±0.05690.1558\pm0.05690.2433±0.08700.2433\pm0.08700.1099±0.04300.1099\pm0.0430
KL0.1030±0.02140.1030\pm0.02140.1671±0.02470.1671\pm0.02470.0785±0.02020.0785\pm0.0202
Distributed ShampooF0.8172±0.29980.8172\pm0.29982.4629±0.54962.4629\pm0.54961.7961±0.39181.7961\pm0.3918
VN0.2905±0.09420.2905\pm0.09420.5722±0.17320.5722\pm0.17320.3280±0.12620.3280\pm0.1262
SQ0.1273±0.02390.1273\pm0.02390.1984±0.03860.1984\pm0.03860.0919±0.03070.0919\pm0.0307
KL0.1028±0.02260.1028\pm0.02260.1588±0.02760.1588\pm0.02760.0706±0.02270.0706\pm0.0227

We have explained this in Section 5.3, and added both tables above to Appendix E.5 (Tables 11 and 12), with the related discussion in Appendix E.4 and E.5.

W3: Learning-rate tuning across optimizers and widths

A3: Thanks for raising this point. Our models use standard parameterization rather than μ\muP. To verify that the observed performance gaps do not come from sharing learning rates, we ran a two-stage per-width, per-method LR sweep across all five widths over 33 seeds (280280 runs in total; please see our General Response above for the setup and validation losses at every tested LR).

While per-method tuning changed the best LRs for certain width–optimizer pairs, the method ranking at every width and our conclusions in Section 6 remain unchanged: SQ and KL outperform VN and F across all five widths, perform similarly at smaller widths, and KL is slightly better at larger widths.

Since learning-rate tuning was the main reason behind your initial score, we respectfully ask you to reassess the paper in light of these new results.

Reviewer_wCewOpenReview ↗

Reply

Thanks for providing the sweeps, I have increased my score to 3.

Thank you very much for taking the time to review our new results and for raising your score. Your feedback helped make our paper much stronger.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (Part 1): Per-Method LR Tuning and 3-Seed GPT-2 Results

We thank all the reviewers for their constructive feedback. To address the concerns of Reviewers wCew and Sw37 on learning-rate tuning and multi-seed evaluation, we reran Section 6 across all five widths (256256 to 15361536) using two-stage per-method LR sweeps over 3 random seeds (280280 runs in total, all run from scratch in an identical environment).

In summary, compared to our initial submission, this changed the best LRs for certain width–optimizer pairs, while the method ranking at every width and our conclusions in Section 6 remain unchanged: SQ and KL outperform VN and F at all five widths (and the gap grows with width); SQ and KL have very close mean losses at widths 256256–768768, and KL achieves lower loss than SQ on all three matched seeds at widths 10241024 and 15361536.

Please find the tuning details and results below.

1. Two-Stage Tuning (280 Runs)

  • Coarse grid (1 seed, 120 runs): Following recent LLM optimizer benchmarks [1, 2], at each width we swept a shared 66-point LR grid spaced by factors of 22 to locate the best LR range for each optimizer (Appendix F, Table 14).
  • Multi-seed grid (3 seeds, 80 configurations; 160 additional runs): Since nearby learning rates can yield similar validation losses on a single seed, for each width and optimizer we re-evaluated four nearby learning rates from the coarse grid across 3 seeds (reusing the coarse-sweep seed plus 2 new seeds) and selected the LR with the lowest 33-seed mean validation loss (General Response (Part 2) below; Appendix F, Table 15).

2. Summary of Best Learning Rates and Validation Losses (Table 4 & Table 13)

Best LR and 33-seed validation loss (mean±SD\text{mean}\pm\text{SD} after 5,1005{,}100 steps) for each width and method. Bold marks the lowest mean validation loss at each width:

Width (Params)MetricF (q=2q=2)VN (q→1q\to 1)SQ (q=12q=\frac{1}{2})KL (q→0q\to 0)
256 (22.3M22.3\text{M})Best LR0.00960.00960.00480.00480.00480.00480.00240.0024
Val. loss3.7835±0.00183.7835\pm0.00183.7462±0.00203.7462\pm0.00203.7405±0.0029\mathbf{3.7405\pm0.0029}3.7436±0.00253.7436\pm0.0025
512 (63.5M63.5\text{M})Best LR0.00240.00240.00240.00240.00240.00240.00240.0024
Val. loss3.4898±0.00313.4898\pm0.00313.4328±0.00183.4328\pm0.00183.4164±0.0007\mathbf{3.4164\pm0.0007}3.4168±0.00103.4168\pm0.0010
768 (123.6M123.6\text{M})Best LR0.00180.00180.00180.00180.00180.00180.00180.0018
Val. loss3.3501±0.00243.3501\pm0.00243.2859±0.00063.2859\pm0.00063.2672±0.00243.2672\pm0.00243.2662±0.0016\mathbf{3.2662\pm0.0016}
1024 (202.5M202.5\text{M})Best LR0.00120.00120.00120.00120.00240.00240.00240.0024
Val. loss3.2698±0.00263.2698\pm0.00263.1970±0.00093.1970\pm0.00093.1778±0.00043.1778\pm0.00043.1751±0.0002\mathbf{3.1751\pm0.0002}
1536 (417.0M417.0\text{M})Best LR0.00080.00080.00080.00080.00160.00160.00160.0016
Val. loss3.1736±0.00273.1736\pm0.00273.0971±0.00433.0971\pm0.00433.0705±0.00103.0705\pm0.00103.0677±0.0010\mathbf{3.0677\pm0.0010}

3. Detailed Observations

  • Gaps over VN and F: SQ and KL achieve lower mean losses than VN and F at all five widths, and the gap grows at every width: from 0.0030.003–0.0060.006 over VN (0.040.04 over F) at width 256256 to 0.0270.027–0.0290.029 over VN (>0.10>0.10 over F) at width 15361536.
  • SQ vs. KL: Under the selected best learning rates, SQ and KL have very close mean losses at widths 256256–768768. At widths 10241024 and 15361536, KL achieves lower loss than SQ on each of the three matched seeds, with mean gaps of 0.00270.0027 and 0.00280.0028, respectively:
Width (Best LR)MethodSeed 1Seed 2Seed 3Mean ±\pm SD
1024 (0.00240.0024)SQ3.17793.17793.17823.17823.17743.17743.1778±0.00043.1778\pm0.0004
KL3.1751\mathbf{3.1751}3.1749\mathbf{3.1749}3.1753\mathbf{3.1753}3.1751±0.0002\mathbf{3.1751\pm0.0002}
1536 (0.00160.0016)SQ3.07053.07053.07153.07153.06943.06943.0705±0.00103.0705\pm0.0010
KL3.0666\mathbf{3.0666}3.0685\mathbf{3.0685}3.0680\mathbf{3.0680}3.0677±0.0010\mathbf{3.0677\pm0.0010}

4. Role of Section 6

Finally, we would like to briefly clarify the role of Section 6. F-, VN-, and KL-Shampoo are existing methods: KL's advantage over VN and F was shown through detailed comparisons in [3], and its strong performance in LLM pretraining was reported in [4, 5]. SQ-Shampoo implements PSGD-Kron's existing whitening objective [6] in the same Shampoo scheme to test our unified trace-power family analysis (especially positioning SQ relative to KL, VN, and F), instead of proposing a new SOTA optimizer. Thus, our main focus is explaining why this performance gap arises and verifying the predicted ordering in training.

References:

[1] Wen et al., "Fantastic Pretraining Optimizers and Where to Find Them," ICLR 2026.

[2] Semenov et al., "Benchmarking Optimizers for Large Language Model Pretraining," arXiv:2509.01440, 2025.

[3] Lin et al., "Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization," ICLR 2026.

[4] Eschenhagen et al., "Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory," arXiv:2602.09314, 2026.

[5] Zhang et al., "Online KL Shampoo," Tilde Research technical blog, 2026.

[6] Li, "Preconditioner on Matrix Lie Group for SGD," ICLR 2019.

AuthorsOpenReview ↗

General Response (Part 2): Full 3-Seed LR Grid Sweep

Takeaway: In all 2020 width–optimizer pairs, each method's best LR (bold) lies strictly inside its 44-point window (mean loss increases on both sides).

Below are the full 33-seed validation losses (mean ±\pm SD after 5,1005{,}100 steps) across the 44-point LR window for each width and method (Appendix F, Table 15; see Table 14 for the 11-seed 66-point coarse sweep):

Width (Params)LRF (q=2q=2)VN (q→1q\to 1)SQ (q=12q=\frac{1}{2})KL (q→0q\to 0)
256 (22.3M22.3\text{M})0.00120.0012-3.7593±0.00103.7593\pm0.0010-3.7558±0.00273.7558\pm0.0027
0.00240.00243.8030±0.00213.8030\pm0.00213.7528±0.00183.7528\pm0.00183.7456±0.00133.7456\pm0.00133.7436±0.0025\mathbf{3.7436\pm0.0025}
0.00480.00483.7893±0.00343.7893\pm0.00343.7462±0.0020\mathbf{3.7462\pm0.0020}3.7405±0.0029\mathbf{3.7405\pm0.0029}3.7473±0.00383.7473\pm0.0038
0.00960.00963.7835±0.0018\mathbf{3.7835\pm0.0018}3.7507±0.00073.7507\pm0.00073.7451±0.00223.7451\pm0.00223.7579±0.00113.7579\pm0.0011
0.01920.01923.7919±0.01213.7919\pm0.0121-3.7577±0.00363.7577\pm0.0036-
512 (63.5M63.5\text{M})0.00060.0006-3.4500±0.00183.4500\pm0.0018--
0.00120.00123.4987±0.00243.4987\pm0.00243.4376±0.00153.4376\pm0.00153.4246±0.00063.4246\pm0.00063.4214±0.00033.4214\pm0.0003
0.00240.00243.4898±0.0031\mathbf{3.4898\pm0.0031}3.4328±0.0018\mathbf{3.4328\pm0.0018}3.4164±0.0007\mathbf{3.4164\pm0.0007}3.4168±0.0010\mathbf{3.4168\pm0.0010}
0.00480.00483.4976±0.01593.4976\pm0.01593.4367±0.00053.4367\pm0.00053.4206±0.00073.4206\pm0.00073.4252±0.00033.4252\pm0.0003
0.00960.00963.5047±0.02323.5047\pm0.0232-3.4288±0.00083.4288\pm0.00083.4421±0.00093.4421\pm0.0009
768 (123.6M123.6\text{M})0.000450.00045-3.3054±0.00173.3054\pm0.0017--
0.00090.00093.3570±0.00233.3570\pm0.00233.2885±0.00183.2885\pm0.00183.2784±0.00123.2784\pm0.00123.2749±0.00243.2749\pm0.0024
0.00180.00183.3501±0.0024\mathbf{3.3501\pm0.0024}3.2859±0.0006\mathbf{3.2859\pm0.0006}3.2672±0.0024\mathbf{3.2672\pm0.0024}3.2662±0.0016\mathbf{3.2662\pm0.0016}
0.00360.00363.3638±0.01323.3638\pm0.01323.2968±0.00513.2968\pm0.00513.2703±0.00283.2703\pm0.00283.2720±0.00333.2720\pm0.0033
0.00720.00723.3813±0.01763.3813\pm0.0176-3.2793±0.00263.2793\pm0.00263.2877±0.00173.2877\pm0.0017
1024 (202.5M202.5\text{M})0.00030.00033.2938±0.00263.2938\pm0.00263.2322±0.00133.2322\pm0.0013--
0.00060.00063.2763±0.00033.2763\pm0.00033.2070±0.00173.2070\pm0.00173.1977±0.00113.1977\pm0.00113.1929±0.00053.1929\pm0.0005
0.00120.00123.2698±0.0026\mathbf{3.2698\pm0.0026}3.1970±0.0009\mathbf{3.1970\pm0.0009}3.1786±0.00093.1786\pm0.00093.1755±0.00053.1755\pm0.0005
0.00240.00243.3180±0.04403.3180\pm0.04403.2081±0.00513.2081\pm0.00513.1778±0.0004\mathbf{3.1778\pm0.0004}3.1751±0.0002\mathbf{3.1751\pm0.0002}
0.00480.0048--3.1833±0.00153.1833\pm0.00153.1856±0.00173.1856\pm0.0017
1536 (417.0M417.0\text{M})0.00020.00023.1984±0.00383.1984\pm0.0038---
0.00040.00043.1805±0.00323.1805\pm0.00323.1079±0.00163.1079\pm0.00163.1028±0.00183.1028\pm0.0018-
0.00080.00083.1736±0.0027\mathbf{3.1736\pm0.0027}3.0971±0.0043\mathbf{3.0971\pm0.0043}3.0782±0.00153.0782\pm0.00153.0746±0.00083.0746\pm0.0008
0.00160.00163.2022±0.01803.2022\pm0.01803.1046±0.01033.1046\pm0.01033.0705±0.0010\mathbf{3.0705\pm0.0010}3.0677±0.0010\mathbf{3.0677\pm0.0010}
0.00320.0032-3.1186±0.00523.1186\pm0.00523.0739±0.00113.0739\pm0.00113.0724±0.00143.0724\pm0.0014
0.00640.0064---3.0937±0.00173.0937\pm0.0017
AuthorsOpenReview ↗

New changes in the revised version

All main changes in the revised paper are highlighted in blue.

  • Main text
    • clarified our motivation and the connection to PSGD via SQ-Shampoo in Sections 1 and 3, and added future work on the q<0q < 0 regime and online eigenvalue correction in Section 7 (Reviewer nTmP)
    • added the explicit definitions of dfd_f and Δf\Delta_f directly in Proposition 2 in Section 3.1 (Reviewer wCew)
    • updated Table 1 and revised Section 3.2 to more explicitly distinguish Proposition 3's theoretical guarantee from our synthetic and real-gradient observations (Reviewer Sw37)
    • clarified in Section 4 that held-out scales are used as an offline evaluation reference (Reviewer nTmP)
    • included a discussion in Section 5.3 on why F and VN have larger standard deviations across matrices in Tables 2 and 3 (Reviewer wCew)
    • updated Figure 1 (right), Table 4, and Section 6 with the two-stage per-method learning-rate sweeps across 33 random seeds for GPT-2 training (Reviewers wCew and Sw37)
  • The appendix
    • expanded Appendix D.3 with additional motivation and references on stochastic quadratic models (Reviewer nTmP)
    • added a per-projection comparison between QQ and VV (Table 11, Appendix E.5) and additional experiments on Block 9 (Table 12, Appendix E.5), along with discussion in Appendices E.4 and E.5 (Reviewer wCew)
    • updated Appendix F with the two-stage tuning setup, summarized the best learning rates and validation losses in Table 13, and added the full sweep results in Tables 14–15 (Reviewers wCew and Sw37)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.