acceptodds
Under review as a conference paper at ICLR 2027

Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization

Abstract

Optimizers that exploit the matrix structure of gradients are central to modern LLM pre-training, with two distinct frontiers: explicit Kronecker-factored preconditioning—most recently KL-Shampoo, which estimates the preconditioner via KL divergence minimization—and orthogonalization of the gradient momentum, exemplified by Muon and analyzed as steepest descent under the spectral norm. The two routes are typically developed in isolation. We make a structural observation about KL-Shampoo's Kronecker preconditioners: their eigenvalue spectra exhibit a spike-and-flat shape—a few dominant eigenvalues followed by an approximately uniform tail—across layers and training stages, holding exactly under a rank- signal-plus-noise gradient model. We exploit this structure by restricting one of KL-Shampoo's Kronecker factors to a parametric family aligned with the spike-and-flat shape: full spectral structure on a tracked -dimensional subspace, single shared eigenvalue across the remaining directions. On these directions, we apply orthogonalization. An identity shows that this orthogonalization recovers the algebraic form of full KL-Shampoo's preconditioner, with the running estimate replaced by an instantaneous one. On four pre-training scales (GPT-2 124M / 350M, LLaMA 134M / 450M), Pro-KLShampoo consistently outperforms KL-Shampoo at every subspace rank we test in validation loss, peak per-GPU memory, and wallclock time to reach KL-Shampoo's final validation loss.

Primary area:
optimization
Keywords:
optimizers, LLM pretraining, preconditioning, Kronecker factorization, KL-Shampoo, Muon, orthogonalization, subspace projection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D3NZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a new optimization algorithm that is effectively a combination of a KL-Shampoo style Kronecker factored preconditioner and a Muon style orthogonalization. The authors identify that the distribution of the eigenvalues in most layers’ preconditions follow a similar distribution: a few top eigenvalues dominate followed by a relatively uniform tail. With this observation, they motivate tracking (one of) the preconditions in a subspace to encapsulate the top eigenvalues, while tracking the tail in the orthogonal complement subspace. They further modify this idea by applying a Muon style orthogonalization to the tail component using the observation its eigenvalues should be approximately similar. The authors do a non-trivial amount of derivations to show the different mathematical forms this algorithm can take an write it down in the most mathematically concise way possible. They additionally solve the KL-Shampoo preconditioner optimization problem for each of the new objects they track in their optimizer state assuming one object (UU) is fixed, and provide a theoretical argument for how to select the optimal UU given the rest of their optimizer state is a solution to the previous problem. The authors further argue that the Muon style orthogonalization step is equivalent to the original form of KL-Shampoo’s preconditioner if only the current mini-batch is considered. The authors conclude their theoretical work by providing a convergence guarantee on the average momentum buffer norm under some (somewhat strong) assumptions. Experimentally, the authors show across two architectures, two scales, and two datasets that their algorithm. They show superior performance in terms of memory consumption, clock time for matched loss, and iterations. The authors additionally provide some ablations regarding which subspace has more impact and how the orthogonalization step helps versus simply using the original two subspaces.

Critical strengths

I like the idea of this paper, I think it is an effective use of exploiting an empirical observation to develop a new algorithm. The idea of splitting into subspaces is somewhat intuitive, and the orthogonalization step relates to other approaches in modern optimization literature. The theory the authors provide appears sound and some results appear to make intuitive sense in their conclusions. The authors additionally provide some interesting commentary on relating their method to other concepts, such as the algebraic equivalence of their whitening method. Overall the paper is fairly well structured and clearly written, notational voices typically make the mathematics read well. Experimentally, the result is faster in terms of wall time and memory which I would expect given the nature of the algorithm, however I do have some questions regarding their other experimental results as follows.

Critical weaknesses

While I do like this paper, as above, I do have some questions about the validity of the results I feel the authors must address for acceptance. I largely find their motivation and theory somewhat convincing, but there is a core inconsistency in the theoretical and experimental results that the authors must address. This work solves a restricted form of the KL-Shampoo preconditioner problem and in fact provides theory for the error on this approximation. Given this, the solution the authors find (at least for their Smok-Hop) variant should live in a subset of the solutions to the full problem. Further, the authors argue that polar decomposition is equivalent to the algebraic form of the KL-Shampoo preconditioner for a single batch. This however includes the fact that (1) [0] shows that the instantaneous version of KL-Shampoo underperforms the full method, and (2) that in practice the true polar decomposition is not used, rather an approximation. Because of these facts, I find it fair to call the proposed algorithm an approximation of KL-Shampoo. This is fine, as the approximation-speed tradeoff of optimization algorithms is clearly a very important area of research. However, when the authors present their experimental results, this is not consistent. In the main text the authors largely focus on time and memory advantages which should be expected for an approximation algorithm, but Figure 3 and Figure 5 show that Smok-Hop and Pro-KLShampoo can consistently outperform KL-Shampoo in terms of both training and validation loss when measured per step. I’m somewhat troubled by this results as it appears to be a case of the approximation outperforming what it is approximating. While the authors do report on their experimental procedure quite extensively, I did not find any commentary on why they suspect this result to be the case, rather a focus on the wall time improvements.

A more minor weakness is the architecture scale is a bit limited, largely on the account that on models that use larger batch size the performance of these sorts of algorithms can very significantly, but I understand if there are some computational limitations.

[0] Wu Lin, Scott C Lowe, Felix Dangel, Runa Eschenhagen, Zikun Xu, and Roger B Grosse. Understanding and improving shampoo and soap via kullback-leibler minimization. arXiv preprint arXiv:2509.03378, 2025.

Questions for authors
  • Does the claim of algebraic equivalence of the decomposition with this form of whitening actually hold numerically or matter that much? In particular, [0] notes that the instantaneous version of KL-Shampoo is far inferior to the EMA versions, and thus I wonder if this instantaneous equivalence actually signals much improvement.
  • Do the updates that KL-Shampoo and Pro-KLShampoo find appear empirically similar? Given theory predicts they should be in low rank settings or under instantaneous updates, one may expect similar update directions.
  • My primary question is resolving the approximation and yet over performance of KL-Shampoo as details in the weakness section, if the authors can provide insights into why this takes place I would likely raise my score.

[0] Wu Lin, Scott C Lowe, Felix Dangel, Runa Eschenhagen, Zikun Xu, and Roger B Grosse. Understanding and improving shampoo and soap via kullback-leibler minimization. arXiv preprint arXiv:2509.03378, 2025.

4 comments
AuthorsOpenReview ↗

Response 1/3

We thank the reviewer for the careful reading.

W1 and Q3 (first part). How can the method outperform KL-Shampoo if it approximates KL-Shampoo?

Pro-KLShampoo is an approximation of KL-Shampoo plus one added step, the orthogonalization of the complement, and with this step it is ahead of KL-Shampoo at every scale. The approximation alone (Smok-Hop) does not consistently outperform KL-Shampoo, as shown below. KL-Shampoo's right-side preconditioner, averaged over past steps, has a few large eigenvalues and a nearly flat tail (App. E). Pro-KLShampoo tracks the eigenspace of the large eigenvalues and applies KL-Shampoo's formula there, while the added orthogonalization acts on its complement.

the solution the authors find (at least for their Smok-Hop) variant should live in a subset of the solutions to the full problem.

We agree. Smok-Hop is within the restricted family, and it does not consistently outperform KL-Shampoo. In Table 2 it is ahead of KL-Shampoo by 10% of Pro-KLShampoo's gain at GPT-2 124M and by 17% at LLaMA-134M, and behind by 30% at LLaMA-450M. With the same Nesterov momentum and a swept learning rate, Smok-Hop is behind Pro-KLShampoo by 0.0185 at LLaMA-134M and 0.0270 at LLaMA-450M (our response to Reviewer dNqQ).

(1) [0] shows that the instantaneous version of KL-Shampoo underperforms the full method

The two papers use "instantaneous" for different things. In [0] (Sec. 4, Fig. 4), the instantaneous variant still builds the preconditioners from an average over past steps and only changes how their eigenvalues are estimated. In [1] (Proposition 4.8), as in our identity (12), it means computing KL-Shampoo's statistics from the current gradient alone instead of an average over past steps.

In the second sense the reviewer's point holds: run this way on a fixed gradient, KL-Shampoo converges to spectral descent [1, Proposition 4.8], which with momentum is the update of Muon, and Muon is behind KL-Shampoo at three of the four scales in Table 1. Pro-KLShampoo does this only for its right-side preconditioner on the complement and, like Muon, computes it from the current momentum. Its left preconditioner and its right-side preconditioner on the tracked subspace are averaged over past steps, as in KL-Shampoo. This combination is ahead of both Muon and KL-Shampoo at every scale in Table 1.

(2) that in practice the true polar decomposition is not used, rather an approximation

We use five Newton–Schulz steps with the coefficients be the same as Muon's (App. K). The iteration is an odd matrix polynomial, so its output has the same singular vectors as the exact polar factor, with singular values pushed toward one instead of set to one.

W2. Model scale and batch size

A more minor weakness is the architecture scale is a bit limited, largely on the account that on models that use larger batch size the performance of these sorts of algorithms can very significantly

We extended the evaluation to LLaMA-1.3B trained on 26B tokens, with sequence length 1024 and batch size 2000, about 2.05M tokens per iteration (previously at most batch size 1620, about 1.66M tokens, for LLaMA-450M).

MethodValidation loss
Pro-KLShampoo (r=128r = 128, αkl=0.005\alpha_{\mathrm{kl}} = 0.005, lr 0.02)2.5116
KL-Shampoo (lr 0.002)2.5206
Muon (lr 0.02)2.5268

Due to the limited computational resources, learning rates were selected by short screening runs for each method, {0.01, 0.02, 0.03} for Muon, {0.001, 0.002, 0.003} for KL-Shampoo, and three (learning rate, αkl\alpha_{\mathrm{kl}}) pairs for Pro-KLShampoo, (0.01, 0.005), (0.02, 0.005), and (0.02, 0.003). The selected configurations were trained for the full budget, with one run each.

In the revision, Sec. 3.3 will state that we use "instantaneous" in the sense of [1], and Table 1, App. E, and the Limitations will include LLaMA-1.3B, with the batch range extended to 2.05M tokens per step.

[0] Lin et al., arXiv 2509.03378.

[1] Eschenhagen, Cai, Lee, Shi. Clarifying Shampoo. arXiv 2602.09314.

AuthorsOpenReview ↗

Response 2/3

W1 and Q3 (continued)

I find it fair to call the proposed algorithm an approximation of KL-Shampoo.

Fig. 3 and Fig. 5 show that Smok-Hop and Pro-KLShampoo can consistently outperform KL-Shampoo [...] the approximation outperforming what it is approximating.

My primary question is resolving the approximation and yet over performance of KL-Shampoo

We view Pro-KLShampoo as an approximation of KL-Shampoo plus one added step, the orthogonalization of the complement. Write Xt=L−1/2GtP⊥X_t = L^{-1/2} G_t P_\perp for the left-whitened gradient of step tt projected on the complement of a rank-rr subspace, each algorithm with its own LL and P⊥P_\perp. Up to scale, KL-Shampoo's update on the complement is Xt(E[Xt⊤Xt])†/2X_t (\mathbb{E}[X_t^\top X_t])^{\dagger/2} (Eq. (11), expectation over the stochastic gradient at full-KL stationarity), and Pro-KLShampoo's update is polar(Xt)=Xt(Xt⊤Xt)†/2\mathrm{polar}(X_t) = X_t (X_t^\top X_t)^{\dagger/2} (Eq. (12)). In practice each method applies its formula to its own momentum in place of GtG_t, and KL-Shampoo replaces E[Gt⊤L−1Gt]\mathbb{E}[G_t^\top L^{-1} G_t] by its right factor RR, an exponential moving average of Gs⊤Ls−1GsG_s^\top L_s^{-1} G_s over past steps ss in the full space. We call RR the running statistic and Xt⊤XtX_t^\top X_t, with the momentum in place of GtG_t, the current statistic.

The eigenvalues of RR differ on the subspace while they are nearly equal on the complement (App. E, tail ratio medians at most 1.195), so on the complement RR is close to a scalar. Hence KL-Shampoo's update there is close to the complement term of the restricted update in (6) (i.e., μ⊥−1/2Xt\mu_\perp^{-1/2} X_t) which Smok-Hop uses (cosine 0.927 to 0.998 between the two, see Q2 in the next comment).

Against μ⊥−1/2Xt\mu_\perp^{-1/2} X_t, the step capolar(Xt)c_a \mathrm{polar}(X_t) changes the singular values, while αkl\alpha_{\mathrm{kl}} keeps the ratio between the complement and subspace updates close to that of KL-Shampoo.

On the complement the running statistic is nearly a multiple of the identity (App. E), so KL-Shampoo's step nearly preserves the singular values of XtX_t up to scale, and its update there is close to left-whitened gradient descent with momentum. The polar factor instead makes them equal, so Pro-KLShampoo's update there is left-whitened spectral descent with momentum. The two differ because these singular values (of the left-whitened gradient, momentum in practice) are unequal, with mean AM/GM ratios of their squares of 3.42 / 1.91 (q_proj in attention / up_proj in MLP) at LLaMA-134M and 5.45 / 2.66 at LLaMA-450M, over all records of one Pro-KLShampoo run per scale. By identity (12) the difference lies in the statistic: the polar step is KL-Shampoo's complement formula with the current statistic in place of the running statistic. The running statistic is an average of squares of left-whitened gradients, while the current statistic is the square of the current left-whitened momentum, which is itself an average of gradients; a flat average of squares does not make the square of an average flat.

The ratio between the subspace update and the complement update is controlled by αkl\alpha_{\mathrm{kl}}, which App. J.3 relates to the expected ratio of update (6) at the stationary point. At GPT-2 350M this Frobenius-norm ratio divided by KL-Shampoo's (split by its top-128 eigenspace) has median 0.92 (attention) and 1.02 (MLP).

Why spectral descent outperforms gradient descent is partly explained [2, 3, 4] but still open [1, Sec. 5]. Our conjecture is about the relative step sizes along the singular directions of the left-whitened momentum on the complement. KL-Shampoo's step there is nearly proportional to this momentum, so the directions with large singular values dominate it and those with small ones move little. Wang et al. [4] show that in an associative memory model the singular values of the gradient encode how often each fact appears in the training data, so gradient descent learns rarely seen facts more slowly than frequent ones, while Muon, with equal step sizes, learns all facts at the same rate. If the small singular directions on the complement likewise correspond to rarely seen data, the polar step may help in the same way. Testing it is left to future work.

In the revision, Secs. 3.3, 4.1, and 4.2 will give this explanation, and the measurements will go to a new appendix section.

[1] Eschenhagen, Cai, Lee, Shi. Clarifying Shampoo. arXiv 2602.09314, 2026.

[2] Bernstein, Newhouse. Old Optimizer, New Norm: An Anthology. arXiv 2409.20325, 2024.

[3] Su. Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal? arXiv 2511.00674, 2025.

[4] Wang et al. Muon Outperforms Adam in Tail-End Associative Memory Learning. ICLR 2026, arXiv 2509.26030.

AuthorsOpenReview ↗

Response 3/3

Q1. The algebraic identity and [0]

Does the claim of algebraic equivalence of the decomposition with this form of whitening actually hold numerically or matter that much? In particular, [0] notes that the instantaneous version of KL-Shampoo is far inferior to the EMA versions, and thus I wonder if this instantaneous equivalence actually signals much improvement.

Identity (12) holds exactly for the polar factor. In practice the polar factor is computed by Newton–Schulz, which gives the same singular vectors, with singular values pushed toward one instead of set to one (first comment, under (2)); its output has cosine 0.985 / 0.986 at LLaMA-134M and 0.976 / 0.977 at LLaMA-450M with the exact polar factor (q_proj / up_proj). As stated in Sec. 3.3, the identity recovers the algebraic form of KL-Shampoo's complement update (11) with the stationary statistic replaced by the instantaneous one, and we do not claim that the instantaneous statistic approximates the stationary one in a statistical sense. It is therefore not meant to signal an improvement by itself. The comparison in [0] is a different one (first comment, under (1)), and we discuss where the improvement comes from in our second comment.

Q2. Similarity of the updates

Do the updates that KL-Shampoo and Pro-KLShampoo find appear empirically similar? Given theory predicts they should be in low rank settings or under instantaneous updates, one may expect similar update directions.

The polar step shares the algebraic form of KL-Shampoo's complement update but not its statistic, so we do not expect the two updates to be similar on the complement. We nevertheless compared them, as the reviewer asks. To this end, we computed the two optimizers' own updates along the trajectory of Pro-KLShampoo (KL-Shampoo's update with its own EMA momentum and running statistics maintained alongside). We found that the two updates are similar on Pro-KLShampoo's tracked subspace, with cosine 0.941 / 0.882 at LLaMA-134M and 0.931 / 0.877 at LLaMA-450M (q_proj / up_proj), and that on the complement they differ, with 0.767 / 0.771 and 0.690 / 0.709.

On the complement KL-Shampoo multiplies the left-whitened momentum by the inverse square root of the running statistic. In the low-rank model, when the subspace rank rr is at least the signal rank ρ\rho, the expectation in (11) is flat there (Lemma 1), so this only rescales the momentum, as Smok-Hop's scalar tail does. The polar step multiplies by the inverse square root of the current statistic instead (Q1), which is not flat (AM/GM ratios in our second comment), so it also changes the direction of the momentum.

To test this, we compared the complement steps alone. For q_proj and up_proj across layers and steps, we applied KL-Shampoo's complement step, the scalar-tail step, and the polar step (exact polar factor) to the same left-whitened momentum, on two complements: of KL-Shampoo's top-128 eigenspace (its own tail) and of Pro-KLShampoo's tracked subspace (where the polar step acts). The table gives the mean cosine of the scalar-tail and polar updates with KL-Shampoo's complement update (q_proj / up_proj).

UpdateComplement ofLLaMA-134MLLaMA-450M
Scalar tailKL-Shampoo's top-128 eigenspace0.998 / 0.9730.993 / 0.944
Scalar tailPro-KLShampoo's tracked subspace0.990 / 0.9700.975 / 0.927
PolarKL-Shampoo's top-128 eigenspace0.792 / 0.8130.698 / 0.753
PolarPro-KLShampoo's tracked subspace0.802 / 0.8130.720 / 0.753

The table shows that the scalar-tail update is close to KL-Shampoo's complement update while the polar update is not, which largely explains the lower cosine on the complement in the first paragraph under Q2.

Sec. 4.2's statement that the scalar approximation on the complement appears insufficient refers to Smok-Hop. Since KL-Shampoo's complement update is close to the same scalar step, the question is whether this whitening of the complement is enough, and the polar step replaces it by whitening with the current statistic, which is far from flat.

We also checked that Pro-KLShampoo's tracked subspace captures the spike of KL-Shampoo's right factor, i.e., its directions with large eigenvalues. We divide the mean eigenvalue of this factor on a subspace by its mean on the complement. The ratio is 12 to 41 for the top-128 eigenspace, which maximizes it, and for Pro-KLShampoo's subspace it is at least 0.969 (q_proj) and 0.883 (up_proj) times that.

[0] Lin et al., arXiv 2509.03378, as cited by the reviewer.

As promised, the revised PDF (changes in blue) now addresses your main question, how Pro-KLShampoo can outperform KL-Shampoo.

  • W1/Q3 (outperforming KL-Shampoo). Sec. 3.3 and Sec. 4.2 state that the two methods differ on the complement, where KL-Shampoo's update is close to scaled gradient descent and Pro-KLShampoo's is spectral descent, both left-preconditioned. Appendix P gives our conjecture for why spectral descent may help there.
  • Q1 (the identity and "instantaneous"). Sec. 3.3 states that we use "instantaneous" in the sense of Eschenhagen et al. (2026, Proposition 4.8), and Appendix P compares the Newton–Schulz output with the exact polar factor.
  • Q2 (similarity of the updates). Sec. 4.2 and the new Appendix P compare the two optimizers' updates on the tracked subspace and on the complement. The updates are similar on the tracked subspace, which captures the spike of KL-Shampoo's right factor, and differ on the complement, where KL-Shampoo's update is close to the scalar-tail term but not to the polar step (Table 18).
  • W2 (model scale and batch size). LLaMA-1.3B with 2.05M tokens per step is added to Table 1, App. E and the Limitations, with its validation-loss curves in Figure 9.

The discussion closes on November 18. We would be glad to hear whether this answers your question.

Reviewer_ABMhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors observed that the eigenvalues of KL-Shampoo’s preconditioners have a spike-and-flat shape with a few large values and the rest nearly equal, and they show with a lemma that this shape arises when gradients are low rank plus isotropic noise. Based on this, they keep one of the two preconditioners in full only on a small tracked subspace and use a single scalar for the remaining directions. They then replace that scalar with Muon-style orthogonalization (motivated by an identity showing that orthogonalization has the same algebraic form as full KL-Shampoo on those directions). Finally, they prove convergence for an idealized version of the resulting method, Pro-KLShampoo, and empirically, on GPT-2 and LLaMA models at four scales, it reaches lower validation loss than KL-Shampoo, with less memory and less wall-clock time.

Critical strengths
  • The spike-and-flat observation is measured across layers, depths, training stages and two architectures, and it motivates the method very evidently
  • The theory is solid and consistent, and the assumptions are mostly standard and reasonable.
  • The range for the mixing weight (αkl\alpha_{kl}) is derived from the theory and feels very principled
  • The models are trained to a full, standard token budget, and the gains hold under a longer training protocol in results in the appendix.
  • The method degrades very nicely in the sense that what the tracked subspace misses is still updated through the orthogonalized complement, so even a small rank beats KL-Shampoo.
  • The ablations show that both parts of the update (the preconditioned subspace and the orthogonalized complement) are useful.
Critical weaknesses
  • I think the theory motivates the design but doesn’t explain the empirical gains. For instance, the convergence result holds for any subspace but doesn’t use the spike-and-flat structure or the link to KL-Shampoo. I don’t think this is disqualifying, but the connection between theory and results is unclear to me.
  • The empirical gains over KL-Shampoo are only partly explained, as on one model the validation loss is lower at similar training loss, and on the other the method makes more progress per step, and in both cases the reason is left open.
  • On the empirical results, the proposed method gets roughly two to three times as many configurations as KL-Shampoo, mainly because its extra mixing weight is swept on top of the learning rate and weight decay. Since each configuration reports the best of two seeds, more configurations also means more chances at a lucky run. I think this is more important on GPT-2, where the gains are small, thus the tuning budget is unequal
  • The main results report the best of two seeds, with no error bars, and this is mentioned only in the appendix. There is one multi-seed experiment in table 6, which helps but covers a single setting.
  • The closest prior method (COSMOS) is compared and distinguished only in the appendix, and the main text has no related work discussion so feel like poor engagement with literature, more specifically the positioning relative to prior work is just not very clear from the paper.
Questions for authors
  • The spike-and-flat structure is measured only on the two smaller models, does it hold at the larger models trained here, especially in the MLP layers, where the tail is least flat? is there any reason to expect it to change at larger scale?
  • The lemma on the spike-and-flat structure needs the gradient noise to be isotropic, how realistic do the authors think this is during LLM training, and could the condition be stated in the main text?
  • For the theoretical convergence result the authors assume every stochastic gradient is bounded. Is gradient clipping used, or is there evidence that this holds in practice?
  • On the convergence result the authors prove for an idealized version of the algorithm, while the experiments use a version with momentum and approximate orthogonalization. Which part of the proof would break for the version that is run? Would momentum remove the noise floor in the bound as it does in recent analyses of Muon (cited in the paper, e.g https://arxiv.org/abs/2509.15816)?
6 comments
AuthorsOpenReview ↗

Response 1/3

We thank the reviewer for the detailed assessment. We answer the five weaknesses and the four questions in order, in three comments.

W1. Theory and empirical gains

I think the theory motivates the design but doesn't explain the empirical gains. For instance, the convergence result holds for any subspace but doesn't use the spike-and-flat structure or the link to KL-Shampoo.

The theory is used to design the algorithm, not to show the improvement. Lemma 1 and Claim 1 show that the spike-and-flat restriction costs nothing in the KL objective under the low-rank model when the subspace rank rr is at least the signal rank ρ\rho, Claims 2 and 3 give its factors and the subspace to track, and identity (12) gives the complement step. Theorem 1 gives an upper bound on the stationarity measure of the idealized update for any orthonormal UU, but it does not use the spike-and-flat structure. The bound depends on the smoothness of the loss, whereas the spike-and-flat structure is a property of the gradient statistics that the preconditioner estimates; using the structure would require an assumption linking the two, which we leave to future work. The improvement is shown by the experiments, and W2 below shows where the two methods' updates differ. The revision will state the role of each result at the start of Sec. 3.

W2. Explaining the gains over KL-Shampoo

The empirical gains over KL-Shampoo are only partly explained, as on one model the validation loss is lower at similar training loss, and on the other the method makes more progress per step, and in both cases the reason is left open.

The two methods differ mainly on the complement, where KL-Shampoo's update is close to left-whitened gradient descent and Pro-KLShampoo's is left-whitened spectral descent. On the tracked subspace Pro-KLShampoo uses KL-Shampoo's formula with its own estimate of KL-Shampoo's right factor, which is averaged over past steps. On the complement this factor has nearly equal eigenvalues, with median tail ratios of at most 1.195 (Q1 in our third comment and App. E), so KL-Shampoo's update there is nearly proportional to the left-whitened momentum (cosine 0.927 to 0.998), which is left-whitened gradient descent with momentum. Pro-KLShampoo's complement update is the polar factor of the left-whitened momentum (computed approximately by Newton–Schulz), which is left-whitened spectral descent with momentum. Why spectral descent outperforms gradient descent is partly explained [2, 3, 4] but still open [1, Sec. 5]. Our second and third comments to Reviewer D3NZ give the details, the measurements, and our conjecture.

The GPT-2 observation, a lower validation loss at similar training loss, is a question of generalization, which we leave to future work. The revision will add this explanation to Secs. 3.3, 4.1, and 4.2 and the measurement as an appendix section.

[1] Eschenhagen, Cai, Lee, Shi. Clarifying Shampoo. arXiv 2602.09314, 2026.

[2] Bernstein, Newhouse. Old Optimizer, New Norm: An Anthology. arXiv 2409.20325, 2024.

[3] Su. Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal? arXiv 2511.00674, 2025.

[4] Wang et al. Muon Outperforms Adam in Tail-End Associative Memory Learning. ICLR 2026, arXiv 2509.26030.

AuthorsOpenReview ↗

Response 2/3

W3. Unequal tuning budget

the proposed method gets roughly two to three times as many configurations as KL-Shampoo, mainly because its extra mixing weight is swept on top of the learning rate and weight decay. Since each configuration reports the best of two seeds, more configurations also means more chances at a lucky run.

With αkl\alpha_{\mathrm{kl}} fixed at 0.01, Pro-KLShampoo is below KL-Shampoo at all four scales, with no more configurations per rank than the baselines. The count in the review is right for the full grid, which also sweeps αkl\alpha_{\mathrm{kl}}. The table gives the number of configurations swept for each method in Tables 7 to 10, and for each rank for Pro-KLShampoo.

ScaleKL-ShampooMuonPro-KLShampoo, αkl=0.01\alpha_{\mathrm{kl}} = 0.01Pro-KLShampoo, full gridSelected αkl\alpha_{\mathrm{kl}}
GPT-2 124M181515450.01
GPT-2 350M182015450.01
LLaMA-134M54390.01
LLaMA-450M33260.005

At LLaMA-450M the run with αkl=0.01\alpha_{\mathrm{kl}} = 0.01 (lr 0.02, r=128r = 128) reaches 2.7482, against 2.7650 for KL-Shampoo and 2.7453 for the best Pro-KLShampoo run in Table 1 (r=64r = 64). The default 0.01 (App. K) lies inside the range [0.0014, 0.016] derived in Sec. 3.3.

At GPT-2, where the gains are smallest, the advantage does not rest on Pro-KLShampoo's best cell. At GPT-2 350M all six Table 12 cells with αkl\alpha_{\mathrm{kl}} in {0.005, 0.01} are below every seed of KL-Shampoo's selected configuration (Table 6). At GPT-2 124M, under the 10,000-iteration protocol of Lin et al., momentum and β2\beta_2 were swept over the same six cells for both methods (Table 11), and Pro-KLShampoo is lower in every cell. The best cells are 3.2146 and 3.2198, a gap of 0.0052, about the same as the 0.0051 in Table 1.

W4. Best of two seeds, no error bars

The main results report the best of two seeds, with no error bars, and this is mentioned only in the appendix. There is one multi-seed experiment in table 6, which helps but covers a single setting.

We added a second multi-seed setting, LLaMA-134M, at the selected hyperparameters of each method. In both settings every Pro-KLShampoo run is below every KL-Shampoo run. The table lists every run.

SettingPro-KLShampoo (r=128r = 128)KL-Shampoo
GPT-2 350M (Table 6)3.0344, 3.0346, 3.0354, 3.03643.0384, 3.0397, 3.0406, 3.0418
LLaMA-134M (sweep seeds and additional runs)3.0315, 3.0334, 3.0335, 3.0335, 3.03463.0494, 3.0504, 3.0524, 3.0525, 3.0568

At LLaMA-134M the worst Pro-KLShampoo run is 0.0148 below the best KL-Shampoo run, the exact one-sided rank-sum test gives p=1/252≈0.004p = 1/252 \approx 0.004, and the standard deviations are 0.0011 and 0.0028. At GPT-2 350M the worst Pro-KLShampoo run is 0.0020 below the best KL-Shampoo run, the same test gives p=0.014p = 0.014 (Table 6), and the standard deviations are 0.0009 and 0.0014. The revision will state in Sec. 4 that the main results report, for every method, the best of two seeds.

W5. Related work and COSMOS only in the appendix

The closest prior method (COSMOS) is compared and distinguished only in the appendix, and the main text has no related work discussion

App. N compares the two methods at matched rank (r=64r = 64, 128128), using the official COSMOS release with the learning rate and rank swept per scale. We observe that Pro-KLShampoo has lower validation loss at three of the four scales, by 0.004 to 0.026. At LLaMA-450M, COSMOS is lower by 0.005 without weight decay, and Pro-KLShampoo is lower by 0.012 with weight decay (r=64r = 64), which was swept for COSMOS only.

COSMOS combines SOAP-style estimation inside a tracked subspace with Muon on the remaining directions, which it regards as less critical but too costly to handle with SOAP. Pro-KLShampoo is motivated by the spike-and-flat structure of KL-Shampoo's preconditioner (App. E), so we restrict KL-Shampoo's objective to this family, which gives the tracked subspace and its factors (Claims 2 and 3). On the complement it uses the polar factor of the left-whitened momentum, which by identity (12) recovers the algebraic form of KL-Shampoo's complement update, whereas COSMOS orthogonalizes the momentum without left whitening.

The revision will give the protocol and these results in Sec. 4, with a pointer to App. N for the full table, and will add a related-work discussion of subspace-based optimizers in the main text.

AuthorsOpenReview ↗

Response 3/3

Q1. Spike-and-flat structure at the larger scales

does it hold at the larger models trained here, especially in the MLP layers, where the tail is least flat? is there any reason to expect it to change at larger scale?

It holds at the two larger scales where we recorded the spectra, GPT-2 350M and LLaMA-1.3B. The 1.3B model is a new experiment, described in our response to Reviewer D3NZ. The table follows the format of Table 3 (medians, early / late).

ScaleLayer typeρ0\rho_0ρ32\rho_{32}ρ128\rho_{128}
GPT-2 350MAttention7.147 / 10.0701.100 / 1.0591.012 / 1.013
GPT-2 350MMLP1.797 / 1.5201.311 / 1.2081.182 / 1.121
LLaMA-1.3BAttention10.849 / 18.2561.107 / 1.0261.031 / 1.003
LLaMA-1.3BMLP1.163 / 1.1231.082 / 1.0601.052 / 1.034

The MLP tail stays at the same level as the scale grows. Its ρ128\rho_{128} is 1.182 / 1.121 at GPT-2 350M against 1.195 / 1.186 at GPT-2 124M, and 1.052 / 1.034 at LLaMA-1.3B against 1.082 / 1.029 at LLaMA-134M.

Q2. Isotropic noise in the lemma

how realistic do the authors think this is during LLM training, and could the condition be stated in the main text?

Yes, Sec. 3 will state the condition, i.i.d. mean-zero noise entries with a common variance. We do not expect it to hold exactly, and the method does not rely on it. Claim 1 bounds the KL gap of the restricted family through the AM/GM ratio of the tail for any spectrum. Under separable anisotropic noise the same calculation gives a spike plus a floor shaped by the noise covariance (App. I). Even for MLP, the least flat layer type, the leading 128 directions remove a median of 70.8% to 84.4% of each matrix's full-spectrum log AM/GM gap across GPT-2 124M, LLaMA-134M and LLaMA-1.3B and all training stages.

Q3. Bounded stochastic gradients

Is gradient clipping used, or is there evidence that this holds in practice?

No gradient clipping is used in any experiment. Over a complete GPT-2 124M run of 10,000 steps, the largest observed gradient operator norm is 0.176. In the proof this assumption enters through the upper bounds on the preconditioner factors (Lemma 2), the constant σkl\sigma_{\mathrm{kl}} of Theorem 1, and Corollary 2.

Q4. The proof and the version that is run

Which part of the proof would break for the version that is run? Would momentum remove the noise floor in the bound as it does in recent analyses of Muon

Algorithm 2 differs from the analyzed Algorithm 1 in the points listed in App. K. The main ones for the proof are the following three.

Newton–Schulz (P3). The proof uses two properties of the exact polar factor, ⟨M,polar(M)⟩=∥M∥∗\langle M, \mathrm{polar}(M)\rangle = \Vert M\Vert_* and ∥polar(M)∥op≤1\Vert\mathrm{polar}(M)\Vert_{\mathrm{op}} \le 1. The Newton–Schulz output has the same singular vectors but not unit singular values, so neither property holds exactly, and we have not carried the error through the proof.

Momentum (P1). The descent step takes the conditional expectation of the update given the past and uses that GtG_t is unbiased. With momentum the update depends on past gradients and this step no longer applies. The analyses of Muon with momentum in App. A handle this dependence for Muon's update alone, while our update also contains the subspace term.

Maintained spectral estimates (P4). As in practical Shampoo-type optimizers such as SOAP and KL-Shampoo, the eigenbasis is refreshed only periodically (every 10 steps here) and is stale in between, and the eigenvalues are EMA estimates in this basis. The KL-Shampoo baseline in our experiments uses the same scheme. The transported QSQ_S is in addition not exactly orthonormal unless the QR refresh is applied at every step (App. K). The applied matrices are therefore not exactly the inverse square roots of LtL_t and StS_t that the proof uses.

On the noise floor, the analyses of Muon cited in the paper remove the fixed-batch term in two ways, by coupled step-size and momentum schedules or variance reduction (Chang et al., arXiv 2509.15816), and by batch growth (Nagashima and Iiduka, arXiv 2601.19400). The second route is discussed after Theorem 1 (Sec. 3.4), whose noise term is the average of the noise levels σF,t\sigma_{F,t} over the TT steps. This average is a floor for a fixed batch and decays as O(T−1/2)O(T^{-1/2}) for a batch growing linearly in tt. We have not proved the first route for our update and leave it, together with the three points above, to future work.

Summary of revisions

In the revision, besides the changes under W1, W2, W4 and W5, Sec. 3 will state the noise condition of Lemma 1, App. K will state where each of the three points above affects the proof, and Sec. 4 will give the number of configurations swept for each method and the LLaMA-134M multi-seed runs. Table 3 in App. E will include the GPT-2 350M and LLaMA-1.3B rows.

We are happy to provide further detail or run additional experiments, as far as our compute allows.

Reviewer_dNqQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies optimizers in the intersection of Kronecker-factored preconditioning and (gradient) momentum orthogonalization. The authors propose Pro-KL Shampoo, which is based on the empirical observations of spike-and-flat eigenvalue structure in LL and RR matrices, and rank ρ\rho signal plus noise model. Pro-KL Shampoo maintains a subspace for the spike, and uses orthogonalization for the flat tail. Some theoretical guarantees are given, and Pro-KL Shampoo consistently outperforms baselines in 4 GPT-2 and LLaMA training experiments measured in final validation loss.

Critical strengths
  1. Pro-KL Shampoo is well motivated by experimental observations and supported by a low rank noise model.
  2. Pro-KL Shampoo is more memory efficient.
  3. Pro-KL Shampoo achieves lower validation loss than baseline methods and show shorter per-step wall-clock time.
  4. Several ablation studies show that the dominant subspace is low rank and validate the effectiveness of the core idea of Pro-KL Shampoo.
Critical weaknesses
  1. The validation loss gains over baselines are small, and there is no comparisons on the search budgets. One may question the baselines are not equally tuned as Pro-KL Shampoo.
  2. The convergence theorem is based on fairly strong assumptions including bounded gradient operator norm, and spectral lower bound. The latter naturally fails when gradient converges to zero.
  3. I think the comparisons between Smok-Hop and Pro-KL Shampoo needs more discussions/ablations. The current experiments used different setups for both algorithms while an ideal ablation should differentiate only on scalar scaling for Smok-Hop and orthogonalization for Pro-KL Shampoo.
Questions for authors

n.a.

2 comments

We thank the reviewer for the positive assessment and constructive feedback. We answer the three weaknesses in order.

W1. Small gains and search budgets

The validation loss gains over baselines are small, and there is no comparisons on the search budgets. One may question the baselines are not equally tuned as Pro-KL Shampoo.

With αkl\alpha_{\mathrm{kl}} fixed at 0.01, Pro-KLShampoo is below KL-Shampoo at all four scales, with no more configurations per rank than the baselines. The table gives the number of configurations swept for each method in Tables 7 to 10, and for each rank for Pro-KLShampoo.

ScaleKL-ShampooMuonPro-KLShampoo, αkl=0.01\alpha_{\mathrm{kl}} = 0.01Pro-KLShampoo, full gridSelected αkl\alpha_{\mathrm{kl}}
GPT-2 124M181515450.01
GPT-2 350M182015450.01
LLaMA-134M54390.01
LLaMA-450M33260.005

The full grid of Pro-KLShampoo is larger because the mixing weight αkl\alpha_{\mathrm{kl}} is swept. At LLaMA-450M the run with αkl=0.01\alpha_{\mathrm{kl}} = 0.01 (lr 0.02, r=128r = 128) reaches 2.7482, against 2.7650 for KL-Shampoo and 2.7453 for the best Pro-KLShampoo run in Table 1 (r=64r = 64). The default 0.01 (App. K) lies inside the range [0.0014, 0.016] derived in Sec. 3.3.

The gains are smallest at the two GPT-2 scales, and three results support them there. At GPT-2 350M, four seeds per method give 3.0352±0.00093.0352 \pm 0.0009 against 3.0401±0.00143.0401 \pm 0.0014 with disjoint ranges (Table 6, p=0.014p = 0.014). At the same scale the advantage does not rest on Pro-KLShampoo's best cell, as all six Table 12 cells with αkl\alpha_{\mathrm{kl}} in {0.005, 0.01} are below every seed of KL-Shampoo's selected configuration. At GPT-2 124M, under the 10,000-iteration protocol of Lin et al., with momentum and β2\beta_2 swept over the same six cells for both methods, Pro-KLShampoo is lower in every cell (Table 11). We also added multi-seed runs at LLaMA-134M at the selected configuration of each method, where five Pro-KLShampoo runs (3.0315 to 3.0346) and five KL-Shampoo runs (3.0494 to 3.0568) are separated by 0.0148.

W2. Assumptions of the convergence theorem

The convergence theorem is based on fairly strong assumptions including bounded gradient operator norm, and spectral lower bound. The latter naturally fails when gradient converges to zero.

Both are assumptions of Theorem 1 (App. J), and the lower bound cannot hold if the gradient statistics tend to zero, so Theorem 1 is a statement for runs in which these statistics stay bounded below. For a fixed batch, the setting of our experiments, it gives convergence to a noise-dependent neighborhood of size O(B−1/2)O(B^{-1/2}) rather than to a stationary point, as Sec. 3.4 states. For the bounded gradient operator norm, no gradient clipping is used in our experiments, and the largest observed gradient operator norm over a complete GPT-2 124M run of 10,000 steps is 0.176. The revision will state this scope next to Theorem 1.

W3. The ablation between Smok-Hop and Pro-KLShampoo

The current experiments used different setups for both algorithms while an ideal ablation should differentiate only on scalar scaling for Smok-Hop and orthogonalization for Pro-KL Shampoo.

We ran this ablation at the two LLaMA scales. Smok-Hop in Table 2 uses KL-Shampoo's EMA momentum and even uses KL-Shampoo's learning rate without extra sweep at GPT-2 124M and LLaMA-450M. At both LLaMA scales, we reran it at r=128r = 128 with Pro-KLShampoo's selected configuration, including its Nesterov momentum, and swept Smok-Hop's learning rate. At both scales the best learning rate lies inside the swept range. The two methods then differ only in the complement step, the mixing weight and the tuned learning rate. Pro-KLShampoo needs the mixing weight because orthogonalization changes the scale of the complement. The table reports the final validation loss.

ScaleSmok-Hop (Nesterov), with learning ratePro-KLShampooKL-Shampoo
LLaMA-134M3.0591 (1e-4), 3.0520 (2e-4), 3.1060 (5e-4)3.03353.0504
LLaMA-450M2.7940 (5e-5), 2.7739 (1e-4), 2.7961 (2e-4)2.74692.7650

Pro-KLShampoo is ahead of the best Smok-Hop cell by 0.0185 at LLaMA-134M and by 0.0270 at LLaMA-450M. With EMA momentum (Table 2) Smok-Hop reaches 3.0476 and 2.7704, so the gap does not come from the momentum. At LLaMA-450M a further run at lr 5e-4 had a loss spike at step 809 and was stopped at step 1750, and the run at lr 2e-4 had one loss spike at step 836 and recovered.

Our response to Reviewer D3NZ compares the two complement steps on the same input, and this will enter Sec. 4.1.

In the revision, Sec. 4 will report the configuration counts of the table under W1 and the multi-seed runs at LLaMA-134M, and the Nesterov ablation will be added to Table 2 with its sweep in App. O.

We are happy to provide further detail during the discussion.

As promised, the revised PDF (changes in blue) includes the following changes for your three points.

  • W3 (ablation between Smok-Hop and Pro-KLShampoo). Table 2 and Figure 3 now include Smok-Hop with Pro-KLShampoo's Nesterov momentum and its own learning-rate sweep (Table 17, App. O).
  • W1 (gains and search budgets). With αkl\alpha_{\mathrm{kl}} fixed at its default 0.01, Pro-KLShampoo sweeps about as many configurations per rank as the baselines and still has lower validation loss than KL-Shampoo at all four scales (Sec. 4; Table 16, App. O). At LLaMA-134M, five runs per method (different seeds) give disjoint ranges (Sec. 4.1; Table 6, App. O).
  • W2 (assumptions of the convergence theorem). Sec. 3.4 states the scope of Theorem 1, and App. J reports the largest gradient operator norm over a complete 10,000-step run and states that no gradient clipping is used.

We would welcome your view on whether these changes address your concerns before the discussion closes on November 18.

Comments on the paper

1 comment

We thank all reviewers for their careful reviews. As promised in our responses, we have uploaded a revised PDF, with added and changed text in blue. Below we list the revisions and the reviewers who asked for them.

  • Sec. 3, just before Sec. 3.1: stated the role of each theoretical result (ABMh W1).
  • Sec. 3, paragraph "Low rank gradient supports the spike-and-flat": stated the noise condition of Lemma 1 (ABMh Q2).
  • Sec. 3.3, Sec. 4.2 and the new Appendix P: stated where KL-Shampoo and Pro-KLShampoo differ (scaled left-preconditioned gradient descent vs. left-preconditioned spectral descent on the complement), compared their updates on the tracked subspace and on the complement, and gave our conjecture for why spectral descent may help on the complement (D3NZ W1/Q3, Q1, Q2; ABMh W2).
  • Sec. 3.3: stated that "instantaneous" is used in the sense of Eschenhagen et al. (2026, Proposition 4.8) (D3NZ W1/Q3, Q1).
  • Sec. 3.4 and App. J: stated the scope of Theorem 1, and reported the largest gradient operator norm over a 10,000-step run without gradient clipping (dNqQ W2; ABMh Q3).
  • App. K: stated where each main difference of the practical algorithm affects the proof (ABMh Q4).
  • Table 1, Sec. 4.1, App. O, Figure 9 and the Limitations: added LLaMA-1.3B with 2.05M tokens per step (D3NZ W2).
  • Sec. 4 and Table 16: added the number of configurations swept per method and the results with αkl\alpha_{\mathrm{kl}} fixed at its default 0.01 (ABMh W3; dNqQ W1).
  • Sec. 4, Sec. 4.1 and Table 6: stated the best-of-two-seeds protocol and reported five runs per method (different seeds) at LLaMA-134M (ABMh W4; dNqQ W1).
  • Sec. 4.1: added the comparison with COSMOS and named other subspace-based optimizers (ABMh W5).
  • Table 2, Figure 3 and Table 17: added Smok-Hop with Pro-KLShampoo's Nesterov momentum and its own learning-rate sweep (dNqQ W3).
  • Table 3 (App. E): added the tail ratios at a larger scale of each architecture, GPT-2 350M and LLaMA-1.3B (ABMh Q1).

We are glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.