- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes a new optimization algorithm that is effectively a combination of a KL-Shampoo style Kronecker factored preconditioner and a Muon style orthogonalization. The authors identify that the distribution of the eigenvalues in most layers’ preconditions follow a similar distribution: a few top eigenvalues dominate followed by a relatively uniform tail. With this observation, they motivate tracking (one of) the preconditions in a subspace to encapsulate the top eigenvalues, while tracking the tail in the orthogonal complement subspace. They further modify this idea by applying a Muon style orthogonalization to the tail component using the observation its eigenvalues should be approximately similar. The authors do a non-trivial amount of derivations to show the different mathematical forms this algorithm can take an write it down in the most mathematically concise way possible. They additionally solve the KL-Shampoo preconditioner optimization problem for each of the new objects they track in their optimizer state assuming one object () is fixed, and provide a theoretical argument for how to select the optimal given the rest of their optimizer state is a solution to the previous problem. The authors further argue that the Muon style orthogonalization step is equivalent to the original form of KL-Shampoo’s preconditioner if only the current mini-batch is considered. The authors conclude their theoretical work by providing a convergence guarantee on the average momentum buffer norm under some (somewhat strong) assumptions. Experimentally, the authors show across two architectures, two scales, and two datasets that their algorithm. They show superior performance in terms of memory consumption, clock time for matched loss, and iterations. The authors additionally provide some ablations regarding which subspace has more impact and how the orthogonalization step helps versus simply using the original two subspaces.
Critical strengths
I like the idea of this paper, I think it is an effective use of exploiting an empirical observation to develop a new algorithm. The idea of splitting into subspaces is somewhat intuitive, and the orthogonalization step relates to other approaches in modern optimization literature. The theory the authors provide appears sound and some results appear to make intuitive sense in their conclusions. The authors additionally provide some interesting commentary on relating their method to other concepts, such as the algebraic equivalence of their whitening method. Overall the paper is fairly well structured and clearly written, notational voices typically make the mathematics read well. Experimentally, the result is faster in terms of wall time and memory which I would expect given the nature of the algorithm, however I do have some questions regarding their other experimental results as follows.
Critical weaknesses
While I do like this paper, as above, I do have some questions about the validity of the results I feel the authors must address for acceptance. I largely find their motivation and theory somewhat convincing, but there is a core inconsistency in the theoretical and experimental results that the authors must address. This work solves a restricted form of the KL-Shampoo preconditioner problem and in fact provides theory for the error on this approximation. Given this, the solution the authors find (at least for their Smok-Hop) variant should live in a subset of the solutions to the full problem. Further, the authors argue that polar decomposition is equivalent to the algebraic form of the KL-Shampoo preconditioner for a single batch. This however includes the fact that (1) [0] shows that the instantaneous version of KL-Shampoo underperforms the full method, and (2) that in practice the true polar decomposition is not used, rather an approximation. Because of these facts, I find it fair to call the proposed algorithm an approximation of KL-Shampoo. This is fine, as the approximation-speed tradeoff of optimization algorithms is clearly a very important area of research. However, when the authors present their experimental results, this is not consistent. In the main text the authors largely focus on time and memory advantages which should be expected for an approximation algorithm, but Figure 3 and Figure 5 show that Smok-Hop and Pro-KLShampoo can consistently outperform KL-Shampoo in terms of both training and validation loss when measured per step. I’m somewhat troubled by this results as it appears to be a case of the approximation outperforming what it is approximating. While the authors do report on their experimental procedure quite extensively, I did not find any commentary on why they suspect this result to be the case, rather a focus on the wall time improvements.
A more minor weakness is the architecture scale is a bit limited, largely on the account that on models that use larger batch size the performance of these sorts of algorithms can very significantly, but I understand if there are some computational limitations.
[0] Wu Lin, Scott C Lowe, Felix Dangel, Runa Eschenhagen, Zikun Xu, and Roger B Grosse. Understanding and improving shampoo and soap via kullback-leibler minimization. arXiv preprint arXiv:2509.03378, 2025.
Questions for authors
- Does the claim of algebraic equivalence of the decomposition with this form of whitening actually hold numerically or matter that much? In particular, [0] notes that the instantaneous version of KL-Shampoo is far inferior to the EMA versions, and thus I wonder if this instantaneous equivalence actually signals much improvement.
- Do the updates that KL-Shampoo and Pro-KLShampoo find appear empirically similar? Given theory predicts they should be in low rank settings or under instantaneous updates, one may expect similar update directions.
- My primary question is resolving the approximation and yet over performance of KL-Shampoo as details in the weakness section, if the authors can provide insights into why this takes place I would likely raise my score.
[0] Wu Lin, Scott C Lowe, Felix Dangel, Runa Eschenhagen, Zikun Xu, and Roger B Grosse. Understanding and improving shampoo and soap via kullback-leibler minimization. arXiv preprint arXiv:2509.03378, 2025.
4 comments
Response 2/3
W1 and Q3 (continued)
I find it fair to call the proposed algorithm an approximation of KL-Shampoo.
Fig. 3 and Fig. 5 show that Smok-Hop and Pro-KLShampoo can consistently outperform KL-Shampoo [...] the approximation outperforming what it is approximating.
My primary question is resolving the approximation and yet over performance of KL-Shampoo
We view Pro-KLShampoo as an approximation of KL-Shampoo plus one added step, the orthogonalization of the complement. Write for the left-whitened gradient of step projected on the complement of a rank- subspace, each algorithm with its own and . Up to scale, KL-Shampoo's update on the complement is (Eq. (11), expectation over the stochastic gradient at full-KL stationarity), and Pro-KLShampoo's update is (Eq. (12)). In practice each method applies its formula to its own momentum in place of , and KL-Shampoo replaces by its right factor , an exponential moving average of over past steps in the full space. We call the running statistic and , with the momentum in place of , the current statistic.
The eigenvalues of differ on the subspace while they are nearly equal on the complement (App. E, tail ratio medians at most 1.195), so on the complement is close to a scalar. Hence KL-Shampoo's update there is close to the complement term of the restricted update in (6) (i.e., ) which Smok-Hop uses (cosine 0.927 to 0.998 between the two, see Q2 in the next comment).
Against , the step changes the singular values, while keeps the ratio between the complement and subspace updates close to that of KL-Shampoo.
On the complement the running statistic is nearly a multiple of the identity (App. E), so KL-Shampoo's step nearly preserves the singular values of up to scale, and its update there is close to left-whitened gradient descent with momentum. The polar factor instead makes them equal, so Pro-KLShampoo's update there is left-whitened spectral descent with momentum. The two differ because these singular values (of the left-whitened gradient, momentum in practice) are unequal, with mean AM/GM ratios of their squares of 3.42 / 1.91 (q_proj in attention / up_proj in MLP) at LLaMA-134M and 5.45 / 2.66 at LLaMA-450M, over all records of one Pro-KLShampoo run per scale. By identity (12) the difference lies in the statistic: the polar step is KL-Shampoo's complement formula with the current statistic in place of the running statistic. The running statistic is an average of squares of left-whitened gradients, while the current statistic is the square of the current left-whitened momentum, which is itself an average of gradients; a flat average of squares does not make the square of an average flat.
The ratio between the subspace update and the complement update is controlled by , which App. J.3 relates to the expected ratio of update (6) at the stationary point. At GPT-2 350M this Frobenius-norm ratio divided by KL-Shampoo's (split by its top-128 eigenspace) has median 0.92 (attention) and 1.02 (MLP).
Why spectral descent outperforms gradient descent is partly explained [2, 3, 4] but still open [1, Sec. 5]. Our conjecture is about the relative step sizes along the singular directions of the left-whitened momentum on the complement. KL-Shampoo's step there is nearly proportional to this momentum, so the directions with large singular values dominate it and those with small ones move little. Wang et al. [4] show that in an associative memory model the singular values of the gradient encode how often each fact appears in the training data, so gradient descent learns rarely seen facts more slowly than frequent ones, while Muon, with equal step sizes, learns all facts at the same rate. If the small singular directions on the complement likewise correspond to rarely seen data, the polar step may help in the same way. Testing it is left to future work.
In the revision, Secs. 3.3, 4.1, and 4.2 will give this explanation, and the measurements will go to a new appendix section.
[1] Eschenhagen, Cai, Lee, Shi. Clarifying Shampoo. arXiv 2602.09314, 2026.
[2] Bernstein, Newhouse. Old Optimizer, New Norm: An Anthology. arXiv 2409.20325, 2024.
[3] Su. Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal? arXiv 2511.00674, 2025.
[4] Wang et al. Muon Outperforms Adam in Tail-End Associative Memory Learning. ICLR 2026, arXiv 2509.26030.
Response 3/3
Q1. The algebraic identity and [0]
Does the claim of algebraic equivalence of the decomposition with this form of whitening actually hold numerically or matter that much? In particular, [0] notes that the instantaneous version of KL-Shampoo is far inferior to the EMA versions, and thus I wonder if this instantaneous equivalence actually signals much improvement.
Identity (12) holds exactly for the polar factor. In practice the polar factor is computed by Newton–Schulz, which gives the same singular vectors, with singular values pushed toward one instead of set to one (first comment, under (2)); its output has cosine 0.985 / 0.986 at LLaMA-134M and 0.976 / 0.977 at LLaMA-450M with the exact polar factor (q_proj / up_proj). As stated in Sec. 3.3, the identity recovers the algebraic form of KL-Shampoo's complement update (11) with the stationary statistic replaced by the instantaneous one, and we do not claim that the instantaneous statistic approximates the stationary one in a statistical sense. It is therefore not meant to signal an improvement by itself. The comparison in [0] is a different one (first comment, under (1)), and we discuss where the improvement comes from in our second comment.
Q2. Similarity of the updates
Do the updates that KL-Shampoo and Pro-KLShampoo find appear empirically similar? Given theory predicts they should be in low rank settings or under instantaneous updates, one may expect similar update directions.
The polar step shares the algebraic form of KL-Shampoo's complement update but not its statistic, so we do not expect the two updates to be similar on the complement. We nevertheless compared them, as the reviewer asks. To this end, we computed the two optimizers' own updates along the trajectory of Pro-KLShampoo (KL-Shampoo's update with its own EMA momentum and running statistics maintained alongside). We found that the two updates are similar on Pro-KLShampoo's tracked subspace, with cosine 0.941 / 0.882 at LLaMA-134M and 0.931 / 0.877 at LLaMA-450M (q_proj / up_proj), and that on the complement they differ, with 0.767 / 0.771 and 0.690 / 0.709.
On the complement KL-Shampoo multiplies the left-whitened momentum by the inverse square root of the running statistic. In the low-rank model, when the subspace rank is at least the signal rank , the expectation in (11) is flat there (Lemma 1), so this only rescales the momentum, as Smok-Hop's scalar tail does. The polar step multiplies by the inverse square root of the current statistic instead (Q1), which is not flat (AM/GM ratios in our second comment), so it also changes the direction of the momentum.
To test this, we compared the complement steps alone. For q_proj and up_proj across layers and steps, we applied KL-Shampoo's complement step, the scalar-tail step, and the polar step (exact polar factor) to the same left-whitened momentum, on two complements: of KL-Shampoo's top-128 eigenspace (its own tail) and of Pro-KLShampoo's tracked subspace (where the polar step acts). The table gives the mean cosine of the scalar-tail and polar updates with KL-Shampoo's complement update (q_proj / up_proj).
| Update | Complement of | LLaMA-134M | LLaMA-450M |
|---|---|---|---|
| Scalar tail | KL-Shampoo's top-128 eigenspace | 0.998 / 0.973 | 0.993 / 0.944 |
| Scalar tail | Pro-KLShampoo's tracked subspace | 0.990 / 0.970 | 0.975 / 0.927 |
| Polar | KL-Shampoo's top-128 eigenspace | 0.792 / 0.813 | 0.698 / 0.753 |
| Polar | Pro-KLShampoo's tracked subspace | 0.802 / 0.813 | 0.720 / 0.753 |
The table shows that the scalar-tail update is close to KL-Shampoo's complement update while the polar update is not, which largely explains the lower cosine on the complement in the first paragraph under Q2.
Sec. 4.2's statement that the scalar approximation on the complement appears insufficient refers to Smok-Hop. Since KL-Shampoo's complement update is close to the same scalar step, the question is whether this whitening of the complement is enough, and the polar step replaces it by whitening with the current statistic, which is far from flat.
We also checked that Pro-KLShampoo's tracked subspace captures the spike of KL-Shampoo's right factor, i.e., its directions with large eigenvalues. We divide the mean eigenvalue of this factor on a subspace by its mean on the complement. The ratio is 12 to 41 for the top-128 eigenspace, which maximizes it, and for Pro-KLShampoo's subspace it is at least 0.969 (q_proj) and 0.883 (up_proj) times that.
[0] Lin et al., arXiv 2509.03378, as cited by the reviewer.
As promised, the revised PDF (changes in blue) now addresses your main question, how Pro-KLShampoo can outperform KL-Shampoo.
- W1/Q3 (outperforming KL-Shampoo). Sec. 3.3 and Sec. 4.2 state that the two methods differ on the complement, where KL-Shampoo's update is close to scaled gradient descent and Pro-KLShampoo's is spectral descent, both left-preconditioned. Appendix P gives our conjecture for why spectral descent may help there.
- Q1 (the identity and "instantaneous"). Sec. 3.3 states that we use "instantaneous" in the sense of Eschenhagen et al. (2026, Proposition 4.8), and Appendix P compares the Newton–Schulz output with the exact polar factor.
- Q2 (similarity of the updates). Sec. 4.2 and the new Appendix P compare the two optimizers' updates on the tracked subspace and on the complement. The updates are similar on the tracked subspace, which captures the spike of KL-Shampoo's right factor, and differ on the complement, where KL-Shampoo's update is close to the scalar-tail term but not to the polar step (Table 18).
- W2 (model scale and batch size). LLaMA-1.3B with 2.05M tokens per step is added to Table 1, App. E and the Limitations, with its validation-loss curves in Figure 9.
The discussion closes on November 18. We would be glad to hear whether this answers your question.
Response 1/3
We thank the reviewer for the careful reading.
W1 and Q3 (first part). How can the method outperform KL-Shampoo if it approximates KL-Shampoo?
Pro-KLShampoo is an approximation of KL-Shampoo plus one added step, the orthogonalization of the complement, and with this step it is ahead of KL-Shampoo at every scale. The approximation alone (Smok-Hop) does not consistently outperform KL-Shampoo, as shown below. KL-Shampoo's right-side preconditioner, averaged over past steps, has a few large eigenvalues and a nearly flat tail (App. E). Pro-KLShampoo tracks the eigenspace of the large eigenvalues and applies KL-Shampoo's formula there, while the added orthogonalization acts on its complement.
We agree. Smok-Hop is within the restricted family, and it does not consistently outperform KL-Shampoo. In Table 2 it is ahead of KL-Shampoo by 10% of Pro-KLShampoo's gain at GPT-2 124M and by 17% at LLaMA-134M, and behind by 30% at LLaMA-450M. With the same Nesterov momentum and a swept learning rate, Smok-Hop is behind Pro-KLShampoo by 0.0185 at LLaMA-134M and 0.0270 at LLaMA-450M (our response to Reviewer dNqQ).
The two papers use "instantaneous" for different things. In [0] (Sec. 4, Fig. 4), the instantaneous variant still builds the preconditioners from an average over past steps and only changes how their eigenvalues are estimated. In [1] (Proposition 4.8), as in our identity (12), it means computing KL-Shampoo's statistics from the current gradient alone instead of an average over past steps.
In the second sense the reviewer's point holds: run this way on a fixed gradient, KL-Shampoo converges to spectral descent [1, Proposition 4.8], which with momentum is the update of Muon, and Muon is behind KL-Shampoo at three of the four scales in Table 1. Pro-KLShampoo does this only for its right-side preconditioner on the complement and, like Muon, computes it from the current momentum. Its left preconditioner and its right-side preconditioner on the tracked subspace are averaged over past steps, as in KL-Shampoo. This combination is ahead of both Muon and KL-Shampoo at every scale in Table 1.
We use five Newton–Schulz steps with the coefficients be the same as Muon's (App. K). The iteration is an odd matrix polynomial, so its output has the same singular vectors as the exact polar factor, with singular values pushed toward one instead of set to one.
W2. Model scale and batch size
We extended the evaluation to LLaMA-1.3B trained on 26B tokens, with sequence length 1024 and batch size 2000, about 2.05M tokens per iteration (previously at most batch size 1620, about 1.66M tokens, for LLaMA-450M).
Due to the limited computational resources, learning rates were selected by short screening runs for each method, {0.01, 0.02, 0.03} for Muon, {0.001, 0.002, 0.003} for KL-Shampoo, and three (learning rate, αkl) pairs for Pro-KLShampoo, (0.01, 0.005), (0.02, 0.005), and (0.02, 0.003). The selected configurations were trained for the full budget, with one run each.
In the revision, Sec. 3.3 will state that we use "instantaneous" in the sense of [1], and Table 1, App. E, and the Limitations will include LLaMA-1.3B, with the batch range extended to 2.05M tokens per step.
[0] Lin et al., arXiv 2509.03378.
[1] Eschenhagen, Cai, Lee, Shi. Clarifying Shampoo. arXiv 2602.09314.