- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper store each Shampoo factor in the current basis, P = QᵀSQ, instead of storing S itself. P is symmetric and close to diagonal, so the QR used to refresh the basis gets cheaper, and one can refresh only a block of basis vectors while the whole basis stays orthogonal. Together with a custom mixed-precision QR, the paper reports much faster basis updates for KL-Shampoo and better BF16 test loss, mostly for SOAP and KL-SOAP.
Critical strengths
Storing the factor in its own basis is a simple ideas that drops easily into existing Shampoo-type code, and the full-basis update is mathematically the same as before. The subspace update keeps the whole basis orthogonal without any extra re-orthogonalization, which is neat. I also liked that better BF16 behavior comes out as a side effect, and the runtime study shows the loss/time trade-off over refresh interval and subspace size clearly, with tuned Muon as a reference. Overall, it’s a good paper.
Critical weaknesses
-
The speed story is mostly told through a per-step timing breakdown on one 8000×8000 matrix, which is bigger than any weight in the trained models. The end-to-end runtime plots only compare the new variants with each other and with Muon, so we never see how much wall-clock time the reparametrization saves over the original KL-Shampoo or SOAP. The paper also says several times that the gain holds with plain PyTorch QR, but PyTorch QR is never timed on P. Could you add an old vs new run with the same QR and refresh interval?
-
The full-basis variants are supposed to give the same iterates, and in FP32 KL-Shampoo does match. SOAP and KL-SOAP come out a bit better with the new version even in FP32, which hints that something besides storing P differs. The custom QR also uses FP16 matrix products inside and zeros out off-diagonal entries for inactive directions, so it isn't quite the same computation. Since it isn't clear which QR the BF16 table uses, I can't tell whether the BF16 gain comes from storing P or from the QR. Does the gain hold with PyTorch QR on P?
-
All subspace and runtime experiments use KL-Shampoo only, while the paper sells the framework for SOAP and KL-SOAP too. I understand compute is limited, but SOAP also keeps per-coordinate second moments tied to the basis, so it's not obvious the same trade-off carries over. Do subspace updates give a similar time/loss trade-off for SOAP?
-
Part of the speedup comes from a QR kernel adapted from public GPU MODE submissions, and the subspace update is openly a block Jacobi variant, so keeping P around is the main new piece. Purifying Shampoo (Eschenhagen et al.) already forms QᵀSQ and uses how far it is from diagonal to decide when each factor's basis needs a refresh. That's not the same as picking which block to rotate, but it's close in spirit and the paper doesn't discuss it. It would help to discuss this and say whether the two ideas could be combined, for example using P to decide both when and where to refresh.
Questions for authors
Please see the weaknesses!
5 comments
Response to VKtF
- Part of the speedup comes from a QR kernel adapted from public GPU MODE submissions, and the subspace update is openly a block Jacobi variant, so keeping P around is the main new piece. Purifying Shampoo (Eschenhagen et al.) already forms QᵀSQ and uses how far it is from diagonal to decide when each factor's basis needs a refresh. [...] but it's close in spirit and the paper doesn't discuss it.
Thank you for highlighting this connection. We agree that Purifying Shampoo deserves explicit discussion and will clarify its relationship to our work. We would also like to distinguish our contributions from the existing components on which we build.
Our reparametrization was originally inspired by classical block Jacobi methods. Its contribution is not simply retaining an intermediate matrix: we maintain directly as optimizer state, updating it consistently as both the factor and basis evolve, and develop structure-aware QR and dynamically selected subspace updates within this maintained representation. This provides a unified framework without repeatedly reconstructing or storing both and .
1. Relation to block Jacobi methods: subspace QR for evolving factors. Classical block Jacobi eigensolvers repeatedly apply subspace eigendecomposition to a fixed matrix, seeking to diagonalize it through successive local updates. In contrast, we use a single subspace QR step per basis refresh for continually evolving Shampoo-based factors, without an inner loop to diagonalize the current factor. The distinction concerns both the local computation, QR rather than eigendecomposition, and the objective, tracking an evolving factor rather than solving a fixed eigenproblem. Maintaining provides direct access to the blocks needed for dynamically selected updates. The resulting local rotations preserve full-basis orthogonality and maintain a consistent factor representation. Our goal is to improve preconditioned training directions at lower cost, rather than fully diagonalize each changing factor.
2. Differences from generic GPU MODE implementations: structure-aware QR. We explicitly build on and credit the public GPU MODE submissions. The generic QR implementations we adapt were optimized for NVIDIA B200 GPUs and do not explicitly exploit symmetric positive-definite (SPD) input structure. We make them structure-aware and extend their scope to earlier-generation GPUs, including NVIDIA A100 and H100. Unlike the generally nonsymmetric input , the stored directly exposes SPD and potentially near-diagonal structure. Our implementation exploits these properties to select and execute cheaper factorization paths, with numerical checks and fallbacks for unsuitable inputs.
The new ablation in our general response shows the reparametrized method with our structure-aware QR is faster than the original parametrization with our adaptation of generic GPU MODE QR. Thus, the measured QR acceleration is not simply due to adopting an optimized generic kernel: it comes from combining the maintained representation with a solver that exploits its structure.
Response to VKtF
3. Differences from Purifying Shampoo: maintained state and specialized QR updates. Eschenhagen et al. form and use its off-diagonal content to determine when individual factors require a basis refresh; their warm-started QR procedure also operates on this transformed matrix. Their emphasis is on controlling approximation error through adaptive full-basis refreshes, and the same criterion can terminate a multistep QR inner loop. Their main experiments primarily use adaptive eigendecomposition.
Our distinction lies in both the maintained representation and the structure-aware update algorithms developed around it. We maintain as the evolving state through which factor estimation and basis updates are performed, and design QR to exploit its SPD and potentially near-diagonal structure. This goes beyond keeping available for refresh decisions: the representation and specialized solver together reduce factorization cost, while dynamic subspace selection reduces the decomposition size.
We also focus on a single QR step per refresh, whereas their QR variant allows multiple steps to meet an approximation-error tolerance. Consistent with SOAP, our experiments support the effectiveness of a single step as a lower-cost alternative to multistep QR or eigendecomposition. Their cited AlgoPerf implementation stores the factors in FP32, and the paper does not report a BFP16 factor-storage evaluation. We additionally evaluate BFP16 storage of both and , addressing memory efficiency and reduced-precision robustness alongside computational cost.
It would help to discuss this and say whether the two ideas could be combined, for example using P to decide both when and where to refresh.
The approaches can naturally be combined to decide both when and where to refresh. Their relative off-diagonal criterion could be evaluated directly from the stored to determine whether a refresh is needed, after which our block-selection strategy would choose the subspace and our QR implementation would compute the update. Maintaining avoids reconstructing for these decisions. A single subspace step need not satisfy their full-basis stopping tolerance, so the combined policy’s accuracy–runtime trade-off remains to be evaluated. We will add this discussion, clarify the block-Jacobi motivation of our approach, and distinguish this compatible extension from our evaluated results.
Response to VKtF
We thank the reviewer for th detailed and constructive feedback. We address each point below.
We have added Figure 6 to compare the end-to-end wall-clock runtimes of the original and reparametrized KL-Shampoo methods. It covers three BFP16 configurations: the original parametrization with standard PyTorch QR, the reparametrization with standard PyTorch QR, and the reparametrization with our customized QR. With standard PyTorch QR, the reparametrization is more robust under BFP16, while the wall-clock acceleration comes from combining the reparametrization with our customized QR. In terms of steps, our customized QR does not differ from standard PyTorch QR, as expected.
This end-to-end result is consistent with the per-call QR timings in our general response: standard PyTorch QR takes essentially the same time on SQ and P, whereas our customized QR on P is substantially faster.
Please see Figure 6 for the original-versus-reparametrized training comparison, where both parametrizations use standard PyTorch QR and the same refresh interval, and our General Response to All Reviewers: Standard-QR ablation and attribution of speedups for the comparison using the same standard PyTorch QR routine. The general response also separates the gains from generic QR optimization from those obtained by exploiting the structure exposed by the new parametrization.
Yes. Table 2 in Appendix F uses standard PyTorch QR for both parametrizations. We have updated its caption to make this explicit. Thus, the improved robustness under BFP16 storage is observed without our customized QR implementation.
For completeness, we have also added customized-QR results for SOAP:
SOAP on nanoGPT (123M), with update interval T=10:
Arrow labels show the difference between the BFP16 and FP32 means; positive values indicate increased loss.
The new parametrization reduces the loss increase under BFP16 storage with both solvers: from 0.013 to 0.005 with PyTorch QR, and from 0.014 to 0.005 with customized QR.
We have not completed the corresponding customized-QR precision ablations for KL-Shampoo and KL-SOAP within the rebuttal compute budget. We will extend Table 2 when those results become available; the existing standard-QR results already address whether the numerical benefit depends on our customized solver. We note that Figure 6 also already displays empiracle robustness to BF16 training for KL-Shampoo on Llama3 (598M) when comparing parameterizations, we will update the table when we complete corresponding FP32 training runs.
To address this directly, we have added SOAP subspace-update experiments on nanoGPT, rather than relying solely on the KL-Shampoo results.
Entries report loss, with runtime per step in parentheses.
SOAP also exhibits a controllable runtime–loss trade-off: smaller blocks and longer update intervals reduce runtime, with modest changes in loss across the tested settings. For example, moving from B=1/2, T=3 to B=1/3, T=5 reduces the reported runtime from 661.51 ms to 600.39 ms, while loss changes from 3.252 to 3.257. These experiments directly evaluate the trade-off in SOAP, including its coordinatewise second-moment preconditioning, rather than extrapolating from KL-Shampoo.