acceptodds
Under review as a conference paper at ICLR 2027

Reparametrizing Shampoo and SOAP

Abstract

Shampoo-based methods, such as KL-Shampoo and SOAP, have demonstrated strong performance in neural network training, but their QR-based updates become costly for large preconditioning matrices. To address this, we propose a preconditioner reparametrization that accelerates full-basis QR computation and enables faster partial basis updates. By maintaining preconditioning factors in the current orthogonal basis, our approach exposes useful input structure to facilitate QR decomposition and allows dynamically selected basis vectors to be cheaply updated via subspace QR while preserving the full-basis orthogonality. To further accelerate QR computation, we provide a customized implementation supporting mixed-precision computation and batched execution. The resulting framework unifies full-basis and subspace updates and applies broadly to QR-based updates, including KL-Shampoo, SOAP, and KL-SOAP. Empirically, our approach reduces computational overhead and improves the training performance of SOAP-type methods in mixed precision, enabling KL-SOAP to match or exceed KL-Shampoo. Overall, our approach improves the time efficiency of Shampoo-based methods.

Primary area:
optimization
Keywords:
optimization, deep learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VKtFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper store each Shampoo factor in the current basis, P = QᵀSQ, instead of storing S itself. P is symmetric and close to diagonal, so the QR used to refresh the basis gets cheaper, and one can refresh only a block of basis vectors while the whole basis stays orthogonal. Together with a custom mixed-precision QR, the paper reports much faster basis updates for KL-Shampoo and better BF16 test loss, mostly for SOAP and KL-SOAP.

Critical strengths

Storing the factor in its own basis is a simple ideas that drops easily into existing Shampoo-type code, and the full-basis update is mathematically the same as before. The subspace update keeps the whole basis orthogonal without any extra re-orthogonalization, which is neat. I also liked that better BF16 behavior comes out as a side effect, and the runtime study shows the loss/time trade-off over refresh interval and subspace size clearly, with tuned Muon as a reference. Overall, it’s a good paper.

Critical weaknesses
  1. The speed story is mostly told through a per-step timing breakdown on one 8000×8000 matrix, which is bigger than any weight in the trained models. The end-to-end runtime plots only compare the new variants with each other and with Muon, so we never see how much wall-clock time the reparametrization saves over the original KL-Shampoo or SOAP. The paper also says several times that the gain holds with plain PyTorch QR, but PyTorch QR is never timed on P. Could you add an old vs new run with the same QR and refresh interval?

  2. The full-basis variants are supposed to give the same iterates, and in FP32 KL-Shampoo does match. SOAP and KL-SOAP come out a bit better with the new version even in FP32, which hints that something besides storing P differs. The custom QR also uses FP16 matrix products inside and zeros out off-diagonal entries for inactive directions, so it isn't quite the same computation. Since it isn't clear which QR the BF16 table uses, I can't tell whether the BF16 gain comes from storing P or from the QR. Does the gain hold with PyTorch QR on P?

  3. All subspace and runtime experiments use KL-Shampoo only, while the paper sells the framework for SOAP and KL-SOAP too. I understand compute is limited, but SOAP also keeps per-coordinate second moments tied to the basis, so it's not obvious the same trade-off carries over. Do subspace updates give a similar time/loss trade-off for SOAP?

  4. Part of the speedup comes from a QR kernel adapted from public GPU MODE submissions, and the subspace update is openly a block Jacobi variant, so keeping P around is the main new piece. Purifying Shampoo (Eschenhagen et al.) already forms QᵀSQ and uses how far it is from diagonal to decide when each factor's basis needs a refresh. That's not the same as picking which block to rotate, but it's close in spirit and the paper doesn't discuss it. It would help to discuss this and say whether the two ideas could be combined, for example using P to decide both when and where to refresh.

Questions for authors

Please see the weaknesses!

5 comments
AuthorsOpenReview ↗

Response to VKtF

We thank the reviewer for th detailed and constructive feedback. We address each point below.

1.1 The end-to-end runtime plots only compare the new variants with each other and with Muon, so we never see how much wall-clock time the reparametrization saves over the original KL-Shampoo or SOAP.

We have added Figure 6 to compare the end-to-end wall-clock runtimes of the original and reparametrized KL-Shampoo methods. It covers three BFP16 configurations: the original parametrization with standard PyTorch QR, the reparametrization with standard PyTorch QR, and the reparametrization with our customized QR. With standard PyTorch QR, the reparametrization is more robust under BFP16, while the wall-clock acceleration comes from combining the reparametrization with our customized QR. In terms of steps, our customized QR does not differ from standard PyTorch QR, as expected.

This end-to-end result is consistent with the per-call QR timings in our general response: standard PyTorch QR takes essentially the same time on SQSQ and PP, whereas our customized QR on PP is substantially faster.

1.2 Could you add an old vs new run with the same QR and refresh interval?

Please see Figure 6 for the original-versus-reparametrized training comparison, where both parametrizations use standard PyTorch QR and the same refresh interval, and our General Response to All Reviewers: Standard-QR ablation and attribution of speedups for the comparison using the same standard PyTorch QR routine. The general response also separates the gains from generic QR optimization from those obtained by exploiting the structure exposed by the new parametrization.

  1. Since it isn't clear which QR the BF16 table uses, I can't tell whether the BF16 gain comes from storing P or from the QR. Does the gain hold with PyTorch QR on P?

Yes. Table 2 in Appendix F uses standard PyTorch QR for both parametrizations. We have updated its caption to make this explicit. Thus, the improved robustness under BFP16 storage is observed without our customized QR implementation.

For completeness, we have also added customized-QR results for SOAP:

SOAP on nanoGPT (123M), with update interval T=10T=10:

ParametrizationPyTorch QR: FP32 → BFP16Customized QR: FP32 → BFP16
Original (old)3.248→+0.0133.2613.248 \xrightarrow{+0.013} 3.2613.248→+0.0143.2623.248 \xrightarrow{+0.014} 3.262
Reparametrized (new)3.247→+0.0053.2523.247 \xrightarrow{+0.005} 3.2523.248→+0.0053.2533.248 \xrightarrow{+0.005} 3.253

Arrow labels show the difference between the BFP16 and FP32 means; positive values indicate increased loss.

The new parametrization reduces the loss increase under BFP16 storage with both solvers: from 0.013 to 0.005 with PyTorch QR, and from 0.014 to 0.005 with customized QR.

We have not completed the corresponding customized-QR precision ablations for KL-Shampoo and KL-SOAP within the rebuttal compute budget. We will extend Table 2 when those results become available; the existing standard-QR results already address whether the numerical benefit depends on our customized solver. We note that Figure 6 also already displays empiracle robustness to BF16 training for KL-Shampoo on Llama3 (598M) when comparing parameterizations, we will update the table when we complete corresponding FP32 training runs.

  1. All subspace and runtime experiments use KL-Shampoo only, while the paper sells the framework for SOAP and KL-SOAP too. I understand compute is limited, but SOAP also keeps per-coordinate second moments tied to the basis, so it's not obvious the same trade-off carries over. Do subspace updates give a similar time/loss trade-off for SOAP?

To address this directly, we have added SOAP subspace-update experiments on nanoGPT, rather than relying solely on the KL-Shampoo results.

Block size\Update interval\text{Block size}\backslash\text{Update interval}T=3T=3T=4T=4T=5T=5
B=1/2B=1/23.252 (661.51 ms)3.253 (633.25 ms)3.254 (616.17 ms)
B=1/3B=1/33.253 (637.31 ms)3.254 (614.48 ms)3.257 (600.39 ms)

Entries report loss, with runtime per step in parentheses.

SOAP also exhibits a controllable runtime–loss trade-off: smaller blocks and longer update intervals reduce runtime, with modest changes in loss across the tested settings. For example, moving from B=1/2, T=3B=1/2,\ T=3 to B=1/3, T=5B=1/3,\ T=5 reduces the reported runtime from 661.51 ms to 600.39 ms, while loss changes from 3.252 to 3.257. These experiments directly evaluate the trade-off in SOAP, including its coordinatewise second-moment preconditioning, rather than extrapolating from KL-Shampoo.

AuthorsOpenReview ↗

Response to VKtF

  1. Part of the speedup comes from a QR kernel adapted from public GPU MODE submissions, and the subspace update is openly a block Jacobi variant, so keeping P around is the main new piece. Purifying Shampoo (Eschenhagen et al.) already forms QᵀSQ and uses how far it is from diagonal to decide when each factor's basis needs a refresh. [...] but it's close in spirit and the paper doesn't discuss it.

Thank you for highlighting this connection. We agree that Purifying Shampoo deserves explicit discussion and will clarify its relationship to our work. We would also like to distinguish our contributions from the existing components on which we build.

Our reparametrization was originally inspired by classical block Jacobi methods. Its contribution is not simply retaining an intermediate matrix: we maintain P=Q⊤SQP=Q^\top SQ directly as optimizer state, updating it consistently as both the factor and basis evolve, and develop structure-aware QR and dynamically selected subspace updates within this maintained representation. This provides a unified framework without repeatedly reconstructing PP or storing both SS and PP.

1. Relation to block Jacobi methods: subspace QR for evolving factors. Classical block Jacobi eigensolvers repeatedly apply subspace eigendecomposition to a fixed matrix, seeking to diagonalize it through successive local updates. In contrast, we use a single subspace QR step per basis refresh for continually evolving Shampoo-based factors, without an inner loop to diagonalize the current factor. The distinction concerns both the local computation, QR rather than eigendecomposition, and the objective, tracking an evolving factor rather than solving a fixed eigenproblem. Maintaining PP provides direct access to the blocks needed for dynamically selected updates. The resulting local rotations preserve full-basis orthogonality and maintain a consistent factor representation. Our goal is to improve preconditioned training directions at lower cost, rather than fully diagonalize each changing factor.

2. Differences from generic GPU MODE implementations: structure-aware QR. We explicitly build on and credit the public GPU MODE submissions. The generic QR implementations we adapt were optimized for NVIDIA B200 GPUs and do not explicitly exploit symmetric positive-definite (SPD) input structure. We make them structure-aware and extend their scope to earlier-generation GPUs, including NVIDIA A100 and H100. Unlike the generally nonsymmetric input SQSQ, the stored PP directly exposes SPD and potentially near-diagonal structure. Our implementation exploits these properties to select and execute cheaper factorization paths, with numerical checks and fallbacks for unsuitable inputs.

The new ablation in our general response shows the reparametrized method with our structure-aware QR is faster than the original parametrization with our adaptation of generic GPU MODE QR. Thus, the measured QR acceleration is not simply due to adopting an optimized generic kernel: it comes from combining the maintained representation with a solver that exploits its structure.

AuthorsOpenReview ↗

Response to VKtF

3. Differences from Purifying Shampoo: maintained state and specialized QR updates. Eschenhagen et al. form Q⊤SQQ^\top SQ and use its off-diagonal content to determine when individual factors require a basis refresh; their warm-started QR procedure also operates on this transformed matrix. Their emphasis is on controlling approximation error through adaptive full-basis refreshes, and the same criterion can terminate a multistep QR inner loop. Their main experiments primarily use adaptive eigendecomposition.

Our distinction lies in both the maintained representation and the structure-aware update algorithms developed around it. We maintain PP as the evolving state through which factor estimation and basis updates are performed, and design QR to exploit its SPD and potentially near-diagonal structure. This goes beyond keeping PP available for refresh decisions: the representation and specialized solver together reduce factorization cost, while dynamic subspace selection reduces the decomposition size.

We also focus on a single QR step per refresh, whereas their QR variant allows multiple steps to meet an approximation-error tolerance. Consistent with SOAP, our experiments support the effectiveness of a single step as a lower-cost alternative to multistep QR or eigendecomposition. Their cited AlgoPerf implementation stores the factors in FP32, and the paper does not report a BFP16 factor-storage evaluation. We additionally evaluate BFP16 storage of both PP and QQ, addressing memory efficiency and reduced-precision robustness alongside computational cost.

It would help to discuss this and say whether the two ideas could be combined, for example using P to decide both when and where to refresh.

The approaches can naturally be combined to decide both when and where to refresh. Their relative off-diagonal criterion could be evaluated directly from the stored PP to determine whether a refresh is needed, after which our block-selection strategy would choose the subspace and our QR implementation would compute the update. Maintaining PP avoids reconstructing Q⊤SQQ^\top SQ for these decisions. A single subspace step need not satisfy their full-basis stopping tolerance, so the combined policy’s accuracy–runtime trade-off remains to be evaluated. We will add this discussion, clarify the block-Jacobi motivation of our approach, and distinguish this compatible extension from our evaluated results.

Reviewer_Qqk7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose to use reparametrization to accelerate QR computation in Shampoo and SOAP. They also make some improvements in practical implementation to further improve the efficiency. The experiments show actual speedup in test loss reduction.

Critical strengths
  1. Overall the idea is reasonable and being clearly represented to the readers.
  2. The method sees consistent gains across different settings.
Critical weaknesses
  1. Some might argue the improvement is somehow engineering and the topic and the significance seems not as common in the field of machine learning.
Questions for authors

While it is nice to see end to end gains, it is nice to see speed up breakdown for different types/shapes of matrices. It is also would be nice to see breakdown of the speedup into faster steps and potentially less steps due to enhanced numerical precision. Some metrics to show the enhanced numerical precision will also be desired if that is a significant phenomenon. Do the authors have any materials related to these?

2 comments
AuthorsOpenReview ↗

Response to Qqk7

We thank the reviewer for the positive assessment of our work. We address each point below.

  1. Some might argue the improvement is somehow engineering and the topic and the significance seems not as common in the field of machine learning.

Our work is algorithm–system co-design: the reparametrization changes how optimizer state is maintained and updated, exposing structure that enables more efficient computation. The contribution therefore goes beyond optimizing an existing kernel. Muon’s use of Newton–Schulz iterations rather than an explicit SVD illustrates a similar principle: the numerical formulation is integral to an optimizer’s practical efficiency. Our work applies this principle to Shampoo- and SOAP-type methods through a maintained representation that supports structure-aware QR and efficient subspace updates.

Algorithmic contributions beyond the implementation. Our reparametrization maintains P=Q⊤SQP=Q^\top SQ directly as optimizer state, supporting full-basis updates and dynamically selected subspace updates. The latter reduce the decomposition size while preserving full-basis orthogonality. These formulations work with standard QR routines and do not require our customized solver. However, a solver-independent formulation does not guarantee a solver-independent speedup: our new ablation shows no meaningful acceleration of standard PyTorch QR calls from changing the representation alone in the tested setting. We have corrected our earlier claim accordingly.

The measured QR speedup comes from structural specialization. Our structure-aware QR, adapted from the public GPU MODE implementations, exploits the symmetric positive-definite and potentially near-diagonal structure of PP to select and execute cheaper factorization paths. Maintaining PP makes this structure directly available without repeated reconstruction. The reparametrized method with structure-aware QR outperforms the original parametrization with our adaptation of generic GPU MODE QR, not merely a baseline using PyTorch QR. Thus, the gain is not simply attributable to adopting an optimized generic kernel: it comes from combining the maintained representation with a numerical algorithm that exploits its structure.

Relevance to machine learning. We address the computational cost of training with Shampoo-based optimizers, not decomposition speed in isolation. Basis updates are a practical bottleneck in our evaluated workloads; reducing their cost makes these methods more accessible for large-scale training. Our objective is to compute useful preconditioned training directions efficiently, rather than fully diagonalize each evolving factor. The reparametrization also improves robustness under BFP16 storage, including with standard PyTorch QR, as discussed below. This numerical benefit is distinct from the solver-dependent runtime gains. Together, these contributions address both computational efficiency and reduced-precision robustness—two practical concerns in large-scale ML optimization.

While it is nice to see end to end gains, it is nice to see speed up breakdown for different types/shapes of matrices. It is also would be nice to see breakdown of the speedup into faster steps and potentially less steps due to enhanced numerical precision. Some metrics to show the enhanced numerical precision will also be desired if that is a significant phenomenon. Do the authors have any materials related to these?

Runtime breakdown. Please see our General Response to All Reviewers: Standard-QR ablation and attribution of speedups, which compares solvers across five matrix sizes, including four used in our model experiments. These measurements report milliseconds per solver call and therefore isolate faster computation from any potential reduction in training steps.

Reduced-precision robustness. Please see Table 2 in Appendix F and the SOAP comparison in the general response. We measure the increase in mean loss when switching from FP32 to BFP16 storage. For example, for SOAP on nanoGPT with standard PyTorch QR, this increase falls from 0.013 under the original parametrization to 0.005 under the new parametrization. A similar improvement is observed with customized QR. These results support improved robustness to reduced-precision storage.

Overall, I am satisfied with the authors' response. I might consider raising my score during the later AC-reviewer discussion phase.

The four-point rating scale is kind of the problem here. I am willing to support the acceptance of the paper (like raising to a virtual 3.5 rating), but due to the paper's scope (a solid improvement in a small focused scope), direction (the engineering-ish nature), and some part of my unfamiliarity, I am not comfortable recommending any reward beyond that.

Reviewer_EoZYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a preconditioner reparametrization that enable cheaper basis updates in QR computation in Shampoo-based optimization methods. While maintaining preconditioning factors in the current orthogonal basis, the proposed approach exposes useful input structure to facilitate QR decomposition, allowing dynamically selected basis vectors via QR. Empirically, it reduces computational overhead and improves the training performance of SOAP-typy methods in mixed precision.

Critical strengths
  1. Clear reformulation and theretical framework. The paper is easy to follow and related studies is well conducted. It establishes the equivalence of the original and reprarametrized full-basis QR updates in exact arithmetic. It enables selected basis vectors to be updated in stead of full-basis, providing a useful and practical framework for reducing overhead.

  2. Empirical Evidance. Extensive experiments and ablations are conducted to compare the test loss and wall-clock time, together with subspace size, update frequency, and selection strategy.

Critical weaknesses
  1. Missing ablation. The paper claims that reparametrization alone accelerates basis updates in standard PyTorch QR (lines 85–86 and 165–166). However, the computation of reparametrized method is not reported using standard QR. Including this comparison would help verify the claim independently of the customized QR implementation.
Questions for authors
  1. What precision is used to store the basis, preconditioning factors, and momentum?
3 comments
AuthorsOpenReview ↗

Response to EoZY

We thank the reviewer for the positive assessment of our work. We address each point below.

However, the computation of reparametrized method is not reported using standard QR. Including this comparison would help verify the claim independently of the customized QR implementation.

Please see our General Response to All Reviewers: Standard-QR ablation and attribution of speedups for the added comparison and the corresponding correction to our runtime claim. We provide results of this ablation in Appendix F, Figure 6 and Appendix F, Table 3.

  1. What precision is used to store the basis, preconditioning factors, and momentum?

We store the orthogonal bases, preconditioning factors, and momentum in BFP16. As shown in Table 2 (Appendix F), our reparametrization improves empirical robustness to this reduced-precision storage compared with the original parametrization.

Dear Reviewer EoZY,

We thank you for your initial review and hope our response has addressed any questions and concerns you had. We would appreciate any feedback on our revisions or further questions if you have them as the discussion period comes to a close.

Thank you for your response and the additional experiments. I think it is worth accepting. In the later AC discussion phase, I will consider whether to keep or raise my score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers: Standard-QR ablation and attribution of speedups

We thank the reviewers for identifying this missing ablation. We have added an original-versus-reparametrized comparison using standard PyTorch QR (torch.linalg.qr) to distinguish the effect of the representation from that of the QR implementation.

Runtime comparison across solvers and matrix shapes (also see Table 3, Appendix F of the paper)

Solver\Shape\text{Solver}\backslash\text{Shape}8000×80008000 \times 80005760×57605760 \times 5760 (llama3-big)4032×40324032 \times 4032 (llama3-mid)3072×30723072 \times 3072 (nanoGPT)2304×23042304 \times 2304 (llama3-small)
torch_eigen(S) (Original param.)581.2253.697.056.534.3
torch_qr(SQ) (Original param.)104.1 (1x)56.3 (1x)31.3 (1x)20.4 (1x)13.3 (1x)
customized_qr(SQ) (Our adaptation of GPU MODE QR)32.5 (3.2x)20.1 (2.8x)11.4 (2.7x)7.2 (2.8x)5.0 (2.7x)
torch_qr(P) (New param.)104.0 (1x)56.4 (1x)31.2 (1x)20.3 (1x)13.2 (1x)
customized_qr(P) (Our structure-aware QR)17.3 (6.0x)10.8 (5.2x)5.6 (5.6x)4.5 (4.5x)3.2 (4.2x)

Runtime is reported in milliseconds per solver call on the same machine with H100 (HBM3). Matrix shapes apply to SS, PP, and QQ. Speedups in parentheses are relative to torch_qr(SQ) at the same size. As suggested by the reviewers, we include four additional matrix sizes used in our experiments.

The ablation shows that the full-basis reparametrization alone does not provide a meaningful speedup with standard PyTorch QR across the tested sizes. We acknowledge that our earlier claim suggested otherwise and have corrected it in the paper. We emphasize that this overstatement does not impact any of our experimental results, rather we refine our claim that reparameterization improves BFP16 training robustness and exposes useful structure that our QR implementation exploits for significant acceleration.

A substantial speedup is obtained by combining the reparametrization with our structure-aware QR. Importantly, this combination also outperforms the original parametrization paired with our adaptation of generic GPU MODE QR. Our implementation builds on the public GPU MODE submissions but additionally exploits the symmetric positive-definite and potentially near-diagonal structure exposed by the maintained P=Q⊤SQP=Q^\top SQ. The generic implementations we adapt do not explicitly exploit these properties. Thus, the gains are not simply attributable to replacing PyTorch QR with an optimized generic kernel: the reparametrization makes the structure directly available without repeated reconstruction, and the specialized solver exploits it to select and execute cheaper factorization paths. Exposing structure does not by itself guarantee faster computation when the solver treats the input as a general dense matrix.

Numerical robustness under BFP16 storage

The reparametrization also improves robustness to BFP16 storage, including when standard PyTorch QR is used (see the comparison below and Table 2 in Appendix F). This benefit is reflected in the smaller increase in loss when moving from FP32 to BFP16 and is distinct from the solver-dependent runtime gains.

SOAP on nanoGPT (123M), with update interval T=10T=10:

ParametrizationPyTorch QR: FP32 → BFP16Customized QR: FP32 → BFP16
Original (old)3.248→+0.0133.2613.248 \xrightarrow{+0.013} 3.2613.248→+0.0143.2623.248 \xrightarrow{+0.014} 3.262
Reparametrized (new)3.247→+0.0053.2523.247 \xrightarrow{+0.005} 3.2523.248→+0.0053.2533.248 \xrightarrow{+0.005} 3.253

Arrow labels indicate the difference between the BFP16 and FP32 means; positive values indicate increased loss.

The correction to our standard-QR runtime claim does not alter our algorithmic contributions. The persistent-state reparametrization, its exact-arithmetic equivalence to the conventional full-basis QR update, and the dynamically selected subspace updates that preserve full-basis orthogonality remain unchanged. These formulations are independent of the particular QR solver; their mathematical validity is distinct from the runtime benefits realized by a specific implementation.

Our main empirical conclusions for the combined method also remain unchanged. Most of our experiments use the customized, structure-aware QR solver, and the reported results for that configuration are unaffected. We have revised the paper to distinguish three points: the solver-independent algorithmic formulation, the improved robustness under BFP16 storage, and the measured full-basis QR acceleration from combining the reparametrization with structure-aware QR. We no longer attribute the latter to the representation alone.

We appreciate the reviewers’ suggestions, which have helped us identify and communicate the source of the computational gains more precisely.

AuthorsOpenReview ↗

Changes in the revised submission

All main changes in the revised paper are highlighted in blue.

  • Main text

    • Corrected our claim of a full-basis speedup with standard PyTorch QR and clarified that the measured QR acceleration comes from combining the reparametrization with structure-aware QR (lines 89–93 and 172–173; all reviewers).
  • Appendix

    • Added a table (Table 3, Appendix F) comparing runtimes across solvers and matrix shapes (all reviewers).
    • Updated the caption of Table 2 in Appendix F to clarify that the results were obtained using standard PyTorch QR (Reviewer VKtF).
    • Added Figure 6 (Appendix F) to compare the wall-clock runtimes of the original and reparametrized KL-Shampoo methods on llama3-big (Reviewer EoZY, VKtF).
AuthorsOpenReview ↗

Post-Rebuttal Summary by Authors

Dear AC and Reviewers,

We thank the reviewers for their positive assessment of our work and hope we have addressed their concerns in our rebuttal. To summarize our contributions in the rebuttal, we did the following:

  • Refined our claim on where we accelerate training, showing that a combination of the strengthened numerical robustness and exposure of useful structure that our QR solver exploits.
  • Provided additional ablations to isolate the impact of our reparameterization alone by comparing the combinations of parametrizations, torch.linalg.qr\texttt{torch.linalg.qr}, and our custom solver.
  • Added Figure 6, including full training runs on Llama3 598M which compare the standard parametrization of KL-Shampoo, our reparameterization with torch.linalg.qr\texttt{torch.linalg.qr}, and our reparameterization with custom QR.
  • Provided further empirical evidence to support the generality of contributions with reparameterized SOAP.
  • Clarified the precision we train and store optimizer state in.
  • Discussed connections to relevant literature such as Eschenhagen et al. (2025) and how their approach could complement ours.

We note that due to the new ICLR four-point scale, some reviewers may be unsure about changing their score. We do, however, hope that our contributions during the discussion period have helped reviewers address any concerns they may have had and that their commentary after the Reviewer-Author discussion period selects this.

We note to the AC that all reviewers have recommended acceptance both in their initial review and their comments post-rebuttal, and we will further address concerns they pointed out in a final revision should the work be accepted.

We thank the reviewers and AC for their time and effort assessing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.