acceptodds
Under review as a conference paper at ICLR 2027

Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon

Abstract

Memory-efficient subspace optimization has emerged as a practical approach to reducing the memory cost of LLM training while preserving optimization performance. Previous algorithms typically rely on the leading singular subspace of stochastic gradients, yet the gradient noise may dominate the subspace in later training stages and can prevent convergence. To overcome this limitation, we adopt a residual correction framework, which maintains the optimizer states in the subspace and adds the orthogonal residual gradient to the parameter update. We first integrate this scheme into momentum SGD to develop SMRC, which attains an convergence rate for a fixed orthonormal basis, matching that of full-space momentum SGD. Our analysis introduces a subspace virtual iterate to resolve the coupling between the momentum and the residual correction. Furthermore, we extend SMRC to a multi-stage variant which periodically recomputes bases to track the evolving dominant subspace and retains the same rate. Since each basis refresh breaks the consistency of the virtual iterates, we devise a boundary correction approach that aligns them across stages. Beyond momentum SGD, we incorporate our framework into Muon and propose PMRC, which applies the matrix sign to the projected momentum. Through a weighted momentum tracking analysis that accounts for the residual gradient, we establish an convergence rate for PMRC, which matches the rate of full-space Muon. We also design a multi-stage PMRC with periodically refreshed bases, which maintains a convergence rate of the same order. Extensive experimental results confirm the effectiveness and efficiency of our algorithms.

Primary area:
optimization
Keywords:
Memory-efficient optimization, GaLore, Momentum SGD, Muon

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wrr1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors analyze how to reduce the memory usage in LLM training. At the same time, the authors want to preserve the convergence guarantees and practical performance. The authors present two algorithms, Subspace Momentum SGD with Residual Correction (SMRC) and Projected Muon with Residual Correction (PMRC). Both algorithms has the O(T−1/4)O(T^{-1/4}) convergence rate under the standard assumptions.

Critical strengths

First, in this paper, the authors give the theoretical proofs for the convergence guarantees, and the theoretical analysis only replies to the standard assumptions. The convergence of SMRC accommodates predictable, gradient-dependent subspaces, which addresses the limitations of Galore.

Second, the convergence analysis seems non-trivial, especially for the PMRC. It is more challenging since the matrix sign is not linear. Therefore, the authors develop a weighted momentum tracking analysis and control the interaction between nuclear-norm and Frobenius-norm.

Third, the algorithm is straightforward and easy to follow, and the paper presents substantial experimental results. In addition, the authors give a detailed discussion on related work in Appendix A. It makes it clear what the novelties are.

Critical weaknesses

First, I believe this paper gives sufficient and novel theoretical analysis, but the algorithmic procedure seems standard. For example, the residual correction was analyzed in prior works. Fira combines subspace optimization with residual gradient correction, and Nguyen & Nguyen (2025) analyzes the projected momentum with SGD updates. Therefore, it seems that making the momentum in a subspace while applying SGD outside that subspace is related to prior works.

Second, I believe that there is a gap between the theoretical result and the experiment. The practical implementation uses γ^t\widehat{\gamma}_t, which is not shown in the theoretical PMRC algorithm. The paper shows the convergence of PMRC, but it is unknown whether or not the analysis can be applied to the practical implementation.

Questions for authors

Please see weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer wrr1

Many thanks for your constructive review and positive assessment of our paper. We have uploaded our revised paper, with the changes highlighted in blue for ease of reference.


W1. I believe this paper gives sufficient and novel theoretical analysis, but the algorithmic procedure seems standard.

Response. We thank the reviewer for recognizing our theoretical contributions. Although residual correction has been investigated in prior work, our study advances both algorithmic design and theoretical analysis, with the key distinctions outlined below.

  • Fira incorporates residual correction into GaLore to recover the discarded gradient information. Fira is built upon AdamW and employs a heuristically designed approach. More importantly, it does not establish convergence guarantees. In contrast, our work provides rigorous convergence guarantees for both momentum SGD and Muon with residual correction.

  • Although Nguyen & Nguyen (2025) also combine projected momentum SGD with residual correction, we identify a mathematical error in their convergence analysis. Specifically, the proof of Lemma D.1 omits a nonnegative term from an upper bound, which invalidates a key step of their proof. Consequently, their claimed convergence guarantee is not established by the existing analysis, as detailed in Appendix A.1. Moreover, their analysis is restricted to a fixed projection basis and only considers an unweighted residual update. In contrast, we establish convergence guarantees for multi-stage SMRC with periodically refreshed subspaces by developing a boundary correction technique that ensures the consistency of virtual iterates across stage transitions.

  • We further extend the residual correction framework to Muon and propose PMRC, which applies the matrix sign to the projected momentum. The nonlinear dependence of the update prevents an application of the virtual-iterate analysis developed for SMRC. To address this challenge, we develop a weighted momentum tracking analysis that accounts for the interaction between nuclear-norm and Frobenius-norm geometries. Based on this analysis, we establish convergence guarantees for both single-stage and multi-stage PMRC under a mixed stationarity measure.


W2. There is a gap between the theoretical result and the experiment.

Response. We thank the reviewer for pointing out this issue. We acknowledge that there is a minor difference between the theoretical analysis and the practical implementation. To address this concern, we extend the analysis of Theorem 4 to accommodate the residual cap, with the detailed proof provided in Appendix E.2.1 (page 31). Under Assumptions 1–4, with a fixed ρ\rho and the parameter settings in Theorem 4, we can ensure

1N∑k=1K∑t=1TE[∣Pk⊤∇F(Wk,t)∣∗+∣Qk∇F(Wk,t)∣F]≤O(N−1/4)+12γN∑k=1K∑t=1TE[(γ−γ^k,t)2∣QkGk,t∣F2]\frac{1}{N}\sum_ {k=1}^{K}\sum_ {t=1}^{T} \mathbb{E}\left[\\|P_ k^\top\nabla F(W_ {k,t})\\|_ * +\\|Q_ k\nabla F(W_ {k,t})\\|_ F\right]\le O(N^{-1/4}) + \frac{1}{2\gamma N} \sum_ {k=1}^{K}\sum_ {t=1}^{T} \mathbb{E}\left[(\gamma-\widehat{\gamma}_ {k,t})^2 \\|Q_kG_{k,t}\\|_ F^2 \right].

The second term characterizes the effect of the residual cap. In particular, if 1N∑k=1K∑t=1TE[(γ−γ^k,t)2∣QkGk,t∣F2]=O(1)\frac{1}{N}\sum_ {k=1}^{K}\sum_ {t=1}^{T}\mathbb{E}[(\gamma-\widehat{\gamma}_ {k,t})^2\\|Q_kG_{k,t}\\|_ F^2]=O(1), the capped PMRC retains the original O(N−1/4)O(N^{-1/4}) convergence rate.

To further evaluate the effect of the residual cap, we conduct additional experiments to examine the cap activation frequency and the magnitude of the coefficient reduction during training, with the cap activation frequency calculated separately for each interval rather than over all training steps.

Training Steps (out of 9900)Cap Frequency (\\%)Coefficient Reduction (\\%)
1-200018.8628.62
2001-40002.734.83
4001-60002.053.43
6001-80002.043.25
8001-99002.043.26

These results further support the conclusion that the residual cap primarily improves training stability without substantially altering the original PMRC updates.


References.

[1] Fira: Can We Achieve Full-rank Training of LLMs under Low-rank Constraint? NeurIPS, 2025.

[2] Lean and Mean Adaptive Optimization via Subset-norm and Subspace-momentum with Convergence Guarantees. ICML, 2025.


We hope that our responses address your concerns and are happy to answer any further questions during the reviewer–author discussion period.

Reviewer_xBSLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies memory-efficient optimization for large language model training. The central idea is to maintain optimizer momentum within a low-dimensional subspace while applying an SGD-style correction using the gradient component in the orthogonal complement. The authors instantiate this framework in two algorithms, SMRC and PMRC that are low-rank SGD with momentum and muon with residual gradient corrections. Authors provide theoretical convergence guarantee for the proposed algorithms and in order to account the change of subspaces in the theory, they introduce virtual iterate.

Critical strengths

1- The paper correctly identifies that while the low-rank momentum and the residual components in each step are orthogonal, they remain coupled, as the residual update changes future iterates and therefore affects the gradients accumulated in projected momentum.

2- The authors also identify the need to incorporate the jumps when the subspace changes, and the virtual-iterate construction for SMRC is a useful analytical device, while existing theory did not provide this guarantee for periodically refreshed SVD subspaces.

3- The combination of nuclear-norm and Frobenius-norm in PMRC was non-trivial, and they analyzed this mixed-geometry in their theorems.

4- the provided rank, residual weight and residual cap ablations on PMRC are informative about the algorithmic design choices.

Critical weaknesses

1- The paper contribution is more theoretical, as the residual correction has been previously discussed in the literature, including Fira which is mentioned by the paper, and is not entirely new. Also the extension of residual correction to Muon is important in terms of the geometric mixture. However, the algorithmic novelty of SMRC appears incremental relative to Fira and Nguyen & Nguyen, and the framing should be around the paper’s main contributions.

2- A substantial portion of paper and theories are around SMRC and its multi-stage variants, however in the main experiments they are not evaluated, and remain limited to synthetic experiments in the appendix. It is especially useful to see the benefit of proposed boundary correction in the multi-stage SMRC in real pre-training and fine-tuning experiments.

3- The subspace is estimated using the top-rr singular vectors, and while the paper mentions GoLore, it does not directly discuss whether capturing the most dominant directions in their algorithmic setting (with the residual-correction) is still needed or not. An ablation for the subspace estimation method to compare SVD with other alternatives, specially random subspaces would be helpful in positioning the paper.

4- It would be helpful to see the pre-training result on larger models (i.e. 7B+) as the behavior might be different when the models scales, also including multiple-seed results at least for smaller models would further strengthen the performance claim.

Questions for authors

Please refer to weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer xBSL (Part 1)

Thanks for your interest in our work. We have uploaded our revised paper, with the changes highlighted in blue for ease of reference.


W1. The algorithmic novelty of SMRC appears incremental relative to Fira and Nguyen & Nguyen.

Response. We acknowledge that residual correction has been explored in prior works. However, we would like to emphasize that our work makes contributions to both algorithmic design and theoretical analysis.

  • Fira incorporates residual correction into GaLore to recover the discarded gradient information. Fira is built upon AdamW and employs a heuristically designed approach. More importantly, it does not establish convergence guarantees. In contrast, our work provides rigorous convergence guarantees for both momentum SGD and Muon with residual correction.
  • Although Nguyen & Nguyen (2025) also combine projected momentum SGD with residual correction, we identify a mathematical error in their convergence analysis. Specifically, the proof of Lemma D.1 omits a nonnegative term from an upper bound, which invalidates a key step of their proof. Consequently, their claimed convergence guarantee is not established by the existing analysis, as detailed in Appendix A.1. Moreover, their analysis is restricted to a fixed projection basis and only considers an unweighted residual update. In contrast, we establish convergence guarantees for multi-stage SMRC with periodically refreshed subspaces by developing a boundary correction technique that ensures the consistency of virtual iterates across stage transitions.
  • We further extend the residual correction framework to Muon and propose PMRC, which applies the matrix sign to the projected momentum. The nonlinear dependence of the update prevents an application of the virtual-iterate analysis developed for SMRC. To address this challenge, we develop a weighted momentum tracking analysis that accounts for the interaction between nuclear-norm and Frobenius-norm geometries. Based on this analysis, we establish convergence guarantees for both single-stage and multi-stage PMRC under a mixed stationarity measure.

W2. It is useful to see the benefit of proposed boundary correction in the multi-stage SMRC.

Response. We thank the reviewer for this valuable suggestion. We introduce boundary correction to address the challenge in the convergence analysis of multi-stage SMRC. Specifically, when the projection subspace is refreshed, the terminal virtual iterate of the previous stage does not coincide with the initial virtual iterate of the next stage. To resolve this, we develop a boundary correction that aligns the virtual iterates at stage transitions.

We agree that an empirical evaluation of boundary correction in LLM training would further strengthen our work. Accordingly, we conduct additional pretraining experiments on LLaMA-60M and LLaMA-150M using the C4 dataset, comparing multi-stage SMRC with and without boundary correction. Following [3], we use a batch size of 64, a sequence length of 512, a projection rank of r=256r=256 and a refresh interval of 100 for both methods. We report the corresponding training and validation losses.

ModelMethodTraining LossValidation Loss
LLaMA-60MMulti-stage SMRC w/o boundary correction3.5943.539
LLaMA-60MMulti-stage SMRC3.5253.524
LLaMA-150MMulti-stage SMRC w/o boundary correction3.3543.353
LLaMA-150MMulti-stage SMRC3.2283.231

To further evaluate the effect of boundary correction, we conduct additional SFT experiments on Qwen3-8B and report both prompt-level and instruction-level strict accuracy (\\%) on the IFEval benchmark.

MethodPrompt-levelInstruction-level
Multi-stage SMRC w/o boundary correction55.0865.47
Multi-stage SMRC56.0165.83

The results show that boundary correction provides improvements in practical performance.

AuthorsOpenReview ↗

Response to Reviewer xBSL (Part 2)


W3. An ablation for the subspace estimation method to compare SVD with other alternatives.

Response. We would like to clarify that our convergence guarantees do not require SVD-based projection and hold for any orthonormal basis. SVD-based projection can better capture the dominant gradient directions, which is beneficial for practical performance. To investigate the effect of different subspace update strategies, we conduct additional ablation experiments on both LLM pretraining and SFT tasks. First, we evaluate their performance on LLaMA-60M and LLaMA-150M pretraining tasks. The results are summarized below.

ModelLoss TypeSVDRandom Projection
LLaMA-60MTraining Loss3.4063.476
LLaMA-60MValidation Loss3.3803.447
LLaMA-150MTraining Loss3.1173.156
LLaMA-150MValidation Loss3.0953.166

We further compare these two subspace update strategies on SFT tasks using three models and report both prompt-level and instruction-level strict accuracy (\\%) on the IFEval benchmark.

ModelMetricSVDRandom Projection
Qwen3-8BPrompt-level59.8956.75
Qwen3-8BInstruction-level69.0667.03
LLaMA-3.1-8BPrompt-level37.5235.12
LLaMA-3.1-8BInstruction-level50.2447.24
OLMo-2-1124-7BPrompt-level27.3627.17
OLMo-2-1124-7BInstruction-level38.9738.49

The results show that SVD-based subspace updates consistently outperform random projection in both pretraining and SFT tasks, which is consistent with the ability of SVD to capture dominant gradient directions.


W4. It would be helpful to see the pre-training result on larger models.

Response. We thank the reviewer for this suggestion. We agree that larger-scale experiments would provide a more comprehensive assessment of the optimizer’s behavior. However, pretraining a 7B-size model with the compute-optimal token budget is computationally expensive. In experiments, we find that a single run with Muon requires more than 24 days on 16 NVIDIA H20 GPUs, limiting the accessibility of such experiments to the broader research community. Given the time constraints, we pretrain LLaMA-7B on 1B tokens, which already exceeds the training budget used in prior work [4]. We report the corresponding training and validation losses below.

MethodTraining LossValidation Loss
AdamW3.6273.649
Muon3.4323.452
GaLore4.6924.699
GoLore5.0755.081
Fira4.6604.667
GUM4.4024.413
PMRC4.3344.343

We also report the zero-shot accuracy (\\%) on seven benchmarks.

OptimizerARC-EARC-COBQAHellaSwagPIQASIQAWinoGrandeAvg.
AdamW28.5421.4214.0026.6656.5835.7750.5133.35
Muon33.5417.1512.8027.4859.9636.1350.9934.01
GaLore27.9019.9714.2025.6954.1934.5449.3332.26
GoLore26.9419.8013.4025.7353.1034.7550.9132.09
Fira27.9019.2813.8025.6853.6534.5449.8032.09
GUM28.4518.6913.2026.0554.1334.8549.7232.16
PMRC28.0719.7114.4026.3254.1934.7549.3332.40

The results show that PMRC achieves the lowest loss among all memory-efficient optimizers, demonstrating its effectiveness at larger model scales. We will include additional 7B pretraining experiments with larger token budgets in a future version.

AuthorsOpenReview ↗

Response to Reviewer xBSL (Part 3)

W5. Including multiple-seed results for smaller models.

Response. We conduct additional pretraining experiments on LLaMA-60M, LLaMA-150M and LLaMA-350M using four random seeds (42, 43, 44 and 45) to evaluate the robustness of our results. We report the mean and standard deviation of the training and validation losses and further details can be found in Section 6.3 in the revised paper (page 10).

Loss TypeSizeAdamWMuonGaLoreGoLoreFiraGUMPMRC
Training60M3.439 ± 0.0343.365 ± 0.0283.469 ± 0.0253.518 ± 0.0503.584 ± 0.0313.449 ± 0.0273.389 ± 0.026
Training150M3.129 ± 0.0323.061 ± 0.0263.191 ± 0.0243.195 ± 0.0323.159 ± 0.0243.141 ± 0.0263.089 ± 0.030
Training350M2.856 ± 0.0112.698 ± 0.0052.926 ± 0.0422.993 ± 0.0152.852 ± 0.0192.853 ± 0.0132.750 ± 0.014
Validation60M3.434 ± 0.0263.357 ± 0.0033.464 ± 0.0043.513 ± 0.0253.578 ± 0.0023.435 ± 0.0083.384 ± 0.005
Validation150M3.130 ± 0.0063.064 ± 0.0023.204 ± 0.0043.203 ± 0.0033.162 ± 0.0013.154 ± 0.0023.092 ± 0.001
Validation350M2.894 ± 0.0052.745 ± 0.0022.964 ± 0.0133.018 ± 0.0202.906 ± 0.0032.882 ± 0.0012.788 ± 0.001

PMRC achieves the lowest mean training and validation losses among all memory-efficient methods across the three model sizes, further confirming the consistent performance observed.


We hope that our responses address your concerns, and we would be grateful if you would reconsider your assessment. We would be happy to answer any further questions during the reviewer–author discussion period.


References.

[1] Fira: Can We Achieve Full-rank Training of LLMs under Low-rank Constraint? NeurIPS, 2025.

[2] Lean and Mean Adaptive Optimization via Subset-norm and Subspace-momentum with Convergence Guarantees. ICML, 2025.

[3] Is Your Batch Size the Problem? Revisiting the Adam-SGD Gap in Language Modeling. arXiv, 2506.12543.

[4] SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training. NeurIPS, 2025.

Reviewer_wmX3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes two optimization methods that reduce optimizer-state memory by storing momentum in a low-dimensional subspace. Following Fira (Chen et al., 2025), both methods add a scaled correction based on the component of the current gradient orthogonal to the chosen subspace. SMRC adapts SGD with momentum by accumulating momentum only in the chosen subspace. PMRC applies Muon’s matrix-sign operation to the projected momentum before forming the update. The paper gives convergence guarantees for fixed and periodically refreshed bases and evaluates a practical version of PMRC on language-model pretraining and fine-tuning.

Critical strengths

The authors provide convergence guarantees for both proposed algorithms under standard stochastic assumptions, under which prior methods such as GaLore can fail to converge, and the proofs themselves seem non-trivial. They select the subspace from the dominant gradient directions, which perform better than GoLore’s random subspaces in the reported GaLore–GoLore comparisons. The residual correction itself comes from Fira (Chen et al., 2025a), which provides no convergence guarantee for its optimizer, so the main theoretical contribution is proving convergence of the resulting algorithms, including with Muon’s matrix-sign update. Compared with GUM, PMRC keeps all momentum in the projected space, avoiding full-size momentum buffers for sampled layers.

PMRC achieves lower pretraining validation loss than all tested memory-efficient baselines at each of the four model sizes. At 1.3B, it improves on GUM at nearly the same peak GPU memory.

Critical weaknesses

The theory uses a fixed weight for the residual correction. However, the experiments also use a cap that reduces this weight whenever the residual becomes too large relative to the projected Muon direction. Although the authors explain that this helps avoid instability, it is unclear how often the cap is active and how much it reduces the residual weight. If it frequently reduces the weight substantially, the residual update would largely depend on the current gradient and momentum, which the theory does not cover.

Questions for authors

Could you report validation loss against training time for PMRC, GUM, and full Muon at 1.3B? Table 3 covers only the basis refresh.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wmX3

Many thanks for your constructive review and positive assessment of our paper. We have uploaded our revised paper, with the changes highlighted in blue for ease of reference.


W1. It is unclear how often the cap is active and how much it reduces the residual weight.

Response. We thank the reviewer for this insightful comment. We introduce the residual cap because we observe loss fluctuations during the early stages of PMRC training, when the residual update is larger than the projected Muon update. To stabilize training, we introduce a residual cap to control the magnitude of the residual correction while retaining the original coefficient γ\gamma when the cap is inactive. As shown in the ablation study in Appendix E.2.4, the uncapped PMRC exhibits loss fluctuations during early training, while both variants achieve comparable losses in the later stages. We also provide the frequency of cap activation and the magnitude of the coefficient reduction in the following table. The cap activation frequency is calculated separately for each interval rather than over all training steps.

Training Steps (out of 9900)Cap Frequency (\\%)Coefficient Reduction (\\%)
1-200018.86%28.62%
2001-40002.73%4.83%
4001-60002.05%3.43%
6001-80002.04%3.25%
8001-99002.04%3.26%

The results show that the cap produces a coefficient reduction during early training, whereas the reduction decreases in later stages. After the first 20% of training steps, the cap is activated in only approximately 2% of the subsequent iterations, with a coefficient reduction of 3%–5%. Together with the ablation results, these findings indicate that the residual cap primarily improves early-stage training stability without substantially affecting the training performance.

We further extend the analysis of Theorem 4 to PMRC with the residual cap. Under Assumptions 1–4, with a fixed ρ\rho and the parameter choices in Theorem 4, we can ensure

1N∑k=1K∑t=1TE[∣Pk⊤∇F(Wk,t)∣∗+∣Qk∇F(Wk,t)∣F]≤O(N−1/4)+12γN∑k=1K∑t=1TE[(γ−γ^k,t)2∣QkGk,t∣F2]\frac{1}{N}\sum_ {k=1}^{K}\sum_ {t=1}^{T} \mathbb{E}\left[\\|P_ k^\top\nabla F(W_ {k,t})\\|_ * +\\|Q_ k\nabla F(W_ {k,t})\\|_ F \right]\le O(N^{-1/4}) + \frac{1}{2\gamma N} \sum_ {k=1}^{K}\sum_ {t=1}^{T} \mathbb{E}\left[(\gamma-\widehat{\gamma}_ {k,t})^2 \\|Q_ kG_ {k,t}\\|_F^2 \right].

The second term characterizes the effect of the residual cap. In particular, the original O(N−1/4)O(N^{-1/4}) convergence rate is preserved when 1N∑k=1K∑t=1TE[(γ−γ^k,t)2∣QkGk,t∣F2]=O(1)\frac{1}{N}\sum_ {k=1}^{K}\sum_ {t=1}^{T} \mathbb{E}\left[ (\gamma-\widehat{\gamma}_ {k,t})^2 \\|Q_ kG_ {k,t}\\|_ F^2 \right]=O(1). Our additional experiments also provide empirical support for this analysis.


Q1. Could you report validation loss against training time for PMRC, GUM, and full Muon at 1.3B?

Response. We thank the reviewer for this suggestion. We provide an additional comparison of validation loss against wall-clock training time for PMRC, GUM and full Muon on LLaMA-1.3B in the revised paper (refer to Figure 3 on page 33). The total training times of PMRC, GUM and full Muon are 47.0147.01, 46.9246.92 and 48.3048.30 hours, respectively. PMRC requires less training time than full Muon, as the Newton-Schulz iterations are performed only on the low-rank subspace and the projection basis can be efficiently updated using our approximate SVD method. For a fair comparison, we also use the same approximate SVD implementation for GUM. Consequently, PMRC and GUM exhibit nearly identical training times.


We hope that our responses address your concerns and are happy to answer any further questions during the reviewer–author discussion period.

Thank you for the response. The cap frequency results and the wall-clock comparison with GUM address the questions I raised. For the capped update, I would clarify that the original rate is recovered under the assumption that the cap changes the residual update during training. I keep my score at 3.

Dear Reviewer wmX3,

We are grateful that our rebuttal has addressed your question and we sincerely appreciate your positive assessment of our work.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Dear Reviewers,

We sincerely thank you for your constructive comments and questions, which have helped us improve our manuscript. We have addressed your comments point by point in our individual responses and uploaded a revised manuscript with all changes highlighted in blue for ease of reference.

We hope that our detailed responses and revisions adequately address your concerns. We would greatly appreciate your feedback on whether our revisions have sufficiently addressed the concerns raised in your reviews. We would be happy to provide further clarification or address any remaining concerns during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.