acceptodds
Under review as a conference paper at ICLR 2027

Beyond Instantaneous Gradients: Momentum-Guided Adaptive Gradient Masking for Sparse Fine-Tuning of LLMs

Abstract

Sparse fine-tuning of large language models (LLMs) requires selecting the parameters that contribute most to downstream adaptation under a limited update budget. Existing gradient-based selection methods rely primarily on gradient statistics from the current optimization stage, making their importance estimates susceptible to mini-batch noise, outliers, and short-term fluctuations in optimization. We propose Momentum-Guided Adaptive Gradient Masking (MAGM) to address this issue. MAGM uses an exponential moving average to integrate gradient information across training steps, yielding a history-aware importance estimate that captures sustained parameter contributions. These importance scores guide layer-wise budget allocation and within-layer parameter selection under a fixed global update budget. Experiments show that MAGM achieves the highest average scores among the compared methods in code generation, mathematical reasoning, and general-domain evaluation. On DeepSeek-Coder-Base-6.7B and MISTRAL-7B, MAGM improves the average code-generation score by 1.63% and 0.52%, respectively, over the best-performing algorithm. In mathematical reasoning, MAGM achieves average performance comparable to the best-performing baseline on both MISTRAL-7B and LLAMA3-8B. On general-domain tasks, the gains over the best-performing algorithm are 2.0% for supervised fine-tuning and 2.1% for direct preference optimization. These results indicate that MAGM enables more stable parameter importance estimation and more effective budget allocation under a limited parameter update budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Sparse Fine-Tuning, Gradient Masking, Momentum-Guided Importance Estimation, Layer-Adaptive Budget Allocation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8YzJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies sparse parameter optimization in LLM training. For a given budget, only partial parameters are updated with rest frozen. The authors propose a momentum method where the past gradients are accounted in selecting the parameters. An per-layer budget allocation and thresholding is introduced to selecting parameters. Experiments are conducted on multiple models and domains to validate the performance.

Critical strengths
  1. The approach is simple and straightforward. Using historical gradient information, aka momentum, is also well-applied in gradient updates of LLMs.

  2. The presentation is easy to follow. The experiments covers multiple models and different domains.

Critical weaknesses
  1. Key reference is missing. Chen Y, Wang S, Zhang Y, Lin Z, Zhang H, Sun W, Ding T, Sun R. Mofo: Momentum-filtered optimizer for mitigating forgetting in llm fine-tuning. arXiv preprint arXiv:2407.20999. 2024 Jul 30. The method in Mofo is highly related to this paper. The contribution on top of Mofo needs to be justified, compared, and baselined. Statistical evidence must be provided to justify any incremental performance gains, if there's any.

  2. It's unclear how AdamW is combined with sparse update. In AdamW, even if the current gradient is zero, aka masked, it can still get moved by momentum. The author needs to be clear what it means by sparse update and how adamW works together with it under the setting.

Questions for authors
  1. AdamW and actual update mechanism.

  2. Comparison with Mofo baseline. Novelty should be explained.

  3. Need clarification on memory usage, both analytical and empirically. The authors should be very clear where the benefit of sparse update and prove by evidence.

  4. Statistical significance is unclear with one seed and marginal improvement over baselines. Many difference are less than 1%, which might be noice, as opposed to real improvement.

  5. In Table 1, 2, mask ratio should be provided as an additional column.

2 comments

Reviewer 3

Comment 1. How does gradient masking interact with AdamW, and do masked parameters remain unchanged?

Response: Thank you for pointing out this ambiguity. Masking the current gradient does not guarantee that the corresponding parameter remains unchanged, since residual momentum and decoupled weight decay in AdamW can still produce parameter changes.

Evidence: This clarification does not alter MAGM’s selection mechanism: masking removes the current gradient contribution at unselected coordinates from AdamW’s moment updates, thereby affecting the optimizer states and subsequent parameter trajectories.

Revision place: Sections 3.1 (p. 3) and 3.3 (p. 5) clarify that masking suppresses the current gradient contribution rather than freezing parameters.

Comment 2. How does MAGM differ from MoFO, and how does it compare experimentally?

Response: Thank you for highlighting MoFO. MoFO selects a predefined fraction of coordinates within each parameter block based on momentum magnitudes. Each block’s selection budget is determined by its size and the predefined fraction; historical momentum determines the within-block ranking but does not guide cross-layer budget allocation. MAGM first aggregates historical importance to assess layer contributions and adaptively allocates layer-wise budgets under a fixed global budget. It then selects coordinates within each tensor according to their importance scores. MAGM therefore determines both how many coordinates each layer retains and which coordinates are selected, allowing the limited budget to be redistributed according to layer contributions. Its core contribution is to connect cross-layer budget allocation with local parameter selection through historical importance. We have added citations to MoFO (Chen et al., 2025) in Section 2 (Related Work, p. 2) and Section 4.1.1 (Baselines, p. 6), together with a discussion of its differences from MAGM. The full bibliographic entry appears on p. 11, and the additional baseline results are reported in Table 1 (p. 7).

Evidence: We have also conducted additional experiments with MoFO as a baseline. MAGM achieves the highest average performance among the compared methods in all four evaluated task–backbone settings. Table 1. Code-generation and mathematical-reasoning performance. All scores are percentages. Code backbone: Mistral-7B; Math backbone: Mistral-7B.

\text{Method}&\text{Mask (\\%)}&\text{HumanEval}&\text{HumanEval+}& \text{MBPP}&\text{MBPP+}&\text{Code Avg.}&\text{GSM8K}&\text{MATH}& \text{Math Avg.}\\\\ \hline \text{Pre-trained}&-&28&23.2&53.7&42.3&36.8&-&-&-\\\\ \text{HFT}&70&61.6&\mathbf{58.9}&48.3&43.1&52.98&62.3&18.1&40.2\\\\ \text{SFT}&0&61&57.3&\mathbf{50}&42.6&52.7&62.1&17.9&40.0\\\\ \text{MoFO}&70&60.3&56.1&48.7&42.1&51.8&62.5&18.2&40.4\\\\ \text{GMT}&70&59.8&55.5&48.7&\mathbf{43.4}&51.9&62.7&\mathbf{18.5}&40.6\\\\ \text{MAGM}&70&\mathbf{62.8}&57.9&\mathbf{50}&\mathbf{43.4}&\mathbf{53.5}& \mathbf{62.9}&18.4&\mathbf{40.7}\\\\ \end{array}$$ Code backbone: DeepSeek-Coder-Base-6.7B; Math backbone: Llama3-8B. $$\begin{array}{lrrrrrrrrr} \text{Method}&\text{Mask (\\%)}&\text{HumanEval}&\text{HumanEval+}& \text{MBPP}&\text{MBPP+}&\text{Code Avg.}&\text{GSM8K}&\text{MATH}& \text{Math Avg.}\\\\ \hline \text{Pre-trained}&-&47.0&39.0&69.3&55.3&52.65&-&-&-\\\\ \text{HFT}&70&64.0&56.7&75.7&63.5&65.0&74.9&24.7&49.8\\\\ \text{SFT}&0&65.2&60.4&75.4&63&66&61.1&18.42&39.76\\\\ \text{MoFO}&70&64.2&57.1&75.5&64.3&65.3&75.6&24.9&50.3\\\\ \text{GMT}&70&64.6&58.5&\mathbf{76.2}&65.6&66.225&76.5&\mathbf{25.74}& 51.12\\\\ \text{MAGM}&70&\mathbf{67.7}&\mathbf{62.2}&75.4&\mathbf{66.1}& \mathbf{67.85}&\mathbf{76.88}&25.4&\mathbf{51.14}\\\\ \end{array}$$ Revision place: Section 2 (p. 2) adds the MoFO discussion; Section 4.1.1 (p. 6) adds the baseline description; Table 1 (p. 7) adds the MoFO results. The MoFO reference appears on p. 11. Comment 3. What accounts for the memory overhead, and what is the benefit of sparse updates? Response: To reduce GPU memory consumption, we optimized threshold computation to avoid materializing complete sorting results. We also reduced unnecessary FP32 copies, processed tensors sequentially, and released temporary variables promptly. Evidence: Together, these implementation optimizations reduced MAGM’s GPU memory consumption to 46.3 GB, bringing it closer to that of the other compared methods. Moreover, our experiments show that the proposed method maintains performance even at extreme masking ratios and achieves better downstream performance than the baselines in most settings. Revision place: Section 4.3.3 and Table 6 (p. 10) report the optimized memory measurement and implementation explanation. Section 4.3.1 (pp. 8-9) clarifies the mask-ratio interpretation.

Reviewer 3

Comment 4. Are the gains robust, and is statistical significance established across random seeds?

Response: Given the substantial training time required for additional runs with different random seeds, we supplemented the evaluation with multiple ablation experiments to examine MAGM’s sensitivity to its hyperparameters.

Evidence: These experiments show that MAGM maintains its performance advantage across most of the tested hyperparameter settings, providing additional evidence of the robustness of its design. Table 4. Code-generation performance of DeepSeek-Coder-Base-6.7B under different allocation coefficients α and momentum coefficients β. Bold denotes the best benchmark scores and average.

\alpha&\beta&\text{HumanEval}&\text{HumanEval+}&\text{MBPP}&\text{MBPP+}& \text{Avg.}\\\\ \hline 1.0&0.9&67.7&62.2&75.4&\mathbf{66.1}&67.85\\\\ 0.5&0.9&\mathbf{70.7}&\mathbf{64.0}&76.2&65.6&\mathbf{69.13}\\\\ 2.0&0.9&67.7&61.6&75.9&65.3&67.63\\\\ 1.0&0.95&68.9&62.8&\mathbf{77.0}&65.1&68.45\\\\ 1.0&0.99&70.1&62.2&75.9&65.1&68.33\\\\ \end{array}$$ Revision place: Section 4.3.2 and Table 4 (p. 9) add the hyperparameter-sensitivity results. Comment 5. Please state the mask ratios explicitly in Tables 1 and 2. Response: In Tables 1 and 2, all fine-tuning methods except full-parameter SFT were evaluated at the same mask ratio of 70%. Our previous description stated that the experimental conditions were matched without explicitly reporting this ratio, leaving the setting insufficiently clear. Evidence: A Mask Ratio (%) column now reports the setting in both tables; full-parameter SFT uses 0%. Table 2. General-domain performance on Llama2-7B. All scores are percentages. Pre-trained model $$\begin{array}{lrrrrrrrr} \text{Method}&\text{Mask (\\%)}&\text{MMLU}&\text{GSM8K}&\text{BBH}& \text{TyDiQA}&\text{TruthfulQA}&\text{HumanEval}&\text{Avg.}\\\\ \hline \text{Pre-trained}&-&31.9&10&33.1&43.5&34.3&7.3&26.7\\\\ \end{array}$$ Supervised Fine-tuning (SFT) on TÜLU V2 $$\begin{array}{lrrrrrrrr} \text{Method}&\text{Mask (\\%)}&\text{MMLU}&\text{GSM8K}&\text{BBH}& \text{TyDiQA}&\text{TruthfulQA}&\text{HumanEval}&\text{Avg.}\\\\ \hline \text{SFT}&0&36.5&19.5&28.8&\mathbf{52.0}&43.6&24.6&34.2\\\\ \text{HFT}&70&\mathbf{37.9}&14.0&\mathbf{34.2}&\mathbf{52.0}&41.7&24.6& 34.1\\\\ \text{GMT}&70&35.9&20.0&30.5&51.9&43.1&24.6&34.3\\\\ \text{MAGM}&70&36.9&\mathbf{25.0}&33.4&\mathbf{52.0}&\mathbf{43.8}& \mathbf{26.4}&\mathbf{36.3}\\\\ \end{array}$$ Direct Preference Optimization (DPO) on UltraFeedback $$\begin{array}{lrrrrrrrr} \text{Method}&\text{Mask (\\%)}&\text{MMLU}&\text{GSM8K}&\text{BBH}& \text{TyDiQA}&\text{TruthfulQA}&\text{HumanEval}&\text{Avg.}\\\\ \hline \text{SFT}&0&\mathbf{39.3}&18.5&28.9&51.9&43.5&14.6&32.8\\\\ \text{HFT}&70&37.5&15.0&\mathbf{33.8}&51.8&41.7&\mathbf{17.7}&32.9\\\\ \text{GMT}&70&38.7&18.5&31.0&51.9&43.1&15.0&33.0\\\\ \text{MAGM}&70&39.2&\mathbf{27.0}&33.6&\mathbf{52.0}&\mathbf{43.8}&15.2& \mathbf{35.1}\\\\ \end{array}$$ All results use Llama2-7B. Evaluation protocols: MMLU, 0-shot EM; GSM8K, 8-shot CoT EM; BBH, 3-shot CoT EM; TyDiQA, 1-shot F1; TruthfulQA, 0-shot MC2; HumanEval, 0-shot Pass@10. Revision place: Section 4.1.1 (p. 6) and the Mask Ratio (%) columns in Tables 1 and 2 (pp. 7-8).
Reviewer_MJ94OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAGM, a sparse fine-tuning method that selects which parameters to update under a fixed budget. It addresses the instability of instantaneous gradient-based importance by using an EMA of gradient magnitudes for selection, combined with layer-adaptive budget allocation. The method is evaluated on code generation, mathematical reasoning, and general-domain tasks (SFT and DPO).

Critical strengths

1. Simple and reasonable idea. Combining history-aware importance estimation with layer-wise budget allocation is intuitive, and the two components are complementary in principle.

2. Broad empirical coverage. The paper evaluates on code generation, mathematical reasoning, and general-domain tasks, covering both SFT and DPO, and compares against SFT, HFT, and GMT under matched update budgets.

Critical weaknesses

1. Incremental combination, limited novelty. GMT already uses first-order gradient information as an importance indicator, and allocating sparse budgets by layer importance is common in pruning and structured sparsity. The contribution is essentially adding EMA smoothing and a layer-wise allocation rule on top of existing ideas, without a new mechanism or insight.

2. Large overhead without clear performance gain. MAGM increases GPU memory by 30.4% (44.8→58.4 GB) and per-step time slightly, while the reported gains are small—especially in mathematical reasoning, where MAGM is essentially on par with GMT (e.g., +0.02 average on Llama3-8B, and GMT is better on MATH).

3. Necessity of layer allocation is unclear. MAGM-UA (global selection, no layer allocation) is very close to the full method (e.g., 67.6 vs. 67.85 average on code generation, 40.0 vs. 40.7 on math). This suggests the marginal value of layer-adaptive allocation is limited, yet it is presented as a main contribution.

4. Relationship to Adam's second moment is not discussed. The EMA of gradient magnitudes is closely related to Adam's v_t (EMA of squared gradients). It is unclear why one should maintain a separate importance statistic instead of using Adam's second moment directly.

Questions for authors

please see weakness.

1 comment

Reviewer 2

Comment 1. Incremental combination and limited novelty. What is the contribution beyond EMA smoothing and layer-wise allocation?

Response: Thank you for raising this concern. We would like to clarify that MAGM’s contribution lies in a history-guided, globally budget-constrained selection mechanism, rather than introducing EMA or layer-wise sparsity allocation individually.

Evidence: First, MAGM changes the information used for selection. GMT uses gradient magnitudes from the current optimization step, including within-step gradient accumulation. MAGM maintains a signed-gradient EMA across optimization steps and uses its magnitude as the selection score, incorporating historical gradient information into importance estimation. Second, the same historical statistic drives two connected decisions: how much gradient-retention budget each layer receives, and which coordinates are retained within the allocated ratios. Layer-level aggregation determines bounded allocations under a fixed global target budget, while coordinate-level scores determine fine-grained selection. Therefore, MAGM’s contribution is the specific coupling of importance estimation and hierarchical budget allocation.

Revision place: New supporting analyses in Section 4.3.2 (p. 9), Table 4 (p. 9), and Table 5 (p. 10).

Comment 2. Large overhead without clear performance gain. How is the additional memory justified?

Response: We reduce the additional GPU memory overhead by minimizing unnecessary FP32 copies, processing tensors sequentially, and promptly releasing temporary variables, bringing memory usage closer to that of the other methods.

Evidence: Although MAGM performs similarly to GMT on MATH, it achieves higher mathematical-reasoning averages on both evaluated backbones. MAGM also delivers clear improvements in the reported code-generation and general-domain averages. Table 6 reports 46.3 GB for optimized MAGM, compared with 58.4 GB originally and 44.8 GB for GMT; Tables 1 and 2 report the performance comparisons.

Revision place: Section 4.3.3 and Table 6 (p. 10) add the optimized memory measurement and implementation explanation. Performance comparisons remain in Tables 1 and 2 (pp. 7-8). Comment 3. Necessity of layer allocation is unclear. What evidence supports its contribution? Response: To further examine the contribution of layer allocation, we conducted additional controlled experiments varying the allocation coefficient α while fixing the EMA coefficient β=0.9.

Evidence: As shown in the table, the average scores are 69.13, 67.85, and 67.63 for α=0.5, 1.0, and 2.0, respectively. Changing α from 1.0 to 0.5 improves the average by 1.28 percentage points. Since α controls the concentration of the layer-wise budget allocation, these results show that, with the EMA setting and global budget unchanged, how the budget is distributed across layers affects downstream performance.

\alpha&\beta&\text{HE}&\text{HE+}& \text{MBPP}&\text{MBPP+}&\text{Avg.}\\\\ \hline 1.0&0.9&67.7&62.2&75.4&\mathbf{66.1}&67.85\\\\ 0.5&0.9&\mathbf{70.7}&\mathbf{64.0}&76.2&65.6& \mathbf{69.13}\\\\ 2.0&0.9&67.7&61.6&75.9&65.3&67.63\\\\ 1.0&0.95&68.9&62.8&\mathbf{77.0}&65.1&68.45\\\\ 1.0&0.99&70.1&62.2&75.9&65.1&68.33\\\\ \end{array}$$ HE denotes HumanEval; HE+ denotes HumanEval+. Bold marks the best benchmark scores and average. All results use DeepSeek-Coder-Base-6.7B. Revision place: Section 4.3.2, the new hyperparameter-sensitivity paragraph and Table 4 (p. 9). Comment 4. Relationship to Adam's second moment. Why maintain a separate importance statistic? Response: We therefore retained a single EMA buffer for importance estimation to reduce storage overhead. Evidence: In preliminary experiments, we introduced an additional second-moment statistic into importance estimation, but observed no improvement in downstream performance while incurring additional GPU memory overhead.
Reviewer_fikPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Momentum-Guided Adaptive Gradient Masking (MAGM), a sparse fine-tuning method for large language models. Average parameter importance within each Transformer layer determines its share of a global update budget through a power-weighted allocation rule with box constraints and parameters are selected within each layer according to importance quantiles, and the resulting binary masks are applied to the current gradients.The method is evaluated on code generation, mathematical reasoning, supervised instruction tuning, and direct preference optimization.

Critical strengths
  • MAGM combines a per-parameter EMA importance estimate with explicit layer-wise allocation under a fixed nominal budget.
  • The method is evaluated on coding, mathematical reasoning, SFT, and DPO.
Critical weaknesses
  • The actual sparse-update guarantee is unclear under AdamW.
  • The performance improvements are not yet convincingly connected to the claimed stability mechanism, especially given the additional memory requirements.
Questions for authors
  • How are masked parameters guaranteed to remain unchanged under AdamW?
  • How is the initial dense-update phase included in the sparsity and compute budget?
  • Does EMA improve parameter selection because the importance rankings become more reliable?
  • Report adaptation behavior across training stages, or test a controlled task-distribution shift during training
  • Compare against a memory-matched GMT configuration or introduce lower-precision/compressed importance buffers.
2 comments

Reviewer 1

Comment 1. How are masked parameters guaranteed to remain unchanged under AdamW?

Response: The reviewer asks whether masking guarantees unchanged parameters. MAGM masks current gradients before AdamW; it does not completely freeze the corresponding parameters.

Evidence: At unselected coordinates, the current gradient contribution to AdamW's moment updates is suppressed. Residual momentum and decoupled weight decay can still change the parameters. Masking therefore affects the optimizer states and subsequent parameter trajectories without guaranteeing unchanged parameters.

Revision place: Sections 3.1 and 3.3 (pp. 3 and 5) clarify gradient masking and the AdamW update mechanism.

Comment 2. How is the initial dense-update phase included in the sparsity and compute budget?

Response: The reviewer asks whether dense initialization is included in the reported budget. It is included in both the total training steps and computational accounting, rather than being an additional stage outside the reported step budget.

Evidence: During initialization, gradients are computed, historical importance is accumulated, and unmasked optimizer updates are performed. Let TT be the actual number of optimizer steps up to the evaluated checkpoint, WW the number of unmasked initialization steps, PP the number of eligible gradient coordinates, and rr the subsequent target mask ratio. The retained-coordinate budget and training-wide average retained fraction are Bkeep=WP+(T−W)(1−r)P,B_{\mathrm{keep}}=WP+(T-W)(1-r)P,

=\frac{W+(T-W)(1-r)}{T}.$$ The average masking ratio is $\bar r=(1-W/T)r$. These expressions use the target ratio; exact accounting uses realized masked-coordinate counts at each step. Computational cost includes both phases: $$C_{\mathrm{total}}= \sum_{t=1}^{W}C_t^{\mathrm{dense}} +\sum_{t=W+1}^{T}C_t^{\mathrm{masked}}.$$ The recorded training runtime includes dense initialization. Revision place: Section 4.1 (p. 6) distinguishes the masked-phase target from the training-wide masking ratio and includes initialization in the step and runtime budgets. Comment 3. Does EMA improve parameter selection because the importance rankings become more reliable? Response: Thank you for raising this question. Improving the reliability of importance rankings is the core rationale behind using EMA for parameter selection. Current gradient magnitudes are affected by mini-batch composition and training fluctuations, so a large gradient within a single accumulation interval does not necessarily indicate sustained importance. MAGM aggregates gradient magnitudes across consecutive optimization steps through EMA, reducing the influence of local fluctuations on importance estimation. This allows parameter rankings to better reflect sustained gradient contributions and reduces selection errors caused by short-term fluctuations. Evidence: Our theoretical analysis shows that EMA reduces the noise variance of importance estimates under the stated assumptions. The connection to local Top-k selection can be understood through an estimation-error condition: if the maximum absolute estimation error is smaller than half the importance gap between the k-th and (k+1)-th coordinates, the estimated scores recover the same selected set as the target importance scores. When the noise reduction from historical aggregation outweighs its tracking bias, EMA can reduce estimation error and thereby improve selection reliability, rather than merely slowing changes in the rankings. The component ablation results, in which removing historical importance estimation reduces downstream performance, are consistent with this mechanism, although they do not directly measure ranking accuracy. Revision place: Appendix A.1.1 provides the EMA noise-variance analysis. Section 4.3.2 and Table 3 (p. 9) report the component ablation results.

Reviewer 1

Comment 4. Report adaptation behavior across training stages, or test a controlled task-distribution shift during training.

Response: The reviewer requests evidence of adaptation during training. We report the allocated mask ratios of six selected Transformer layers at different training progress points under a fixed global budget.

Evidence: The table provides cross-layer and within-layer comparisons. At the same training point, different layers receive different gradient-retention densities; across training, the budget assigned to an individual layer changes. Values below are allocated mask ratios (%).

\text{Progress (\\%)}&\text{Layer 0}&\text{Layer 2}&\text{Layer 3}&\text{Layer 13}&\text{Layer 14}&\text{Layer 29}\\\\ \hline 5&91.19&75.59&72.48&59.75&60.78&73.05\\\\ 20&91.31&73.64&72.91&59.37&59.93&72.53\\\\ 40&91.47&75.79&73.99&59.45&59.85&71.86\\\\ 50^{\dagger}&74.37&74.10&72.73&61.52&61.89&72.02\\\\ 60&91.72&74.46&73.06&59.74&60.06&71.85\\\\ 80&92.35&76.72&74.74&59.68&59.86&71.66\\\\ 100.00&92.04&74.24&73.33&59.54&59.95&71.33 \end{array}$$ $L_i$ denotes Layer $i$; the dagger marks the highlighted snapshot. Revision place: Section 4.3.2 and Table 5 (pp. 9-10) report layer-wise allocation and adaptation across training. Comment 5. Compare against a memory-matched GMT configuration or introduce lower-precision/compressed importance buffers. Response: The reviewer requests a closer memory comparison or reduced importance-buffer overhead. We reduce memory consumption by minimizing FP32 copies, processing tensors sequentially, and promptly releasing temporary variables. Evidence: These implementation optimizations reduce MAGM's reported GPU memory consumption to 46.3 GB, bringing it closer to that of the other compared methods. Reducing precision did not affect model performance in these experiments. Revision place: Section 4.3.3 and Table 6 (p. 10) describe the implementation optimizations and report the memory measurements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.