- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies sparse parameter optimization in LLM training. For a given budget, only partial parameters are updated with rest frozen. The authors propose a momentum method where the past gradients are accounted in selecting the parameters. An per-layer budget allocation and thresholding is introduced to selecting parameters. Experiments are conducted on multiple models and domains to validate the performance.
Critical strengths
-
The approach is simple and straightforward. Using historical gradient information, aka momentum, is also well-applied in gradient updates of LLMs.
-
The presentation is easy to follow. The experiments covers multiple models and different domains.
Critical weaknesses
-
Key reference is missing. Chen Y, Wang S, Zhang Y, Lin Z, Zhang H, Sun W, Ding T, Sun R. Mofo: Momentum-filtered optimizer for mitigating forgetting in llm fine-tuning. arXiv preprint arXiv:2407.20999. 2024 Jul 30. The method in Mofo is highly related to this paper. The contribution on top of Mofo needs to be justified, compared, and baselined. Statistical evidence must be provided to justify any incremental performance gains, if there's any.
-
It's unclear how AdamW is combined with sparse update. In AdamW, even if the current gradient is zero, aka masked, it can still get moved by momentum. The author needs to be clear what it means by sparse update and how adamW works together with it under the setting.
Questions for authors
-
AdamW and actual update mechanism.
-
Comparison with Mofo baseline. Novelty should be explained.
-
Need clarification on memory usage, both analytical and empirically. The authors should be very clear where the benefit of sparse update and prove by evidence.
-
Statistical significance is unclear with one seed and marginal improvement over baselines. Many difference are less than 1%, which might be noice, as opposed to real improvement.
-
In Table 1, 2, mask ratio should be provided as an additional column.
2 comments
Reviewer 3
Comment 4. Are the gains robust, and is statistical significance established across random seeds?
Response: Given the substantial training time required for additional runs with different random seeds, we supplemented the evaluation with multiple ablation experiments to examine MAGM’s sensitivity to its hyperparameters.
Evidence: These experiments show that MAGM maintains its performance advantage across most of the tested hyperparameter settings, providing additional evidence of the robustness of its design. Table 4. Code-generation performance of DeepSeek-Coder-Base-6.7B under different allocation coefficients α and momentum coefficients β. Bold denotes the best benchmark scores and average.
\alpha&\beta&\text{HumanEval}&\text{HumanEval+}&\text{MBPP}&\text{MBPP+}& \text{Avg.}\\\\ \hline 1.0&0.9&67.7&62.2&75.4&\mathbf{66.1}&67.85\\\\ 0.5&0.9&\mathbf{70.7}&\mathbf{64.0}&76.2&65.6&\mathbf{69.13}\\\\ 2.0&0.9&67.7&61.6&75.9&65.3&67.63\\\\ 1.0&0.95&68.9&62.8&\mathbf{77.0}&65.1&68.45\\\\ 1.0&0.99&70.1&62.2&75.9&65.1&68.33\\\\ \end{array}$$ Revision place: Section 4.3.2 and Table 4 (p. 9) add the hyperparameter-sensitivity results. Comment 5. Please state the mask ratios explicitly in Tables 1 and 2. Response: In Tables 1 and 2, all fine-tuning methods except full-parameter SFT were evaluated at the same mask ratio of 70%. Our previous description stated that the experimental conditions were matched without explicitly reporting this ratio, leaving the setting insufficiently clear. Evidence: A Mask Ratio (%) column now reports the setting in both tables; full-parameter SFT uses 0%. Table 2. General-domain performance on Llama2-7B. All scores are percentages. Pre-trained model $$\begin{array}{lrrrrrrrr} \text{Method}&\text{Mask (\\%)}&\text{MMLU}&\text{GSM8K}&\text{BBH}& \text{TyDiQA}&\text{TruthfulQA}&\text{HumanEval}&\text{Avg.}\\\\ \hline \text{Pre-trained}&-&31.9&10&33.1&43.5&34.3&7.3&26.7\\\\ \end{array}$$ Supervised Fine-tuning (SFT) on TÜLU V2 $$\begin{array}{lrrrrrrrr} \text{Method}&\text{Mask (\\%)}&\text{MMLU}&\text{GSM8K}&\text{BBH}& \text{TyDiQA}&\text{TruthfulQA}&\text{HumanEval}&\text{Avg.}\\\\ \hline \text{SFT}&0&36.5&19.5&28.8&\mathbf{52.0}&43.6&24.6&34.2\\\\ \text{HFT}&70&\mathbf{37.9}&14.0&\mathbf{34.2}&\mathbf{52.0}&41.7&24.6& 34.1\\\\ \text{GMT}&70&35.9&20.0&30.5&51.9&43.1&24.6&34.3\\\\ \text{MAGM}&70&36.9&\mathbf{25.0}&33.4&\mathbf{52.0}&\mathbf{43.8}& \mathbf{26.4}&\mathbf{36.3}\\\\ \end{array}$$ Direct Preference Optimization (DPO) on UltraFeedback $$\begin{array}{lrrrrrrrr} \text{Method}&\text{Mask (\\%)}&\text{MMLU}&\text{GSM8K}&\text{BBH}& \text{TyDiQA}&\text{TruthfulQA}&\text{HumanEval}&\text{Avg.}\\\\ \hline \text{SFT}&0&\mathbf{39.3}&18.5&28.9&51.9&43.5&14.6&32.8\\\\ \text{HFT}&70&37.5&15.0&\mathbf{33.8}&51.8&41.7&\mathbf{17.7}&32.9\\\\ \text{GMT}&70&38.7&18.5&31.0&51.9&43.1&15.0&33.0\\\\ \text{MAGM}&70&39.2&\mathbf{27.0}&33.6&\mathbf{52.0}&\mathbf{43.8}&15.2& \mathbf{35.1}\\\\ \end{array}$$ All results use Llama2-7B. Evaluation protocols: MMLU, 0-shot EM; GSM8K, 8-shot CoT EM; BBH, 3-shot CoT EM; TyDiQA, 1-shot F1; TruthfulQA, 0-shot MC2; HumanEval, 0-shot Pass@10. Revision place: Section 4.1.1 (p. 6) and the Mask Ratio (%) columns in Tables 1 and 2 (pp. 7-8).
Reviewer 3
Comment 1. How does gradient masking interact with AdamW, and do masked parameters remain unchanged?
Response: Thank you for pointing out this ambiguity. Masking the current gradient does not guarantee that the corresponding parameter remains unchanged, since residual momentum and decoupled weight decay in AdamW can still produce parameter changes.
Evidence: This clarification does not alter MAGM’s selection mechanism: masking removes the current gradient contribution at unselected coordinates from AdamW’s moment updates, thereby affecting the optimizer states and subsequent parameter trajectories.
Revision place: Sections 3.1 (p. 3) and 3.3 (p. 5) clarify that masking suppresses the current gradient contribution rather than freezing parameters.
Comment 2. How does MAGM differ from MoFO, and how does it compare experimentally?
Response: Thank you for highlighting MoFO. MoFO selects a predefined fraction of coordinates within each parameter block based on momentum magnitudes. Each block’s selection budget is determined by its size and the predefined fraction; historical momentum determines the within-block ranking but does not guide cross-layer budget allocation. MAGM first aggregates historical importance to assess layer contributions and adaptively allocates layer-wise budgets under a fixed global budget. It then selects coordinates within each tensor according to their importance scores. MAGM therefore determines both how many coordinates each layer retains and which coordinates are selected, allowing the limited budget to be redistributed according to layer contributions. Its core contribution is to connect cross-layer budget allocation with local parameter selection through historical importance. We have added citations to MoFO (Chen et al., 2025) in Section 2 (Related Work, p. 2) and Section 4.1.1 (Baselines, p. 6), together with a discussion of its differences from MAGM. The full bibliographic entry appears on p. 11, and the additional baseline results are reported in Table 1 (p. 7).
Evidence: We have also conducted additional experiments with MoFO as a baseline. MAGM achieves the highest average performance among the compared methods in all four evaluated task–backbone settings. Table 1. Code-generation and mathematical-reasoning performance. All scores are percentages. Code backbone: Mistral-7B; Math backbone: Mistral-7B.
\text{Method}&\text{Mask (\\%)}&\text{HumanEval}&\text{HumanEval+}& \text{MBPP}&\text{MBPP+}&\text{Code Avg.}&\text{GSM8K}&\text{MATH}& \text{Math Avg.}\\\\ \hline \text{Pre-trained}&-&28&23.2&53.7&42.3&36.8&-&-&-\\\\ \text{HFT}&70&61.6&\mathbf{58.9}&48.3&43.1&52.98&62.3&18.1&40.2\\\\ \text{SFT}&0&61&57.3&\mathbf{50}&42.6&52.7&62.1&17.9&40.0\\\\ \text{MoFO}&70&60.3&56.1&48.7&42.1&51.8&62.5&18.2&40.4\\\\ \text{GMT}&70&59.8&55.5&48.7&\mathbf{43.4}&51.9&62.7&\mathbf{18.5}&40.6\\\\ \text{MAGM}&70&\mathbf{62.8}&57.9&\mathbf{50}&\mathbf{43.4}&\mathbf{53.5}& \mathbf{62.9}&18.4&\mathbf{40.7}\\\\ \end{array}$$ Code backbone: DeepSeek-Coder-Base-6.7B; Math backbone: Llama3-8B. $$\begin{array}{lrrrrrrrrr} \text{Method}&\text{Mask (\\%)}&\text{HumanEval}&\text{HumanEval+}& \text{MBPP}&\text{MBPP+}&\text{Code Avg.}&\text{GSM8K}&\text{MATH}& \text{Math Avg.}\\\\ \hline \text{Pre-trained}&-&47.0&39.0&69.3&55.3&52.65&-&-&-\\\\ \text{HFT}&70&64.0&56.7&75.7&63.5&65.0&74.9&24.7&49.8\\\\ \text{SFT}&0&65.2&60.4&75.4&63&66&61.1&18.42&39.76\\\\ \text{MoFO}&70&64.2&57.1&75.5&64.3&65.3&75.6&24.9&50.3\\\\ \text{GMT}&70&64.6&58.5&\mathbf{76.2}&65.6&66.225&76.5&\mathbf{25.74}& 51.12\\\\ \text{MAGM}&70&\mathbf{67.7}&\mathbf{62.2}&75.4&\mathbf{66.1}& \mathbf{67.85}&\mathbf{76.88}&25.4&\mathbf{51.14}\\\\ \end{array}$$ Revision place: Section 2 (p. 2) adds the MoFO discussion; Section 4.1.1 (p. 6) adds the baseline description; Table 1 (p. 7) adds the MoFO results. The MoFO reference appears on p. 11. Comment 3. What accounts for the memory overhead, and what is the benefit of sparse updates? Response: To reduce GPU memory consumption, we optimized threshold computation to avoid materializing complete sorting results. We also reduced unnecessary FP32 copies, processed tensors sequentially, and released temporary variables promptly. Evidence: Together, these implementation optimizations reduced MAGM’s GPU memory consumption to 46.3 GB, bringing it closer to that of the other compared methods. Moreover, our experiments show that the proposed method maintains performance even at extreme masking ratios and achieves better downstream performance than the baselines in most settings. Revision place: Section 4.3.3 and Table 6 (p. 10) report the optimized memory measurement and implementation explanation. Section 4.3.1 (pp. 8-9) clarifies the mask-ratio interpretation.