- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes DGMM, extending GMT-style gradient masking for full-parameter LLM fine-tuning by assigning gradient retention budgets to individual layers. It combines directional sign bias, temporal stability, and inter-layer correlation into a quality score with manually chosen weights. Experiments on Mistral-7B, DeepSeek-Coder-6.7B, and Llama-2-7B compare DGMM with SFT, HFT, and GMT across several tasks and report some improvements.
Critical strengths
The paper addresses a practical question: how to allocate gradient-mask budgets across layers and adapt them during training, rather than applying a single global ratio.
Critical weaknesses
- DGMM's core design, quality-guided masking, is not sufficiently validated or isolated in the experiments. DGMM differs from GMT in three main ways: the overall masking ratio, the addition of warm-up and a stage-based schedule, and quality-guided adaptive masking. However, the experiments lack corresponding ablations. I could not find specific values or visualization for the actual masking ratios in each layer anywhere in the paper. According to Eq. (7), the masking ratio seems to remain between 1% and 11%. The masking appears to be almost uniform.
- I have serious doubts about the credibility of the experiments. Below is a comparison of the same SFT baselines and tasks in this paper and the GMT paper [1].
| Model / benchmark | SFT in this paper | SFT in the GMT paper |
|---|---|---|
| Mistral-7B HumanEval | 32.3 | 68.3 |
| Mistral-7B GSM8K | 16.0 | 75.3 |
| DeepSeek-Coder-6.7B HumanEval | 47.0 | 76.8 |
| Llama-2-7B (Tülu-V2) MMLU | 39.12 | 48.5 |
[1] Li, Haoling, et al. "Enhancing large language model performance with gradient-based parameter selection." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 23. 2025.
Questions for authors
- What is the "external mask ratio" varied in Fig. 2?
- How many random seeds were used for each experiment? I did not see any analysis of variability across runs. DGMM performs poorly in most cases in Tables 1–2. How do the authors explain this?
- Why is DGMM's MMLU score 39.23 in Table 2 but 41.40 in Table 4? There are many other internal inconsistencies; for example, Fig. 2 also does not agree with Table 1.
- Minor issues. For example, “layers.DGMM integrates”.
7 comments
3.2. How should the discrepancy between the SFT results in this manuscript and those reported in the original GMT paper be interpreted?
Response: Thank you for identifying this issue. We distinguish the provenance of our GMT implementation from the SFT result discrepancy. Our GMT baseline is an independent implementation of the published method; this fact does not explain differences in SFT results, because dense SFT does not use the GMT masking mechanism. We agree that cross-paper SFT differences require a separate comparison of model checkpoints, training data and preprocessing, configurations, and evaluation protocols. We do not treat the GMT implementation detail as an explanation for the SFT discrepancy.
Evidence: Section 4.2 describes the experimental baselines and the independent implementation of GMT. The revised text explicitly states that this clarification concerns only GMT and does not explain differences between dense SFT results reported across studies.
Revision place: Section 4.2 (p. 6) now separates the GMT implementation issue from the cross-paper SFT discrepancy and states that the latter requires a separate comparison of checkpoints, training data, preprocessing, configurations, and evaluation protocols. We do not claim that this cross-paper audit has been completed.
3.3. How are the numerical differences across the tables and figures reconciled, including the MMLU values and the relationship between Figure 2 and Table 1?
Response: We appreciate the reviewer’s careful checking. We agree that numerical consistency must be established from the underlying evaluation records rather than explained by assumption. We have clarified the specific MMLU difference between Table 2 and Table 4: the values come from two separate experimental runs and do not refer to the same checkpoint or run. This clarification should not be taken to mean that every possible table–figure discrepancy has been independently re-evaluated. Figure 2 presents external mask-ratio sensitivity, whereas Table 1 reports task-specific benchmark results; any remaining numerical correspondence should be checked against the original evaluation records.
Evidence: Table 2 (p. 7) reports a DGMM MMLU value of 39.23% in the DPO comparison, while Table 4 (p. 9) reports 41.40% for the full DGMM ablation configuration. The revised discussion in Section 4.3.2 identifies these as separate runs/checkpoints and explains that the 2.17-percentage-point difference should not be interpreted as an inconsistency within one evaluation result. Figure 2 (p. 8) is the external mask-ratio sensitivity plot.
Revision place: Section 4.3.2 (p. 8) now explicitly clarifies that the 39.23% and 41.40% MMLU values come from separate experimental runs and different checkpoints. This revision addresses the Table 2–Table 4 MMLU distinction. We have not claimed that it independently resolves any separate numerical discrepancy between Figure 2 and Table 1.
3.1. Does the manuscript report the actual layer-wise retention behavior, rather than only the target retention rule or external mask-ratio sweep?
Response: We agree that target retention budgets and realized retention behavior should be distinguished. Equations (7)–(9) define the raw, EMA-smoothed, and stage-constrained retention values assigned to eligible layers. Figure 2, by contrast, evaluates performance under different external mask-ratio budgets and should not be interpreted as a direct visualization of the realized retention ratio of every layer during training. The revised manuscript does not yet provide a complete per-layer realized-retention trajectory.
Evidence: Section 3.3 defines K_(l,raw), K_(l,EMA), and K_l in Eqs. (7)–(9) and gives the stage-aware schedule. Section 4.4 and Figure 2 (p. 8) report performance sensitivity to an external mask-ratio budget. Neither provides a complete per-layer realized-retention trajectory.
Revision place: The retention definitions and stage schedule are specified in Section 3.3 (pp. 4–5), while Figure 2 (p. 8) remains an external mask-ratio sensitivity analysis. No new per-layer realized-retention trajectory plot has been added in this revision, and we do not claim that this issue has been fully resolved.