- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents an empirical study of eight sparse fine-tuning methods on mathematical reasoning and code generation tasks, using full fine-tuning as a reference. It compares task performance and general capability preservation and includes ablation studies to provide a comprehensive analysis of this paradigm. Experimental results suggest that sparse fine-tuning can improve task performance while reducing forgetting compared with full fine-tuning. The findings also indicate that the optimal update ratio varies across methods and tasks, and that the benefits of sparse fine-tuning grow with the capability of the base model.
Critical strengths
This is a comprehensive empirical study in the field of sparse fine-tuning. The paper compares eight sparse fine-tuning methods under a unified evaluation framework across different models and tasks, which provides a useful reference for understanding the strengths and limitations of these methods. Moreover, the paper conducts ablation studies on different aspects, including the neuron-update ratio, update location, and mask construction strategy. The overall experimental effort is also substantial, covering multiple models, tasks, benchmarks, and controlled analyses. I believe this would be helpful to the community.
Critical weaknesses
The paper shows that the optimal neuron-update ratio varies across methods and tasks. However, adaptive methods such as NGFT and NMKE also use different neuron-selection criteria from the fixed-ratio methods. Therefore, it is difficult to determine whether their advantage comes from adaptive ratio selection or from the selection criterion itself.
Questions for authors
-
Some ablation experiments show very narrow differences. For example, in the update-location ablation, the performance differences among FFN-only, MHA-only, and MHA+FFN are only around 0.1–0.3 points in several settings. Based on these small differences, the authors conclude that neuron position does not strongly correlate with fine-tuning effectiveness and that selection location has only a minor influence. I believe this conclusion would be more convincing if multiple runs were included.
-
There appears to be an incorrectly calculated average in Table 12. For Prefix Tuning on the math task, the reported benchmark scores are 85.1, 48.8, 20.3, and 19.0. Their average should be 43.3 rather than the reported 41.8. Moreover, in Section 4.6, this value is used to support the claim that Prefix Tuning underperforms FFT (42.2). Please verify this number and the corresponding conclusion.
-
Please refer to the major weakness.
3 comments
Response to authors
Thank you for the detailed response and additional experiments. The revisions address my concerns, and I appreciate the authors’ effort to improve the paper. The question numbering in the response differs from my original review, but this does not affect my assessment. I maintain my rating of 3 (weak acceptance).
Official Comment by Authors
Thank you for your reply!
Additional experiments and results:
We provide the additional experiments promised in our previous response. We have now completed four independent runs of the Llama 3.1 8B model on the mathematical reasoning task and added the mean and standard deviation to Table 16. The results show that variability can depend on the update location. In particular, MHA-only configurations exhibit higher standard deviations in some cases: for NGFT, the standard deviation on Minerva reaches 0.8, compared with maximum standard deviations of 0.3 for FFN-only and 0.5 for MHA+FFN configurations. We have added this observation to the table caption and interpret it as evidence that MHA-only updates can be less stable on certain benchmarks.
We have incorporated all revisions into the latest version of the manuscript. We hope this finding further enriches your understanding of sparse fine-tuning. We sincerely thank you again for your support.
| Model | Method | GSM8K | MATH500 | Minerva | Olympiad | MMLU (P) | HellaSwag | ARC | TyDiQA |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-Math-7B | NCFT (FFN) | 88.6 ± 0.1 | 66.8 ± 0.2 | 29.4 ± 0.0 | 36.3 ± 0.1 | 42.2 ± 0.3 | 65.7 ± 0.2 | 51.0 ± 0.1 | 38.1 ± 0.3 |
| NCFT (MHA+FFN) | 88.4 ± 0.1 | 66.7 ± 0.2 | 29.4 ± 0.2 | 36.0 ± 0.1 | 42.5 ± 0.2 | 65.1 ± 0.2 | 51.7 ± 0.3 | 37.8 ± 0.4 | |
| NCFT (MHA) | 88.5 ± 0.4 | 66.7 ± 0.2 | 29.4 ± 0.2 | 36.1 ± 0.3 | 41.9 ± 0.6 | 66.2 ± 0.4 | 51.2 ± 0.5 | 38.4 ± 0.4 | |
| NGFT (FFN) | 89.2 ± 0.1 | 68.0 ± 0.2 | 29.8 ± 0.0 | 37.3 ± 0.2 | 42.0 ± 0.2 | 65.4 ± 0.2 | 50.2 ± 0.1 | 46.1 ± 0.2 | |
| NGFT (MHA+FFN) | 89.0 ± 0.1 | 67.9 ± 0.2 | 29.4 ± 0.2 | 37.0 ± 0.1 | 42.6 ± 0.3 | 65.9 ± 0.2 | 49.8 ± 0.1 | 46.7 ± 0.3 | |
| NGFT (MHA) | 89.1 ± 0.2 | 67.7 ± 0.2 | 29.4 ± 0.4 | 37.2 ± 0.4 | 42.4 ± 0.5 | 64.7 ± 0.4 | 51.0 ± 0.2 | 45.8 ± 0.5 | |
| Llama3.1-8B | NCFT (FFN) | 85.7 ± 0.2 | 50.0 ± 0.2 | 22.4 ± 0.0 | 18.7 ± 0.1 | 44.1 ± 0.2 | 78.8 ± 0.1 | 49.3 ± 0.2 | 60.1 ± 0.1 |
| NCFT (MHA+FFN) | 85.9 ± 0.3 | 50.3 ± 0.0 | 22.1 ± 0.0 | 18.7 ± 0.1 | 43.5 ± 0.3 | 79.4 ± 0.1 | 49.0 ± 0.1 | 60.8 ± 0.1 | |
| NCFT (MHA) | 85.7 ± 0.3 | 49.7 ± 0.2 | 22.1 ± 0.0 | 18.2 ± 0.2 | 44.8 ± 0.5 | 78.3 ± 0.5 | 49.9 ± 0.2 | 59.4 ± 0.5 | |
| NGFT (FFN) | 84.6 ± 0.2 | 52.4 ± 0.2 | 23.5 ± 0.2 | 18.9 ± 0.2 | 43.0 ± 0.3 | 78.6 ± 0.2 | 50.2 ± 0.3 | 57.6 ± 0.2 | |
| NGFT (MHA+FFN) | 84.5 ± 0.2 | 52.6 ± 0.2 | 23.5 ± 0.4 | 18.9 ± 0.2 | 43.5 ± 0.5 | 77.8 ± 0.1 | 50.1 ± 0.1 | 58.0 ± 0.3 | |
| NGFT (MHA) | 84.9 ± 0.2 | 52.3 ± 0.4 | 23.5 ± 0.8 | 18.7 ± 0.1 | 42.6 ± 0.5 | 78.1 ± 0.4 | 50.5 ± 0.5 | 57.0 ± 0.7 |
Official Comment by Authors
Thank you very much for your insightful suggestions, which have greatly helped improve the quality of our manuscript. We have revised the manuscript in accordance with your comments. To help you quickly understand the revisions, we provide a brief summary below. Further details can be found in the revised manuscript, where all changes are highlighted in blue for your convenience.
Q1. There appears to be an incorrectly calculated average in Table 12. For Prefix Tuning on the math task, the reported benchmark scores are 85.1, 48.8, 20.3, and 19.0. Their average should be 43.3 rather than the reported 41.8. Moreover, in Section 4.6, this value is used to support the claim that Prefix Tuning underperforms FFT (42.2). Please verify this number and the corresponding conclusion.
A1. Thank you for catching this. We have corrected the error and revised the conclusion in §4.6. The sentence "BitFit (41.3) and Prefix Tuning (41.8) both fall below FFT (42.2)" now states that only BitFit (41.3) falls below FFT (42.2). We also corrected the reference to the strongest PEFT method from Pfeiffer Adapter to Prefix Tuning, which outperforms FFT by 1.1 points. Finally, we have updated Table 4 and redrawn the Prefix Tuning bar to show 43.3.
Q2. The paper shows that the optimal neuron-update ratio varies across methods and tasks. However, adaptive methods such as NGFT and NMKE also use different neuron-selection criteria from the fixed-ratio methods. Therefore, it is difficult to determine whether their advantage comes from adaptive ratio selection or from the selection criterion itself.
A2. We thank the reviewer for highlighting this important confound. To disentangle the adaptive ratio from the selection criterion, we have added a controlled experiment in Appendix H (Table 13). We hold NGFT’s criterion fixed, rank all positively activated neurons by CV in ascending order, and retain the lowest-CV ρ%, with ρ ∈ {1, 3, 5, 7, 8.2, 10}. The experiment uses Llama3.1-8B on math, where NGFT adaptively selects 8.2%. The results show that both components contribute.
(1) The criterion alone is insufficient: performance varies by 4.0 points across ratios (40.9 at 1% versus 44.9 at 8.2%), and a poorly chosen ratio underperforms FFT (42.2).
(2) The adaptive ratio is effective: the self-selected 8.2% achieves the highest Math score (44.9) in the sweep without manual search. A well-chosen fixed ratio comes close (44.8 at 5%), suggesting that the adaptive ratio’s main benefit is avoiding per-task tuning rather than substantially outperforming a tuned ratio.
Thus, the criterion determines which neurons to update, while the adaptive ratio determines how many; together, they achieve the best observed performance. We have revised Section 4.3 and added Appendix H accordingly.
Q3. The update-location ablation shows only 0.1–0.3-point differences among FFN-only, MHA-only, and MHA+FFN in several settings. Multiple runs are needed to support the conclusion that selection location has little effect on fine-tuning performance.
A3. Thank you for raising this important concern. We have already completed three independent training runs of the update-location ablation on Qwen2.5-Math-7B, and we report the standard deviations in the table below (also added to Table 16 of the revised paper). We do observe some fluctuation in the results introduced by retraining the base model, but so far the gaps among the three locations remain small and within the range of the standard deviations, so our current conclusion still holds; that said, we prefer to state it with caution.
Since these experiments involve extensive base-model training, we need additional time to complete the remaining runs (including Llama3.1-8B), which we expect within one to two days. This also gives us an opportunity to further analyze whether there are other experimental phenomena we have not previously observed. Once completed, we will immediately incorporate the results and any new findings into the final version.