acceptodds
Under review as a conference paper at ICLR 2027

Understanding Sparse Fine-Tuning: Lessons and Insights from a Unified Empirical Study

Abstract

Sparse fine-tuning updates a small subset of task-critical neurons and has been reported to outperform full fine-tuning (FFT) while resisting catastrophic forgetting. Existing methods, however, differ in the selection criterion, module, schedule, and neuron-update ratio, and why sparse fine-tuning works remains poorly understood. We select eight representative sparse fine-tuning methods and conduct a unified empirical study on mathematical reasoning and code generation, using FFT as a reference, establishing a performance leaderboard of sparse fine-tuning methods. We further ablate the key design choices underlying sparse fine-tuning to provide a comprehensive analysis of this paradigm. We characterise the role of the sparse masked update: it acts as a capacity allocator that concentrates learning on task-critical neurons, and as a regulariser that pins the remaining parameters to their pre-trained values. This dual role mitigates the trade-off between task performance and anti-forgetting in FFT: sparse methods improve over FFT by points on average while retaining – of the base model's general capability across all tested families, scales, and tasks. We further find that: (1) The advantage of sparse fine-tuning over FFT grows with the capability of the base model; (2) Task-relevant neurons reside in both the MHA and the FFN of the backbone, so the selection location has only a marginal effect; (3) The optimal neuron-update ratio is method- and task-dependent, making adaptive neuron selection essential; (4) Dynamic mask regeneration trades general capability for a modest task gain, at a recurring compute and memory cost. Our analysis clarifies the design choices of sparse fine-tuning and identifies future research directions.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Large language model, Sparse fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GZL3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an empirical study of eight sparse fine-tuning methods on mathematical reasoning and code generation tasks, using full fine-tuning as a reference. It compares task performance and general capability preservation and includes ablation studies to provide a comprehensive analysis of this paradigm. Experimental results suggest that sparse fine-tuning can improve task performance while reducing forgetting compared with full fine-tuning. The findings also indicate that the optimal update ratio varies across methods and tasks, and that the benefits of sparse fine-tuning grow with the capability of the base model.

Critical strengths

This is a comprehensive empirical study in the field of sparse fine-tuning. The paper compares eight sparse fine-tuning methods under a unified evaluation framework across different models and tasks, which provides a useful reference for understanding the strengths and limitations of these methods. Moreover, the paper conducts ablation studies on different aspects, including the neuron-update ratio, update location, and mask construction strategy. The overall experimental effort is also substantial, covering multiple models, tasks, benchmarks, and controlled analyses. I believe this would be helpful to the community.

Critical weaknesses

The paper shows that the optimal neuron-update ratio varies across methods and tasks. However, adaptive methods such as NGFT and NMKE also use different neuron-selection criteria from the fixed-ratio methods. Therefore, it is difficult to determine whether their advantage comes from adaptive ratio selection or from the selection criterion itself.

Questions for authors
  1. Some ablation experiments show very narrow differences. For example, in the update-location ablation, the performance differences among FFN-only, MHA-only, and MHA+FFN are only around 0.1–0.3 points in several settings. Based on these small differences, the authors conclude that neuron position does not strongly correlate with fine-tuning effectiveness and that selection location has only a minor influence. I believe this conclusion would be more convincing if multiple runs were included.

  2. There appears to be an incorrectly calculated average in Table 12. For Prefix Tuning on the math task, the reported benchmark scores are 85.1, 48.8, 20.3, and 19.0. Their average should be 43.3 rather than the reported 41.8. Moreover, in Section 4.6, this value is used to support the claim that Prefix Tuning underperforms FFT (42.2). Please verify this number and the corresponding conclusion.

  3. Please refer to the major weakness.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you very much for your insightful suggestions, which have greatly helped improve the quality of our manuscript. We have revised the manuscript in accordance with your comments. To help you quickly understand the revisions, we provide a brief summary below. Further details can be found in the revised manuscript, where all changes are highlighted in blue for your convenience.

Q1. There appears to be an incorrectly calculated average in Table 12. For Prefix Tuning on the math task, the reported benchmark scores are 85.1, 48.8, 20.3, and 19.0. Their average should be 43.3 rather than the reported 41.8. Moreover, in Section 4.6, this value is used to support the claim that Prefix Tuning underperforms FFT (42.2). Please verify this number and the corresponding conclusion.

A1. Thank you for catching this. We have corrected the error and revised the conclusion in §4.6. The sentence "BitFit (41.3) and Prefix Tuning (41.8) both fall below FFT (42.2)" now states that only BitFit (41.3) falls below FFT (42.2). We also corrected the reference to the strongest PEFT method from Pfeiffer Adapter to Prefix Tuning, which outperforms FFT by 1.1 points. Finally, we have updated Table 4 and redrawn the Prefix Tuning bar to show 43.3.

Q2. The paper shows that the optimal neuron-update ratio varies across methods and tasks. However, adaptive methods such as NGFT and NMKE also use different neuron-selection criteria from the fixed-ratio methods. Therefore, it is difficult to determine whether their advantage comes from adaptive ratio selection or from the selection criterion itself.

A2. We thank the reviewer for highlighting this important confound. To disentangle the adaptive ratio from the selection criterion, we have added a controlled experiment in Appendix H (Table 13). We hold NGFT’s criterion fixed, rank all positively activated neurons by CV in ascending order, and retain the lowest-CV ρ%, with ρ ∈ {1, 3, 5, 7, 8.2, 10}. The experiment uses Llama3.1-8B on math, where NGFT adaptively selects 8.2%. The results show that both components contribute.

(1) The criterion alone is insufficient: performance varies by 4.0 points across ratios (40.9 at 1% versus 44.9 at 8.2%), and a poorly chosen ratio underperforms FFT (42.2).

(2) The adaptive ratio is effective: the self-selected 8.2% achieves the highest Math score (44.9) in the sweep without manual search. A well-chosen fixed ratio comes close (44.8 at 5%), suggesting that the adaptive ratio’s main benefit is avoiding per-task tuning rather than substantially outperforming a tuned ratio.

Thus, the criterion determines which neurons to update, while the adaptive ratio determines how many; together, they achieve the best observed performance. We have revised Section 4.3 and added Appendix H accordingly.

Q3. The update-location ablation shows only 0.1–0.3-point differences among FFN-only, MHA-only, and MHA+FFN in several settings. Multiple runs are needed to support the conclusion that selection location has little effect on fine-tuning performance.

A3. Thank you for raising this important concern. We have already completed three independent training runs of the update-location ablation on Qwen2.5-Math-7B, and we report the standard deviations in the table below (also added to Table 16 of the revised paper). We do observe some fluctuation in the results introduced by retraining the base model, but so far the gaps among the three locations remain small and within the range of the standard deviations, so our current conclusion still holds; that said, we prefer to state it with caution.

Since these experiments involve extensive base-model training, we need additional time to complete the remaining runs (including Llama3.1-8B), which we expect within one to two days. This also gives us an opportunity to further analyze whether there are other experimental phenomena we have not previously observed. Once completed, we will immediately incorporate the results and any new findings into the final version.

ModelMethodGSM8KMATH500MinervaOlympiadMMLU (P)HellaSwagARCTyDiQA
Qwen2.5-Math-7BNCFT (FFN)88.6 ± 0.166.8 ± 0.229.4 ± 0.036.3 ± 0.142.2 ± 0.365.7 ± 0.251.0 ± 0.138.1 ± 0.3
NCFT (MHA+FFN)88.4 ± 0.166.7 ± 0.229.4 ± 0.236.0 ± 0.142.5 ± 0.265.1 ± 0.251.7 ± 0.337.8 ± 0.4
NCFT (MHA)88.5 ± 0.466.7 ± 0.229.4 ± 0.236.1 ± 0.341.9 ± 0.666.2 ± 0.451.2 ± 0.538.4 ± 0.4
NGFT (FFN)89.2 ± 0.168.0 ± 0.229.8 ± 0.037.3 ± 0.242.0 ± 0.265.4 ± 0.250.2 ± 0.146.1 ± 0.2
NGFT (MHA+FFN)89.0 ± 0.167.9 ± 0.229.4 ± 0.237.0 ± 0.142.6 ± 0.365.9 ± 0.249.8 ± 0.146.7 ± 0.3
NGFT (MHA)89.1 ± 0.267.7 ± 0.229.4 ± 0.437.2 ± 0.442.4 ± 0.564.7 ± 0.451.0 ± 0.245.8 ± 0.5
Reviewer_GZL3OpenReview ↗

Response to authors

Thank you for the detailed response and additional experiments. The revisions address my concerns, and I appreciate the authors’ effort to improve the paper. The question numbering in the response differs from my original review, but this does not affect my assessment. I maintain my rating of 3 (weak acceptance).

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your reply!

Additional experiments and results:

We provide the additional experiments promised in our previous response. We have now completed four independent runs of the Llama 3.1 8B model on the mathematical reasoning task and added the mean and standard deviation to Table 16. The results show that variability can depend on the update location. In particular, MHA-only configurations exhibit higher standard deviations in some cases: for NGFT, the standard deviation on Minerva reaches 0.8, compared with maximum standard deviations of 0.3 for FFN-only and 0.5 for MHA+FFN configurations. We have added this observation to the table caption and interpret it as evidence that MHA-only updates can be less stable on certain benchmarks.

We have incorporated all revisions into the latest version of the manuscript. We hope this finding further enriches your understanding of sparse fine-tuning. We sincerely thank you again for your support.

ModelMethodGSM8KMATH500MinervaOlympiadMMLU (P)HellaSwagARCTyDiQA
Qwen2.5-Math-7BNCFT (FFN)88.6 ± 0.166.8 ± 0.229.4 ± 0.036.3 ± 0.142.2 ± 0.365.7 ± 0.251.0 ± 0.138.1 ± 0.3
NCFT (MHA+FFN)88.4 ± 0.166.7 ± 0.229.4 ± 0.236.0 ± 0.142.5 ± 0.265.1 ± 0.251.7 ± 0.337.8 ± 0.4
NCFT (MHA)88.5 ± 0.466.7 ± 0.229.4 ± 0.236.1 ± 0.341.9 ± 0.666.2 ± 0.451.2 ± 0.538.4 ± 0.4
NGFT (FFN)89.2 ± 0.168.0 ± 0.229.8 ± 0.037.3 ± 0.242.0 ± 0.265.4 ± 0.250.2 ± 0.146.1 ± 0.2
NGFT (MHA+FFN)89.0 ± 0.167.9 ± 0.229.4 ± 0.237.0 ± 0.142.6 ± 0.365.9 ± 0.249.8 ± 0.146.7 ± 0.3
NGFT (MHA)89.1 ± 0.267.7 ± 0.229.4 ± 0.437.2 ± 0.442.4 ± 0.564.7 ± 0.451.0 ± 0.245.8 ± 0.5
Llama3.1-8BNCFT (FFN)85.7 ± 0.250.0 ± 0.222.4 ± 0.018.7 ± 0.144.1 ± 0.278.8 ± 0.149.3 ± 0.260.1 ± 0.1
NCFT (MHA+FFN)85.9 ± 0.350.3 ± 0.022.1 ± 0.018.7 ± 0.143.5 ± 0.379.4 ± 0.149.0 ± 0.160.8 ± 0.1
NCFT (MHA)85.7 ± 0.349.7 ± 0.222.1 ± 0.018.2 ± 0.244.8 ± 0.578.3 ± 0.549.9 ± 0.259.4 ± 0.5
NGFT (FFN)84.6 ± 0.252.4 ± 0.223.5 ± 0.218.9 ± 0.243.0 ± 0.378.6 ± 0.250.2 ± 0.357.6 ± 0.2
NGFT (MHA+FFN)84.5 ± 0.252.6 ± 0.223.5 ± 0.418.9 ± 0.243.5 ± 0.577.8 ± 0.150.1 ± 0.158.0 ± 0.3
NGFT (MHA)84.9 ± 0.252.3 ± 0.423.5 ± 0.818.7 ± 0.142.6 ± 0.578.1 ± 0.450.5 ± 0.557.0 ± 0.7
Reviewer_M2XrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors conduct an analysis of various sparse finetuning methods to draw unified insights by analyzing multiple methods and comparing them against the full finetuning baseline. Their key findings conclude that sparse finetuning is always better at preventing catastrophic forgetting while improving performance on the target task via finetuning. Even when compared to PEFT methods like LoRA, their experiments show improved performance in comparison. Experiments are conducted on math and coding tasks for language models up to size 8B.

The paper provides a consolidated understanding of sparse finetuning methods, however, I have some concerns as listed in the weaknesses. I would be happy to raise my score if those concerns are adequately addressed.

Critical strengths
  1. The authors attempt at unifying the wide range of existing sparse finetuning methods under one framework and studying their gains is a valuable and useful effort for the community.
  2. The insights about static vs dynamic masking, sparse finetuning being better than PEFT make a strong case for preferring sparse finetuning methods over others.
  3. The ablation in Sec 4.3 is particularly useful, which suggests carefully tuning the sparsity ratio per task to maximize benefits (and it holds true across methods and model sizes).
Critical weaknesses
  1. I am not sure how the eight methods for this study (Table 1) were chosen. Is there a reason this subset of methods was chosen and does each method represent a broader class of methods? It seems a bit ad hoc how these methods have been chosen for the study without any explanation. It also misses some work which might be relevant in this context, for example [1] and [2] among others. If the authors can provide a motivation for choosing the subset of individual methods and how they represent the broader work in the area, it would help motivate the study.
  2. The conjecture in Sec 4.3 states that general models react to S2FT differently from domain specialised models, but there is no analysis provided to back this claim. The only evidence is the performance on one task in Table 3. It will help to add evidence about how many neurons have low activations and which ones are trained and so on to strengthen this claim.
  3. The analysis in Figure 3 about mask regeneration is difficult to follow. In what cases does a dynamic mask have a positive effect of performance and is the recurring overhead due to the dynamic mask a real bottleneck. This overhead vs performance plot would be a better alternative to consider to make claims regarding this comparison.
  4. The insights in Sec 4.4 that update neurons need not be restricted to a specific part of the model (MHA or FFN) are not new and have been studied before (also in [2]). But I appreciate that the authors want to consolidate this insight in their analysis. To this end, it will help to look at the contribution of specific layers as well instead of just MHA vs FFN. Additionally it seems the finetuning only FFN in some cases seems to help more, does this mean that we should always choose finetuning the FFN part, it is not very clear?
  5. Random masks have been shown to be quite effective in [1] and [2] but the authors claim the opposite in sec 4.7, could they explain or provide experiments to clarify this? Additionally in sec 4.7 it is not clear how sparse finetuning behaves as a regularizer and why?

[1] Modoranu, Ionut-Vlad, et al. "Microadam: Accurate adaptive optimization with low space overhead and provable convergence." NeurIPS 2024.

[2] Zhou, Chao, et al. "Pay attention to small weights." NeurIPS 2025.

Questions for authors

I have included the questions in the weaknesses, please see above.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you very much for your insightful suggestions, which have greatly helped improve the quality of our manuscript. We have revised the manuscript accordingly and conducted a number of additional experiments. To help you quickly grasp the revisions, we provide only a brief summary below, rather than elaborating on all experimental details and result tables one by one. Further details can be found in the revised manuscript, where all changes are highlighted in blue for your convenience.

W1. The selection of the eight methods in Table 1 seems ad hoc. Please justify the selection criteria, clarify whether each method represents a broader class of approaches, and address relevant work that appears to be missing (e.g., [1], [2]).

A1. Thank you for the question. We agree that the rationale for selecting the methods should be clearly explained. We group the methods in Table 1 into four categories according to their neuron-selection criteria: activation-based, gradient-based, weight-based, and composite methods that combine two criteria. Our selection aims to cover influential and representative approaches from each category.

Regarding the two studies you mentioned, [1] targets memory-efficient training via gradient sparsification, but its error-compensation mechanism updates a large number of neurons, diverging from the sparse fine-tuning setting of this work. In contrast, NanoAdam in [2] is directly relevant to sparse fine-tuning, and we have added experiments and analysis of it in Sections 4.1, 4.2, and 4.5.

Our evaluation shows that: (1) NanoAdam is competitive overall; (2) like S²FT, it primarily updates under-utilized neurons and tends to achieve larger gains when fine-tuning a base model than when adapting a model already specialised for a particular task; and (3) NanoAdam also uses dynamically generated masks. Consistent with the trend observed for other methods, replacing its dynamic mask with a static one reduces the number of neurons updated, resulting in a modest decline in task performance alongside a slight improvement in general capability.

W2. The conjecture in Sec 4.3 states that general models react to S2FT differently from domain specialised models, but there is no analysis provided to back this claim. The only evidence is the performance on one task in Table 3. It will help to add evidence about how many neurons have low activations and which ones are trained and so on to strengthen this claim.

A2. Thank you for catching this. To address this concern, we conducted additional experiments and analyses, which are now included in Section 4.3 and the Appendix D.

1. Comparison with a general-purpose base model. We evaluated Qwen2.5-7B on the mathematics task and compared its results with those of the math-specialised Qwen2.5-Math-7B. S²FT improves the Math average over the base model by 12.3 points on Qwen2.5-7B, compared with 9.2 points on Qwen2.5-Math-7B. The difference is particularly pronounced on Minerva, where S²FT yields a 12.9-point gain on Qwen2.5-7B, compared with 1.9 points on Qwen2.5-Math-7B. These results support the hypothesis that low-activation neurons provide more available adaptation capacity in the general-purpose model.

2. Analysis of low-activation neurons. We compared the low-activation neurons identified in Qwen2.5-7B and Qwen2.5-Math-7B. The overlap is only 50.2%, indicating that many neurons with low activation in the general-purpose model are no longer low-activation in the specialised model. In addition, the mean activation of Qwen2.5-Math-7B is 24% higher than that of Qwen2.5-7B. Together, these findings suggest that domain specialisation may activate some neurons that are less utilised in the general-purpose model, providing internal evidence consistent with our conjecture.

[1] Modoranu, Ionut-Vlad, et al. "Microadam: Accurate adaptive optimization with low space overhead and provable convergence." NeurIPS 2024.

[2] Zhou, Chao, et al. "Pay attention to small weights." NeurIPS 2025.

AuthorsOpenReview ↗

Official Comment by Authors

W3. The analysis in Figure 3 about mask regeneration is difficult to follow. In what cases does a dynamic mask have a positive effect of performance and is the recurring overhead due to the dynamic mask a real bottleneck. This overhead vs performance plot would be a better alternative to consider to make claims regarding this comparison.

A3. We thank the reviewer for this helpful suggestion. We have revised Section 4.5 and redesigned Figure 3 to make the performance–overhead trade-off easier to interpret. The updated figure directly compares each method’s task-performance and general-capability changes with the relative training-time change when switching between static and dynamic masks.

The results show that dynamic masks do not provide a uniform benefit. Across the originally static methods, regeneration improves task performance by an average of 0.2 points, with gains ranging from 0.0 to 0.4 points across methods and tasks. It also reduces or leaves unchanged general capability, with a maximum reduction of 0.7 points. Conversely, freezing the masks of the originally dynamic methods lowers task performance by 0.1–0.7 points while improving general capability by 0.1–1.2 points. Dynamic regeneration increases training time by 12%–33% (20.2% on average), with the largest increase for NCFT; freezing masks reduces training time by 9% for GMT, 21% for NMKE, and 10% for NanoAdam.

We have clarified that this overhead is measurable but should not be interpreted as a universal bottleneck: its practical importance depends on whether the modest task-performance gains justify the additional training time and potential loss in general capability. The revised figure and discussion present these trade-offs directly, allowing readers to assess the benefit of dynamic masks in each setting (Section 4.5).

W4. The insight in Sec. 4.4 that updated neurons need not be restricted to MHA or FFN is not new (also studied in [1]). To strengthen the analysis, please examine the contribution of specific layers rather than only MHA vs. FFN. Moreover, FFN-only finetuning sometimes performs better; does this imply FFN should always be preferred? This needs clarification.

A4. Thank you for recognising the value of consolidating this finding and for suggesting a layer-wise analysis. We have revised Section 4.4 and added the analysis in Appendix E to clarify both the distribution of selected neurons and our practical recommendation.

(1) Motivation for the MHA–FFN comparison. Our aim is to resolve a practical discrepancy across existing methods: as Table 1 shows, different methods select neurons from different modules, making it unclear whether the selection location materially affects their performance. We therefore compare MHA-only, FFN-only, and MHA+FFN selection under controlled settings across methods. While [1] selects neurons from both MHA and FFN, it does not provide a controlled ablation comparing these three choices. Our aim is to isolate this design choice through controlled comparisons, rather than claim that applying sparse updates to different modules is itself novel.

(2) Additional layer-wise analysis. Following your suggestion, we have added a layer-wise analysis in Appendix E for NCFT, NGFT, and S²FT on mathematical reasoning and code generation. Two key findings emerge: (i) all three methods concentrate their selections in early and middle layers, although their allocation across individual layers differs; (ii) NCFT and NGFT exhibit similar layer-wise distributions despite their different scoring criteria, including a pronounced resurgence in the final layer that is absent in S²FT. These broad patterns are also similar across the two tasks.

(3) Clarifying the recommendation for FFN-only selection. Section 4.4 now explicitly identifies the approximate performance ordering of FFN > MHA+FFN > MHA. Accordingly, we recommend prioritising FFN neurons as an effective default for sparse fine-tuning.

[1] Zhou, Chao, et al. "Pay attention to small weights." NeurIPS 2025.

AuthorsOpenReview ↗

Official Comment by Authors

W5. Random masks have been shown to be quite effective in [1] and [2] but the authors claim the opposite in sec 4.7, could they explain or provide experiments to clarify this? Additionally in sec 4.7 it is not clear how sparse finetuning behaves as a regularizer and why?

A5. Thank you for highlighting this ambiguity. We agree that random masks can be effective. The latter part of section 4.7 aims to distinguish the benefits of restricting parameter updates from the additional gains of informed neuron selection.

(1) Random-mask experiments (Appendix F) on Llama3.1-8B use three independent seeds with the same 10% neuron-update ratio. Random masks achieve mathematical reasoning scores of 42.0–42.9, comparable to FFT’s 42.2, while retaining substantially higher general capability scores (57.1–57.2 versus 44.4). Nevertheless, most criterion-guided methods outperform all three random masks on mathematical reasoning. This aligns with [1] and [2], where small-weight and task-specific neuron selection, respectively, outperform random selection. Thus, random masks provide an effective baseline, while suitable selection criteria can further improve task adaptation.

(2) By “regulariser,” we mean a constraint that helps preserve pre-trained capabilities: freezing unselected parameters reduces adaptation degrees of freedom and may limit interference with existing knowledge. This interpretation is already highlighted in Section 4.7: "the data points of sparse fine-tuning all lie close to the ΔGen = 0 baseline, whereas the FFT stars lie far from it." Freezing unselected parameters restricts adaptation to a smaller parameter subspace and may reduce interference with previously acquired knowledge.

[1] Zhou, Chao, et al. “Pay attention to small weights.” NeurIPS 2025.

[2] Yongqi Leng et al. “Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neurons.” COLING 2025.


Thank you for your time and thoughtful feedback. If the response can alleviate your concerns and promote your positive view of the paper, we would appreciate it if you could strengthen the recommendation. Thank you again for your support!

Reviewer_GxKtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper compares eight existing sparse fine-tuning methods for LLMs on mathematical reasoning and code generation tasks. They also study how neuron selection, update ratio, and masking strategies affect performance and catastrophic forgetting. Their results show that sparse fine-tuning can improve task performance and preserve the general capabilities of the pretrained models.

Critical strengths

I think comparing several sparse fine-tuning methods under the same experimental setup is useful. The paper also provides some interesting observations about neuron selection and catastrophic forgetting.

Critical weaknesses
  1. The paper mainly compares existing methods, and I am not really sure about the novelty of the work.
  2. The comparison with LoRA and other PEFT methods is limited to one model. I would like to see a broader comparison across the different models used in the paper.
Questions for authors
  1. Could the authors clarify what the main new contribution of this work is beyond comparing existing methods?
  2. Could the authors provide more comparisons with LoRA and other PEFT methods across different models and tasks?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you very much for your insightful suggestions, which have greatly helped improve the quality of our manuscript. We have revised the manuscript in accordance with your comments. To help you quickly understand the revisions, we provide a brief summary below. Further details can be found in the revised manuscript, where all changes are highlighted in blue for your convenience.


W1 & Q1. Could the authors clarify what the main new contribution of this work is beyond comparing existing methods?

A1. Thank you for the question. We appreciate the opportunity to clarify our contribution. Our study goes beyond comparing existing baselines and has three objectives: (1) to establish a performance leaderboard under unified experimental conditions, helping researchers evaluate and select sparse fine-tuning methods; (2) to disentangle the effects of design choices that differ across existing methods, including mask schedules, update locations, and neuron-update ratios; and (3) to uncover empirical insights that are difficult to obtain from individual method studies. Specifically, our contributions are:

1. Unified leaderboard(§4.1, 4.2). We evaluate nine representative sparse fine-tuning methods across model families and scales on mathematical reasoning and code generation, jointly measuring task performance and general-capability retention to support method evaluation and selection.

2. Controlled component analyses (§4.3–4.5). We find that: (1) selection location (MHA vs. FFN) has a relatively small effect; (2) the optimal neuron-update ratio depends on the method and task, motivating adaptive selection; and (3) dynamic masks yield modest task gains at the cost of general-capability retention and recurring computational overhead.

3. Further insights (§4.2, 4.7 and Appendices D–F).  (1) Sparse fine-tuning’s advantage over FFT tends to grow with base-model task capability. (2) Our results suggest that highly activated neurons are more useful targets in stronger, domain-specialised models, while low-activation or small-magnitude parameters can be more competitive in less task-specialised models. (3) We propose a unifying interpretation of sparse masks as both capacity allocators and regularisers, supported by random-mask controls that help distinguish the effects of sparsity itself from those of neuron selection.

W2 & Q2. Could the authors provide more comparisons with LoRA and other PEFT methods across different models and tasks?

A2. We have extended our comparative experiments with LoRA, BitFit, prefix tuning, and Pfeiffer adapters to mathematical reasoning with Qwen2.5-Math-7B and code generation with Qwen2.5-Coder-3B. The additional results, comparison plots, and analyses are presented in Appendix G, with corresponding references added to Section 4.6.

(1) On both Qwen models, LoRA is the best-performing PEFT baseline, achieving average scores of 53.6 on the mathematical reasoning tasks and 45.1 on the code generation tasks.

(2) On the mathematical reasoning tasks, NeFT achieves a score of 55.9, outperforming LoRA by 2.3 points while maintaining comparable general capabilities (52.0 vs. LoRA 52.4).

(3) On the code generation tasks, NGFT achieves a score of 50.8, surpassing LoRA by 5.7 points and attaining a higher general-capability score (54.6 vs. LoRA 53.4).

Together with the results on Llama3.1-8B, these additional comparisons across model families and tasks provide consistent evidence that sparse fine-tuning can yield greater task-performance improvements than the PEFT methods evaluated in this study while preserving comparable general capabilities.


Thank you for your time and thoughtful feedback. If the response can alleviate your concerns and promote your positive view of the paper, we would appreciate it if you could strengthen the recommendation. Thank you again for your support!

AuthorsOpenReview ↗

Official Comment by Authors

Additional experiments and results:

Prior work has shown that PEFT methods are particularly effective in low-resource settings [1,2]. To investigate this issue further, we conducted an additional experiment on BioInstruct [3]. BioInstruct is a biomedical question-answering dataset with 25,000 samples. We used 20,000 samples for training and 5,000 for testing. All experiments used Llama3.1-8B.

The advantage of sparse fine-tuning over PEFT is much smaller in this setting. LoRA achieves the highest score among the PEFT methods, with a BioInstruct score of 40.5. NGFT performs best among the sparse methods, with a score of 41.4. This corresponds to a gain of only 0.9 points. NeFT performs slightly worse than LoRA (40.3 vs 40.5). In contrast, sparse methods outperform LoRA by up to 2.6 points on Llama3.1-8B in the large-scale setting (Sec 4.6). These results show that sparse fine-tuning remains competitive with PEFT on small-scale data, but its performance advantage is more modest than that observed with large-scale data.

ModelMethodBioInstructGeneral capability
Llama3.1-8BMMLU (P)HellaSwagARCTyDiQA
LoRA40.541.374.747.656.2
BitFit38.741.175.947.455.9
Prefix Tuning39.240.674.148.254.1
Pfeiffer Adapter40.339.874.446.354.8
NCFT41.140.575.248.554.6
NEFT40.341.075.848.952.2
NGFT41.440.975.049.453.9
NMKE41.240.774.947.755.0

We hope this new comparison experiment addresses your concern and deepens your understanding of sparse fine-tuning's performance across different data scales. As the discussion period is coming to an end, we would kindly invite you to share any updated thoughts.

[1] "LoRA Learns Less and Forgets Less", Transactions on Machine Learning Research, 2024.

[2] "When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method", ICLR 2024.

[3] "BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing", Journal of the American Medical Informatics Association, 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Overview and Clarified Contributions

Dear SACs, ACs, and Reviewers,

We sincerely thank you for your time and for your expert reviews. Your constructive feedback has helped us further improve the manuscript. We are grateful that all reviewers recognised the value to the community of bringing diverse sparse fine-tuning methods into a unified evaluation framework and systematically comparing them across multiple models and tasks under identical experimental settings. We also appreciate your recognition of the extensive experimental effort and the well-supported conclusions drawn from comparative analyses spanning multiple model families, tasks, and benchmarks.


Research Motivation

Compared with extensively studied fine-tuning paradigms such as FFT and PEFT, sparse fine-tuning, despite the proliferation of methods, still lacks a systematic, paradigm-level understanding of when and why it works. Existing methods differ in their neuron-selection criteria, selection locations, mask-construction schedules, and neuron-update ratios. Some studies even reach seemingly contradictory conclusions, yet their methods all outperform FFT. The community therefore needs fair, systematic comparisons, ablations, and analyses to clarify the design space and underlying mechanisms of sparse fine-tuning.


Contributions and Findings

(1) The first unified performance leaderboard for sparse fine-tuning. We evaluate nine representative methods spanning four broad categories of neuron-selection criteria: activation-based, gradient-based, weight-based, and composite criteria. All methods are compared under identical experimental conditions, providing a reference for both practitioners and researchers working on sparse fine-tuning.

(2) Controlled experiments that disentangle key design choices. Controlled ablations of nine sparse fine-tuning methods reveal that: (i) optimal neuron-update ratios vary by method and task, motivating adaptive selection; (ii) selection location has a minor effect, with a consistent ranking of FFN > MHA+FFN > MHA; and (iii) dynamic mask regeneration yields an average task gain of only 0.2 points, while reducing general capability by up to 0.7 points and increasing training time by 12%–33%.

(3) New empirical insights that are difficult to obtain from studies of individual methods. We find that: (i) the advantage of sparse fine-tuning over FFT grows with the base model’s task capability; and (ii) in stronger, domain-specialised models, high-activation neurons are more effective update targets, whereas in less task-specialised models, low-activation neurons or low-magnitude parameters are more competitive.

(4) The first mechanistic account of why sparse fine-tuning works. We characterise the two complementary roles of the sparse mask: as a capacity allocator, it concentrates the learning budget on task-critical neurons; as a regulariser, it pins the remaining parameters to their pre-trained values. We further include random-mask control experiments to distinguish the effects of the sparse update constraint itself from those of the neuron-selection criterion.


Reviewer Concerns and Revisions

(1) Clarity of contributions and rationale for method selection (Reviewers GxKt and M2Xr). We have clarified the three core contributions: a unified performance leaderboard, controlled ablations of key design components, and mechanistic insights. We have also explained the rationale for method selection in terms of the four broad categories of neuron-selection criteria and added experiments and analyses for NanoAdam, a recent baseline closely related to our study.

(2) Broader comparisons with PEFT baselines (Reviewer GxKt). We have expanded the comparisons to include LoRA, BitFit, Prefix Tuning, and Pfeiffer Adapter, with experiments on Qwen2.5-Math-7B and Qwen2.5-Coder-3B (Appendix G). We have also added an empirical analysis under limited-data settings.

(3) Stronger mechanistic and empirical evidence (Reviewer M2Xr). We have added an analysis of differences in neuron activations between general-purpose and domain-specialised models (Appendix D) and revised the figure illustrating the performance–overhead trade-off of dynamic mask regeneration (Section 4.4).

(4) Statistical reliability and numerical corrections (Reviewer GZL3). We have added standard deviations across repeated runs, which also revealed instability when updating MHA alone (Appendix I). In addition, we have introduced a controlled experiment to disentangle the contribution of the adaptive neuron-update ratio from that of the selection criterion (Appendix H).


We have responded to every review comment individually and incorporated your valuable suggestions into the revised manuscript. All revisions are highlighted in blue. We once again thank the SACs, ACs, and reviewers for their valuable feedback and support.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.