- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Mask Fine-Tuning (MFT), a post-fine-tuning approach that improves the performance of well-trained large language models by selectively removing model parameters rather than updating their weights. Specifically, MFT freezes the parameters of a fully fine-tuned model and learns a binary mask using the original fine-tuning objective and dataset. The mask is optimized through a straight-through gradient estimator to identify parameters whose removal can improve downstream task performance.
Experiments across multiple LLM backbones and domains, including mathematical reasoning, code generation, and instruction following, demonstrate consistent improvements over fully fine-tuned models. The authors further investigate the effects of masking ratios, layer selection, and loss landscapes to analyze the effectiveness of the proposed approach.
Critical strengths
S1. An interesting perspective on LLM optimization beyond parameter updates and model scaling.
The paper presents an interesting perspective on improving modern LLMs by shifting the focus from conventional optimization strategies, such as additional training and model scaling, toward selectively removing existing parameters. Rather than assuming that all learned parameters contribute positively to model performance, the proposed approach explores whether eliminating certain parameters can further enhance a well-trained model. This perspective provides a meaningful alternative direction for investigating LLM optimization.
S2. Consistent empirical improvements across different models and downstream tasks.
The experimental results demonstrate consistent improvements over fully fine-tuned models across multiple backbones and downstream domains. In particular, the comparisons with random and magnitude-based masking suggest that the observed improvements are not simply attributable to parameter removal, but depend on learning an appropriate mask.
Critical weaknesses
W1. Lack of analysis on general capability preservation and potential task-specific specialization.
The paper emphasizes that MFT differs from conventional pruning by focusing on performance improvement rather than model compression. Specifically, the authors argue that selectively removing parameters that are irrelevant or potentially detrimental to model performance can further enhance a well-trained LLM.
However, the experiments primarily evaluate improvements on the target tasks used for mask optimization, without sufficiently investigating how the learned masks affect the model’s general capabilities.
Since LLM parameters are shared across different capabilities, removing parameters to improve a specific task may negatively affect other capabilities. Therefore, the reported improvements do not necessarily indicate an overall enhancement of the model. Instead, the proposed approach may primarily achieve task-specific specialization at the expense of general capability preservation.
Without comprehensive cross-domain evaluations, it remains unclear whether MFT genuinely improves the overall capabilities of LLMs or simply redistributes their existing capabilities toward the target tasks.
W2. Limited evaluation on recent LLM architectures and questionable model characterization.
Although the paper aims to introduce a general optimization paradigm for modern LLMs, its experiments rely on a relatively limited selection of model families, including older architectures such as LLaMA2-7B.
While the inclusion of LLaMA3.1 and Qwen3 provides some architectural diversity, the experimental evidence remains insufficient to establish whether the proposed approach generalizes to more recent and advanced LLMs.
Additionally, the paper repeatedly characterizes its target models as “GPT-like” language models. This terminology is somewhat broad and imprecise, given that the experiments specifically focus on decoder-only Transformer architectures. A more precise characterization of the supported model architectures would improve the technical clarity of the paper.
Questions for authors
Q1. What is the fundamental technical distinction between MFT and existing pruning approaches?
The authors emphasize that the primary distinction between MFT and conventional pruning lies in their optimization objectives: pruning focuses on model compression, whereas MFT aims to improve model performance.
However, both approaches involve identifying and removing model parameters through masking. Moreover, existing pruning methods can also employ task-specific objectives and learnable masks.
Could the authors clarify what fundamentally distinguishes MFT from existing learnable masking and pruning approaches at the methodological level, beyond the difference in intended objectives?
In particular, it would be helpful to explain whether MFT introduces a distinct optimization mechanism or primarily represents a different application of existing masking techniques.
Q2. How can the authors demonstrate that MFT achieves general capability improvement rather than task-specific specialization?
The reported results demonstrate that MFT improves performance on the target domains used for mask optimization. However, these improvements alone do not establish whether the model’s general capabilities are preserved.
Could the authors provide cross-domain evaluations to investigate whether masks optimized for one task negatively affect performance on other tasks?
For example, evaluating a math-optimized mask on coding, instruction following, and general reasoning benchmarks would help determine whether MFT produces genuine capability improvements or merely shifts the model’s performance toward a specific domain.
Such evaluations would be particularly important for supporting the paper’s broader claim that selectively removing parameters can enhance well-trained LLMs.
5 comments
3. Methodological distinction from learnable masking and pruning
MFT establishes a subsequent optimization stage over the weight support of a sufficiently fine-tuned checkpoint. Its methodological contribution is the post-FFT refinement protocol and the demonstrated ability to improve that trained checkpoint while preserving all of its weight values. Specifically, MFT tests whether changing the retained weight support can further improve performance after weight optimization has already produced a competitive FFT checkpoint.
The distinction from representative learned masking studies becomes clearer. For example, Zhou et al.'s learned supermasks operate on fixed random initializations, while their lottery ticket analyses also examine resetting and retraining pruned networks. Ramanujan et al. use learned scores, top-k selection, and STE to identify effective subnetworks in randomly weighted networks. Wortsman et al. learn task-specific masks over a fixed random backbone for continual learning and task identification. Compared to ours, these settings explore different starting points and learning problems.
Figure 2 illustrates the additional optimization stage. Figure 1 and Table 1 evaluate its value by comparing it with Best FFT, continued FFT, random masking, and L1 masking. These comparisons establish that the reported gains depend on the learned support and can exceed the performance of the starting trained checkpoint. They provide the empirical basis for treating mask optimization as a further stage of domain-specific LLM refinement.
Conclusion
We hope these clarifications and additional experiments address the reviewer’s concerns about model coverage, cross-domain behavior, and the methodological contribution. We welcome further questions and look forward to continuing the discussion.
We thank the reviewer again for the careful assessment. We would appreciate your feedback on our rebuttal and revised manuscript, particularly the additional Gemma-3-1B-IT experiments and ARC-Challenge and HellaSwag evaluations. Could you let us know whether these additions and our methodological clarifications address your concerns? We welcome the opportunity to address any remaining questions during the discussion. If our responses resolve your concerns, we would be grateful if you could consider raising your score.
We thank the reviewer for the careful assessment and for highlighting MFT’s perspective on model optimization. Below, we address the weaknesses and questions, drawing on evidence from the submission and additional experiments on model coverage and general capability evaluation.
1. General capability preservation and domain-specific specialization
MFT is designed for domain-specific post-training improvement. It starts from a checkpoint that has already undergone sufficient FFT and learns a mask to further improve performance in a specific domain. The contribution is the additional target-domain improvement over Best FFT. Preserving performance across every non-target domain is outside this optimization objective.
For the specific example raised by the reviewer, Table 4 already evaluates the same mathematics-optimized mask on mathematics, coding, and instruction following. We reproduce the corresponding LLaMA3.1-8B results below.
We additionally evaluated scientific reasoning and commonsense performance using ARC-Challenge and HellaSwag. These experiments use the original LLaMA2-7B Best FFT and final MFT checkpoints for each domain. Each pair uses identical prompts, scoring procedures, and evaluation settings. Evaluations are zero-shot and use the complete evaluation sets.
These additional evaluations show that MFT retains almost the same Best FFT performance on the evaluated scientific reasoning and commonsense tasks. Together with the target-domain improvements, these results indicate that domain-specific mask optimization can further enhance the intended capability while substantially retaining the general capability.
2. Model coverage and architectural characterization
The original evaluation includes LLaMA2-7B, LLaMA3.1-8B, Qwen3-1.7B, and Qwen3-14B in Section 3.1 and Table 1. Appendix Table 7 further evaluates two multimodal configurations, LLaVA with CLIP and Qwen2.5-0.5B, and LLaVA with SigLIP and Gemma-2B. Both configurations improve over Best FFT across all six evaluated multimodal benchmarks. Appendix Table 8 additionally evaluates DeepSeek-Coder-1.3B on MBPP. Thus, the submitted evidence extends beyond the main LLaMA and Qwen language model experiments.
To expand this coverage further, we conduct an additional coding-domain experiment on Gemma-3-1B-IT, evaluated on HumanEval. The following table includes the preliminary layer-group ablations and the final MFT result.
Following the original experimental procedure, the preliminary layer-group comparisons use 10% of the coding training data. Layers 24 to 25 achieve the highest score in this comparison and are selected for final mask optimization using the full dataset. This group contains two layers because Gemma-3-1B-IT has 26 layers in total. Final MFT starts from Best FFT and freezes all model weights throughout optimization, improving HumanEval by 3.61 percentage points.
This experiment extends the evidence to a Gemma language model and demonstrates further improvement after domain-specific FFT, complementing the Gemma-based multimodal experiment already included in the appendix.
Regarding terminology, the language backbones studied here are autoregressive, decoder-only Transformer models. In Section 2, “GPT-like” refers to this autoregressive modeling setup. The mask operates on the linear mappings described in Equations 3 and 4.