acceptodds
Under review as a conference paper at ICLR 2027

Boosting Large Language Models with Mask Fine-Tuning

Abstract

The large language model (LLM) is usually kept integral in the mainstream optimization protocol. No works have questioned whether maintaining the integrity of the model is indispensable for promising performance. In this work, we introduce Mask Fine-Tuning (MFT), a brand-new LLM fine-tuning paradigm to show that properly breaking the structural integrity of the model can surprisingly lead to improved performance without model weights update. Specifically, MFT learns and applies a set of binary masks on well-optimized models supervised by the typical LLM fine-tuning objective. Based on full fine-tuned models, MFT uses the same fine-tuning datasets to gain consistent performance boosts across various domains and backbones (e.g., 2.60 / 4.15 average gain in IFEval with LLaMA2-7B / 3.1-8B). Detailed ablations and analyses study the proposed MFT from different perspectives such as sparse ratio, loss surface, etc. Additionally, MFT is compatible for collaborating with other LLM optimization procedures for general model enhancement by deploying it on well-trained models. Further, this study extends the functionality of masking operation from its conventional network pruning context for model compression into a general model capability scope.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparsity-Aware Training, Mask-based Fine-tuning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Gjk3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Mask Fine-Tuning (MFT), a post-fine-tuning approach that improves the performance of well-trained large language models by selectively removing model parameters rather than updating their weights. Specifically, MFT freezes the parameters of a fully fine-tuned model and learns a binary mask using the original fine-tuning objective and dataset. The mask is optimized through a straight-through gradient estimator to identify parameters whose removal can improve downstream task performance.

Experiments across multiple LLM backbones and domains, including mathematical reasoning, code generation, and instruction following, demonstrate consistent improvements over fully fine-tuned models. The authors further investigate the effects of masking ratios, layer selection, and loss landscapes to analyze the effectiveness of the proposed approach.

Critical strengths

S1. An interesting perspective on LLM optimization beyond parameter updates and model scaling.

The paper presents an interesting perspective on improving modern LLMs by shifting the focus from conventional optimization strategies, such as additional training and model scaling, toward selectively removing existing parameters. Rather than assuming that all learned parameters contribute positively to model performance, the proposed approach explores whether eliminating certain parameters can further enhance a well-trained model. This perspective provides a meaningful alternative direction for investigating LLM optimization.

S2. Consistent empirical improvements across different models and downstream tasks.

The experimental results demonstrate consistent improvements over fully fine-tuned models across multiple backbones and downstream domains. In particular, the comparisons with random and magnitude-based masking suggest that the observed improvements are not simply attributable to parameter removal, but depend on learning an appropriate mask.

Critical weaknesses

W1. Lack of analysis on general capability preservation and potential task-specific specialization.

The paper emphasizes that MFT differs from conventional pruning by focusing on performance improvement rather than model compression. Specifically, the authors argue that selectively removing parameters that are irrelevant or potentially detrimental to model performance can further enhance a well-trained LLM.

However, the experiments primarily evaluate improvements on the target tasks used for mask optimization, without sufficiently investigating how the learned masks affect the model’s general capabilities.

Since LLM parameters are shared across different capabilities, removing parameters to improve a specific task may negatively affect other capabilities. Therefore, the reported improvements do not necessarily indicate an overall enhancement of the model. Instead, the proposed approach may primarily achieve task-specific specialization at the expense of general capability preservation.

Without comprehensive cross-domain evaluations, it remains unclear whether MFT genuinely improves the overall capabilities of LLMs or simply redistributes their existing capabilities toward the target tasks.

W2. Limited evaluation on recent LLM architectures and questionable model characterization.

Although the paper aims to introduce a general optimization paradigm for modern LLMs, its experiments rely on a relatively limited selection of model families, including older architectures such as LLaMA2-7B.

While the inclusion of LLaMA3.1 and Qwen3 provides some architectural diversity, the experimental evidence remains insufficient to establish whether the proposed approach generalizes to more recent and advanced LLMs.

Additionally, the paper repeatedly characterizes its target models as “GPT-like” language models. This terminology is somewhat broad and imprecise, given that the experiments specifically focus on decoder-only Transformer architectures. A more precise characterization of the supported model architectures would improve the technical clarity of the paper.

Questions for authors

Q1. What is the fundamental technical distinction between MFT and existing pruning approaches?

The authors emphasize that the primary distinction between MFT and conventional pruning lies in their optimization objectives: pruning focuses on model compression, whereas MFT aims to improve model performance.

However, both approaches involve identifying and removing model parameters through masking. Moreover, existing pruning methods can also employ task-specific objectives and learnable masks.

Could the authors clarify what fundamentally distinguishes MFT from existing learnable masking and pruning approaches at the methodological level, beyond the difference in intended objectives?

In particular, it would be helpful to explain whether MFT introduces a distinct optimization mechanism or primarily represents a different application of existing masking techniques.

Q2. How can the authors demonstrate that MFT achieves general capability improvement rather than task-specific specialization?

The reported results demonstrate that MFT improves performance on the target domains used for mask optimization. However, these improvements alone do not establish whether the model’s general capabilities are preserved.

Could the authors provide cross-domain evaluations to investigate whether masks optimized for one task negatively affect performance on other tasks?

For example, evaluating a math-optimized mask on coding, instruction following, and general reasoning benchmarks would help determine whether MFT produces genuine capability improvements or merely shifts the model’s performance toward a specific domain.

Such evaluations would be particularly important for supporting the paper’s broader claim that selectively removing parameters can enhance well-trained LLMs.

5 comments

We thank the reviewer for the careful assessment and for highlighting MFT’s perspective on model optimization. Below, we address the weaknesses and questions, drawing on evidence from the submission and additional experiments on model coverage and general capability evaluation.

1. General capability preservation and domain-specific specialization

MFT is designed for domain-specific post-training improvement. It starts from a checkpoint that has already undergone sufficient FFT and learns a mask to further improve performance in a specific domain. The contribution is the additional target-domain improvement over Best FFT. Preserving performance across every non-target domain is outside this optimization objective.

For the specific example raised by the reviewer, Table 4 already evaluates the same mathematics-optimized mask on mathematics, coding, and instruction following. We reproduce the corresponding LLaMA3.1-8B results below.

Evaluation domainBenchmarkMathematics Best FFTMathematics MFT
MathematicsGSM8K77.077.3
MathematicsMath24.324.9
CodingHumanEval35.436.0
CodingHumanEval+29.932.2
Instruction followingIF-Eval31.531.1
Instruction followingAlpaca-Eval0.60.5

We additionally evaluated scientific reasoning and commonsense performance using ARC-Challenge and HellaSwag. These experiments use the original LLaMA2-7B Best FFT and final MFT checkpoints for each domain. Each pair uses identical prompts, scoring procedures, and evaluation settings. Evaluations are zero-shot and use the complete evaluation sets.

DomainModelARC-Challenge accARC-Challenge acc_normHellaSwag accHellaSwag acc_norm
-Pretrained42.7544.8857.0976.17
MathBest FFT43.7744.5458.6076.72
MathMFT43.6544.3258.5676.60
CodingBest FFT46.1648.6359.7678.38
CodingMFT45.7048.1759.5478.28
Instruction followingBest FFT46.2550.1759.7278.21
Instruction followingMFT46.0649.8559.0477.84

These additional evaluations show that MFT retains almost the same Best FFT performance on the evaluated scientific reasoning and commonsense tasks. Together with the target-domain improvements, these results indicate that domain-specific mask optimization can further enhance the intended capability while substantially retaining the general capability.

2. Model coverage and architectural characterization

The original evaluation includes LLaMA2-7B, LLaMA3.1-8B, Qwen3-1.7B, and Qwen3-14B in Section 3.1 and Table 1. Appendix Table 7 further evaluates two multimodal configurations, LLaVA with CLIP and Qwen2.5-0.5B, and LLaVA with SigLIP and Gemma-2B. Both configurations improve over Best FFT across all six evaluated multimodal benchmarks. Appendix Table 8 additionally evaluates DeepSeek-Coder-1.3B on MBPP. Thus, the submitted evidence extends beyond the main LLaMA and Qwen language model experiments.

To expand this coverage further, we conduct an additional coding-domain experiment on Gemma-3-1B-IT, evaluated on HumanEval. The following table includes the preliminary layer-group ablations and the final MFT result.

ConfigurationMasked layersData ratioHumanEval
Gemma-3-1B-IT--41.46
Best FFTNoneFull dataset56.71
MFT layer ablation0 to 310%48.78
MFT layer ablation4 to 710%48.78
MFT layer ablation8 to 1110%46.95
MFT layer ablation12 to 1510%46.34
MFT layer ablation16 to 1910%46.95
MFT layer ablation20 to 2310%53.66
MFT layer ablation24 to 2510%56.10
Final MFT24 to 25Full dataset60.32

Following the original experimental procedure, the preliminary layer-group comparisons use 10% of the coding training data. Layers 24 to 25 achieve the highest score in this comparison and are selected for final mask optimization using the full dataset. This group contains two layers because Gemma-3-1B-IT has 26 layers in total. Final MFT starts from Best FFT and freezes all model weights throughout optimization, improving HumanEval by 3.61 percentage points.

This experiment extends the evidence to a Gemma language model and demonstrates further improvement after domain-specific FFT, complementing the Gemma-based multimodal experiment already included in the appendix.

Regarding terminology, the language backbones studied here are autoregressive, decoder-only Transformer models. In Section 2, “GPT-like” refers to this autoregressive modeling setup. The mask operates on the linear mappings described in Equations 3 and 4.

3. Methodological distinction from learnable masking and pruning

MFT establishes a subsequent optimization stage over the weight support of a sufficiently fine-tuned checkpoint. Its methodological contribution is the post-FFT refinement protocol and the demonstrated ability to improve that trained checkpoint while preserving all of its weight values. Specifically, MFT tests whether changing the retained weight support can further improve performance after weight optimization has already produced a competitive FFT checkpoint.

The distinction from representative learned masking studies becomes clearer. For example, Zhou et al.'s learned supermasks operate on fixed random initializations, while their lottery ticket analyses also examine resetting and retraining pruned networks. Ramanujan et al. use learned scores, top-k selection, and STE to identify effective subnetworks in randomly weighted networks. Wortsman et al. learn task-specific masks over a fixed random backbone for continual learning and task identification. Compared to ours, these settings explore different starting points and learning problems.

Figure 2 illustrates the additional optimization stage. Figure 1 and Table 1 evaluate its value by comparing it with Best FFT, continued FFT, random masking, and L1 masking. These comparisons establish that the reported gains depend on the learned support and can exceed the performance of the starting trained checkpoint. They provide the empirical basis for treating mask optimization as a further stage of domain-specific LLM refinement.

Conclusion

We hope these clarifications and additional experiments address the reviewer’s concerns about model coverage, cross-domain behavior, and the methodological contribution. We welcome further questions and look forward to continuing the discussion.

We thank the reviewer again for the careful assessment. We would appreciate your feedback on our rebuttal and revised manuscript, particularly the additional Gemma-3-1B-IT experiments and ARC-Challenge and HellaSwag evaluations. Could you let us know whether these additions and our methodological clarifications address your concerns? We welcome the opportunity to address any remaining questions during the discussion. If our responses resolve your concerns, we would be grateful if you could consider raising your score.

Reviewer_neA1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MFT starts from an already fully fine-tuned model and continues training while learning a binary mask over the weights of few consecutive layers, using a straight-through estimator with a 10% intra-group sparsity target. Only unmasked weights receive updates, so MFT is a sparse continuation of FFT, which the paper describes as a structure-level post-training refinement whose effect it explains as loss-landscape flattening. A PAC-Bayes/description-length argument gives the masked model a lower test-loss bound.

Critical strengths
  1. The paper is well-established. The paper is well-structured, with clear figures and easy-to-understand wording that presents its methods. The methods are somewhat simple but effective.

  2. The results replicate across four backbones from two families. Random and L1 masking both start from Best FFT, and the model performance ends up below MFT. In addition, the Qwen3-14B and LLaMA3.1-8B IF-Eval gains of +6.5 and +6.0 on the benchmark where the base models already score highest, making the results significant.

  3. The proposed mechanism is measured thoroughly. The experiment reports lower training loss for MFT than for FFT across all three domains, providing direct evidence for the account rather than an appeal to it.

Critical weaknesses
  1. The best configuration requires choosing which four layers to mask. MFT requires specifically determining which layers to mask for different model backbones and different domains. In addition, larger groups such as 16 layers achieve “a significant performance drop.”

  2. The experiments do not separate mask learning from dense continued training of the same four layers. Random and L1 masks are not the actual ablations. For example, a random 10% subset is noisier than a dense update and can perform worse for reasons unrelated to mask quality. Without the dense four-layer run, the result is equally consistent with “updating four layers and freezing the rest helps, and the mask is incidental”. Therefore, an ablation study that uses the same token budget and the same four layers, but without a mask, should be considered.

  3. The theoretical argument cannot explain the configuration that makes the method work. With p the masked-out fraction, the bound's sparsity term is d[H(p) − bp], which falls monotonically as p rises: more masking is always better on the bound. Empirically, the 16-layer groups are worse than the four-layer groups, so the theory does not identify the sparsity or layer count where the gains appear and cannot guide the choice that decides the result.

Questions for authors
  1. What happens if the layer group is chosen without looking at the target domain, for example, by transferring LLaMA2-7B's math group to coding, or by using one group for all three domains?

  2. What is the spread of MFT performance across all 4-layer groups for one model and domain? Is 4-7 only works on LLaMA2-7B math, and how many groups were examined before the reported one was selected?

  3. Please add the dense 4-layer baseline: starting from Best FFT, continue training the same four layers without a mask, at the same token budget.

  4. Does MFT change inference cost? If the mask is unstructured and the deployed weights are dense, sparsity is a training-time regulariser, and the complexity analysis only works for the mask rather than the model.

6 comments

We thank the reviewer for the careful assessment and for highlighting MFT's effectiveness. Below, we address the weaknesses and questions.

1. Layer selection and larger groups

Our main experiments use local masking to investigate how sensitive different layers are to masking across domains and to examine whether selectively masking particular layers can further improve an already fine-tuned model. Comparing different layer groups allows us to study this domain-dependent sensitivity (with only 10% training data).

Figures 3 and 8 characterize how performance varies with the location and extent of local masking. The local configuration therefore serves our investigation of layer sensitivity, while global masking provides a complementary setting that does not require selecting a consecutive layer group.

Our initial global exploration in Appendix Table 6 was a preliminary test that directly applied the local masking ratio of 10% across all eligible layers. However, the same percentage removes substantially more weights globally than when restricted to a four-layer group. We obtained the mixed results in Table 6 under this more aggressive masking setting.

In subsequent global experiments, we reduced the masking ratio (around 1%). For masking across both attention and MLP modules, we keep the total number of removed weights roughly comparable to local MFT with 10% sparsity within the selected group. We additionally examined global masking restricted to attention or MLP modules at the same ratio. The following table reports these additional results on Qwen3-1.7B.

BenchmarkZero-shotBest FFTGlobal MFT, attentionGlobal MFT, MLPGlobal MFT, both
GSM8K58.868.873.372.271.6
HumanEval38.450.151.152.450.4
IF-Eval30.356.157.358.257.0

All three global configurations achieve higher reported scores than the corresponding Best FFT baseline on every benchmark. These results show that MFT can improve performance without local layer-group selection at low global sparsity.

2. Dense four-layer baseline

First, MFT freezes all model weights throughout optimization, including the retained weights. The learnable variables are the mask scores described in Equations 4 and 5. Consequently, MFT does not update the unmasked weights.

Following the reviewer’s suggestion, we also evaluated dense continuation on the same four layers on LLaMA2-7B. Each experiment starts from the corresponding domain’s Best FFT checkpoint, uses the full domain training data, updates only the selected layers, and freezes all other weights. The additional token budget after Best FFT, evaluation protocol, and checkpoint selection rules match the corresponding MFT experiment.

MetricSelected layersBest FFTDense four-layer FFTMFT
GSM8K4 - 746.747.047.4
HumanEval20 - 2329.729.932.1
IF-Eval0 - 341.337.444.6

The results show that MFT has higher reported scores on all three metrics. Dense continuation yields modest improvements in mathematics and coding, while its instruction-following performance falls below Best FFT. These comparisons provide additional evidence for the value of mask optimization relative to updating the same local weight subset.

3. Theoretical explanation of effective configurations

The theoretical comparison includes both empirical loss and model description length. Equations 9 and 10 compare the complete upper bounds through the change in training loss and the change in the complexity penalty. Equivalently, for a learned model associated with group G and sparsity p, U(hG,p)=LS(hG,p)+Φ(C(hG,p)).U(h_{G,p}) = L_S(h_{G,p}) + \Phi(C(h_{G,p})). A decrease in the encoding term alone does not imply that this complete bound decreases. Changing the group or sparsity also changes the learned model and its empirical loss. Even groups with the same parameter count can produce different losses. Thus, the complexity term alone does not predict that larger masked groups must perform better.

For the reported MFT configurations, Appendix Table 9 provides the following existing training-loss evidence on LLaMA2-7B.

ModelMathematicsCodingInstruction following
Best FFT0.1010.0980.125
MFT0.0850.0540.060

The analysis compares the resulting learned models under their achieved losses. The layer-group experiments in Figures 3 and 8 and the sparsity experiments in Figures 6 and 10 empirically support these configuration choices. Section 3.3, page 8, explicitly states

We use a 10% masking ratio across all settings (Tab. 1), which may not be optimal.

The theoretical comparison therefore provides a conditional explanation of the observed improvement, while the ablations establish how performance varies with layer placement, group size, and masking ratio.

4. Layer selection without consulting the target domain

To address this question directly, we reused layers 4 to 7, selected for mathematics, in coding and instruction following. Each experiment starts from its own domain’s Best FFT checkpoint and learns a new mask using that domain’s full training data. Only the layer-group choice is transferred.

MetricBest FFTMFT with fixed layers 4 - 7MFT with domain-specific layers
HumanEval29.730.832.1
IF-Eval41.339.844.6

For the coding domain, the group selected for mathematics improves coding performance over Best FFT (though still below domain-specific selection). The observed transfer may reflect partially shared reasoning capabilities. DeepSeekMath [1] reports this interpretation as well, finding that code pretraining improved subsequent mathematical reasoning in its two-stage training experiments.

For instruction following, the fixed group falls below Best FFT, while domain-specific selection improves performance. The results therefore show partial transferability and a dependence on the target domain.

5. Performance variation and the number of candidate groups

Figure 3 already shows the full performance profile across the four-layer candidates. For example, for LLaMA2-7B, the complete four-layer candidate set consisted of the eight nonoverlapping groups shown in Figure 3. No additional four-layer starting positions were searched.

QuantitySetting reported in Figure 3
Number of four-layer groups8
Candidate layers0 - 3, 4 - 7, 8 - 11, 12 - 15, 16 - 19, 20 - 23, 24 - 27, 28 - 31
Fine-tuning data used in the exploratory sweep10%
Masked fraction within each group10%

Figure 3 includes every candidate group, including configurations below the FFT baseline. In the LLaMA2-7B math domain, layers 4 to 7 are the only four-layer group clearly above the baseline. This observation concerns the candidate sweep using a subset of the training data. The main results subsequently use the complete fine-tuning datasets for the selected groups.

6. Inference cost and model complexity

The learned mask is applied to the FFT weights before inference, after which the resulting weights can be used in ordinary dense operations. With dense execution, the tensor shapes and nominal arithmetic cost remain unchanged. Unstructured zeros do not automatically speed up inference. Merging the mask beforehand avoids an additional masking operation in each forward pass.

The complexity analysis concerns the resulting model's description length. Equation 11 includes both the retained weight values and the information needed to identify zero positions. The accompanying text on page 19 states

The first term of CMFT is the encoding cost of non-zero weights, and the second is for the index cost of zero weights.

Thus, the analysis accounts for the encoded model, not just the mask. This description-length quantity is distinct from execution cost under dense kernels. Figures 4 and 9 report training memory, token usage, and training time, rather than inference acceleration.

Conclusion

We hope these clarifications and additional experiments address the reviewer’s questions. We welcome further discussion of the experimental comparisons and theoretical scope, and look forward to addressing any remaining questions during the discussion period.

References

[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Zhihong Shao et al., https://arxiv.org/abs/2402.03300

Thanks to the authors' responses, which address most of my concerns. I'll keep my score.

Reviewer_tvxwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This study investigates whether the structural integrity of a well-trained Large Language Model (LLM) is essential for its performance. Specifically, the authors explore the question of whether disrupting this structural integrity—by masking certain weights—might actually lead to improved performance. In this approach, a checkpoint is first created after the model has undergone sufficient full fine-tuning (FFT). Subsequently, a binary mask is learned to set specific weights to zero, while the original weight checkpoint remains unchanged. To address the non-differentiable nature of the binary mask, the authors employ a straight-through estimator during the learning process. To validate the effectiveness of their proposed method, the authors conducted experiments across domain-specific and mixed-domain settings. Furthermore, they provided additional evidence demonstrating that their approach differs from simple regularization and that it also yields performance gains in multimodal models such as LLaVA.

Critical strengths
  • A key advantage is the simple approach of learning a mask at a relatively low cost, without altering the pre-existing FFT (Fine-Tuning) weights.

  • While the degree of performance improvement is not entirely consistent, the experimental results indicate an enhancement in generalization performance.

  • The method consistently outperforms the best FFT baseline even when applied to multimodal backbones like LLaVA, suggesting its potential for use across various other tasks and settings.

Critical weaknesses
  • The core approach of this study—improving generalization performance by learning a binary mask over fixed weights—has already been widely explored in existing literature; therefore, its novelty is very limited. Zhou et al., 2019, Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask Wortsman et al., 2020, Supermasks in Superposition Ramanujan et al., 2020, What’s hidden in a randomly weighted neural network? In my view, this approach offers no novelty in terms of its core idea compared to the aforementioned works.

  • Since the experiments conducted by the authors are domain-specific, there is a potential for performance degradation when applied across different domains; consequently, further analysis from the perspective of practical, real-world application is required.

Questions for authors
  • It would be helpful if the authors could provide further clarification on how their proposed method differs from approaches frequently used in other existing studies—specifically: Zhou et al., 2019, Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask Wortsman et al., 2020, Supermasks in Superposition Ramanujan et al., 2020, What’s hidden in a randomly weighted neural network? What are the fundamental differences between your method and these approaches?
6 comments

We thank the reviewer for the careful evaluation and for highlighting MFT’s simple design, low adaptation cost, and promising results on multimodal backbones. We address the two concerns below through specific comparisons with each cited study and evidence already presented in the main text and appendix.

1. Differences from the cited work and novelty

The novelty of MFT lies in establishing and validating a further optimization stage after sufficient full fine-tuning. Starting from Best FFT, MFT investigates whether selectively removing weights can improve the same trained checkpoint while keeping its weight values fixed. The cited studies investigate different questions, as detailed below.

Zhou et al. Their learned supermasks operate on fixed random initializations, using Bernoulli sampling with sparsity indirectly influenced by mask initialization. MFT operates on the final Best FFT weights and constructs deterministic masks with a prescribed retention ratio. Their lottery ticket analysis also examines resetting and retraining pruned networks. MFT preserves the FFT weight values throughout mask optimization and evaluates improvement over that same trained checkpoint. Their learned supermask results therefore address obtaining predictive capability from random weights, while our experiments examine further improvement of an already optimized model.

Ramanujan et al. Their edge-popup algorithm uses learned scores, top-k selection, and STE to discover effective subnetworks within randomly weighted networks. Their experiments investigate how this capability varies with network width, depth, and sparsity. MFT investigates the remaining improvement potential of an already optimized LLM. Our main language model experiments remove 10% of weights within selected groups of four layers while leaving other layers intact. Figure 3 examines this layer dependence, and comparisons with Best FFT and continued FFT directly test the benefit of this subsequent optimization stage.

Wortsman et al. SupSup addresses continual learning by learning separate task masks over a shared, fixed random backbone. They focus on retaining task-specific solutions and, when task identity is unavailable, identifying the appropriate task through mask superposition and output entropy. MFT refines a checkpoint that has already undergone FFT for the target task. Its objective and evaluation focus on further improving that checkpoint using existing fine-tuning data. SupSup’s results on sequential task learning and task identification do not establish this post-FFT improvement.

2. Cross-domain performance and practical applicability

MFT is designed for domain-specific post-training improvement. It starts from a checkpoint that has already undergone FFT and learns a mask to further improve performance in a specific domain. Its practical value is therefore assessed by the additional target-domain gains over Best FFT. Preserving performance across every non-target domain is outside this optimization objective.

Tables 3 and 4 explicitly evaluate cross-domain effects, and Appendix B discusses them. The manuscript states:

Therefore, the learned mask may remove weights that are redundant or noisy for the target domain, but those weights may still be useful for other domains.

The following table reproduces the complete evaluation of mathematics-specific refinement on LLaMA3.1-8B from Table 4.

DomainBenchmarkBest FFT trained on mathMFT trained on math
MathematicsGSM8K77.077.3
MathematicsMath24.324.9
CodingHumanEval35.436.0
CodingHumanEval+29.932.2
Instruction followingIF-Eval31.531.1
Instruction followingAlpaca-Eval0.60.5

In this example, the same mathematics-specific mask improves both mathematics and coding scores, while instruction-following scores decline. Such changes in non-target capabilities do not invalidate the demonstrated target-domain improvement. For applications with a defined domain, such as a dedicated coding application, MFT provides an additional optimization stage whose utility depends on performance within the intended workload. Our experiments establish this domain-specific benefit and explicitly report the accompanying cross-domain effects.

Please also refer to our response to reviewer Gjk3 about General capability preservation and domain-specific specialization for more evidence.

Conclusion

We hope this response clarifies MFT’s contribution and practical scope. We welcome further questions and look forward to continuing the discussion.

We would like to supplement our earlier response with additional experiments comparing MFT with adaptations of the three cited masking approaches. These experiments examine their behavior when applied to the same fully fine-tuned LLM checkpoint under controlled training conditions.

All experiments use LLaMA2-7B and start from the same coding FFT checkpoint. The underlying model weights remain frozen. The masking methods use the same coding training data, two training epochs, and additional token budget. Checkpoint selection rules and HumanEval generation and scoring settings are also matched.

Local masking operates on the attention and MLP projection matrices in layers 20 to 23 at 10% sparsity. Global masking operates on the corresponding projections across all 32 layers at 1.25% sparsity. Both settings match the total removal budget. Top-k selection is performed within each projection matrix. For Zhou’s Bernoulli masks, the removal budget matches in expectation and is corroborated by the measured sampling densities.

The local masking configurations are listed below. We kept the most official configurations of each method.

ConfigurationScore initializationOptimizerLearning rate
MFTNormal distribution with standard deviation 0.02AdamW0.000002
Edge-PopupKaiming uniformSGD with momentum 0.90.01
SupSupKaiming uniformAdam0.00001
ZhouConstant logitSGD with momentum 0.9100

Weight decay is zero for all configurations. The Zhou adaptation uses dynamic weight rescaling. These settings make the initialization and optimization differences underlying the comparison explicit.

MethodMasked LayersSparsityHumanEval pass@1
FFT-0%29.2%
MFT20-2310%32.0%
Edge-Popup20-2310%29.7%
SupSup-GG20-2310%29.9%
Zhou20-2310%28.66% ± 0.52
Edge-PopupAll 32 layers1.25%27.2%
SupSup-GGAll 32 layers1.25%29.1%
ZhouAll 32 layers1.25%27.68% ± 0.99

Following the stochastic evaluation approach described in Appendix S6 of Zhou et al., we evaluate the Zhou adaptation over ten independent stochastic evaluation runs and report the mean and standard deviation. For each HumanEval problem, we sample a mask once and hold it fixed throughout generation. The reported standard deviation reflects evaluation variability due to mask sampling.

MFT achieves the highest reported HumanEval score among the tested configurations and improves over the shared FFT starting point. The global comparisons also use a matched total removal budget. These results complement the distinctions discussed in our earlier response by providing direct empirical comparisons of the adapted masking configurations in the post-FFT setting.

We hope these additional experiments clarify both MFT’s contribution and the relationship to the cited approaches. We welcome further questions about the adaptation settings and results, and look forward to continuing the discussion.

I appreciate the authors’ efforts to address several points raised in my initial review. My original score remains unchanged.

Comments on the paper

2 comments

We thank all three reviewers for their careful assessments and constructive discussion. We have provided additional experiments and detailed clarifications in our individual responses and incorporated these findings into the revised manuscript. We summarize the main responses and revisions below.

Responses to the reviewers

Reviewer tvxw

We clarified MFT’s contribution as a domain-specific refinement stage after sufficient FFT and provided specific comparisons with Zhou et al., Edge-Popup, and SupSup. Additional controlled adaptations start from the same coding FFT checkpoint with matched training budgets and evaluation settings. MFT achieves 32.0% HumanEval pass@1 compared with 29.2% for the starting checkpoint and the lower reported scores of the tested adaptations. We also explained the existing cross-domain evaluations and the practical scope of domain-specific refinement.

Reviewer neA1

We clarified that all FFT weights remain frozen and only mask scores are optimized. The additional dense four-layer baseline uses the same layers and additional token budget, with MFT achieving higher reported scores across all three domains. We also added layer-group transfer experiments and low-sparsity global masking results that demonstrate improvements without local layer selection. We specified the complete candidate search and clarified how the theoretical comparison depends on both empirical loss and description length, and distinguished description length from inference cost.

Reviewer Gjk3

We expanded model coverage with Gemma-3-1B-IT, improving HumanEval from 56.71% after Best FFT to 60.32% after MFT. Additional ARC-Challenge and HellaSwag evaluations remain close to the corresponding Best FFT scores, supporting substantial preservation of the evaluated reasoning and commonsense capabilities alongside target-domain gains. We also clarified the relationship to learnable masking and pruning and specified the evaluated language architectures as decoder-only Transformers.

Manuscript revisions

We revised the abstract, introduction, related work, and conclusion to state the contribution and domain-specific scope consistently. The method and analysis sections now make weight freezing, mask optimization, layer selection, inference behavior, and the conditions of the theoretical comparison more explicit.

The appendix now includes the additional masking comparisons, dense four-layer controls, layer-group transfer results, low-sparsity global masking, general capability evaluations, and Gemma experiments with all candidate layer groups. The main text points to these additions, and we have added the corresponding references. Additions are marked in blue and deletions in red.

We hope these responses and revisions clarify the contribution and strengthen its empirical support. We welcome any remaining questions and look forward to continuing the discussion.

As the discussion draws to a close, we thank the reviewers and chairs for the time and effort they devoted to our submission. We especially appreciate the reviewers who followed up on our responses. The questions raised during review helped us clarify MFT’s contribution, design additional experiments, and state more precisely what our results support. We have incorporated these changes into the revised manuscript and are grateful for the feedback that helped improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.