acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models

Abstract

Fine-tuning has become the dominant paradigm for adapting Vision-Language Models (VLMs), yet most approaches rely on explicit weight updates that introduce a fundamental trade-off. Full Fine-Tuning (FFT) may perturb pretrained representations due to cross-modal gradient interference, whereas Parameter-Efficient Fine-Tuning (PEFT) methods rely on additive modules, such as low-rank adapters, which may limit adaptation capacity. In this paper, we rethink VLM adaptation from a structural selection framework that adapts VLMs without modifying backbone weights, and we propose Mask Fine-Tuning (MFT). MFT learns masks that selectively route information through existing pretrained connections, dynamically uncovering subnetworks that better align pretrained representations with downstream objectives. Extensive experiments show that MFT provides an effective structural alternative to both FFT and PEFT, consistently achieving superior performance across multiple vision-language benchmarks without adding knowledge or altering the deployment architecture. Moreover, our analysis with MFT provides new insights into how pretrained VLMs reorganize their internal representational pathways during adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Machine Learning, Vision-Language Models, Structural Adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zxVYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mask Fine-Tuning (MFT), a structural adaptation approach for vision-language models that learns weight masks while keeping pretrained parameters frozen. The method introduces both hard binary masks and continuous soft masks to selectively attenuate pretrained connections without modifying the original weights. The authors further provide theoretical analyses of generalization and evaluate MFT against full fine-tuning, parameter-efficient fine-tuning, and pruning methods across multiple vision-language benchmarks. Experimental results demonstrate improved downstream performance and reduced training costs compared with full fine-tuning, while additional analyses examine the learned mask distributions and their structural properties.

Critical strengths
  1. The paper presents a straightforward mask-based adaptation framework that is easy to understand and implement.
  2. Broad experimental evaluation.
Critical weaknesses
  1. Limited methodological novelty over existing mask-based adaptation methods. The core approach of adapting pretrained models through learnable weight masks while freezing the original parameters has been extensively explored, including Zhao et al. (2020), Zheng et al. (2023), and Zhang et al. (2025). More importantly, continuous weight masking has also been investigated in KSM (Yang et al., 2021) and Soft-TransFormers (Kang & Yoo, 2024), while recent Bit-Mask Tuning (Lu et al., 2026) employs a closely related sigmoid-based formulation for generative LLM adaptation. Therefore, the proposed method appears largely to extend existing mask-based adaptation techniques to generative VLMs, without sufficiently establishing the novelty or necessity of its design choices. Also direct comparisons with existing learnable mask-tuning methods are missing.

  2. Insufficient validation of the claimed structural selection mechanism. Although the paper attributes its improvements to uncovering and reorganizing pretrained representational pathways, the supporting analyses mainly examine mask distributions and loss landscapes. These observations do not establish that the learned masks identify functionally meaningful pathways or that the improvements arise specifically from structural selection rather than general multiplicative weight regularization. More direct ablations and causal interventions are needed to substantiate the central claim.

Questions for authors
  1. What is the practical necessity of introducing H-MFT when S-MFT already provides continuous weight adaptation without requiring a predefined sparsity ratio and generally achieves better performance? In addition, what motivates the different score-to-mask mappings used by the two variants?
9 comments

We thank the reviewer for the detailed assessment. Below, we address each point using the submitted manuscript and appendix, together with additional comparisons against mask adaptation methods, regularization baselines, and intervention experiments.

1. Method contributions and relationships to prior mask adaptation methods

Our central question is how effectively generative multimodal instruction tuning can be accomplished by selecting and attenuating frozen pretrained connections. We investigate this question by masking and adapting the language backbone and projector, continuously attenuating and binary-selecting under an explicit removal budget, and systematically evaluating across language models, vision encoders, and downstream tasks.

Section 2.3 already discusses the mask adaptation approaches represented by Zhao et al. and RMT. Zhang et al.'s main procedure applies binary mask learning to models that have already undergone sufficient full fine-tuning.

MethodGQAMMMUPOPEMMESQA-ImageTextVQA
Zhao et al.57.2 ± 0.3231.0 ± 0.3486.2 ± 0.381229.8 ± 12.3758.7 ± 0.9845.1 ± 0.96
RMT52.1 ± 0.6130.3 ± 0.1882.7 ± 0.441241.0 ± 12.3655.6 ± 0.5244.1 ± 0.52
S-MFT Both59.5 ± 0.2234.8 ± 0.3487.0 ± 0.481349.3 ± 8.6059.2 ± 0.1949.4 ± 0.38
H-MFT59.133.886.61345.759.448.0

Please refer to our response 1 to reviewer FUX2, “Method contributions and comparisons with prior mask methods,” for more details.

Among continuous masking methods, KSM learns masks over convolutional kernels, while Soft-TransFormers combines real-valued masks with prompts and task-related parameters for visual continual learning. S-MFT explicitly constrains each effective weight to the segment between zero and its pretrained value and retains continuous attenuation during both training and inference. BMT likewise uses a sigmoid during training, but progressively sharpens it and applies binary masks at evaluation. Continuous masking as the final adaptation representation and continuous relaxation for eventual binary selection therefore serve different objectives. BMT is concurrent work, and we will add the corresponding reference in the camera-ready version.

For direct comparisons, we adapted the official KSM and BMT (no official code for Soft-TransFormers) implementations to Qwen2.5-0.5B with SigLIP, following the original training data and evaluation protocol while preserving their core mechanisms. We trained both additional baselines with five independent training seeds.

MethodGQAMMMUPOPEMMESQA-ImageTextVQA
KSM57.7 ± 0.3428.1 ± 0.3886.0 ± 0.471250.1 ± 8.6353.6 ± 0.2943.4 ± 0.21
BMT53.4 ± 0.2132.3 ± 0.3283.3 ± 0.321208.1 ± 9.2758.2 ± 0.3643.0 ± 0.28
S-MFT Both59.5 ± 0.2234.8 ± 0.3487.0 ± 0.481349.3 ± 8.6059.2 ± 0.1949.4 ± 0.38
H-MFT59.133.886.61345.759.448.0

S-MFT Both reports the five-seed mean and standard deviation from Table 1. H-MFT reports the original Table 2 results at a 0.05% removal rate. S-MFT achieves higher mean scores than both additional baselines on all six metrics. H-MFT also achieves higher reported scores than BMT on all six metrics, showing that the performance advantage extends to the binary-mask setting. These differences can be understood through the available attenuation values, optimization mechanisms, and adaptation coverage.

For KSM, a relevant distinction is the expressivity of continuous attenuation. In the official implementation, a connection is either fully retained with mask value 1 or assigned a normalized attenuation value in [0,0.5]. This parameterization cannot directly express intermediate values between 0.5 and 1. For example, it can retain a connection unchanged or reduce its contribution to half or less, but cannot retain 80% of its contribution. S-MFT covers the entire interval (0,1), allowing finer adjustment of pretrained connections.

This provides a plausible mechanism for the observed advantage. When a useful pretrained connection benefits from moderate attenuation, S-MFT can represent that adjustment directly. Its optimization also follows the actual continuous forward mask through its exact derivative, whereas KSM uses surrogate gradients for discrete retention decisions. Combining finer attenuation with direct differentiation offers a flexible way to adjust existing connections, consistent with the observed improvements across all six metrics.

For BMT, the relevant distinctions include the final adaptation representation and module coverage. Its increasingly sharp sigmoid supports eventual binary retention or deletion. S-MFT optimizes and preserves continuous contributions throughout training and evaluation, allowing connections to retain intermediate influence in the deployed model.

H-MFT provides additional evidence that our approach's effectiveness extends beyond continuous attenuation. BMT determines its final binary mask from whether learned scores exceed zero, so the final removal count follows the learned score distribution. H-MFT specifies the removal budget and learns which connections to remove within that budget. Section 4.2 uses the attenuation distribution learned by S-MFT to inform an approximate budget, and Table 2 validates this choice against neighboring removal rates. For Qwen, 0.05% removal outperforms the evaluated 0.01% and 0.1% settings. This supports a design that preserves most pretrained connections while learning targeted changes within an explicit structural budget.

2. Functional significance of structural selection and its relationship to regularization

The submitted structural analysis includes reproducible unit-level observations. Appendix C.3 and Fig. 11 directly show learned masks over attention heads and MLP neurons across five training seeds. Layer 0, head 7 is consistently strongly suppressed. Each run suppresses approximately 400 MLP neurons, concentrated in the middle layers, particularly layers 10 to 12. Table 7 further compares the performance of unit-level and weight-level masks. These results identify reproducible patterns in unit-level structural selection.

To examine the functional role of the final mask configuration, we intervened on the same Qwen S-MFT Both checkpoint.

InterventionMMMU accuracy
Original learned mask34.44
Restore language masks to all ones29.67
Replace each matrix mask by its mean28.56
Shuffle mask entries within each matrix27.98 ± 1.02
Shuffle within matched weight groups27.36 ± 1.28
Restore projector mask to all ones34.56

Please also refer to our response 5, “Functional structure, regularization, and configuration selection,” for our supplemental intervention experiments and analysis.

We additionally evaluated L2-SP, Mixout, and SPD. Each method retains its official mechanism, selects its configuration under a common limited search protocol with equal candidate counts and per-candidate training steps, and undergoes full training with five independent seeds. All three regularization baselines achieve lower mean scores than S-MFT on the six metrics.

MethodGQAMMMUPOPEMMESQA-ImageTextVQA
L2-SP53.9 ± 0.2832.6 ± 0.3284.2 ± 0.541172.0 ± 6.3756.6 ± 0.2243.7 ± 0.36
Mixout56.9 ± 0.2231.9 ± 0.3785.1 ± 0.471180.6 ± 5.8457.2 ± 0.1644.2 ± 0.38
SPD57.0 ± 0.3132.1 ± 0.4085.4 ± 0.511188.7 ± 6.1256.5 ± 0.1945.2 ± 0.23
S-MFT Both59.5 ± 0.2234.8 ± 0.3487.0 ± 0.481349.3 ± 8.6059.2 ± 0.1949.4 ± 0.38
H-MFT59.133.886.61345.759.448.0

Please see our response to reviewer t6S9 on “Comparisons with regularized and robust fine-tuning” for more details.

Structural selection and regularization can operate together. Restricting the effective-weight family may provide regularization, while supervised training identifies useful connection configurations within that family. The functional significance of those configurations does not require every regularization contribution to be absent. The reproducible unit-level patterns in the appendix and the additional weight-level interventions provide complementary evidence.

3. Practical role of H-MFT

H-MFT addresses an explicit adaptation constraint by selecting connections under a specified removal budget while preserving the pretrained values of all retained connections. It lets us examine how much adaptation deletion alone can achieve and provides a basis for comparing selection procedures at matched sparsity.

The removal budget is empirically grounded in the submitted paper. Section 4.2 and Fig. 6 explain that the attenuation distribution learned by S-MFT informs an approximate sparsity level, which is then evaluated against neighboring removal rates. For Qwen, Table 2 reports MMMU scores of 33.8 at 0.05% removal, compared with 31.7 at 0.01% and 31.4 at 0.1%. Removing fewer connections therefore does not necessarily improve performance. S-MFT supplies a budget reference, and H-MFT learns the deletion locations after that budget is fixed.

At the same 0.05% removal rate, H-MFT achieves higher scores on the six metrics than Wanda and SparseGPT. These results support studying task-supervised selection under a fixed removal count. The additional BMT comparison further shows H-MFT's practical competitiveness among binary-mask methods.

H-MFT also provides a deployment option. Appendix C.4 explains that multiple tasks can share a pretrained backbone while storing their respective binary masks, which can be merged into the model weights for deployment. Retained connections require no additional continuous attenuation coefficients.

H-MFT therefore serves concrete purposes in explicit removal-budget control, preservation of retained weight values, binary task-mask storage, and mechanism analysis. S-MFT provides more flexible continuous attenuation, with the two variants addressing different adaptation requirements.

Reviewer_t6S9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether generative VLMs can be adapted by selecting or attenuating existing connections rather than optimizing unrestricted weight updates. For each eligible language-model or projector matrix W, it introduces trainable scores S and uses effective weights W' = W ⊙ M. H-MFT constructs a fixed-sparsity binary mask and trains scores with a straight-through estimator. S-MFT uses M = sigmoid(S/T), allowing every eligible connection to decrease in magnitude while preserving its pretrained sign. Masks can be merged into W offline.

The empirical study compares attention-only, MLP-only, and combined masking with full fine-tuning and several PEFT methods across four compact language backbones and six benchmarks. Additional experiments change the vision tower, scale to larger models, vary data quantity, and inspect layer and mask patterns. The theory establishes inclusion in an unrestricted update class at matched radius, a compression-based sufficient condition for a tighter bound, and bounds on the score-to-weight map's sensitivity.

Critical strengths

1. Evaluation across multiple models and tasks.

The experiments cover four compact language backbones and six benchmarks, with five-seed means and standard deviations in Table 1. Additional results vary the vision encoder and extend the evaluation to larger models. This coverage supports the empirical finding that restricting adaptation to weight attenuation can be competitive with unrestricted fine-tuning across several settings.

2. Analysis of adaptation scope.

The attention-only, MLP-only, and combined configurations examine how the location and extent of masking affect accuracy and training cost. The layer and data-efficiency studies further characterize where adaptation is effective and how performance depends on supervision. These ablations help assess the trade-offs among masking configurations.

3. No additional inference module.

The learned mask can be merged into the pretrained weights before deployment, preserving the original architecture and avoiding an additional inference module. This is a practical advantage, particularly when inference resources are constrained.

4. Explicit characterization of the adaptation constraint.

The formulation makes the permitted updates precise: eligible weights retain their pretrained sign and can decrease in magnitude. The theoretical analysis establishes inclusion in the unrestricted update class at a matched radius and states conditions for a compression-based generalization bound. This provides a formal account of the method's inductive bias, although its connection to the observed gains remains incomplete.

Critical weaknesses

1. Limited methodological novelty relative to prior mask-based adaptation.

Learning masks over frozen pretrained weights has been explored in prior work. [1–3] The contribution here centers on soft weight attenuation and its application to generative VLMs, but the methodological advance over existing approaches remains insufficiently established. In particular, the sigmoid-based multiplicative formulation warrants a clearer comparison with prior continuous gating and weight-scaling methods. A matched comparison with a task-supervised mask learner would help identify which aspects of MFT account for the gains.

2. Missing comparisons with regularized or robust fine-tuning.

MFT constrains pretrained weights through learned attenuation, which may itself provide a useful regularization effect. However, the reported comparisons do not include regularization-based or robust fine-tuning methods, such as L2-SP, Mixout, and SPD. Comparisons under the same training setup and comparable tuning budgets would help establish whether learned structural selection offers an advantage over existing approaches to constraining adaptation. The pruning controls alone do not resolve this question. [4–6]

References

[1] Zhao et al. Masking as an Efficient Alternative to Finetuning for Pretrained Language Models. 2020. https://aclanthology.org/2020.emnlp-main.174/

[2] Zheng et al. Regularized Mask Tuning. 2023. https://arxiv.org/abs/2307.15049

[3] Zhang et al. Boosting Large Language Models with Mask Fine-Tuning. 2025. https://arxiv.org/abs/2503.22764

[4] Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit Inductive Bias for Transfer Learning with Convolutional Networks. ICML 2018. https://arxiv.org/abs/1802.01483

[5] Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models. ICLR 2020. https://arxiv.org/abs/1909.11299

[6] Junjiao Tian, Chengyue Huang, and Zsolt Kira. Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models. NeurIPS 2024. https://proceedings.neurips.cc/paper_files/paper/2024/hash/281a6b49c66b94f81f3b1ae8f1a0dea4-Abstract-Conference.html

Questions for authors
  1. What methodological differences from prior mask-based adaptation are essential to the gains, and can they be evaluated under a matched training pipeline?

  2. How does MFT compare with regularization-based or robust fine-tuning methods, such as L2-SP, Mixout, and SPD, under the same training setup and comparable hyperparameter-tuning budgets?

7 comments

We thank the reviewer for the assessment. Below, we address both questions using the submitted manuscript and appendix, together with additional experiments on mask adaptation and regularized fine-tuning.

1. Methodological distinctions from prior mask adaptation and comparisons under common training settings

Our work studies generative multimodal instruction tuning directly from pretrained initialization, jointly learning the contributions of connections in the language model and projector while keeping pretrained weights frozen. H-MFT learns connection locations under a specified removal budget, while S-MFT learns continuous attenuation at individual weight positions.

For S-MFT, the effective weights and masks satisfy

Weff=W0⊙M,M=σ(S/T),∂M∂S=M(1−M)T.W_{\mathrm{eff}}=W_0\odot M,\qquad M=\sigma(S/T),\qquad\frac{\partial M}{\partial S}=\frac{M(1-M)}{T}.

Each effective weight lies between zero and its pretrained value, preserving its pretrained sign without exceeding its pretrained magnitude. This parameterization allows a connection to retain an intermediate contribution and updates scores through the derivative of the actual forward mask.

For Zhang et al.'s MFT method, Section 2.3 already discusses the differences in training pipelines. That work is primarily about post-training, starting from a fully fine-tuned model and learning binary masks to further improve it. Our method performs multimodal instruction tuning directly from pretrained initialization and investigates adaptation under continuous attenuation constraints. The distinctions concern the starting checkpoint, adaptation pipeline, and permitted effective-weight changes.

To directly compare task-supervised mask learners, we adapted Zhao et al. and RMT to a common Qwen2.5-0.5B model with the SigLIP setting, aligning the pretrained initialization, full SFT dataset, effective batch size, and mask coverage. Masks cover language attention layers, MLP layers, and the projector. We preserve the binary-mask and gradient mechanisms of the official implementations.

MethodGQAMMMUPOPEMMESQA-ImageTextVQA
Zhao et al.57.2 ± 0.3231.0 ± 0.3486.2 ± 0.381229.8 ± 12.3758.7 ± 0.9845.1 ± 0.96
RMT52.1 ± 0.6130.3 ± 0.1882.7 ± 0.441241.0 ± 12.3655.6 ± 0.5244.1 ± 0.52
S-MFT Both59.5 ± 0.2234.8 ± 0.3487.0 ± 0.481349.3 ± 8.6059.2 ± 0.1949.4 ± 0.38
H-MFT59.133.886.61345.759.448.0

Please refer to our response 1 to reviewer FUX2, “Method contributions and comparisons with prior mask methods,” for more details.

Regarding continuous gating and scaling methods, Appendix Table 6 already includes IA³. IA³ uses learned vectors to scale attention keys and values and the inner activations of feedforward networks, whereas S-MFT learns bounded attenuation coefficients at individual weight positions. S-MFT Both has higher reported means on all six metrics. The submitted manuscript therefore already provides a direct comparison with a representative continuous scaling method.

These findings support the roles of the weight-level constraint, removal budget, and task-supervised selection. We connect the mechanistic distinctions to the observed performance while distinguishing overall method comparisons from the isolated contribution of any single component.

2. Comparisons with regularized and robust fine-tuning

We additionally evaluated L2-SP, Mixout, and SPD using the same Qwen2.5-0.5B with the SigLIP setting as in the original experiments. All three methods preserve their official mechanisms and use the common pretrained initialization, full SFT dataset, and one-epoch training protocol, together with the original evaluation settings for all six benchmarks.

To provide consistent configuration-selection opportunities for the additional methods, we specified three candidates per method in advance. Search used the same settings as our main experiments.

MethodSearch candidatesConfiguration
L2-SP{0.01,0.1,1}=0.01
Mixoutp{0.7,0.8,0.9}p=0.7
SPD{0.5,1,2}=0.5

After selection, each method underwent five independent full-training runs.

MethodGQAMMMUPOPEMMESQA-ImageTextVQA
L2-SP53.9 ± 0.2832.6 ± 0.3284.2 ± 0.541172.0 ± 6.3756.6 ± 0.2243.7 ± 0.36
Mixout56.9 ± 0.2231.9 ± 0.3785.1 ± 0.471180.6 ± 5.8457.2 ± 0.1644.2 ± 0.38
SPD57.0 ± 0.3132.1 ± 0.4085.4 ± 0.511188.7 ± 6.1256.5 ± 0.1945.2 ± 0.23
S-MFT Both59.5 ± 0.2234.8 ± 0.3487.0 ± 0.481349.3 ± 8.6059.2 ± 0.1949.4 ± 0.38
H-MFT59.133.886.61345.759.448.0

S-MFT Both achieve higher reported means than all three additional methods on every benchmark. These results directly compare performance with representative regularized and robust fine-tuning methods.

The results can be interpreted through the different constraints these methods impose. L2-SP penalizes distance from pretrained initialization, Mixout constrains adaptation through stochastic mixing of current and reference parameters, and SPD selectively constrains how far different layers move from pretrained initialization. S-MFT imposes direction and magnitude constraints at each eligible weight while learning connection-specific attenuation through task supervision. It combines a restricted effective-weight family with fine-grained adaptation capacity, providing a mechanistic explanation consistent with the observed advantage.

The H-MFT results also exceed the reported scores of these three additional baselines on all six metrics. The advantage is therefore present in both the binary and continuous masking variants. Please also refer to our response 5 about “Functional structure, regularization, and configuration selection” for our supplemented intervention experiments and the analysis.

Restricting adaptation and learning a useful connection configuration can operate together. The additional L2-SP, Mixout, and SPD comparisons provide direct performance evidence, while the original matched-sparsity controls and additional interventions support the importance of the selected configuration. These findings support the effectiveness of our structural selection approach in this generative VLM setting without requiring any regularization contribution to be absent. We will add the corresponding reference in the camera-ready version.

Conclusion

We hope this response clarifies MFT’s contribution and practical scope. We welcome further questions and look forward to continuing the discussion.

In response to the questions, we completed the L2-SP, Mixout, and SPD comparisons using five independent training seeds and incorporated the results into the manuscript. Both MFT variants achieve higher reported scores on all six benchmarks. Our response also discusses the methodological distinctions and existing continuous-scaling comparisons. We would appreciate your feedback on these additions and whether any questions remain unresolved. If our responses resolve your concerns, we would be grateful if you could consider raising your score.

Reviewer_FUX2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mask Fine-Tuning (MFT), which adapts vision-language models by learning binary or continuous masks over frozen pretrained weights. Experiments across multiple backbones and benchmarks show improvements over full fine-tuning and several PEFT methods.

Critical strengths
  1. The method is simple and can be merged into model weights without additional inference overhead.
  2. Experiments cover diverse language backbones, two vision encoders, and larger models up to 14B parameters.
  3. The binary and soft masking comparisons, layer-wise ablations, and repeated runs provide useful empirical evidence.
Critical weaknesses
  1. Novelty over existing mask tuning needs stronger justification. Learning masks over frozen pretrained weights is an established approach. The main distinction appears to be its application to generative VLM instruction tuning. Direct comparisons with prior mask-learning methods adapted to the same setting would better establish the contribution.

  2. The “without adding knowledge” interpretation is too strong. Masks are learned from supervised multimodal data and therefore encode task-specific information. Moreover, soft masking changes effective weight values through continuous attenuation. The results support constrained adaptation, but do not establish that performance gains solely reveal pre-existing capabilities.

  3. The theoretical results do not establish a practical generalization advantage. Hypothesis-class inclusion gives a non-increasing complexity term, but does not guarantee better population risk. The compression bound additionally requires small approximation distortion, which is not empirically quantified. Its bounded-loss assumptions also need to be related more clearly to the training objective.

  4. The efficiency comparison needs clarification. S-MFT Both trains scores corresponding to approximately 73% of the model parameters and reduces reported peak memory only from 61.2 GB to 55.1 GB. Table 1 reports accuracy after one epoch but time to convergence, mixing different evaluation points. Matched-quality and matched-budget comparisons would be more informative.

  5. The structural interpretation remains largely descriptive. Mask distributions and layer patterns do not establish the functional roles of selected connections. Restoration, removal, or mask-swapping interventions would strengthen the claims about internal pathways. The hyperparameter-selection protocol should also clarify whether MMMU evaluation data influenced configuration choices.

Questions for authors
  1. How does MFT compare with prior mask-tuning methods under identical training conditions?
  2. Do its advantages persist with equally tuned, matched-budget baselines?
  3. Can intervention experiments distinguish meaningful pathway selection from regularization effects?
8 comments

We thank the reviewer for the detailed assessment. Below, we address each point using the submitted manuscript and appendix, along with additional comparisons with prior mask methods and intervention results.

1. Method contributions and comparisons with prior mask methods

To distinguish the effectiveness of our structural selection approach from the additional flexibility of continuous masks, we include our H-MFT results alongside S-MFT and the two additional binary-mask baselines. We adapt Zhao et al. and RMT to Qwen2.5-0.5B with SigLIP. We preserve the core mechanisms of the official implementations. The following table reports the additional baselines and both MFT variants across five training seeds. We keep training conditions fair. The results show that H-MFT retains higher reported scores on all six metrics than both hard-mask baselines.

MethodGQAMMMUPOPEMMESQA-ImageTextVQA
Zhao et al.57.2 ± 0.3231.0 ± 0.3486.2 ± 0.381229.8 ± 12.3758.7 ± 0.9845.1 ± 0.96
RMT52.1 ± 0.6130.3 ± 0.1882.7 ± 0.441241.0 ± 12.3655.6 ± 0.5244.1 ± 0.52
S-MFT Both59.5 ± 0.2234.8 ± 0.3487.0 ± 0.481349.3 ± 8.6059.2 ± 0.1949.4 ± 0.38
H-MFT59.133.886.61345.759.448.0

The hard-mask comparison highlights two connected aspects of our design, namely how the removal budget is determined and how connections are selected within that budget. As described in Section 4.2, the H-MFT sparsity level is informed by an approximate estimate from the attenuation distribution learned by S-MFT. The strong performance also shows that the learned attenuation distribution is meaningful. Our results and analysis in Table 2 also show that the selected removal rate is grounded in the learned soft-mask structure and supported empirically by neighboring sparsity settings, and that simply removing fewer connections does not necessarily improve performance.

The relevant distinction from fixed-threshold masking is that a fixed score threshold does not itself impose a fixed removal budget. In Zhao et al. and RMT, the number of removed connections depends on how many learned scores fall below the threshold. Initialization, learning rate, and regularization can affect the scale and distribution of those scores. Even when their relative ordering is unchanged, a change in overall score magnitude can alter how many connections cross the threshold. Consequently, the amount of structural removal is coupled to optimizing the score distribution.

H-MFT provides explicit control over this interaction. Once the sparsity budget is specified, ranking the learned score magnitudes determines which connections are removed. A uniform rescaling that preserves this ranking does not change the selected connections or the removal count. For generative adaptation, it preserves most pretrained connections while making targeted structural changes, without letting changes in the global score scale alter the amount of removal.

Selection quality remains important within a given budget. At the same 0.05% sparsity, Table 2 reports that H-MFT performs better than Wanda and SparseGPT. These controls support the importance of the selection and adaptation procedure beyond the removal rate alone.

S-MFT provides an additional control mechanism through continuous attenuation. A connection can retain an intermediate contribution while preserving its pretrained sign and remaining bounded by its pretrained magnitude. Its updates differentiate the actual forward mask through M(1−M)/TM(1-M)/T. These properties explain its strong performance.

2. Interpretation of adaptation through frozen pretrained weights

The mechanism is explicitly defined by Weff=W0⊙MW_{\mathrm{eff}}=W_0\odot M. The pretrained weights W0W_0 remain frozen throughout training, while supervision optimizes the mask MM. Appendix C.5 discusses this constrained effective-weight family, and Appendix C.4 explains how masks are merged offline without additional inference modules.

The intended meaning of “without adding knowledge” is that adaptation can operate through the existing pretrained weight structure without introducing additive adaptation modules or directly optimizing the pretrained weights, which is described explicitly in the supervised objective in Eq. 6:

L(Um;M)=−1L∑i=1Llog⁡P(umi∣um<i;W⊙M).\mathcal{L}(U_m;M)=-\frac{1}{L}\sum_{i=1}^{L}\log P\left(u_m^i\mid u_m^{<i};W\odot M\right).

The learned masks encode task-dependent choices about how pretrained connections contribute to the final output. Our experiments establish that this constrained adaptation family is sufficient to achieve strong downstream performance. The additional interventions below further demonstrate that the learned mask configuration has functional significance.

3. Scope of the generalization analysis

Our theoretical contribution is an explicit account of the tradeoff between adaptation capacity and empirical fit. The practical question is whether restricting updates to the pretrained structure can control complexity while preserving useful adaptation capacity. Section 3.4 compares complexity, and the downstream experiments evaluate the effectiveness of the resulting adaptation family.

The submitted theorem already includes the empirical-fit requirement. Conditional on fixed pretrained weights, an i.i.d. downstream sample, and a loss in [0,1][0,1], Appendix C.1 compares MFT and FFT over the same eligible coordinates and effective update radius. For a radius rr fixed independently of S\mathcal S, the following holds with probability at least 1−δ1-\delta, simultaneously for A∈MFT,FFTA\in\\{\mathrm{MFT},\mathrm{FFT}\\} and all h∈HA(r)h\in\mathcal H_A(r),

R(h)≤R^S(h)+2R^n!(ℓ∘HA(r))+3ln⁡(4/δ)2n.R(h)\le\widehat R_{\mathcal S}(h)+2\widehat{\mathfrak R}_n\\!\left(\ell\circ\mathcal H_A(r)\right)+3\sqrt{\frac{\ln(4/\delta)}{2n}}.

Hypothesis-class inclusion gives MFT a no-larger complexity term under these matched conditions. Consequently, the practical relevance of the complexity reduction depends on maintaining suitable task fit. Table 1 and the data-efficiency experiments in Appendix B.3 provide empirical evidence that the restricted family achieves strong downstream performance, including in limited-data settings. This combination supports the usefulness of structural capacity control without inferring an ordering of population risks from complexity alone.

The compression analysis sharpens this argument by explicitly charging for approximation error. For fixed ss, bb, and a data-independent quantization alphabet, the submitted bound holds with probability at least 1−δ1-\delta,

R(hM)≤R^S(hM)+2α(M,M~)+ln⁡(ds)+bsln⁡2+ln⁡(1/δ)2n.R(h_M)\le\widehat R_{\mathcal S}(h_M)+2\alpha(M,\widetilde M)+\sqrt{\frac{\ln\binom{d}{s}+bs\ln 2+\ln(1/\delta)}{2n}}.

Here, α(M,M~)\alpha(M,\widetilde M) is the worst-case absolute loss difference between the original masked predictor and its coded approximation. The equation makes the substantive requirement explicit. A shorter code helps the complete bound only when its complexity savings are accompanied by suitable empirical fit and sufficiently small distortion. For learned soft masks, this is a sufficient-condition result.

The binary case gives this argument a concrete interpretation. An H-MFT mask is represented exactly by its deleted positions, making α=0\alpha=0 and eliminating the need for value-quantization bits. Appendix C.1 additionally includes the ln⁡∣K∣\ln|\mathcal K| penalty when the removal budget is selected from a predefined finite candidate set. Thus, approximation distortion is a substantive condition for soft-mask compression, while exact binary-mask encoding resolves it directly.

The bounded-loss assumption also relates directly to training. Appendix C.1 permits a bounded evaluation loss or a prespecified clipped and rescaled training loss. To illustrate this relationship, we add the following explanatory definition. For a clipping threshold C>0C > 0 fixed in advance, one may define

ℓC(h,z)=min⁡ℓNLL(h,z),CC.\ell_C(h,z)=\frac{\min\\{\ell_{\mathrm{NLL}}(h,z),C\\}}{C}.

Using ℓC\ell_C in the bound controls the population risk of this bounded loss. The model can still be trained with the original token-level NLL because the theorem applies uniformly to predictors in the stated class and does not require the training objective to equal the bounded loss used for analysis.

4. Baseline configurations, training budgets, and efficiency

Our comparison preserves each method’s own configuration. Baselines follow their respective official implementations and method-specific settings. We do not impose hyperparameters selected for MFT on other methods. The additional experiments extend the official configurations through a limited-budget search.

Our primary training budget controls data exposure. The main experiments train for one epoch on full SFT data. The additional full-training runs also use the original full SFT dataset, with the same model initialization and effective batch size among the aligned settings. Additional computation required to complete this training is part of each method’s actual cost.

Equal training time allows methods with different per-step costs to process different amounts of data. It therefore answers a different question from equal data exposure. Our primary comparison evaluates performance after equivalent opportunities to learn from the training data and reports the computational costs arising from each method.

Regarding memory, Section 5 positions MFT as a structural adaptation method that retains substantial adaptation capacity. S-MFT Both optimizes scores corresponding to approximately 73% of the parameter space and reduces reported peak memory from 61.2 GB for FFT to 55.1 GB, approximately 10.0%. S-MFT Attn provides another capacity-memory tradeoff, optimizing approximately 9% of the parameter space and reducing peak memory to 38.1 GB, approximately 37.7%. These settings serve different adaptation requirements. The absence of additional inference overhead after offline merging is a separate deployment property.

The Table 1 caption specifies that benchmark scores are measured after one epoch, whereas Epochs and Time describe progress and elapsed time at convergence. We will distinguish these quantities more clearly so that convergence time is not interpreted as the measured time required to reach the displayed benchmark score. The table supports the stated comparison under controlled data exposure and reports convergence costs separately. It does not measure performance at a common GPU-time cutoff or the cost of reaching a common accuracy target.

Comments on the paper

2 comments

We thank the reviewers and the Area Chair for their assessment. We addressed the reviewers’ questions through additional experiments and clarifications, and we incorporated these findings into the revised manuscript.

Reviewer FUX2. We added comparisons with Zhao et al. and RMT and interventions examining the functional significance of learned masks. We clarified supervised adaptation through frozen pretrained weights, the empirical-fit and distortion conditions in the generalization analysis, and the relationship between bounded loss and training. We also distinguished equal data exposure from equal computation time and documented configuration selection.

Reviewer t6S9. We clarified the distinction between direct adaptation from pretrained initialization and mask learning after full fine-tuning, supported by the Zhao, RMT, and existing IA³ comparisons. We additionally evaluated L2-SP, Mixout, and SPD using their official mechanisms, a common limited search protocol, and five independent training runs. Both MFT variants achieve higher reported scores than these three baselines on all six benchmarks.

Reviewer zxVY. We added KSM and concurrent BMT comparisons and explained differences in attenuation values, gradient mechanisms, and module coverage. We clarified H-MFT’s roles in controlling removal count, preserving retained weights, and storing binary task masks. We inform its removal budget with S-MFT’s learned attenuation distribution and evaluate it against neighboring rates. The interventions support the functional importance of learned configurations, while both mask variants share the same training objective.

The manuscript revisions focus on four areas.

  1. Additional comparisons. Appendix D and Table 8 report seven additional baselines, each evaluated over five training seeds, alongside the original Zero-shot, FFT, S-MFT, and H-MFT results. Both MFT variants exceed the reported scores of all seven additional baselines on all six metrics.

  2. Functional evidence. Section 5 and Appendix E incorporate restoration, mean replacement, and shuffling experiments. Within-matrix shuffling preserves mask-value distributions but reduces MMMU accuracy from 34.44 to 27.98 ± 1.02. We compute the mean and standard deviation over five perturbation seeds of the same trained model. These results support meaningful structural selection while allowing regularization to contribute.

  3. Methodological and theoretical explanations. We expanded the related-work discussion, explained both mask variants and their mappings, and clarified the interpretation of adaptation through frozen weights. The theoretical discussion distinguishes complexity control from population-risk ordering and states the conditions required by the compression argument.

  4. Experimental reporting. We clarified configuration selection, including MMMU’s role, separated one-epoch benchmark scores from convergence costs, and documented memory tradeoffs, baseline coverage, and intervention decoding settings.

We preserved the original framework and equations. Additions are marked in blue, and deletions are shown in red.

As the discussion period concludes, we sincerely thank the Area Chair, Program Chairs, and reviewers for their time and effort in evaluating our work. We particularly appreciate the reviewers who engaged with our responses and provided further feedback. These exchanges helped us clarify our contributions and strengthen the presentation.

Our rebuttal and revised manuscript include additional baseline comparisons, intervention experiments, and clarifications of the theoretical analysis and evaluation protocol. We hope these materials provide a clear and comprehensive basis for the final assessment.

Thank you again for your consideration and for the opportunity to discuss our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.