- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Mask Fine-Tuning (MFT), a structural adaptation approach for vision-language models that learns weight masks while keeping pretrained parameters frozen. The method introduces both hard binary masks and continuous soft masks to selectively attenuate pretrained connections without modifying the original weights. The authors further provide theoretical analyses of generalization and evaluate MFT against full fine-tuning, parameter-efficient fine-tuning, and pruning methods across multiple vision-language benchmarks. Experimental results demonstrate improved downstream performance and reduced training costs compared with full fine-tuning, while additional analyses examine the learned mask distributions and their structural properties.
Critical strengths
- The paper presents a straightforward mask-based adaptation framework that is easy to understand and implement.
- Broad experimental evaluation.
Critical weaknesses
-
Limited methodological novelty over existing mask-based adaptation methods. The core approach of adapting pretrained models through learnable weight masks while freezing the original parameters has been extensively explored, including Zhao et al. (2020), Zheng et al. (2023), and Zhang et al. (2025). More importantly, continuous weight masking has also been investigated in KSM (Yang et al., 2021) and Soft-TransFormers (Kang & Yoo, 2024), while recent Bit-Mask Tuning (Lu et al., 2026) employs a closely related sigmoid-based formulation for generative LLM adaptation. Therefore, the proposed method appears largely to extend existing mask-based adaptation techniques to generative VLMs, without sufficiently establishing the novelty or necessity of its design choices. Also direct comparisons with existing learnable mask-tuning methods are missing.
-
Insufficient validation of the claimed structural selection mechanism. Although the paper attributes its improvements to uncovering and reorganizing pretrained representational pathways, the supporting analyses mainly examine mask distributions and loss landscapes. These observations do not establish that the learned masks identify functionally meaningful pathways or that the improvements arise specifically from structural selection rather than general multiplicative weight regularization. More direct ablations and causal interventions are needed to substantiate the central claim.
Questions for authors
- What is the practical necessity of introducing H-MFT when S-MFT already provides continuous weight adaptation without requiring a predefined sparsity ratio and generally achieves better performance? In addition, what motivates the different score-to-mask mappings used by the two variants?
9 comments
S-MFT Both reports the five-seed mean and standard deviation from Table 1. H-MFT reports the original Table 2 results at a 0.05% removal rate. S-MFT achieves higher mean scores than both additional baselines on all six metrics. H-MFT also achieves higher reported scores than BMT on all six metrics, showing that the performance advantage extends to the binary-mask setting. These differences can be understood through the available attenuation values, optimization mechanisms, and adaptation coverage.
For KSM, a relevant distinction is the expressivity of continuous attenuation. In the official implementation, a connection is either fully retained with mask value 1 or assigned a normalized attenuation value in [0,0.5]. This parameterization cannot directly express intermediate values between 0.5 and 1. For example, it can retain a connection unchanged or reduce its contribution to half or less, but cannot retain 80% of its contribution. S-MFT covers the entire interval (0,1), allowing finer adjustment of pretrained connections.
This provides a plausible mechanism for the observed advantage. When a useful pretrained connection benefits from moderate attenuation, S-MFT can represent that adjustment directly. Its optimization also follows the actual continuous forward mask through its exact derivative, whereas KSM uses surrogate gradients for discrete retention decisions. Combining finer attenuation with direct differentiation offers a flexible way to adjust existing connections, consistent with the observed improvements across all six metrics.
For BMT, the relevant distinctions include the final adaptation representation and module coverage. Its increasingly sharp sigmoid supports eventual binary retention or deletion. S-MFT optimizes and preserves continuous contributions throughout training and evaluation, allowing connections to retain intermediate influence in the deployed model.
H-MFT provides additional evidence that our approach's effectiveness extends beyond continuous attenuation. BMT determines its final binary mask from whether learned scores exceed zero, so the final removal count follows the learned score distribution. H-MFT specifies the removal budget and learns which connections to remove within that budget. Section 4.2 uses the attenuation distribution learned by S-MFT to inform an approximate budget, and Table 2 validates this choice against neighboring removal rates. For Qwen, 0.05% removal outperforms the evaluated 0.01% and 0.1% settings. This supports a design that preserves most pretrained connections while learning targeted changes within an explicit structural budget.
2. Functional significance of structural selection and its relationship to regularization
The submitted structural analysis includes reproducible unit-level observations. Appendix C.3 and Fig. 11 directly show learned masks over attention heads and MLP neurons across five training seeds. Layer 0, head 7 is consistently strongly suppressed. Each run suppresses approximately 400 MLP neurons, concentrated in the middle layers, particularly layers 10 to 12. Table 7 further compares the performance of unit-level and weight-level masks. These results identify reproducible patterns in unit-level structural selection.
To examine the functional role of the final mask configuration, we intervened on the same Qwen S-MFT Both checkpoint.
| Intervention | MMMU accuracy |
|---|---|
| Original learned mask | 34.44 |
| Restore language masks to all ones | 29.67 |
| Replace each matrix mask by its mean | 28.56 |
| Shuffle mask entries within each matrix | 27.98 ± 1.02 |
| Shuffle within matched weight groups | 27.36 ± 1.28 |
| Restore projector mask to all ones | 34.56 |
Please also refer to our response 5, “Functional structure, regularization, and configuration selection,” for our supplemental intervention experiments and analysis.
We additionally evaluated L2-SP, Mixout, and SPD. Each method retains its official mechanism, selects its configuration under a common limited search protocol with equal candidate counts and per-candidate training steps, and undergoes full training with five independent seeds. All three regularization baselines achieve lower mean scores than S-MFT on the six metrics.
| Method | GQA | MMMU | POPE | MME | SQA-Image | TextVQA |
|---|---|---|---|---|---|---|
| L2-SP | 53.9 ± 0.28 | 32.6 ± 0.32 | 84.2 ± 0.54 | 1172.0 ± 6.37 | 56.6 ± 0.22 | 43.7 ± 0.36 |
| Mixout | 56.9 ± 0.22 | 31.9 ± 0.37 | 85.1 ± 0.47 | 1180.6 ± 5.84 | 57.2 ± 0.16 | 44.2 ± 0.38 |
| SPD | 57.0 ± 0.31 | 32.1 ± 0.40 | 85.4 ± 0.51 | 1188.7 ± 6.12 | 56.5 ± 0.19 | 45.2 ± 0.23 |
| S-MFT Both | 59.5 ± 0.22 | 34.8 ± 0.34 | 87.0 ± 0.48 | 1349.3 ± 8.60 | 59.2 ± 0.19 | 49.4 ± 0.38 |
| H-MFT | 59.1 | 33.8 | 86.6 | 1345.7 | 59.4 | 48.0 |
Please see our response to reviewer t6S9 on “Comparisons with regularized and robust fine-tuning” for more details.
Structural selection and regularization can operate together. Restricting the effective-weight family may provide regularization, while supervised training identifies useful connection configurations within that family. The functional significance of those configurations does not require every regularization contribution to be absent. The reproducible unit-level patterns in the appendix and the additional weight-level interventions provide complementary evidence.
3. Practical role of H-MFT
H-MFT addresses an explicit adaptation constraint by selecting connections under a specified removal budget while preserving the pretrained values of all retained connections. It lets us examine how much adaptation deletion alone can achieve and provides a basis for comparing selection procedures at matched sparsity.
The removal budget is empirically grounded in the submitted paper. Section 4.2 and Fig. 6 explain that the attenuation distribution learned by S-MFT informs an approximate sparsity level, which is then evaluated against neighboring removal rates. For Qwen, Table 2 reports MMMU scores of 33.8 at 0.05% removal, compared with 31.7 at 0.01% and 31.4 at 0.1%. Removing fewer connections therefore does not necessarily improve performance. S-MFT supplies a budget reference, and H-MFT learns the deletion locations after that budget is fixed.
At the same 0.05% removal rate, H-MFT achieves higher scores on the six metrics than Wanda and SparseGPT. These results support studying task-supervised selection under a fixed removal count. The additional BMT comparison further shows H-MFT's practical competitiveness among binary-mask methods.
H-MFT also provides a deployment option. Appendix C.4 explains that multiple tasks can share a pretrained backbone while storing their respective binary masks, which can be merged into the model weights for deployment. Retained connections require no additional continuous attenuation coefficients.
H-MFT therefore serves concrete purposes in explicit removal-budget control, preservation of retained weight values, binary task-mask storage, and mechanism analysis. S-MFT provides more flexible continuous attenuation, with the two variants addressing different adaptation requirements.
We thank the reviewer for the detailed assessment. Below, we address each point using the submitted manuscript and appendix, together with additional comparisons against mask adaptation methods, regularization baselines, and intervention experiments.
1. Method contributions and relationships to prior mask adaptation methods
Our central question is how effectively generative multimodal instruction tuning can be accomplished by selecting and attenuating frozen pretrained connections. We investigate this question by masking and adapting the language backbone and projector, continuously attenuating and binary-selecting under an explicit removal budget, and systematically evaluating across language models, vision encoders, and downstream tasks.
Section 2.3 already discusses the mask adaptation approaches represented by Zhao et al. and RMT. Zhang et al.'s main procedure applies binary mask learning to models that have already undergone sufficient full fine-tuning.
Please refer to our response 1 to reviewer FUX2, “Method contributions and comparisons with prior mask methods,” for more details.
Among continuous masking methods, KSM learns masks over convolutional kernels, while Soft-TransFormers combines real-valued masks with prompts and task-related parameters for visual continual learning. S-MFT explicitly constrains each effective weight to the segment between zero and its pretrained value and retains continuous attenuation during both training and inference. BMT likewise uses a sigmoid during training, but progressively sharpens it and applies binary masks at evaluation. Continuous masking as the final adaptation representation and continuous relaxation for eventual binary selection therefore serve different objectives. BMT is concurrent work, and we will add the corresponding reference in the camera-ready version.
For direct comparisons, we adapted the official KSM and BMT (no official code for Soft-TransFormers) implementations to Qwen2.5-0.5B with SigLIP, following the original training data and evaluation protocol while preserving their core mechanisms. We trained both additional baselines with five independent training seeds.