- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This submission studies how to adapt pretrained multimodal large language models (MLLMs) to downstream tasks when only limited target-domain data are available for supervised fine-tuning. The central idea is to explicitly balance adaptation to the target task against preservation of the pretrained model prior, with the goal of avoiding overfitting to scarce target data while retaining transferable pretrained knowledge. The paper provides theoretical guidance for controlling this trade-off and introduces a practical approximation that avoids training or maintaining a separate reference model.
The approach is evaluated on four benchmarks, including ImageNet-100 as the in-domain task and three additional benchmarks covering out-of-domain generalization and multimodal reasoning.
Critical strengths
S1. Principled formulation with practical implications. The paper provides a theoretically motivated account of how strongly fine-tuning should rely on target-domain evidence versus the pretrained prior under limited-data settings. Importantly, the proposed practical approximation avoids the additional cost of training or maintaining a separate reference model, making the method easier to apply in practice.
S2. Reasonably broad evaluation setting. The selected benchmarks cover not only in-domain adaptation but also out-of-domain generalization and multimodal reasoning. This provides a useful test of whether the proposed method preserves pretrained capabilities rather than simply improving performance on the fine-tuning domain.
Critical weaknesses
W1. The theoretical assumptions may limit the relevance of the main result in the settings where the method is most needed. Theorem 1 assumes that the target-task parameters and pretrained parameters are asymptotically close. This assumption may be reasonable for mild domain adaptation, but it is less clear in substantial cross-domain shifts or tasks requiring significant capability adaptation. Since preserving the pretrained prior is particularly important under such distribution shifts, the paper should better justify when this assumption is expected to hold and investigate how the method behaves when it is violated.
W2. The empirical comparison with data-centric fine-tuning is incomplete. The paper claims advantages over data-centric fine-tuning methods, but the experiments do not appear to include sufficiently strong or recent data-centric baselines. This makes it difficult to determine whether the gains come specifically from the proposed prior-preserving formulation or whether comparable improvements could be obtained through modern data selection, reweighting, augmentation, or curriculum-based approaches. This is particularly important given the limited-data setting considered by the paper.
Questions for authors
Q1. How sensitive is the theoretical result to the assumption that the target and pretrained parameters are asymptotically close? Can the authors characterize the expected failure mode when this assumption is violated, particularly under substantial domain shift?
Q2. Can the authors provide empirical evidence connecting the theory to domain distance? For example, does the optimal strength of the pretrained prior change systematically as the target domain becomes further from the pretraining distribution?
Q3. Why are stronger recent data-centric adaptation baselines not included? In particular, how would the proposed approach compare with methods based on modern data selection, reweighting, augmentation, or curriculum construction under the same limited-data budget?
Q4. Are the proposed model-centric and data-centric strategies complementary? It would be useful to know whether combining the proposed prior-preserving adaptation method with stronger data-centric techniques leads to additional gains.
3 comments
I have carefully reviewed the authors' rebuttal. Some core weaknesses remain unaddressed. Thus, I maintain original scores.
Thank you very much for the time and effort you have dedicated to reviewing our paper. If there are any remaining concerns or unresolved questions, please feel free to point them out, and we will do our best to address them further.
Thank you for the questions about theoretical scope and data-centric comparisons.
Following UOWQ (Zhang et al., 2026a, Section IV-A) and OTQMS (Zhang et al., 2025b, Section 4.2), we adopt the local asymptotic setting
=O(N_{\mathrm{tgt}}^{-1/2}).$$ This places the parameter discrepancy on the target estimation-error scale, allowing transfer bias and estimation variance to enter the leading-order KL-risk expansion together and yielding the optimal balancing weight within this regime. Under substantial shifts, higher-order terms can affect the local approximation and the one-step discrepancy estimate. Underestimating the discrepancy can assign excessive weight to conflicting pretrained guidance and hinder target adaptation. > **Q2 Prior strength and task discrepancy** The analytical weight connects prior strength to the Fisher-weighted parameter discrepancy t: $$\alpha^*=\frac{1}{N_{\mathrm{tgt}}(t+1/N_{\mathrm{pre}})}.$$ At fixed sample sizes, the balancing weight decreases as the Fisher-weighted parameter discrepancy t increases. PGIW therefore assigns weaker pretrained guidance to more discrepant tasks. Precisely controlling distance from the original pretraining distribution is difficult in practice. For a cross-task comparison, we use LLaVA-1.5-7B on ScienceQA and M3CoT with their full target training sets. In each setting, t is estimated from gradient and diagonal empirical Fisher statistics on 128 target training examples at the pretrained initialization: | Setting | N_tgt | Estimated t | Applied α* | |---|---:|---:|---:| | LLaVA / ScienceQA | 12,726 | 0.00583856 | 0.01345867 | | LLaVA / M3CoT | 7,863 | 0.00901310 | 0.01411034 | M3CoT requires more demanding multi-step multimodal reasoning than ScienceQA (Chen et al., 2024). In this comparison, it also has a larger estimated parameter discrepancy t. Its smaller target sample size increases the reliance on pretrained guidance, offsetting the reduction associated with larger t; consequently, the applied weights remain close (0.01346 versus 0.01411). This illustrates the joint dependence on discrepancy and sample size across tasks, rather than a controlled change in domain distance. > **Q3 Data-centric comparisons** We added a visual-feature Core-set baseline using greedy k-center selection (Sener and Savarese, 2018). On M3CoT/LLaVA, it selects 1,966 of 7,863 training examples using L2-normalized frozen visual features. LoRA and PGIW share the same fixed Core-set subset and train for two epochs. Final-checkpoint results are shown below, alongside the original full-data PGIW reference. | Method | Training examples | Accuracy (%) | |---|---:|---:| | Core-set + LoRA | 1,966 (25%) | 46.03 | | Core-set + PGIW | 1,966 (25%) | 47.20 | | PGIW, full data (original result) | 7,863 (100%) | 56.86 | All rows use the same 2,318-example test set. The subset rows are retrospectively selected individual runs with different training initializations. Full-data PGIW is a reference at a larger data and training budget. At the same sample count and epoch budget, adding PGIW to the Core-set-selected data improves accuracy from 46.03% to 47.20%, demonstrating a benefit beyond data selection alone. Due to the limited rebuttal time, we have not evaluated additional data-centric methods. If accepted, we will expand the comparisons to include more recent data-centric approaches in the final version. > **Q4 Complementarity with data selection** Yes. On the fixed Core-set subset, PGIW adds 1.17 percentage points over LoRA under the same training budget, supporting the complementarity of data selection and prior-guided weighting. PGIW itself uses importance weighting to adjust each sample's contribution while retaining all available target data, so it does not require a separate selection stage. Core-set + PGIW reaches 47.20%, below PGIW trained directly on the full dataset at the original budget (56.86%, three epochs). Adding PGIW benefits the selected subset, while full-data PGIW makes use of the complete training pool without discarding samples. **References** - Zhang, Q., et al. (2026a). Unified Optimization of Source Weights and Transfer Quantities in Multi-Source Transfer Learning: An Asymptotic Framework. arXiv:2601.10779. - Zhang, Q., et al. (2025b). A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learning. NeurIPS. - Sener, O., and Savarese, S. (2018). Active Learning for Convolutional Neural Networks: A Core-Set Approach. ICLR. - Chen, Q., et al. (2024). M³CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. ACL.