acceptodds
Desk rejected at ICLR 2027

Prior-Guided Importance Weighting for Supervised Fine-Tuning of Multimodal Large Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Supervised fine-tuning (SFT) is critical for adapting pretrained multimodal large language models (MLLMs) to downstream tasks, yet it is often performed with limited target-domain data, making the training outcome highly sensitive to how supervision is utilized. Recent data-centric fine-tuning methods address this challenge by selecting or reweighting target samples, but they often rely on restrictive assumptions or additional components, and their emphasis on better fitting the target distribution may still lead to over-reliance on scarce target supervision. In this work, we view limited-data MLLM fine-tuning as a trade-off between target adaptation and pretrained prior preservation, and propose Prior-Guided Importance Weighting (PGIW), an importance-sampling-based framework that incorporates pretrained model guidance without accessing the original pretraining data. Our theoretical analysis derives an optimal balancing coefficient under an expected Kullback-Leibler divergence risk, while importance weighting enables PGIW to approximate the unavailable pretraining-distribution term using target-domain samples. We further instantiate PGIW as an efficient fine-tuning algorithm for MLLMs. Experiments on multimodal benchmarks demonstrate that PGIW improves target-task performance and cross-domain generalization over standard fine-tuning baselines. The source code is available at https://anonymous.4open.science/r/2026PGIW-67E3/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, supervised fine-tuning, importance weighting, transfer learning, pretrained knowledge preservation, Fisher information

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZRmWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission studies how to adapt pretrained multimodal large language models (MLLMs) to downstream tasks when only limited target-domain data are available for supervised fine-tuning. The central idea is to explicitly balance adaptation to the target task against preservation of the pretrained model prior, with the goal of avoiding overfitting to scarce target data while retaining transferable pretrained knowledge. The paper provides theoretical guidance for controlling this trade-off and introduces a practical approximation that avoids training or maintaining a separate reference model.

The approach is evaluated on four benchmarks, including ImageNet-100 as the in-domain task and three additional benchmarks covering out-of-domain generalization and multimodal reasoning.

Critical strengths

S1. Principled formulation with practical implications. The paper provides a theoretically motivated account of how strongly fine-tuning should rely on target-domain evidence versus the pretrained prior under limited-data settings. Importantly, the proposed practical approximation avoids the additional cost of training or maintaining a separate reference model, making the method easier to apply in practice.

S2. Reasonably broad evaluation setting. The selected benchmarks cover not only in-domain adaptation but also out-of-domain generalization and multimodal reasoning. This provides a useful test of whether the proposed method preserves pretrained capabilities rather than simply improving performance on the fine-tuning domain.

Critical weaknesses

W1. The theoretical assumptions may limit the relevance of the main result in the settings where the method is most needed. Theorem 1 assumes that the target-task parameters and pretrained parameters are asymptotically close. This assumption may be reasonable for mild domain adaptation, but it is less clear in substantial cross-domain shifts or tasks requiring significant capability adaptation. Since preserving the pretrained prior is particularly important under such distribution shifts, the paper should better justify when this assumption is expected to hold and investigate how the method behaves when it is violated.

W2. The empirical comparison with data-centric fine-tuning is incomplete. The paper claims advantages over data-centric fine-tuning methods, but the experiments do not appear to include sufficiently strong or recent data-centric baselines. This makes it difficult to determine whether the gains come specifically from the proposed prior-preserving formulation or whether comparable improvements could be obtained through modern data selection, reweighting, augmentation, or curriculum-based approaches. This is particularly important given the limited-data setting considered by the paper.

Questions for authors

Q1. How sensitive is the theoretical result to the assumption that the target and pretrained parameters are asymptotically close? Can the authors characterize the expected failure mode when this assumption is violated, particularly under substantial domain shift?

Q2. Can the authors provide empirical evidence connecting the theory to domain distance? For example, does the optimal strength of the pretrained prior change systematically as the target domain becomes further from the pretraining distribution?

Q3. Why are stronger recent data-centric adaptation baselines not included? In particular, how would the proposed approach compare with methods based on modern data selection, reweighting, augmentation, or curriculum construction under the same limited-data budget?

Q4. Are the proposed model-centric and data-centric strategies complementary? It would be useful to know whether combining the proposed prior-preserving adaptation method with stronger data-centric techniques leads to additional gains.

3 comments

Thank you for the questions about theoretical scope and data-centric comparisons.

Q1 Local assumption and behavior under large shifts

Following UOWQ (Zhang et al., 2026a, Section IV-A) and OTQMS (Zhang et al., 2025b, Section 4.2), we adopt the local asymptotic setting

=O(N_{\mathrm{tgt}}^{-1/2}).$$ This places the parameter discrepancy on the target estimation-error scale, allowing transfer bias and estimation variance to enter the leading-order KL-risk expansion together and yielding the optimal balancing weight within this regime. Under substantial shifts, higher-order terms can affect the local approximation and the one-step discrepancy estimate. Underestimating the discrepancy can assign excessive weight to conflicting pretrained guidance and hinder target adaptation. > **Q2 Prior strength and task discrepancy** The analytical weight connects prior strength to the Fisher-weighted parameter discrepancy t: $$\alpha^*=\frac{1}{N_{\mathrm{tgt}}(t+1/N_{\mathrm{pre}})}.$$ At fixed sample sizes, the balancing weight decreases as the Fisher-weighted parameter discrepancy t increases. PGIW therefore assigns weaker pretrained guidance to more discrepant tasks. Precisely controlling distance from the original pretraining distribution is difficult in practice. For a cross-task comparison, we use LLaVA-1.5-7B on ScienceQA and M3CoT with their full target training sets. In each setting, t is estimated from gradient and diagonal empirical Fisher statistics on 128 target training examples at the pretrained initialization: | Setting | N_tgt | Estimated t | Applied α* | |---|---:|---:|---:| | LLaVA / ScienceQA | 12,726 | 0.00583856 | 0.01345867 | | LLaVA / M3CoT | 7,863 | 0.00901310 | 0.01411034 | M3CoT requires more demanding multi-step multimodal reasoning than ScienceQA (Chen et al., 2024). In this comparison, it also has a larger estimated parameter discrepancy t. Its smaller target sample size increases the reliance on pretrained guidance, offsetting the reduction associated with larger t; consequently, the applied weights remain close (0.01346 versus 0.01411). This illustrates the joint dependence on discrepancy and sample size across tasks, rather than a controlled change in domain distance. > **Q3 Data-centric comparisons** We added a visual-feature Core-set baseline using greedy k-center selection (Sener and Savarese, 2018). On M3CoT/LLaVA, it selects 1,966 of 7,863 training examples using L2-normalized frozen visual features. LoRA and PGIW share the same fixed Core-set subset and train for two epochs. Final-checkpoint results are shown below, alongside the original full-data PGIW reference. | Method | Training examples | Accuracy (%) | |---|---:|---:| | Core-set + LoRA | 1,966 (25%) | 46.03 | | Core-set + PGIW | 1,966 (25%) | 47.20 | | PGIW, full data (original result) | 7,863 (100%) | 56.86 | All rows use the same 2,318-example test set. The subset rows are retrospectively selected individual runs with different training initializations. Full-data PGIW is a reference at a larger data and training budget. At the same sample count and epoch budget, adding PGIW to the Core-set-selected data improves accuracy from 46.03% to 47.20%, demonstrating a benefit beyond data selection alone. Due to the limited rebuttal time, we have not evaluated additional data-centric methods. If accepted, we will expand the comparisons to include more recent data-centric approaches in the final version. > **Q4 Complementarity with data selection** Yes. On the fixed Core-set subset, PGIW adds 1.17 percentage points over LoRA under the same training budget, supporting the complementarity of data selection and prior-guided weighting. PGIW itself uses importance weighting to adjust each sample's contribution while retaining all available target data, so it does not require a separate selection stage. Core-set + PGIW reaches 47.20%, below PGIW trained directly on the full dataset at the original budget (56.86%, three epochs). Adding PGIW benefits the selected subset, while full-data PGIW makes use of the complete training pool without discarding samples. **References** - Zhang, Q., et al. (2026a). Unified Optimization of Source Weights and Transfer Quantities in Multi-Source Transfer Learning: An Asymptotic Framework. arXiv:2601.10779. - Zhang, Q., et al. (2025b). A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learning. NeurIPS. - Sener, O., and Savarese, S. (2018). Active Learning for Convolutional Neural Networks: A Core-Set Approach. ICLR. - Chen, Q., et al. (2024). M³CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. ACL.

I have carefully reviewed the authors' rebuttal. Some core weaknesses remain unaddressed. Thus, I maintain original scores.

Thank you very much for the time and effort you have dedicated to reviewing our paper. If there are any remaining concerns or unresolved questions, please feel free to point them out, and we will do our best to address them further.

Reviewer_RmWCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper treats limited-data supervised fine-tuning of multimodal LLMs as a trade-off between fitting the target data and preserving the pretrained model.

Starting objective: a transfer objective that mixes target samples with weighted pretraining samples (Eqs. 8–9). Surrogate: pretraining data are unavailable, so the pretraining term is replaced by target samples reweighted with a likelihood ratio. Each sample's loss becomes (1 + α·r)·ℓ, where r = p_pre(y|z) / p_θ(y|z) compares the frozen pretrained model with the current model (Eqs. 20, 23). Theorem 1: minimizing an asymptotic expected KL risk gives a closed-form α* = N_pre / (N_tgt + t·N_pre·N_tgt). Here t = (θ_pre − θ_tgt)ᵀ J (θ_pre − θ_tgt) / d measures the gap between the pretrained and target parameters under the Fisher information. In practice: the paper uses the N_pre → ∞ limit, α* = 1 / (t·N_tgt). It estimates the parameter gap with one natural-gradient step J⁻¹G, using a diagonal empirical Fisher computed on a target subset. Experiments: LoRA fine-tuning of LLaVA-1.5-7B and Qwen2-VL-7B on ImageNet-100, DomainNet (Real as the only source, LLaVA only), ScienceQA, and M3CoT. On the last two, it also compares against L2-SP, EWC, KD, and fixed-coefficient importance weighting.

Critical strengths

Low per-step overhead. One extra frozen forward pass costs +27–40% per step, plus a one-time preparation of about 26–33 minutes (Table 7). A reasonable, clearly credited idea. The paper says what it inherits and acknowledges what target samples cannot recover. Within its own model, the algebra for α* checks out. Consistently favorable point estimates. It wins on all four benchmarks, with both backbones on three of them. DomainNet gains over LoRA are +4.2 to +5.5 points.

Critical weaknesses

The objective is underspecified. The paper never says whether r is detached from the gradient. If it is not, the gradient is [1 + αr(1+ℓ)]∇ℓ; if it is, the gradient is (1+αr)∇ℓ. These are different algorithms. The relationship to KD is never analyzed. If the denominator were the true target conditional, the weighted term's expectation would be the cross-entropy toward the pretrained model, which is exactly a KD term. With the current-model denominator, that equivalence needs extra assumptions. PGIW beats KD by 1.86 points on M3CoT/LLaVA (Table 4), and the paper doesn't explain why. The theory and algorithm diverge. The Eq. 46 cancellation needs the denominator to be the target MLE on the same samples, but the algorithm uses the current model instead. The N_pre → ∞ limit is not shown to be optimal under the paper's assumption that the two sample sizes are comparable. The surrogate ignores the shift in input distribution. α is never reported or calibrated.* The values of t, α*, N_tgt, and the distribution of r are all missing. J⁻¹ is undefined if LoRA is zero-initialized, because some gradients are then exactly zero. The estimate of t can be positive from sampling noise alone, so α* may be inflated or arbitrary. Weak experimental rigor. Results are "averaged over multiple runs," with no count, standard deviation, or confidence intervals. The ScienceQA margins are 0.09 and 0.21 points. LoRA is the only fine-tuning baseline on DomainNet and ImageNet-100. Forgetting is never measured, there are no training-size curves, and every task has classification-style outputs.

Questions for authors

Is r detached from the gradient? If not, which gradient is actually computed? What were N_tgt, the subset size, t, and α* in each experiment? What is the distribution of r? How is J⁻¹ kept well-defined? Do you agree the weighted term reduces to KD when the denominator is the target conditional? Why does PGIW beat KD? Would KD run at coefficient α* separate the effect of the coefficient from the effect of the estimator? Do the Eq. 46 cancellation and the optimality of α* hold for the procedure you actually run? How many runs underlie each number? Please add standard deviations or CIs to Tables 1–4. Can you add KD, EWC, and WiSE-FT on DomainNet and ImageNet-100, training-size curves, and a direct measurement of forgetting? How were the baseline coefficients in Table 6 chosen, and on which split?

3 comments
AuthorsOpenReview ↗

Response to Reviewer RmWC Part 1

Thank you for the detailed questions. We respond in the order of your Questions for Authors, combining the two related questions on KD.

1 Is r detached from the gradient? If not, which gradient is actually computed?

No. The density ratio r is not detached. Its numerator is evaluated using the frozen pretrained model, while the likelihood from the current fine-tuned model in the denominator remains in the computation graph. The balancing weight α is computed before training and held fixed. With ℓθ=−log pθ(y|z) and rθ=p_pre(y|z)/pθ(y|z), the implemented gradient is

=[1+\widehat\alpha r_\theta(1+\ell_\theta)]\nabla_\theta\ell_\theta.$$ Training uses first-token answer supervision and batch-averaged sample losses, with no log-r clipping. > **2 What were N_tgt, the subset size, t, and α* in each experiment?** The subset used to estimate the gradient statistic and diagonal empirical Fisher contains 128 target-domain samples in each setting. | Setting | N_tgt | Estimated t | Applied α* | |---|---:|---:|---:| | LLaVA / ScienceQA | 12,726 | 0.00583856 | 0.01345867 | | LLaVA / M3CoT | 7,863 | 0.00901310 | 0.01411034 | | Qwen / ScienceQA | 12,726 | 0.04961401 | 0.00158381 | | Qwen / M3CoT | 7,863 | 0.05050173 | 0.00251829 | > **3 What is the distribution of r?** The distribution of the model-based density ratio rθ=p_pre(y|z)/pθ(y|z), measured over the full training set at the selected checkpoints, is summarized below. | Setting | Median | P95 | P99 | Maximum | |---|---:|---:|---:|---:| | LLaVA / ScienceQA | 0.75593 | 0.99554 | 1.55868 | 5367.82991 | | LLaVA / M3CoT | 0.59599 | 1.74673 | 3.70085 | 20.67744 | | Qwen / ScienceQA | 0.0079005 | 0.2767230 | 0.4541305 | 0.6308143 | | Qwen / M3CoT | 0.0015701 | 0.1320336 | 0.3467355 | 7.8042498 | The LLaVA/ScienceQA ratio distribution has a pronounced right tail. > **4 How is J⁻¹ kept well-defined?** We use a diagonal empirical Fisher and apply a coordinate-wise floor of 10⁻¹² before inversion. The inverse entries and discrepancy estimate are computed as $$(\widetilde J^{-1})_{kk}=\frac{1}{\max(J_k,10^{-12})}, \qquad \widehat t=\frac{1}{d}\sum_k\frac{G_k^2}{\max(J_k,10^{-12})}.$$ Here G is the mean target-loss gradient and J_k is the diagonal empirical Fisher entry, both evaluated at the pretrained initialization. This handles zero Fisher entries at LoRA initialization; coordinates with G_k=J_k=0 contribute zero to the estimate of t. The dimension d includes all trainable LoRA coordinates: 16,777,216 for LLaVA and 10,092,544 for Qwen. > **5 Does the weighted term reduce to KD when the denominator is the target conditional?** With the true target conditional distribution as a fixed denominator, and the support of the pretrained conditional distribution contained in that target support, the weighted expectation equals cross-entropy under the pretrained conditional distribution. This establishes the temperature-one KD connection. PGIW uses the current fine-tuned model in the ratio and weights observed-label losses; the KD baseline matches the full pretrained-model output distribution. > **6 Why does PGIW beat KD? Would KD run at coefficient α separate the effect of the coefficient from the effect of the estimator?** The practical objectives have different forms. KD adds a full-distribution matching term, $$L_{\mathrm{KD}}=L_{\mathrm{CE}}+\lambda T^2D_{\mathrm{KL}}(p_{\mathrm{pre}}^T\Vert p_\theta^T), \qquad L_{\mathrm{PGIW}}=\mathbb E_{\mathcal D_{\mathrm{tgt}}}[(1+\alpha^*r_\theta)\ell_\theta].$$ PGIW uses the current fine-tuned model in rθ and differentiates through the ratio, giving the gradient in Answer 1. Its sample-wise weighting adapts as the model changes, while KD matches the pretrained-model output distribution. PGIW obtains this form from a weighted-likelihood framework balancing target fitting and pretrained guidance. Local asymptotic analysis of expected KL risk characterizes the global balancing weight for the idealized estimator. The implementation computes α*=1/(N_tgt·t) from target-data gradient/Fisher statistics, adapting prior strength to sample size and estimated parameter discrepancy. This combines an analytically determined global balance with dynamic sample-wise weighting, providing the design rationale for its adaptation benefits. We also tested KD with λ=α*, T=2, and token-averaged losses (Hinton et al., 2015). | LLaVA setting | KD coefficient λ=α* | KD at α*: Accuracy (%) | PGIW Accuracy (%) | |---|---:|---:|---:| | ScienceQA | 0.013458671 | 91.65 | 91.96 | | M3CoT | 0.014110338 | 55.21 | 56.86 | KD entries summarize supplementary runs; PGIW entries retain the original paper results. The reported PGIW accuracies are higher in both settings. This comparison tests the transfer of the PGIW-derived coefficient to KD; the two objectives retain their respective loss scales.
AuthorsOpenReview ↗

Response to Reviewer RmWC Part 2

7 Do the Eq. 46 cancellation and the optimality of α hold for the actual procedure?*

We first derive the ideal objective and its optimal balancing weight, then make implementation choices for practical fine-tuning. The theoretical estimator in Eq. 11 uses a target-model maximum likelihood estimate (MLE), fitted on the same target samples and held fixed in the denominator. The practical algorithm instead uses the current fine-tuned model, which changes during training, for self-consistent importance reweighting. Thus, the objective and balancing weight are derived from the theoretical framework, but the exact cancellation in Eq. 46 requires the fixed-MLE denominator. It does not directly establish the same optimality guarantee for the actual dynamic training procedure.

For t>0,

\qquad \alpha_\infty=\frac{1}{N_{\mathrm{tgt}}t}, \qquad \frac{\alpha_\infty}{\alpha_{\mathrm{finite}}} =1+\frac{1}{N_{\mathrm{pre}}t}.$$ The implementation uses the large-pretraining limit, eliminating the need to specify an effective pretraining sample size. The ratio above makes its approximation condition explicit: N_pre·t must be much larger than one. Eq. 20 uses a shared input reference measure to compare the pretrained and fine-tuned conditional models. It does not estimate the actual pretraining-to-target input-marginal density ratio or recover pretraining probability mass outside target support. > **8 How many runs underlie each number?** We clarify that each accuracy in the original tables corresponds to a single training run. We have supplemented ScienceQA and M3CoT with additional runs and report standard deviations from three runs per method and setting in Answer 9. The original accuracies are retained alongside these supplementary variability statistics. > **9 Standard deviations or confidence intervals for Tables 1–4** Accuracy below retains the paper's reported values. SD is calculated separately across the available run records using denominator n−1. Each SD describes three reported runs. **ScienceQA** | Method | LLaVA Accuracy (%) | SD | Qwen Accuracy (%) | SD | |---|---:|---:|---:|---:| | LoRA | 91.42 | 0.430 | 96.35 | 0.125 | | L2-SP | 91.82 | 0.368 | 95.97 | 0.098 | | EWC | 91.30 | 0.121 | 94.84 | 0.147 | | KD | 91.87 | 0.436 | 96.39 | 0.459 | | Fixed-IW | 91.42 | 0.125 | 96.44 | 0.167 | | PGIW | 91.96 | 0.177 | 96.65 | 0.125 | **M3CoT** | Method | LLaVA Accuracy (%) | SD | Qwen Accuracy (%) | SD | |---|---:|---:|---:|---:| | LoRA | 54.83 | 0.455 | 63.20 | 0.245 | | L2-SP | 55.05 | 0.125 | 63.68 | 1.407 | | EWC | 54.87 | 0.995 | 63.98 | 0.312 | | KD | 55.00 | 0.195 | 63.89 | 0.324 | | Fixed-IW | 55.26 | 0.416 | 63.76 | 1.068 | | PGIW | 56.86 | 0.838 | 64.50 | 0.586 |
AuthorsOpenReview ↗

Response to Reviewer RmWC Part 3

10 Additional baselines, training-size curves, and direct forgetting measurements

ImageNet-100 comparisons

We have expanded the ImageNet-100 comparison with L2-SP, EWC, KD, and Fixed-IW.

MethodLLaVA-1.5-7BQwen2-VL-7B
Zero-shot48.355.2
LoRA59.766.5
L2-SP60.567.0
EWC59.866.8
KD60.967.5
Fixed-IW60.266.9
PGIW61.167.8

In this comparison, PGIW achieves the highest accuracy on both backbones.

DomainNet comparisons

We apologize that our available GPU budget prevented us from completing the additional DomainNet comparisons during the rebuttal period. We recognize the importance of comparing against baselines such as KD and EWC. If the paper is accepted, we commit to adding these comparisons to the final version.

Training size and use of available data

PGIW uses the full available target training set and adjusts its contribution through importance weighting. This design is supported by our local asymptotic framework: substituting the optimal weight into the leading-order KL risk in Appendix B.1, Eqs. 64–67, gives

=\frac{d}{2\left(N_{\mathrm{tgt}}+ \frac{N_{\mathrm{pre}}}{1+tN_{\mathrm{pre}}}\right)}.$$ For fixed task discrepancy and pretraining sample size, this expression decreases with N_tgt, motivating full use of the available target samples. The supplementary random-subset experiments also provide a smaller-budget comparison on M3CoT/LLaVA: | Training data | Examples | LoRA Accuracy (%) | PGIW Accuracy (%) | |---|---:|---:|---:| | Random subset (25%) | 1,966 | 48.40 | 48.84 | | Full training set (original results) | 7,863 | 54.83 | 56.86 | PGIW improves over LoRA at both budgets, reaching 56.86% with the full dataset versus 48.84% with the 25% random subset. The subset entries are selected individual final-checkpoint results. **Forgetting** We evaluated MME Perception, POPE mean F1 across three subsets, and TextVQA soft accuracy. For each method, all three metrics are evaluated on one checkpoint; the methods use separate training runs. | Adaptation setting | Method | MME | POPE F1 | TextVQA (%) | |---|---|---:|---:|---:| | LLaVA / ScienceQA | Base | 1484.96 | 0.840 | 57.86 | | LLaVA / ScienceQA | LoRA | 1430.51 | 0.828 | 57.23 | | LLaVA / ScienceQA | KD | 1408.73 | 0.822 | 57.68 | | LLaVA / ScienceQA | Fixed-IW | 1414.15 | 0.822 | 57.83 | | LLaVA / ScienceQA | PGIW | 1456.23 | 0.836 | 57.85 | | Qwen / M3CoT | Base | 1537.31 | 0.619 | 47.480 | | Qwen / M3CoT | LoRA | 1556.46 | 0.731 | 48.586 | | Qwen / M3CoT | EWC | 1562.83 | 0.682 | 48.160 | | Qwen / M3CoT | KD (λ=1.5) | 1514.89 | 0.632 | 47.760 | | Qwen / M3CoT | Fixed-IW | 1509.92 | 0.707 | 48.476 | | Qwen / M3CoT | PGIW | 1567.24 | 0.711 | 48.932 | PGIW retains scores closest to the base across all three metrics on LLaVA/ScienceQA. On Qwen/M3CoT, it leads the displayed adapted models on MME and TextVQA, while LoRA has higher POPE F1. > **11 How were the Table 6 coefficients chosen, and on which split?** We used the official train/validation/test splits: 12,726/4,241/4,241 for ScienceQA and 7,863/1,108/2,318 for M3CoT. Baseline coefficients were selected based on validation performance. **References** - Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
Reviewer_BJs1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Prior-Guided Importance Weighting (PGIW) for supervised fine-tuning of multimodal LLMs with limited data. The core idea is to keep the pretraining priors by upweighting training samples where the pretrained (zero-shot) model is more confident in the correct label. PGIW can be implemented without pretraining data. Experiments on two models (LLaVA-1.5-7B and Qwen2-VL-7B) and four classification-style benchmarks show gains over a list of baselines.

Critical strengths

Strength 1: The authors propose a simple fix. The method mainly requires changing the loss (plus a one-time pass to compute gradient statistics), and does not require access to the pretraining data.

Strength 2: This method introduces few hyperparameter to be tuned. The balancing coefficient is computed from the data rather than tuned, unlike the coefficients of the compared baselines.

Critical weaknesses

It seems that the baselines are weak. There are existing methods that would preserve priors, including the ones that the paper cites, such as WiSE-FT, but some of them are not compared.

On ImageNet-100 and DomainNet, PGIW is compared only with zero-shot and plain LoRA, not with any prior-preserving method. On DomainNet, PGIW wins over LoRA by a relatively large margin (+4.9 points on average). On ScienceQA and M3CoT, where prior-preserving baselines like L2-SP, EWC and KD are compared, PGIW’s margin is much smaller than that margin (e.g., 0.09 vs. 0.54 points on ScienceQA with LLaVA). It is unclear whether PGIW improves over existing prior-preserving methods.

Questions for authors

Could the authors report more baselines on the ImageNet-100 and DomainNet experiments, such as WISE-FT, KD etc.?

3 comments

Thank you for highlighting the missing prior-preserving comparisons. We have expanded the ImageNet-100 results with L2-SP, EWC, KD, and Fixed-IW.

MethodLLaVA-1.5-7BQwen2-VL-7B
Zero-shot48.355.2
LoRA59.766.5
L2-SP60.567.0
EWC59.866.8
KD60.967.5
Fixed-IW60.266.9
PGIW61.167.8

In this comparison, PGIW achieves the highest accuracy on both backbones.

DomainNet comparisons

We apologize that our available GPU budget prevented us from completing the additional DomainNet comparisons during the rebuttal period. We recognize the importance of comparing against baselines such as KD and EWC. If the paper is accepted, we commit to adding these comparisons to the final version. The current DomainNet results establish gains over ordinary LoRA; comparison with the additional prior-preserving baselines will be reported separately.

Thank you for the additional results. Compared to the other prior-preserving baselines, the margin shrink to 0.2 and 0.3 points on the ImageNet-100. My concern still holds that the gains over existing methods are small (and the significance is unclear), hence the main claim in this paper can be strongly supported. I will keep my score.

Thank you for clarifying your concern. We agree that the ImageNet-100 margins are modest. However, we would emphasize the consistency of the gains: across both backbones on ScienceQA, M3CoT, and ImageNet-100, PGIW improves over the strongest compared prior-preserving baseline in all six settings, with gains reaching 1.60 percentage points on M3CoT/LLaVA. In the cross-domain generalization experiments on DomainNet, PGIW also outperforms LoRA in all five test domains.

We would also highlight our theoretical contribution. Limited-data MLLM fine-tuning requires balancing target adaptation against pretrained prior preservation, yet the strength of prior guidance is often chosen empirically. We extend UOWQ’s (Zhang et al., 2026a) source–target balancing principle from transfer learning to our setting. Specifically, we adapt its theoretical framework, which relies on access to source data, to our scenarios where source data is unavailable. Following the local asymptotic assumptions adopted in UOWQ, we formulate this trade-off through expected KL risk and derive an optimal balancing weight based on target sample size and task discrepancy. Model-based importance weighting incorporates pretrained guidance into target-data training without access to pretraining samples. This yields a practical MLLM fine-tuning algorithm with theoretically guided prior strength.

We believe that the consistent gains, together with this extension of transfer-learning theory to practical MLLM fine-tuning, support the value of PGIW beyond the margin on any single benchmark. We will also explicitly discuss the modest gains and remaining limitations in the experimental analysis.

References

  • Zhang, Q., et al. (2026a). Unified Optimization of Source Weights and Transfer Quantities in Multi-Source Transfer Learning: An Asymptotic Framework. arXiv:2601.10779.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.