acceptodds
Withdrawn from ICLR 2027

Preserve the Language, Use the Vision: Training-Free Data Selection for Vision-Language Instruction Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual instruction tuning is essential for adapting vision-language models (VLMs) to vision-language tasks; however, this process may perturb the language representations acquired by their underlying large language models (LLMs) during language pretraining, thereby degrading their original language capabilities. Existing approaches primarily mitigate this issue through architectural modifications, additional text-only data, or training-time constraints, while largely overlooking the role of vision-language instruction data itself. In this work, we find that different vision-language samples induce substantially different degrees of perturbation to the original language representations of their underlying LLMs, suggesting that data selection can provide an effective way to mitigate language capability degradation. Based on this observation, we propose PLUV, a training-free vision-language data selection framework. PLUV first employs language forgetting risk estimation (LFRE) to select samples that induce less perturbation to the original language representations, and then uses visual dependency estimation (VDE) to measure the actual dependency of each sample on visual information. On text-only and vision-language benchmarks, PLUV achieves 98.6% and 102.0% of the baseline performance, respectively, demonstrating its ability to preserve language capabilities while maintaining strong vision-language performance and thereby achieving a better balance between the two.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Visual Instruction Tuning, Data Selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eVbKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the degradation of existing language capabilities during visual instruction tuning of vision-language models (VLMs). It proposes PLUV, a training-free framework for selecting vision-language instruction data, without modifying the model architecture or introducing additional text data or training constraints. LFRE selects samples with a potentially lower risk of forgetting by comparing the similarity between the dominant singular vectors of the first-layer text attention matrices under text-only and vision-language input conditions. VDE selects samples with stronger visual dependence by randomly ablating image-to-text attention information flow and using ridge regression to estimate the contribution of each attention head to the score assigned to the correct answer. Experiment results show its effectiveness on VL benchmarks.

Critical strengths
  1. The paper investigates language degradation through data selection rather than changes to the model architectures or training settings by jointly considering the stability of language representations and dependence on visual information.
  2. The paper conducts comparisons across a range of text-only and VL benchmarks.
  3. It provides ablation studies and analyses of hyperparameters.
Critical weaknesses
  1. The core assumption underlying LFRE lacks direct validation. For instance, an immediate change in the dominant singular vectors of first-layer text attention caused by adding visual information to the input does not necessarily correspond to forgetting of existing language capabilities after training updates the model parameters.
  2. In Tab. 3 and 4, the substantial difference in dataset size (119287 and 72128) means that these improvements cannot be attributed to the visual dependence estimates. A direct control on the effectiveness of VDE is required.
  3. The evaluated model architectures are limited. It mainly focuses on LLaVA-v1.5-7B. More experiments across different model scales or vision-language architectures are needed to evaluate the method's effectiveness.
  4. The accuracy improvements is somewhat marginal.
  5. There are some typos in the manuscript, e.g., in Tab.1, the acc of PRISM on ARC-C should be bold. More importantly, DataTailor achieves the highest accuracy (46.3), while the proposed PLUV is highlighted (45.4), which raises concerns about the actual effectiveness of the method.
  6. The presentation of the experimental results should be carefully checked, not only Tab. 1 above, but also Tab. 2 (e.g., MME-P), Tab. 3 and Tab.4. It seems that the authors carelessly highlighted all of their own results without actually comparing them.

Based on the comments above, the current manuscript is below the acceptance threshold, and I can only give a Reject at the current stage. I encourage the authors to address these in the rebuttal.

Questions for authors

See weakness above.

Reviewer_mHrjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PLUV, a training-free method for selecting visual instruction-tuning data that aims to preserve an LLM’s language abilities while developing its visual capabilities. It first compares text-to-text attention with and without image input to estimate each sample’s potential forgetting risk, then uses attention-head ablations to identify samples whose answers depend on visual information. The authors evaluate the selected data by fine-tuning LLaVA-v1.5-7B on subsets of LLaVA-665K and VisionFlan-186K. They report aggregate text-only and vision-language performance comparable to or slightly above full-data fine-tuning, using substantially fewer samples, and provide ablations and runtime comparisons.

Critical strengths

The selection objective addresses a useful trade-off. PLUV considers both language preservation and visual dependency, rather than selecting data solely for multimodal performance. The second stage addresses a weakness of the first: samples that barely change language representations may simply not require the image. This gives the two-stage design a clear motivation.

The results suggest substantial data savings without a major performance loss On both LLaVA-665K and VisionFlan-186K, the selected subsets roughly match full-data fine-tuning on aggregate text-only and vision-language scores, using about 72K and 28K samples, respectively. Achieving this without additional text-only training data or architectural changes makes the approach practically useful within the tested setting.

Critical weaknesses

The central forgetting-risk interpretation is not sufficiently validated (Section 3.1; Tables 3–4). LFRE measures how attention changes when an image is introduced into a frozen model, but this does not directly establish how much language capability will be lost after training. Such changes could also reflect useful visual conditioning. The selected-versus-unselected comparison uses 119K versus 546K samples, introducing substantial differences in training exposure. Matched-size, matched-update comparisons would help establish whether LFRE predicts forgetting rather than other properties of the data. This gap weakens the main explanation behind the method.

The aggregate results provide weaker evidence of language preservation than the headline suggests (Table 1). The reported 101.2% is relative to full multimodal fine-tuning, rather than the original language model, and is strongly influenced by the MMLU-Pro improvement from 15.9 to 17.3. PLUV still scores below the original Vicuna model on four of six text-only benchmarks, and below full fine-tuning on MMLU and OpenBookQA. The results support competitive average performance, but do not yet convincingly establish broad language preservation. Per-task retention relative to Vicuna, together with run-to-run uncertainty, would make this claim more persuasive.

Questions for authors

Clarification on the forgetting proxy What evidence links LFRE scores to language degradation after parameter updates, rather than simply to the strength of visual conditioning? Does the existing analysis distinguish these explanations?

Clarification on the ablation controls: Tables 3–4 compare subsets containing 119K and 546K samples, with each trained for one epoch. Were any comparisons matched for sample count or optimization steps? If available, do those results support the same conclusion about LFRE?

Clarification on language preservation: How is the aggregate relative score computed, and how sensitive is the 101.2% result to MMLU-Pro? Are repeated-run results available to establish whether the improvements over full fine-tuning exceed training variability?

Reviewer_SonyOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PLUV, a training-free data selection framework designed to preserve language capabilities during visual instruction tuning. The framework combines Language Forgetting Risk Estimation (LFRE), which measures changes in text-to-text attention representations, with Visual Dependency Estimation (VDE), which estimates the contribution of visual information through attention-head ablation. Experiments on LLaVA-665K and VisionFlan-186K show that the framework achieves competitive performance while using substantially fewer training samples.

Critical strengths

Meaningful problem. Maintaining the original language capabilities of LLMs during multimodal instruction tuning is an important and practically relevant challenge.

Interesting data-centric approach: The proposed two-stage selection strategy jointly accounts for language representation stability and visual dependency, without introducing additional trainable parameters or altering the instruction-tuning procedure.

Encouraging experimental results: PLUV achieves competitive performance on both text-only and vision-language benchmarks while using fewer training samples. The experiments further include multiple data pools, component analyses, and computational-cost comparisons.

Critical weaknesses

Questionable forgetting risk estimation: LFRE assumes that changes in first-layer text-to-text attention caused by introducing visual inputs indicate future language forgetting. However, inference-time attention perturbations do not necessarily correspond to training-induced parameter interference, and the paper lacks direct evidence establishing this relationship.

Insufficiently controlled ablations: The LFRE and VDE ablations compare subsets of substantially different sizes (119K vs. 546K for LFRE, and 72K vs. 119K for VDE). Without size-matched controls or a VDE-only baseline, it is difficult to isolate the benefits of the proposed selection criteria from the effects of reducing training data.

Limited generalization and language preservation evidence: The main experiments focus on LLaVA-v1.5-7B, leaving the effectiveness of the proposed criteria across different model architectures insufficiently validated. Moreover, despite outperforming full fine-tuning on aggregate text-only metrics, PLUV still underperforms the original Vicuna model on several benchmarks, limiting the strength of its language preservation claim.

Inconsistencies in experimental reporting: Appendix D reports that VDE evaluates 624,608 samples, whereas the described two-stage pipeline should apply VDE only to the 119,287 samples retained by LFRE. Additionally, the module runtimes reported in Table 9 (25 h and 16 h) are not clearly reconciled with the 31 h data-selection runtime in Figure 4, raising reproducibility concerns.

Questions for authors

Can the authors provide direct evidence that the LFRE score predicts actual language forgetting after visual instruction tuning, rather than merely measuring attention changes caused by visual inputs?

How do LFRE and VDE perform under matched data-selection budgets, particularly when comparing LFRE-only, VDE-only, and their combination?

Can the authors clarify the discrepancy in the number of samples processed by VDE and explain how the module-level computational costs relate to the total selection time reported in Figure 4?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.