- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the degradation of existing language capabilities during visual instruction tuning of vision-language models (VLMs). It proposes PLUV, a training-free framework for selecting vision-language instruction data, without modifying the model architecture or introducing additional text data or training constraints. LFRE selects samples with a potentially lower risk of forgetting by comparing the similarity between the dominant singular vectors of the first-layer text attention matrices under text-only and vision-language input conditions. VDE selects samples with stronger visual dependence by randomly ablating image-to-text attention information flow and using ridge regression to estimate the contribution of each attention head to the score assigned to the correct answer. Experiment results show its effectiveness on VL benchmarks.
Critical strengths
- The paper investigates language degradation through data selection rather than changes to the model architectures or training settings by jointly considering the stability of language representations and dependence on visual information.
- The paper conducts comparisons across a range of text-only and VL benchmarks.
- It provides ablation studies and analyses of hyperparameters.
Critical weaknesses
- The core assumption underlying LFRE lacks direct validation. For instance, an immediate change in the dominant singular vectors of first-layer text attention caused by adding visual information to the input does not necessarily correspond to forgetting of existing language capabilities after training updates the model parameters.
- In Tab. 3 and 4, the substantial difference in dataset size (119287 and 72128) means that these improvements cannot be attributed to the visual dependence estimates. A direct control on the effectiveness of VDE is required.
- The evaluated model architectures are limited. It mainly focuses on LLaVA-v1.5-7B. More experiments across different model scales or vision-language architectures are needed to evaluate the method's effectiveness.
- The accuracy improvements is somewhat marginal.
- There are some typos in the manuscript, e.g., in Tab.1, the acc of PRISM on ARC-C should be bold. More importantly, DataTailor achieves the highest accuracy (46.3), while the proposed PLUV is highlighted (45.4), which raises concerns about the actual effectiveness of the method.
- The presentation of the experimental results should be carefully checked, not only Tab. 1 above, but also Tab. 2 (e.g., MME-P), Tab. 3 and Tab.4. It seems that the authors carelessly highlighted all of their own results without actually comparing them.
Based on the comments above, the current manuscript is below the acceptance threshold, and I can only give a Reject at the current stage. I encourage the authors to address these in the rebuttal.
Questions for authors
See weakness above.