acceptodds
Under review as a conference paper at ICLR 2027

HQS-Tuning: A Hidden-State Quality-Guided Data Filtering Framework for Efficient Multimodal Instruction Fine-Tuning

Abstract

Instruction fine-tuning is a key technique for enhancing the performance of Multimodal Large Language Models (MLLMs), but massive and noisy datasets often incur prohibitive training costs with diminishing returns. Recent studies suggest that filtering a small amount of high-quality data for instruction fine-tuning can achieve highly efficient and superior training outcomes. However, existing multimodal data filtering approaches predominantly depend on external evaluation models or predefined proxy metrics, without leveraging the internal perception of the target Vision-Language Model (VLM) itself. This limitation often results in a mismatch between the filtering criteria and the actual learning requirements of the model being fine-tuned. To address these issues, we propose a novel perspective: the image-conditioned hidden states of VLMs implicitly reflect their evaluation of multimodal training data quality. Based on this insight, we propose an evaluator-guided two-stage data filtering framework. It extracts the target VLM's hidden states as representative features and builds a lightweight classification model using strict quality-tier supervision to score and select the optimal multimodal training subset. Our extensive experiments are conducted across two mainstream VLMs and eight diverse multimodal benchmarks. 172 In terms of data efficiency, our experiments demonstrate that fine-tuning on less than 1% of the large-scale LLaVA-665K dataset (merely 5K samples) enables our method to outperform models trained on the complete 665K dataset. 173 In comprehensive benchmark evaluations, our approach consistently surpasses multiple state-of-the-art multimodal data selection algorithms (such as COMPACT, ICONS, and Self-Filter) under the same data budgets, maintaining robust performance advantages across varied data scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Instruction Fine-Tuning, Multimodal Data Selection, Hidden-State Representations, Data Quality Assessment, Data-Efficient Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_69QEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HQS-Tuning, a two-stage data filtering framework for efficient multimodal instruction tuning. It uses an external evaluator to construct quality labels and trains a lightweight CNN on frozen MLLM hidden states to predict sample quality. The first-stage scorer identifies candidates for re-evaluation, while the second-stage scorer uses refined quality labels to rank and select training samples.

Experiments on LLaVA-665K using Qwen3.5-9B and Qwen3-VL-4B-Instruct across eight benchmarks report competitive performance with only 5K–65K selected samples. Additional analyses include hidden-state probing, quality-gap experiments, ablation studies, and comparisons across data budgets.

Critical strengths
  1. The paper addresses the practical problem of reducing multimodal instruction-tuning costs through data quality assessment and filtering.

  2. The proposed framework combines frozen MLLM hidden-state representations with a lightweight quality scorer, providing a potentially useful approach to reducing repeated external evaluator calls.

  3. The experiments cover two Qwen-family models and eight multimodal benchmarks, reporting promising performance across different data-selection budgets.

  4. The paper includes hidden-state probing, quality-gap analysis, and component ablations, providing some empirical evidence for the proposed quality prediction and filtering framework.

Critical weaknesses
  1. The paper argues that external quality evaluators may not reflect the target MLLM's learning requirements. However, the proposed CNN is still trained entirely on quality labels provided by an external evaluator. It remains unclear how predicting these labels captures target-model-specific training utility rather than merely reproducing the evaluator's judgments.

  2. LLaVA-665K was originally designed for LLaVA instruction tuning, yet the paper evaluates only Qwen-family models without including the standard LLaVA-v1.5-7B instruction-tuning setting. Since the evaluated Qwen checkpoints already possess instruction-following capabilities, the reported advantage of 5K over Full-665K may partly reflect reduced forgetting or negative transfer. Results for the original checkpoints before additional fine-tuning are needed to distinguish these explanations. The authors should also provide the results from the LLaVA family, such as LLaVA-1.5-7B, like the previous methods (ICONS, Self-Filter, et. al.).

  3. The paper has major inconsistencies with the released code/data. The paper describes Stage-I supervision using VQAv2 samples evaluated by GPT-5.5, whereas the released labels indicate A-OKVQA and a Qwen-based evaluator. Moreover, some failed evaluator calls are labeled as negative examples despite the stated filtering protocol. These discrepancies raise concerns about whether the released implementation corresponds to the experiments reported in the paper.

  4. Stage-II improves aggregate relative performance over Stage-I by only 0.42 percentage points and over direct evaluator-based selection by 0.18 points. However, Stage-II also uses substantially more labeled examples and additional evaluator calls. Without an equal-supervision-budget comparison or repeated runs, the independent contribution of the proposed refinement remains unclear.

  5. Recent and closely related methods, particularly CoIDO [1] and COINCIDE [2], are absent from the related work and main comparisons despite addressing the same visual instruction data-selection setting. Their inclusion under matched data budgets and training protocols would provide a more comprehensive assessment of the proposed method's competitiveness.

  6. Figure 2 could be presented more clearly, and its bottom edge appears to contain an unintended border or artifact from another image. The authors are encouraged to improve the figure's resolution and cropping.

[1] CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance–Diversity Optimization. (NeurIPS), 2025.

[2] Concept-skill Transferability-based Data Selection for Large Vision-Language Models. (EMNLP), 2024.

Questions for authors
  1. The paper argues that external evaluators may not reflect the target MLLM's learning requirements, yet the proposed CNN is trained entirely on external evaluator labels. What evidence demonstrates that the learned scores capture target-model-specific training utility rather than merely reproducing the teacher's quality judgments? Could the authors provide a controlled comparison to substantiate this distinction?

  2. Why is LLaVA-665K evaluated only on Qwen-family models rather than including the standard LLaVA-v1.5-7B instruction-tuning setting? Since the Qwen checkpoints have already undergone post-training, could the authors report their performance before additional fine-tuning? This is important for determining whether the reported 5K-versus-Full advantage reflects effective data selection or reduced forgetting and negative transfer.

  3. The paper reports Stage-I supervision from VQAv2 using GPT-5.5, whereas the supplementary README identifies A-OKVQA and the released labels indicate a Qwen-based evaluator. Which configuration was actually used to produce the reported results? These inconsistencies directly affect the reliability and validity of the reported experimental results and require clarification. Were these invalid evaluator records actually used during CNN training, and if so, how do they affect the reported results?

  4. The paper emphasizes the importance of image-conditioned hidden states for multimodal quality assessment, but does not directly isolate the contribution of visual information. Could the authors compare the proposed method against text-only (question–answer) hidden states and shuffled-image controls under the same supervision and selection budget? This would help establish whether the reported improvements genuinely benefit from multimodal representations rather than primarily relying on textual quality cues.

Reviewer_X4QtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes HQS-Tuning, which uses externally generated quality labels to train a CNN-based scorer on a frozen MLLM’s hidden states, then refines the labels and trains a second scorer. The final scorer ranks multimodal instruction examples and selects a compact subset for subsequent fine-tuning.

Critical strengths

The paper addresses an important problem of selecting multimodal instruction data to improve fine-tuning efficiency.

Critical weaknesses
  1. I am particularly concerned about whether the chosen Qwen-family backbones and main experimental protocol with LLaVA-665K are ideal and adequately isolate the effect of data selection. Given these models’ extensive pretraining and the limited transparency regarding the composition of their pre-training and post-training data, it is difficult to assess their prior exposure to LLaVA-665K or related instruction data. Together with the contamination-related confounds reported for Qwen-family models [1], this uncertainty motivates a closer examination, such as potential overfitting and result significance.

The different outcomes of random selection in the Qwen-based and LLaVA-based settings further reinforce my concerns. In this paper, a randomly selected 65K subset (approximately 10% of LLaVA-665K) easily outperforms the full-data baseline on the reported aggregate metric for both backbones. By comparison, COINCIDE [2] reports that even a random 20% subset still underperforms its full-data baseline. Although the backbones and evaluation settings differ, this discrepancy requires further investigation. I strongly recommend that the authors first benchmark their method against existing data selection approaches using an established LLaVA-based experimental protocol, with matched backbones, selection budgets, training settings, and evaluation suites.

  1. Several highly relevant methods that study selection from LLaVA-665K are missing from the discussion and comparisons, including [2-7].

  2. The technical novelty is not sufficiently established. Although the scoring network-based method Self-Filter is included in the experiments, the methodological differences from Self-Filter and CoIDO [5] need clearer discussion. Moreover, methods such as DataTailor [4] and PRISM [7] already exploit internal multimodal representations for selection.

  3. Although the scorer uses the target model’s hidden states, it is trained separately using externally generated quality labels on VQAv2. Its rankings may therefore reflect the evaluator’s preferences and the calibration data distribution, which may not fully capture the usefulness of examples for downstream fine-tuning. This raises concerns about proxy misalignment and selection bias, as also discussed in related work [7]. The limited evaluated Qwen backbones reinforce this concern.

  4. The efficiency analysis of data selection is missing.

  5. Please improve the resolution and quality of Figure 2.

[1] Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination. AAAI, 2026.

[2] Concept-skill Transferability-based Data Selection for Large Vision-Language Models (COINCIDE). EMNLP, 2024.

[3] Less is More: High-value Data Selection for Visual Instruction Tuning (TIVE). ACM Multimedia, 2025.

[4] Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness. ICCV, 2025.

[5] CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization. NeurIPS, 2025.

[6] Data Selection Matters: Towards Robust Instruction Tuning of Large Multimodal Models. NeurIPS, 2025.

[7] PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection. EMNLP, 2026

Questions for authors

Please see the weakness

Reviewer_16BWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HQS-Tuning, a two-stage CNN-based data selection framework that leverages hidden states from a frozen multimodal language model to improve data efficiency in multimodal instruction tuning. The research problem is well motivated, and the experimental results are competitive. However, the evidence supporting model-specific data selection, the individual contributions of the two-stage design, and the robustness of the reported data efficiency gains remains insufficient.

Critical strengths

The paper addresses the redundancy of multimodal instruction data and explores hidden-state representations for quality-based data selection, offering a practically relevant approach. The combination of hidden-state features, teacher-generated quality supervision, and two-stage filtering has the potential to reduce the cost of directly evaluating large-scale datasets.

Critical weaknesses
  1. Insufficient evidence of model specificity. Both CNN stages are supervised by GPT-5.5-generated quality labels rather than actual training utility for the target model. Moreover, Table 6 shows only minor performance differences across hidden-state sources, leaving the model-specific nature of the selection mechanism insufficiently supported.
  2. Unclear contributions of the two-stage design. In Table 3, Stage II improves over direct teacher-based selection by only 0.18%. Since the training set size, sample distribution, and labeling criteria also differ between stages, the source of this improvement remains unclear.
  3. Potential confounding effects of fine-tuning configurations. The same LoRA configuration and 1 epoch are used across different data budgets, resulting in substantially different numbers of optimization steps. Furthermore, Random-65K also outperforms Full-665K, suggesting that the strength of the full-data baseline may influence the reported data efficiency gains.
  4. Insufficient analysis of cross-dataset generalization and data construction. The quality scorer is trained on VQAv2 but applied to the more diverse LLaVA-665K dataset. Furthermore, while the main text emphasizes high- and low-score candidate mining, Appendix A.1 also includes uncertain and randomly sampled examples without clearly explaining their roles in Stage II.
Questions for authors

1.Table 6 shows similar selection performance across different hidden-state sources. Could the authors analyze the overlap between the selected subsets to further clarify whether the method captures target-model-specific preferences?

2.Under the same teacher-labeling budget, how does Stage I candidate mining compare with random candidate sampling? Could the authors also isolate the contributions of teacher re-evaluation and strict label filtering?

3.Do the advantages of using 5K samples over full-data fine-tuning persist after tuning the learning rate, training epochs, or matching the number of optimization steps? What explains the different optimal data budgets across the two backbone models?

4.Could the authors report multi-seed results for the key experiments and provide an end-to-end computational cost analysis that includes teacher evaluation and hidden-state extraction?

5.Could the authors report selection ratios and downstream performance across different task categories in LLaVA-665K? Additionally, how were the initial 2K samples and the 100K candidate pool sampled, and how were the uncertain and random samples used in Stage II training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.