- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes HQS-Tuning, a two-stage data filtering framework for efficient multimodal instruction tuning. It uses an external evaluator to construct quality labels and trains a lightweight CNN on frozen MLLM hidden states to predict sample quality. The first-stage scorer identifies candidates for re-evaluation, while the second-stage scorer uses refined quality labels to rank and select training samples.
Experiments on LLaVA-665K using Qwen3.5-9B and Qwen3-VL-4B-Instruct across eight benchmarks report competitive performance with only 5K–65K selected samples. Additional analyses include hidden-state probing, quality-gap experiments, ablation studies, and comparisons across data budgets.
Critical strengths
-
The paper addresses the practical problem of reducing multimodal instruction-tuning costs through data quality assessment and filtering.
-
The proposed framework combines frozen MLLM hidden-state representations with a lightweight quality scorer, providing a potentially useful approach to reducing repeated external evaluator calls.
-
The experiments cover two Qwen-family models and eight multimodal benchmarks, reporting promising performance across different data-selection budgets.
-
The paper includes hidden-state probing, quality-gap analysis, and component ablations, providing some empirical evidence for the proposed quality prediction and filtering framework.
Critical weaknesses
-
The paper argues that external quality evaluators may not reflect the target MLLM's learning requirements. However, the proposed CNN is still trained entirely on quality labels provided by an external evaluator. It remains unclear how predicting these labels captures target-model-specific training utility rather than merely reproducing the evaluator's judgments.
-
LLaVA-665K was originally designed for LLaVA instruction tuning, yet the paper evaluates only Qwen-family models without including the standard LLaVA-v1.5-7B instruction-tuning setting. Since the evaluated Qwen checkpoints already possess instruction-following capabilities, the reported advantage of 5K over Full-665K may partly reflect reduced forgetting or negative transfer. Results for the original checkpoints before additional fine-tuning are needed to distinguish these explanations. The authors should also provide the results from the LLaVA family, such as LLaVA-1.5-7B, like the previous methods (ICONS, Self-Filter, et. al.).
-
The paper has major inconsistencies with the released code/data. The paper describes Stage-I supervision using VQAv2 samples evaluated by GPT-5.5, whereas the released labels indicate A-OKVQA and a Qwen-based evaluator. Moreover, some failed evaluator calls are labeled as negative examples despite the stated filtering protocol. These discrepancies raise concerns about whether the released implementation corresponds to the experiments reported in the paper.
-
Stage-II improves aggregate relative performance over Stage-I by only 0.42 percentage points and over direct evaluator-based selection by 0.18 points. However, Stage-II also uses substantially more labeled examples and additional evaluator calls. Without an equal-supervision-budget comparison or repeated runs, the independent contribution of the proposed refinement remains unclear.
-
Recent and closely related methods, particularly CoIDO [1] and COINCIDE [2], are absent from the related work and main comparisons despite addressing the same visual instruction data-selection setting. Their inclusion under matched data budgets and training protocols would provide a more comprehensive assessment of the proposed method's competitiveness.
-
Figure 2 could be presented more clearly, and its bottom edge appears to contain an unintended border or artifact from another image. The authors are encouraged to improve the figure's resolution and cropping.
[1] CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance–Diversity Optimization. (NeurIPS), 2025.
[2] Concept-skill Transferability-based Data Selection for Large Vision-Language Models. (EMNLP), 2024.
Questions for authors
-
The paper argues that external evaluators may not reflect the target MLLM's learning requirements, yet the proposed CNN is trained entirely on external evaluator labels. What evidence demonstrates that the learned scores capture target-model-specific training utility rather than merely reproducing the teacher's quality judgments? Could the authors provide a controlled comparison to substantiate this distinction?
-
Why is LLaVA-665K evaluated only on Qwen-family models rather than including the standard LLaVA-v1.5-7B instruction-tuning setting? Since the Qwen checkpoints have already undergone post-training, could the authors report their performance before additional fine-tuning? This is important for determining whether the reported 5K-versus-Full advantage reflects effective data selection or reduced forgetting and negative transfer.
-
The paper reports Stage-I supervision from VQAv2 using GPT-5.5, whereas the supplementary README identifies A-OKVQA and the released labels indicate a Qwen-based evaluator. Which configuration was actually used to produce the reported results? These inconsistencies directly affect the reliability and validity of the reported experimental results and require clarification. Were these invalid evaluator records actually used during CNN training, and if so, how do they affect the reported results?
-
The paper emphasizes the importance of image-conditioned hidden states for multimodal quality assessment, but does not directly isolate the contribution of visual information. Could the authors compare the proposed method against text-only (question–answer) hidden states and shuffled-image controls under the same supervision and selection budget? This would help establish whether the reported improvements genuinely benefit from multimodal representations rather than primarily relying on textual quality cues.