- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OnceSelect, a reusable data selection framework for efficient multimodal instruction tuning. The key motivation is to reduce repeated selection costs across different datasets and downstream MLLMs. The method first extracts frozen CLIP representations of image–instruction pairs and performs semantic clustering to construct pseudo-labels. A lightweight gated classifier is then trained on centroid-nearest representative samples, with an intermediate checkpoint retained to estimate sample informativeness through prediction uncertainty. During selection, low-confidence candidates are prioritized within each semantic cluster, followed by uncertainty-weighted blockwise k-DPP sampling to balance informativeness, diversity, and semantic coverage.
The authors evaluate OnceSelect on LLaVA-625K, reporting 99.2% of full-data aggregate performance using only 15% of the training data. They further investigate cross-dataset transfer to Vision-Flan and LRV-Sub without retraining the selector, as well as cross-model transfer to Qwen3-VL and InternVL3 using the same selected subset. Additional experiments include component ablations, hyperparameter sensitivity analyses, and computational efficiency comparisons.
Critical strengths
-
The paper addresses the repeated computational cost of data selection across datasets and downstream MLLMs, presenting a practically relevant research problem.
-
OnceSelect achieves 99.2% of full-data aggregate performance using only 15% of LLaVA-625K, outperforming several existing selection baselines.
-
The experiments cover three instruction-tuning datasets and multiple MLLM architectures, providing encouraging empirical evidence for selector and subset reuse in the evaluated settings.
-
The proposed pipeline is straightforward and computationally lightweight. The paper also includes component ablations and selection-time measurements to evaluate its practical effectiveness.
Critical weaknesses
- The proposed method relies on the assumption that source-learned image–instruction patterns can provide meaningful selection signals for other datasets, yet this fundamental premise lacks direct theoretical or empirical justification. Assigning target samples to their nearest source centroids does not establish semantic compatibility. The current experiments on two general-domain target datasets demonstrate empirical effectiveness but do not validate the underlying assumption. Intuitively, this assumption is unlikely to hold for image–text pairs across arbitrary domains, and the paper appears to overstate its applicability without sufficient justification. The authors should provide direct evidence of source–target semantic correspondence or clearly establish the conditions and limitations under which this assumption holds, ideally through additional experiments on substantially different domains (e.g., medical image–text data).
- The MLP classifier is trained entirely on pseudo-labels derived from CLIP-based clustering, without any additional supervision related to actual downstream SFT utility. This raises a fundamental question: why not directly use the geometric information from K-Means for data selection? The paper does not establish whether learned confidence provides additional value beyond simpler measures, such as centroid distance or cluster-boundary proximity, under the same diversity constraints.
- The selector considers only images and instructions, assigning identical scores to samples with identical inputs but potentially correct, incorrect, or corrupted responses. Although input-side informativeness is a reasonable selection criterion, the paper does not investigate its robustness to response-side noise or establish when such uncertainty can reliably reflect the utility of complete SFT samples.
- The 3% centroid-nearest core ratio and 60% checkpoint accuracy target lack sensitivity analyses. The necessity of using a partially converged classifier rather than a fully trained one is also insufficiently demonstrated.
- Relevant methods such as DOSE, DICS, and CoIDO are discussed in the related work but not quantitatively compared in Table 1.
Questions for authors
- What evidence supports the central assumption that semantic patterns learned from one dataset can provide reliable selection signals for other datasets? Does this assumption hold under substantial domain shifts, such as transferring from LLaVA-625K to medical imaging datasets? Importantly, assigning every target sample to its nearest among 128 source centroids is always possible, regardless of whether meaningful semantic overlap exists. How do the authors distinguish genuine shared semantic structure from forced nearest-centroid assignment, and what are the applicability boundaries of the proposed approach?
- What additional information does the learned MLP confidence provide beyond the original CLIP representation geometry, from which its pseudo-labels are derived? Could simpler approaches based on centroid distance or cluster-boundary proximity, combined with the same cluster quotas and k-DPP selection, achieve comparable performance?
- Since the selector only considers images and instructions, two samples with identical inputs but entirely different responses (e.g., one correct and one incorrect or corrupted) would receive identical scores. How does the method account for such differences in supervision quality? What evidence supports using input-side semantic uncertainty as a reliable proxy for the actual utility of complete image–instruction–response triples in downstream SFT?
- The method involves several heuristic choices, particularly the 3% centroid-nearest core ratio and the 60% validation macro-accuracy target, neither of which is accompanied by a sensitivity analysis. Why should a partially converged classifier at approximately 60% accuracy provide a better selection signal than a fully converged classifier? How were these values determined?
- Why is ScalSelect included for LRV-Sub but omitted for Vision-Flan, and included for Qwen3-VL-4B/8B but not InternVL3-8B? Are these omissions due to implementation constraints or differences in evaluation protocols? Clarification is important given that cross-dataset and cross-model transferability are central contributions.
- The paper excludes approximately 40K text-only samples from the original LLaVA-665K dataset. Could the authors clarify the rationale for this exclusion and whether OnceSelect can handle mixed-modality instruction datasets? For example, what would happen if these text-only samples were paired with blank or random-noise placeholder images? How would this affect semantic clustering, data selection, and downstream performance?
- Do the selection-time comparisons include the same preprocessing, feature extraction, caching, and selection stages for all methods? A clearer accounting breakdown would help assess the robustness and practical efficiency advantages of OnceSelect.