acceptodds
Withdrawn from ICLR 2027

OnceSelect: A Reusable Data Selection Framework for Efficient Multimodal Instruction Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal instruction tuning is the de facto training method for adapting multimodal large language models (MLLMs), yet the large-scale image-text datasets it relies on are often highly redundant. Effective data selection is therefore essential for efficient multimodal instruction tuning. However, existing selection methods are often tightly coupled to the specific dataset, target model, or training state used to derive their selection signals. Consequently, applying them to a new dataset or downstream model typically requires retraining the selector or recomputing model-dependent scores, leading to substantial repeated computation. This raises a fundamental practical question: can a selection signal be learned once and reused across datasets and target models? To this end, we propose **OnceSelect**, a reusable data selector for multimodal instruction tuning that is trained only once and then directly applied to unseen datasets. OnceSelect encodes image-instruction pairs in a frozen joint multimodal space, clusters them to obtain pseudo-labels that capture the coarse semantic structure, and trains a lightweight selector and retains an intermediate checkpoint selected according to a fixed validation macro-accuracy target. Samples receiving low confidence under this learned semantic partition are treated as potentially informative candidates. Because the selector is independent of the downstream MLLM, it can be reused across candidate datasets without retraining or model-dependent scoring, while each selected subset can subsequently be reused across model architectures. Using only 15% of LLaVA-625K, OnceSelect retains 99.2% of full-data aggregate performance across nine evaluation metrics. Without selector retraining, it achieves 103.4% and 102.5% relative performance on the unseen Vision-Flan-186K and LRV-Sub-180K, respectively. Moreover, the same 93.7K LLaVA subset achieves 99.9–106.7% relative performance across the evaluated Qwen3-VL and InternVL3 models without model-specific scoring or reselection. Together, these results establish OnceSelect as a reusable and efficient data selection framework across datasets and downstream model architectures. Code and the trained selector will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Model, Multimodal Instruction Tuning, Data Selection, Reusable

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nm2iOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OnceSelect, a reusable data selection framework for efficient multimodal instruction tuning. The key motivation is to reduce repeated selection costs across different datasets and downstream MLLMs. The method first extracts frozen CLIP representations of image–instruction pairs and performs semantic clustering to construct pseudo-labels. A lightweight gated classifier is then trained on centroid-nearest representative samples, with an intermediate checkpoint retained to estimate sample informativeness through prediction uncertainty. During selection, low-confidence candidates are prioritized within each semantic cluster, followed by uncertainty-weighted blockwise k-DPP sampling to balance informativeness, diversity, and semantic coverage.

The authors evaluate OnceSelect on LLaVA-625K, reporting 99.2% of full-data aggregate performance using only 15% of the training data. They further investigate cross-dataset transfer to Vision-Flan and LRV-Sub without retraining the selector, as well as cross-model transfer to Qwen3-VL and InternVL3 using the same selected subset. Additional experiments include component ablations, hyperparameter sensitivity analyses, and computational efficiency comparisons.

Critical strengths
  1. The paper addresses the repeated computational cost of data selection across datasets and downstream MLLMs, presenting a practically relevant research problem.

  2. OnceSelect achieves 99.2% of full-data aggregate performance using only 15% of LLaVA-625K, outperforming several existing selection baselines.

  3. The experiments cover three instruction-tuning datasets and multiple MLLM architectures, providing encouraging empirical evidence for selector and subset reuse in the evaluated settings.

  4. The proposed pipeline is straightforward and computationally lightweight. The paper also includes component ablations and selection-time measurements to evaluate its practical effectiveness.

Critical weaknesses
  1. The proposed method relies on the assumption that source-learned image–instruction patterns can provide meaningful selection signals for other datasets, yet this fundamental premise lacks direct theoretical or empirical justification. Assigning target samples to their nearest source centroids does not establish semantic compatibility. The current experiments on two general-domain target datasets demonstrate empirical effectiveness but do not validate the underlying assumption. Intuitively, this assumption is unlikely to hold for image–text pairs across arbitrary domains, and the paper appears to overstate its applicability without sufficient justification. The authors should provide direct evidence of source–target semantic correspondence or clearly establish the conditions and limitations under which this assumption holds, ideally through additional experiments on substantially different domains (e.g., medical image–text data).
  2. The MLP classifier is trained entirely on pseudo-labels derived from CLIP-based clustering, without any additional supervision related to actual downstream SFT utility. This raises a fundamental question: why not directly use the geometric information from K-Means for data selection? The paper does not establish whether learned confidence provides additional value beyond simpler measures, such as centroid distance or cluster-boundary proximity, under the same diversity constraints.
  3. The selector considers only images and instructions, assigning identical scores to samples with identical inputs but potentially correct, incorrect, or corrupted responses. Although input-side informativeness is a reasonable selection criterion, the paper does not investigate its robustness to response-side noise or establish when such uncertainty can reliably reflect the utility of complete SFT samples.
  4. The 3% centroid-nearest core ratio and 60% checkpoint accuracy target lack sensitivity analyses. The necessity of using a partially converged classifier rather than a fully trained one is also insufficiently demonstrated.
  5. Relevant methods such as DOSE, DICS, and CoIDO are discussed in the related work but not quantitatively compared in Table 1.
Questions for authors
  1. What evidence supports the central assumption that semantic patterns learned from one dataset can provide reliable selection signals for other datasets? Does this assumption hold under substantial domain shifts, such as transferring from LLaVA-625K to medical imaging datasets? Importantly, assigning every target sample to its nearest among 128 source centroids is always possible, regardless of whether meaningful semantic overlap exists. How do the authors distinguish genuine shared semantic structure from forced nearest-centroid assignment, and what are the applicability boundaries of the proposed approach?
  2. What additional information does the learned MLP confidence provide beyond the original CLIP representation geometry, from which its pseudo-labels are derived? Could simpler approaches based on centroid distance or cluster-boundary proximity, combined with the same cluster quotas and k-DPP selection, achieve comparable performance?
  3. Since the selector only considers images and instructions, two samples with identical inputs but entirely different responses (e.g., one correct and one incorrect or corrupted) would receive identical scores. How does the method account for such differences in supervision quality? What evidence supports using input-side semantic uncertainty as a reliable proxy for the actual utility of complete image–instruction–response triples in downstream SFT?
  4. The method involves several heuristic choices, particularly the 3% centroid-nearest core ratio and the 60% validation macro-accuracy target, neither of which is accompanied by a sensitivity analysis. Why should a partially converged classifier at approximately 60% accuracy provide a better selection signal than a fully converged classifier? How were these values determined?
  5. Why is ScalSelect included for LRV-Sub but omitted for Vision-Flan, and included for Qwen3-VL-4B/8B but not InternVL3-8B? Are these omissions due to implementation constraints or differences in evaluation protocols? Clarification is important given that cross-dataset and cross-model transferability are central contributions.
  6. The paper excludes approximately 40K text-only samples from the original LLaVA-665K dataset. Could the authors clarify the rationale for this exclusion and whether OnceSelect can handle mixed-modality instruction datasets? For example, what would happen if these text-only samples were paired with blank or random-noise placeholder images? How would this affect semantic clustering, data selection, and downstream performance?
  7. Do the selection-time comparisons include the same preprocessing, feature extraction, caching, and selection stages for all methods? A clearer accounting breakdown would help assess the robustness and practical efficiency advantages of OnceSelect.
Reviewer_DKjtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OnceSelect, a reusable data selection framework for efficient multimodal instruction tuning. It clusters frozen CLIP image–instruction representations and trains a lightweight classifier on centroid-nearest samples using cluster assignments as pseudo-labels. A partially converged checkpoint provides confidence scores, with low-confidence candidates prioritized and refined through cluster-wise, blockwise k-DPP sampling to balance informativeness, coverage, and diversity. The selector is reused across datasets, and selected subsets are reused across downstream model architectures.

Experiments on LLaVA-625K, Vision-Flan, and LRV-Sub report that retaining 15% of the data achieves aggregate performance close to or above full-data training. Additional experiments evaluate subset transfer to Qwen3-VL and InternVL3, selection costs, component ablations, and sensitivity to selection budgets, cluster counts, and representation backbones.

Critical strengths

A practical framework for reusable data selection. OnceSelect separates selection from the downstream MLLM through frozen representations and a lightweight selector. Reusing the selector across datasets and the selected subset across model architectures addresses a meaningful operational cost in instruction tuning. The framework’s simplicity makes this contribution practically appealing.

Promising empirical evidence under substantial data reduction. Using only 15% of the candidate data, the complete pipeline achieves competitive aggregate performance and outperforms random selection across the evaluated dataset and model settings. These results support the practical utility of the overall approach, even though further controls are needed to establish which proposed components drive the gains.

Critical weaknesses

The experiments do not sufficiently isolate the contribution of the learned confidence signal. OnceSelect combines cluster-proportional allocation, low-confidence selection, and diversity-aware sampling. The existing ablations show that k-DPP helps when confidence-based selection is retained, and that low-confidence selection outperforms high-confidence selection. However, neither establishes that the learned selector adds value beyond coverage and diversity alone. Controls retaining the same clustering and sampling procedure while replacing confidence with random rankings or simple centroid-based scores are missing. This is consequential because the results demonstrate the effectiveness of the complete pipeline without adequately establishing the incremental benefit of its central proposed contribution.

The rationale for partial convergence and the transferability of its uncertainty signal remain insufficiently validated. The selector checkpoint is chosen to achieve approximately 60% validation macro accuracy, but the paper does not compare this choice with fully converged selectors or alternative accuracy targets. Moreover, the reported component ablations focus on the source dataset, leaving unclear whether the confidence signal itself provides benefits on unseen datasets beyond those obtained from frozen representations and diversity sampling. Checkpoint sensitivity and controlled ablations on target datasets are needed to substantiate the claim that partial convergence produces a useful, reusable selection signal.

Questions for authors

I would like to see whether the classifier is needed to obtain these gains. Could you keep the cluster quotas and k-DPP procedure unchanged and replace confidence with random or centroid-based scores? This comparison would be particularly useful on the unseen datasets.

It is unclear to me why 0.60 was chosen as the target validation accuracy. How was this value determined, and what happens if the selector is trained to convergence? A comparison across checkpoints would help explain why partial convergence is beneficial.

For Qwen3-VL and InternVL3, it would help to include the original Instruct models before additional tuning. I wonder whether the gains over full-data tuning partly reflect less forgetting from fewer updates. A comparison at matched training budgets would help resolve this.

Have you checked whether the main gains hold across random seeds? I am particularly interested in the interaction-branch ablation, since its aggregate improvement appears to depend heavily on MME-C.

Reviewer_6XyHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OnceSelect, a data selection method for multimodal instruction tuning. It uses frozen CLIP features to group image–instruction pairs and trains a small classifier using cluster labels. The method selects low-confidence examples and then uses diversity-based sampling to build the final subset. The trained selector can be reused on other datasets, and the selected subset can be reused across target models. Using 15% of LLaVA-625K, the method achieves 99.2% of the full-data score under the paper’s average relative metric. The paper also reports results on two other datasets and on Qwen3-VL and InternVL3 models, together with a large reduction in selection cost.

Critical strengths
  1. The reuse experiments address a useful practical problem. The paper tests both a frozen selector on new datasets and a fixed subset on different target models. These are two useful forms of reuse. The results on Qwen3-VL and InternVL3 make the evidence stronger than an evaluation limited to LLaVA.
  2. The method offers a promising balance between data size, model performance, and selection cost. It compares against several recent methods and reports individual benchmark scores. Using only 15% of the data while remaining close to full-data training on average could make repeated training experiments more affordable.
Critical weaknesses
  1. The added value of the trained confidence scores is not clearly established. A central claim is that low confidence from a partly trained cluster classifier helps identify useful examples. Table 4 compares low-confidence and high-confidence selection, but neither is a control without learned confidence scores. The paper does not compare against diversity sampling within the same clusters without confidence filtering, or a simple score based on distance to the cluster center. It also does not show how results change with the target classifier accuracy of 0.60. These controls matter because they would show whether training the selector is necessary for the reported gains.
  2. The evaluation does not clearly separate method choices from final testing. The appendix explains the default encoder, selection ratio, and cluster count using their results on the reported benchmarks. It is unclear whether these choices were fixed before final evaluation or selected using those results. The paper also reports no variation across random seeds. This matters because the main average gain over ScalSelect is about 1.6 percentage points, and some component results differ substantially across tasks. A separate development setup and a small number of repeated runs would make the comparison more convincing.
Questions for authors
  1. At the same 15% budget, how does OnceSelect compare with diversity sampling within the same clusters without confidence filtering, and with selection based on distance to the cluster center? Does the advantage remain when the target classifier accuracy changes from 0.60?
  2. Which data or benchmarks were used to choose the encoder, cluster count, and selection ratio? Were these choices fixed before the final evaluation? Could you report variation across a few random seeds for OnceSelect and the strongest baseline on the main dataset?
  3. For Qwen3-VL and InternVL3, could you provide the scores before additional instruction tuning and the full training settings? The random-subset comparisons are useful, but the starting scores would help show whether the selected data improves the models or mainly reduces the performance loss caused by further training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.