- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes RankRepair for multimodal instruction data selection. The Rank stage combines alignment and diversity signals derived from frozen model representations to prioritize individual records. The Repair stage identifies underrepresented visual regions using pool-proportional coverage floors and performs replacements.
Critical strengths
- The studied topic of multimodal instruction data selection is important for efficient training.
- The empirical results seem impressive.
Critical weaknesses
-
Some highly relevant work on multimodal instruction data selection using LLaVA-665K are missing [1, 2].
-
The evaluation setting makes comparison with established results difficult. COINCIDE, ICONS, and CoIDO emphasize a 20% selection budget on LLaVA-665K, corresponding to approximately 133K records. The main benchmarks also differs substantially from those used in these studies. Evaluation on a shared benchmarks, with complete per-task results and clearly aligned budget definitions, would make the relative performance easier to assess.
-
The motivation for introducing a new priority score might be unclearer. Although the paper reports comparisons with alternative priorities, it could better explain how the proposed score differs from existing scoring criteria and which specific limitations of those criteria motivate its design. This would help clarify the contribution of the Rank stage within the overall framework.
-
Repair’s compatibility with existing allocation strategies needs further clarification. Some selection methods already incorporate explicit allocation decisions. For example, COINCIDE allocates the selection budget across clusters. It remains unclear how Repair would interact with existing clustering-based selectors.
[1] CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization. NeurIPS, 2025.
[2] Data Selection Matters: Towards Robust Instruction Tuning of Large Multimodal Models. NeurIPS, 2025.
Questions for authors
Could the authors provide a matched comparison with COINCIDE, ICONS, and CoIDO at the commonly used 20% budget (approximately 133K records), as well as align the comparison with PRISM at around 250K to benchmarks shared with prior work?
6 comments
W3: Motivation for the Two Rank Signals.
Existing selectors emphasize different criteria: CoIDO learns joint importance–diversity scores, ICONS uses cross-task influence consensus, COINCIDE combines cluster transferability with MMD-based selection, and ARDS targets bias-sensitive subgroups. Our motivation is to assess two complementary properties of each image–instruction record: whether the image supports its instruction and whether its visual content is locally redundant. Alignment alone cannot identify visual redundancy, while diversity alone may favor distinctive but poorly aligned records.
To examine this tradeoff, we apply horizontal affine shear to approximately 1% of training images while keeping their instructions unchanged. The shear progressively shifts image rows horizontally, with maximum edge displacements of 5%, 10%, and 15% of image height across three severity levels. We then recompute the features and priorities. Despite increasing local-diversity percentiles by 1.51–2.26 percentage points (pp), these perturbations reduce alignment percentiles by 18.41–19.91 pp and final ranking percentiles by 9.70–10.37 pp. This suggests that alignment can offset corruption-induced increases in diversity, preventing them from necessarily raising selection priority.
The original Table 2 further supports the complementarity of the two signals: Rank-only achieves 47.18, exceeding alignment-only (46.31) and diversity-only (45.96) by 0.87 and 1.22 points, respectively. Thus, Rank combines complementary record-level signals without training an additional scorer, while Repair separately addresses coverage shortfalls remaining after global Top-B selection.
W4: Compatibility with COINCIDE's Cluster Budgets.
| Method | GQA | VizWiz | SQA-I | TextVQA | POPE | MME | MMBench-EN | MMBench-CN | LLaVA-B | Rel. (%) |
|---|---|---|---|---|---|---|---|---|---|---|
| COINCIDE | 60.38 | 52.79 | 66.14 | 53.36 | 85.69 | 1492.00 | 59.86 | 54.80 | 64.40 | 97.53 |
| COINCIDE + Repair | 60.32 | 53.17 | 66.53 | 52.08 | 85.90 | 1509.00 | 58.99 | 55.92 | 66.30 | 97.96 |
COINCIDE determines how many records to select from each cluster, while Repair can improve which records are selected within those clusters without changing their budgets.
In this experiment, we keep the clustering, per-cluster quotas, and text-only subset fixed. Swaps are restricted to records within the same cluster to address coverage shortfalls across its radial strata, with an edit cap of 2% of the visual records.
We use the original method's distribution-discrepancy metric, maximum mean discrepancy (MMD), to measure the difference between the selected subset and the original cluster distribution, and constrain the increase in this discrepancy induced by swaps.
With this quota-preserving adaptation of Repair, the nine-benchmark relative performance improves from 97.53% to 97.96% (+0.43 percentage points). Performance improves on six benchmarks (VizWiz, SQA-I, POPE, MME, MMBench-CN, and LLaVA-Bench) and declines on the remaining three. Unlike the original priority-loss constraint, this adaptation uses within-cluster swaps with an MMD-based cost constraint. These results support the compatibility of post-selection coverage repair with existing cluster-budget allocation.
Together, these results support the competitiveness of RankRepair under shared evaluation settings and the applicability of bounded repair beyond our proposed Rank priorities. We will incorporate these additional experiments and clarifications into the revised manuscript.
Dear Reviewer,
As the discussion period is approaching its deadline, We would like to kindly ask whether our responses have adequately addressed your concerns. If you have any remaining questions or concerns, we would be more than happy to discuss them further.
Thank you for your time and consideration.
Shared-Budget Comparisons, Rank Motivation, and Repair Compatibility
We thank the reviewer for recognizing the importance of the problem and the empirical results. We provide additional per-task comparisons under shared budgets and clarify the motivation for Rank and the compatibility of Repair with existing budget-allocation strategies.
W1: Missing Related Work.
We will include CoIDO and Data Selection Matters (ARDS) in the revision and clarify their respective objectives.
While CoIDO focuses on importance–diversity optimization and ARDS targets positional bias and spurious correlations, neither explicitly addresses regional coverage shortfalls after global selection. This motivates RankRepair to separate record-level priority estimation from bounded post-selection coverage adjustment, preserving ranking priorities while correcting underrepresented regions.
W2 & Q1: Shared-Budget and Benchmark Comparisons.
To address the concern about mismatched selection budgets and evaluation benchmarks, we provide additional comparisons at two commonly used budgets: 133K (approximately 20% of LLaVA-665K) against COINCIDE, ICONS, and CoIDO, and approximately 250K against PRISM, using the benchmarks shared with prior work.
133K comparison. We evaluate nine shared benchmarks following CoIDO, excluding VQAv2 due to the unavailability of its official evaluation service.
RankRepair achieves the highest average relative performance (98.69%), exceeding ICONS (98.34%), COINCIDE (97.53%), and CoIDO (96.04%) by 0.35, 1.16, and 2.65 percentage points, respectively. It also achieves the highest scores among these subset methods on GQA, VizWiz, and MME. Although the gains vary across tasks, these results demonstrate that RankRepair remains competitive under the commonly used 20% selection budget.
The Full Data and CoIDO results are taken from CoIDO's reported full-parameter fine-tuning experiments, while the remaining results are obtained through local full-parameter fine-tuning. Relative performance is recomputed over the nine displayed benchmarks.
250K comparison. We further compare RankRepair with PRISM using the eleven benchmarks reported in PRISM's Table 1.
The Full-Finetune and PRISM results are taken from the original PRISM paper. RankRepair achieves 103.20% relative performance, compared with the 101.7% reported by PRISM, a difference of 1.50 percentage points. It outperforms PRISM on five of the eleven benchmarks, including MM-Vet, MME-P, and MMMU, while PRISM remains stronger on the other six. This indicates that RankRepair maintains competitive aggregate performance at a larger selection budget rather than benefiting only from the smaller 100K setting.
Together, these comparisons address the previously mismatched evaluation settings and provide additional evidence of RankRepair's competitiveness across selection budgets and shared benchmark suites.