acceptodds
Desk rejected at ICLR 2027

RankRepair: Decoupling Record Priority and Budget Allocation for Training-Free Multimodal Data Selection

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Training vision-language models (VLMs) often requires substantial computational resources and large-scale data, making the selection of compact, high-quality instruction datasets essential for efficient multimodal learning. Selecting such subsets requires balancing individual record priority against the effective allocation of a limited data budget across the overall candidate pool. However, incorporating diversity or global structure into record-level scores does not by itself ensure adequate regional coverage in the selected subset. Regions can remain underrepresented after selection, motivating an explicit adjustment of subset-level allocation. To address this challenge, we introduce RankRepair, a novel two-stage framework that decouples record priority estimation from subset-level allocation adjustment. In the first stage (Rank), complete records are scored by integrating image–instruction alignment with local visual diversity to select an initial high-priority subset. In the second stage (Repair), the framework measures regional shortfalls relative to pool-proportional coverage floors and systematically mitigates them through record swaps while keeping priority evaluations fixed. By imposing explicit bounds on both the number of allowed swaps and the maximum priority loss per swap, RankRepair limits departure from the initial quality-based selection, ensuring each accepted swap strictly reduces the aggregate coverage deficit by one unit. Extensive evaluations on LLaVA-1.5 Mix-665K demonstrate that a 100K training subset selected by RankRepair outperforms the evaluated equal-budget baselines in mean performance across ten multimodal benchmarks. The selected subset retains 96.76% of the full-data mean performance, exceeding the strongest equal-budget baseline by 3.16 percentage points, supporting bounded coverage adjustment as an effective complement to quality-based ranking for visual instruction data selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Training-Free, Multimodal Data Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fsUyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RankRepair for multimodal instruction data selection. The Rank stage combines alignment and diversity signals derived from frozen model representations to prioritize individual records. The Repair stage identifies underrepresented visual regions using pool-proportional coverage floors and performs replacements.

Critical strengths
  1. The studied topic of multimodal instruction data selection is important for efficient training.
  2. The empirical results seem impressive.
Critical weaknesses
  1. Some highly relevant work on multimodal instruction data selection using LLaVA-665K are missing [1, 2].

  2. The evaluation setting makes comparison with established results difficult. COINCIDE, ICONS, and CoIDO emphasize a 20% selection budget on LLaVA-665K, corresponding to approximately 133K records. The main benchmarks also differs substantially from those used in these studies. Evaluation on a shared benchmarks, with complete per-task results and clearly aligned budget definitions, would make the relative performance easier to assess.

  3. The motivation for introducing a new priority score might be unclearer. Although the paper reports comparisons with alternative priorities, it could better explain how the proposed score differs from existing scoring criteria and which specific limitations of those criteria motivate its design. This would help clarify the contribution of the Rank stage within the overall framework.

  4. Repair’s compatibility with existing allocation strategies needs further clarification. Some selection methods already incorporate explicit allocation decisions. For example, COINCIDE allocates the selection budget across clusters. It remains unclear how Repair would interact with existing clustering-based selectors.

[1] CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization. NeurIPS, 2025.

[2] Data Selection Matters: Towards Robust Instruction Tuning of Large Multimodal Models. NeurIPS, 2025.

Questions for authors

Could the authors provide a matched comparison with COINCIDE, ICONS, and CoIDO at the commonly used 20% budget (approximately 133K records), as well as align the comparison with PRISM at around 250K to benchmarks shared with prior work?

6 comments
AuthorsOpenReview ↗

Shared-Budget Comparisons, Rank Motivation, and Repair Compatibility

We thank the reviewer for recognizing the importance of the problem and the empirical results. We provide additional per-task comparisons under shared budgets and clarify the motivation for Rank and the compatibility of Repair with existing budget-allocation strategies.

W1: Missing Related Work.

We will include CoIDO and Data Selection Matters (ARDS) in the revision and clarify their respective objectives.

MethodMain approachDifference from our work
CoIDOLearns a scoring model and jointly optimizes importance and diversityOur method constructs priorities from frozen representations and separately repairs post-selection coverage shortfalls
ARDSPrioritizes training samples similar to perturbation-sensitive subgroups to mitigate position bias and spurious correlationsOur method focuses on the tradeoff between record priority and subset coverage, with explicit bounds on repair cost

While CoIDO focuses on importance–diversity optimization and ARDS targets positional bias and spurious correlations, neither explicitly addresses regional coverage shortfalls after global selection. This motivates RankRepair to separate record-level priority estimation from bounded post-selection coverage adjustment, preserving ranking priorities while correcting underrepresented regions.

W2 & Q1: Shared-Budget and Benchmark Comparisons.

To address the concern about mismatched selection budgets and evaluation benchmarks, we provide additional comparisons at two commonly used budgets: 133K (approximately 20% of LLaVA-665K) against COINCIDE, ICONS, and CoIDO, and approximately 250K against PRISM, using the benchmarks shared with prior work.

133K comparison. We evaluate nine shared benchmarks following CoIDO, excluding VQAv2 due to the unavailability of its official evaluation service.

MethodGQAVizWizSQA-ITextVQAPOPEMMEMMBench-ENMMBench-CNLLaVA-BRel. (%)
Full Data62.050.066.858.285.91510.764.358.365.4100.0
ICONS60.7949.0469.8153.1885.021456.0063.5257.0466.1098.34
CoIDO57.847.467.854.681.11402.960.655.068.896.04
COINCIDE60.3852.7966.1453.3685.691492.0059.8654.8064.4097.53
RankRepair60.8553.0168.7952.8585.461501.0061.0655.7765.9398.69

RankRepair achieves the highest average relative performance (98.69%), exceeding ICONS (98.34%), COINCIDE (97.53%), and CoIDO (96.04%) by 0.35, 1.16, and 2.65 percentage points, respectively. It also achieves the highest scores among these subset methods on GQA, VizWiz, and MME. Although the gains vary across tasks, these results demonstrate that RankRepair remains competitive under the commonly used 20% selection budget.

The Full Data and CoIDO results are taken from CoIDO's reported full-parameter fine-tuning experiments, while the remaining results are obtained through local full-parameter fine-tuning. Relative performance is recomputed over the nine displayed benchmarks.

250K comparison. We further compare RankRepair with PRISM using the eleven benchmarks reported in PRISM's Table 1.

MethodSQASQA-IVizWizPOPE-PPOPE-RPOPE-AMM-VetMMBenchMME-PMME-CMMMURel. (%)
Full-Finetune69.466.850.086.187.384.231.164.31510.7311.935.4100
PRISM71.369.150.187.788.785.532.065.21470.0330.034.7101.7
RankRepair70.569.452.185.987.884.3637.263.571514324.636.2103.20

The Full-Finetune and PRISM results are taken from the original PRISM paper. RankRepair achieves 103.20% relative performance, compared with the 101.7% reported by PRISM, a difference of 1.50 percentage points. It outperforms PRISM on five of the eleven benchmarks, including MM-Vet, MME-P, and MMMU, while PRISM remains stronger on the other six. This indicates that RankRepair maintains competitive aggregate performance at a larger selection budget rather than benefiting only from the smaller 100K setting.

Together, these comparisons address the previously mismatched evaluation settings and provide additional evidence of RankRepair's competitiveness across selection budgets and shared benchmark suites.

W3: Motivation for the Two Rank Signals.

Existing selectors emphasize different criteria: CoIDO learns joint importance–diversity scores, ICONS uses cross-task influence consensus, COINCIDE combines cluster transferability with MMD-based selection, and ARDS targets bias-sensitive subgroups. Our motivation is to assess two complementary properties of each image–instruction record: whether the image supports its instruction and whether its visual content is locally redundant. Alignment alone cannot identify visual redundancy, while diversity alone may favor distinctive but poorly aligned records.

To examine this tradeoff, we apply horizontal affine shear to approximately 1% of training images while keeping their instructions unchanged. The shear progressively shifts image rows horizontally, with maximum edge displacements of 5%, 10%, and 15% of image height across three severity levels. We then recompute the features and priorities. Despite increasing local-diversity percentiles by 1.51–2.26 percentage points (pp), these perturbations reduce alignment percentiles by 18.41–19.91 pp and final ranking percentiles by 9.70–10.37 pp. This suggests that alignment can offset corruption-induced increases in diversity, preventing them from necessarily raising selection priority.

The original Table 2 further supports the complementarity of the two signals: Rank-only achieves 47.18, exceeding alignment-only (46.31) and diversity-only (45.96) by 0.87 and 1.22 points, respectively. Thus, Rank combines complementary record-level signals without training an additional scorer, while Repair separately addresses coverage shortfalls remaining after global Top-B selection.

W4: Compatibility with COINCIDE's Cluster Budgets.

MethodGQAVizWizSQA-ITextVQAPOPEMMEMMBench-ENMMBench-CNLLaVA-BRel. (%)
COINCIDE60.3852.7966.1453.3685.691492.0059.8654.8064.4097.53
COINCIDE + Repair60.3253.1766.5352.0885.901509.0058.9955.9266.3097.96

COINCIDE determines how many records to select from each cluster, while Repair can improve which records are selected within those clusters without changing their budgets.

In this experiment, we keep the clustering, per-cluster quotas, and text-only subset fixed. Swaps are restricted to records within the same cluster to address coverage shortfalls across its radial strata, with an edit cap of 2% of the visual records.

We use the original method's distribution-discrepancy metric, maximum mean discrepancy (MMD), to measure the difference between the selected subset and the original cluster distribution, and constrain the increase in this discrepancy induced by swaps.

With this quota-preserving adaptation of Repair, the nine-benchmark relative performance improves from 97.53% to 97.96% (+0.43 percentage points). Performance improves on six benchmarks (VizWiz, SQA-I, POPE, MME, MMBench-CN, and LLaVA-Bench) and declines on the remaining three. Unlike the original priority-loss constraint, this adaptation uses within-cluster swaps with an MMD-based cost constraint. These results support the compatibility of post-selection coverage repair with existing cluster-budget allocation.

Together, these results support the competitiveness of RankRepair under shared evaluation settings and the applicability of bounded repair beyond our proposed Rank priorities. We will incorporate these additional experiments and clarifications into the revised manuscript.

Dear Reviewer,

As the discussion period is approaching its deadline, We would like to kindly ask whether our responses have adequately addressed your concerns. If you have any remaining questions or concerns, we would be more than happy to discuss them further.

Thank you for your time and consideration.

Reviewer_cm7YOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose a data-selection framework RankRepair. RankRepair selects multimodal training data in two stages. It first ranks records using image–instruction alignment and local visual diversity. It then swaps a small number of records to reduce regional coverage deficits. The repair limits both the number of swaps and the priority loss. Experiments show that its 100K subset achieves the highest mean score among the evaluated equal-budget baselines.

Critical strengths
  1. Writing is clear, charts are easy to follow.
  2. The matched controls help separate targeted repair from generic subset changes. Table 1 reports the highest mean score among equal-budget baselines across three training runs.
Critical weaknesses
  1. There is a rather specific issue. The subset overlaps in Table 15 provide limited evidence of partition robustness. Each setting changes at most 2% of the same initial subset, so any two outputs must overlap by at least about 96%. Most partition settings also lack downstream results. The stability of the actual swaps and their performance gains remains unclear.
  2. I have some concerns about the novelty of this work. Ranking samples by quality and diversity is common in data selection. Balancing scores and group coverage is also well studied in constrained ranking. The paper extends these ideas through post-selection repair with limits on the number of swaps and score loss. However, the technical advance over existing methods needs clearer explanation.
Questions for authors
  1. How stable are the actual swaps and downstream gains across different partitions?
  2. What is the main technical advance of bounded repair over existing data selection and constrained ranking methods?
3 comments
AuthorsOpenReview ↗

Swap Stability, Downstream Gains, and Bounded Repair

We thank the reviewer for recognizing the clarity of our presentation and the value of our matched controls. We further distinguish bounded modification, swap stability, and downstream improvements to provide a more precise assessment of Repair.

W1 & Q1: Swap Stability and Downstream Gains.

We agree that the 2% edit cap inherently produces high overall subset overlap. We therefore directly examine downstream performance under different partitions and the gains relative to Rank-only.

FactorSettingsMean
Clusters64 / 128 / 256 / 102447.96 / 48.15 / 48.35 / 47.93
Radial strata1 / 3 / 547.03 / 48.15 / 47.88
Clustering seed42 / 123 / 340748.15 / 47.99 / 48.30

Bold values indicate the default configuration. All additional downstream evaluations use training seed 42, independently of the clustering seed. Across cluster counts and clustering seeds, Mean varies by 0.42 and 0.31 points, respectively. The default 3-stratum configuration achieves 48.15, compared with 47.03 without stratification and 47.88 with 5 strata.

Relative to Rank-only (47.18) under the same training seed, all evaluated stratified configurations yield positive gains (+0.70 to +1.17), whereas removing stratification yields -0.15. This supports limited sensitivity to the evaluated cluster counts and initialization seeds while highlighting the importance of radial stratification.

We further examine the actual records modified by Repair. Taking the shared initial subset S0S_0 as the reference, we define the inserted and removed sets for variant ii as

O_i = S_0 \setminus S_i.$$ We separately compute their pairwise Jaccard similarities, defined as intersection size divided by union size. The following table reports all ten pairwise comparisons among five partition variants. K256, R1, R5, S123, and S3407 denote configurations that change only the default cluster count to 256, the number of radial strata to 1 or 5, or the clustering seed to 123 or 3407, respectively. All other settings remain at their defaults (128 clusters, 3 strata, and seed 42). | Partition A | Partition B | Inserted-set Jaccard | Removed-set Jaccard | |---|---|---:|---:| | K256 | R1 | 31.77% | 80.22% | | K256 | R5 | 38.58% | 90.56% | | K256 | S123 | 37.36% | 90.27% | | K256 | S3407 | 38.42% | 90.37% | | R1 | R5 | 48.97% | 84.38% | | R1 | S123 | 37.81% | 80.13% | | R1 | S3407 | 37.71% | 80.05% | | R5 | S123 | 41.45% | 90.66% | | R5 | S3407 | 40.55% | 89.60% | | S123 | S3407 | 41.93% | 88.74% | These comparisons show substantially higher consistency for removed records than for inserted records (80.05–90.66% versus 31.77–48.97%). This difference is consistent with the asymmetric roles of removal and insertion in Repair. For removal, Rank priorities remain fixed across partitions. Repair selects the lowest-priority eligible records from regions whose occupancy exceeds their coverage floors. As a result, similar low-priority records can repeatedly become eligible for removal despite changes in the partition. Under the default setting, 1,877 swaps incur an average priority loss of approximately $0.022$ per swap (2.2 percentile points), below the per-swap bound of $0.05$. The resulting subset-level priority loss is $L_q=4.39\times10^{-4}$. These results indicate that Repair modifies the initial selection at a limited priority cost. For insertion, however, Repair selects high-priority unselected records from deficient regions. Changes in clustering, radial stratification, or initialization alter regional membership and coverage deficits, leading to different insertion candidates. The lower inserted-set Jaccard therefore reflects the partition-dependent nature of coverage correction rather than instability in the underlying Rank priorities. Comparisons involving the unstratified R1 configuration also show lower removed-set Jaccard (80.05–84.38%) than the remaining comparisons (88.74–90.66%). Consistently, all evaluated configurations retaining radial stratification improve over Rank-only (47.18) by +0.70 to +1.17 points, while removing stratification yields −0.15. Together, these results demonstrate relatively stable removal decisions, adaptive insertion behavior, and downstream gains across the evaluated stratified configurations, while highlighting the importance of radial stratification.

W2 & Q2: Technical Contribution of Bounded Repair.

Our main contribution is to separate record ranking from post-selection coverage adjustment. Rank combines image–instruction correspondence and local visual diversity, while Repair identifies coverage shortfalls after selection and corrects them at a bounded cost.

The distinction from existing approaches is summarized below.

MethodExplicit image–instruction pair alignmentDiversity / non-redundancyBudget allocation and repair
COINCIDE [1]×Within-cluster differences: similarity among records in a clusterAdaptive cluster-budget allocation before selection
DataTailor [2]×Within-cluster uniqueness: differences between a record and others in its clusterAdaptive task-budget allocation before selection
PRISM [3]×Global redundancy reduction: similarity between records and the overall pool×
RankRepair (ours)✓Mean Euclidean distance to the kk nearest neighbors in visual PCA spacePost-ranking repair using pool-proportional coverage floors, with bounded priority loss and swap count

Here, × indicates the absence of the corresponding explicit mechanism. Budget allocation refers to task- or region-level allocation rather than the overall subset-size constraint.

Rank first extracts alignment and diversity features to estimate record priorities and selects the global Top-B subset. Repair then asks which regions are underrepresented in the selected subset and how much modification cost can be tolerated.

For example, even if a cluster is allocated 20 records, those records may concentrate within a particular part of the cluster. Coverage floors can identify such shortfalls without requiring every region to reach an exact quota.

Bounded Repair provides verifiable guarantees on both coverage correction and modification cost. Each accepted swap reduces the aggregate coverage deficit by exactly one. Given a visual-record budget BB, at most ⌊εB⌋\lfloor\varepsilon B\rfloor swaps are allowed, with the priority loss of each swap bounded by τ\tau. Therefore, the average priority loss over the visual budget is bounded above by ετ\varepsilon\tau.

Initial selectorCoverage deficit (before → after)Mean (before → after)Gain
PRISM6,486 → 6,12346.53 → 46.91+0.38
DataTailor6,024 → 4,14746.22 → 46.76+0.54
Rank (ours)5,812 → 3,93547.18 → 48.15+0.97

All coverage deficits are measured using the same audit partition. These results show that Repair reduces coverage shortfalls and improves downstream performance across different initial selectors, while explicitly bounding the extent and cost of modification. We will incorporate the swap-stability and downstream partition analyses into the revision and further clarify this distinction from existing budget-allocation methods.

Thanks for taking our concerns seriously and adding these experiments. The swap-level analysis and downstream results give a much clearer picture than overall subset overlap alone. It is reassuring to see gains across the tested settings with radial stratification. The explanation also helps clarify what bounded repair adds to existing methods. I still see the novelty as modest, but the new evidence strengthens my positive view. I will keep my score at 3.

Reviewer_1MKTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RankRepair, a two-stage framework for multimodal data selection. In the first stage, Rank scores complete records by combining image–instruction alignment with local visual diversity to select an initial high-priority subset. In the second stage, Repair addresses coverage gaps in underrepresented visual regions through a limited number of swaps with bounded score loss, largely preserving the initial selection. Evaluations on LLaVA-1.5 Mix-665K show that the selected 100K training subset outperforms equal-budget baselines in average performance across ten multimodal benchmarks. RankRepair achieves a mean score of 48.05, compared with 46.48 for PRISM, corresponding to 96.76% and 93.60% of the full-data performance, respectively.

Critical strengths
  1. Data selection remains an important problem in large-model training. The separation between Rank and Repair is sensible: it preserves the initial ranking while allowing bounded adjustments to improve coverage, making the approach practical and interpretable.
  2. The experiments are extensive and convincing. The main comparison includes three training seeds, while additional analyses include Rank-only, Matched perturbation, Source-matched control, Full-floor enforcement, and an Exact min-cost flow oracle. These controls help clarify the contribution of the proposed repair stage.
  3. The method achieves strong performance with substantially less training data. In the 100K setting, it retains 96.76% of the full-data average performance across ten benchmarks, suggesting practical value for reducing training costs.
Critical weaknesses
  1. The partition-robustness evidence in Appendix D.1 (Table 15) is weaker than the high overlap suggests. Under the replacement cap in Section 3.4, each variant replaces at most 2% of the same initial Rank-only subset S0S_0. Any two resulting subsets therefore satisfy $\frac{|S_1 \cap S_2|}{B} \geq 1-0.02-0.02=0.96 $. Thus, the reported 96.1–96.9% overlap is largely guaranteed by the limited replacement budget and does not, by itself, provide strong evidence of robustness to the partition choice. Additional downstream evaluations across partition settings would strengthen this claim.

  2. How robust is RankRepair to a small fraction of corrupted training images, such as those with color distortions or geometric warping? Could these samples receive high local-diversity scores because they differ from nearby samples, making them more likely to be selected despite their lower quality? A small controlled corruption experiment would help clarify this potential limitation.

  3. The novelty lies mainly in the particular decoupled and bounded procedure. Cluster-based allocation, multimodal quality scoring, and constrained representation are established [1–3].

[1] Concept-skill Transferability-based Data Selection for Large Vision-Language Models. arXiv:2406.10995.

[2] Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness. ICCV 2025.

[3] PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection. arXiv:2502.12119.

Questions for authors
  1. Given that the 2% replacement cap already guarantees at least 96% overlap between variants, what evidence beyond subset overlap supports robustness to the partition choice? Do alternative radial partitions and clustering seeds preserve downstream performance?

  2. How does RankRepair behave when a small fraction of training images contain color distortions or geometric warping? Does the alignment score help prevent visually corrupted samples from being selected because of high local-diversity scores?

  3. What is the main contribution beyond prior methods [1–3]?

4 comments
AuthorsOpenReview ↗

Partition Robustness, Corruption Resistance, and Methodological Novelty

We thank the reviewer for recognizing the value of our two-stage design and matched controls. The central contribution of RankRepair is to address coverage shortfalls in a high-priority subset through a small number of measurable modifications, without reallocating the entire selection budget.

W1 & Q1: Partition Robustness and Downstream Performance.

We appreciate the observation that overall subset overlap is constrained by the edit cap. We therefore provide additional downstream results under different partitions to examine whether the benefits of Repair depend on a particular partition configuration.

Clustering groups records with similar visual representations, while radial stratification sorts records by their distances to the cluster centroid and divides each cluster into approximately equal-sized strata. A single stratum means no within-cluster subdivision. The partition is used only for coverage assessment and Repair; it does not affect Rank priorities.

We independently vary the number of clusters, radial strata, and clustering seeds, keeping the remaining settings at their defaults: 128 clusters, 3 strata, and clustering seed 42.

FactorSettingsMean
Clusters64 / 128 / 256 / 102447.96 / 48.15 / 48.35 / 47.93
Radial strata1 / 3 / 547.03 / 48.15 / 47.88
Clustering seed42 / 123 / 340748.15 / 47.99 / 48.30

Bold values indicate the default configuration. All additional downstream evaluations use training seed 42, independently of the clustering seed. The results show that finer partitions do not necessarily improve downstream performance. Increasing the number of clusters from 64 to 256 improves Mean from 47.96 to 48.35, whereas 1024 clusters reduce it to 47.93. The default 3-stratum setting achieves 48.15, outperforming the unstratified setting (47.03) and the 5-stratum setting (47.88). Across clustering seeds, Mean varies by only 0.31 points. These results suggest that performance is relatively insensitive to the evaluated cluster counts and initialization seeds, whereas radial stratification plays an important role.

Compared with Rank-only (47.18) under the same training seed, all evaluated stratified configurations yield positive gains (+0.70 to +1.17), while removing stratification yields -0.15. Increasing partition granularity changes coverage assessment without changing Rank priorities, and excessively fine partitions may spread the limited swap budget across more regions.

We further examine the actual swap sets across five partition variants. Pairwise Jaccard similarities are 80.05–90.66% for removed records and 31.77–48.97% for inserted records, directly measuring swap stability rather than full-subset overlap.

This asymmetry is consistent with the Repair mechanism. Rank priorities remain fixed across partitions, and Repair removes the lowest-priority eligible selected records from regions with sufficient coverage surplus. Consequently, similar low-priority records can remain eligible for removal across different partitions. Under the default configuration, the average priority loss per swap is approximately 0.0220.022 (2.2 percentile points), below the per-swap bound of 0.050.05. In contrast, inserted records are chosen to address partition-dependent coverage deficits. Changing the partition alters these deficits and the corresponding insertion candidates, resulting in lower inserted-set consistency. Thus, Repair maintains relatively stable removal decisions while adapting insertions to regional coverage requirements.

W2 & Q2: Robustness to Corrupted Images.

Local diversity may increase when images are corrupted. The key question is whether such increases also raise the final selection priority of corrupted records.

We apply color or geometric perturbations at three increasing severity levels to approximately 1% of the training images, keeping their paired instructions unchanged. We then recompute the features and fused priorities and compare the final ranking percentiles of affected records before and after perturbation.

For color perturbations, we blend the original image with its grayscale version at grayscale proportions of 25%, 50%, and 75%, followed by contrast reductions of 12.5%, 25%, and 37.5%, respectively. These operations progressively weaken color information and contrast.

For geometric perturbations, we apply horizontal affine shear, progressively shifting image rows horizontally while keeping the center row fixed. The maximum displacement on either side is approximately 5%, 10%, or 15% of the image height, corresponding to relative displacement between the upper and lower boundaries of 10%, 20%, or 30% of the image height. Image dimensions remain unchanged, and empty regions are filled with gray.

PerturbationSeverityAlignment percentile change (pp)Diversity percentile change (pp)Final ranking percentile change (pp)
ColorMild−2.65+0.25−1.92
ColorModerate−6.70−0.05−3.95
ColorSevere−11.25+0.63−5.84
GeometricMild−19.91+1.51−10.37
GeometricModerate−19.49+2.10−9.70
GeometricSevere−18.41+2.26−10.32

All three columns report the average post-perturbation minus pre-perturbation percentile difference for affected records, measured in percentage points (pp). Higher percentiles indicate higher scores or selection priorities, while negative values indicate decreases. For example, −1.92 means that the final ranking percentile decreases by an average of 1.92 percentage points, not that the score decreases by 1.92% in relative terms.

The final ranking percentile is obtained by applying an additional percentile transformation to the fused score. Consequently, its change need not equal the average of the changes in the two component percentiles.

Both perturbation types reduce the average final ranking percentile of affected records. For color perturbations, the decrease becomes larger as severity increases. Thus, the visual differences introduced by corruption do not translate into higher average final selection priorities. Component-level diagnostics further show that all three severity levels of both color and geometric perturbations reduce image–instruction alignment, even when local diversity increases slightly.

Because Rank selects only high-priority records, lower ranking percentiles can reduce the likelihood that corrupted records enter a low-budget subset, particularly for records near the selection cutoff. These findings therefore support a degree of resistance to corrupted-image selection under the evaluated perturbations.

W3 & Q3: Distinction from Existing Methods.

RankRepair combines image–instruction correspondence and visual distinctness in the Rank stage, then addresses coverage shortfalls in the selected subset through Repair. The differences from existing methods are summarized below.

MethodExplicit image–instruction pair alignmentDiversity / non-redundancyBudget allocation and repair
COINCIDE [1]×Within-cluster differences: similarity among records in a clusterAdaptive cluster-budget allocation before selection
DataTailor [2]×Within-cluster uniqueness: differences between a record and others in its clusterAdaptive task-budget allocation before selection
PRISM [3]×Global redundancy reduction: similarity between records and the overall pool×
RankRepair (ours)✓Mean Euclidean distance to the kk nearest neighbors in visual PCA spacePost-ranking repair using pool-proportional coverage floors, with bounded priority loss and swap count

Here, × indicates the absence of the corresponding explicit mechanism. Budget allocation refers to task- or region-level allocation rather than the overall subset-size constraint.

COINCIDE and DataTailor allocate budgets across clusters or tasks before selection, while PRISM emphasizes representation-based redundancy reduction. In contrast, RankRepair first constructs a global priority ranking and then repairs regional coverage shortfalls under explicit swap-count and priority-loss constraints, without imposing exact regional quotas.

Empirically, Repair reduces the aggregate coverage deficit from 5,812 to 3,935 (−32.3%) while retaining at least 98% of the initial visual subset, with the following downstream improvements:

Initial selectorMean before RepairMean after RepairGain
Rank (ours)47.1848.15+0.97
PRISM [3]46.5346.91+0.38
DataTailor [2]46.2246.76+0.54

These results show that bounded post-selection repair can improve existing selectors without changing their original priority functions. We will incorporate the additional partition-sensitivity, swap-stability, and corruption analyses into the revision to clarify both the effectiveness and limitations of RankRepair.

References

[1] Concept-skill Transferability-based Data Selection for Large Vision-Language Models. arXiv:2406.10995.

[2] Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness. ICCV 2025.

[3] PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection. arXiv:2502.12119.

Thank you for the detailed response; you have addressed most of my concerns, so I think I’ll keep my score.

Comments on the paper

1 comment
Area_Chair_QewLOpenReview ↗

Please read the response from authors and discuss

Dear reviewer fellows,

The author rebuttal has been posted. The rebuttal period is a critical part of our review process, allowing authors to clarify misunderstandings and address specific concerns.

Please take a moment to read the author response, engage in the discussion by replying to the authors' responses, and update your review if the rebuttal changes your assessment.

Your active participation is vital to ensuring a fair and thorough evaluation of the submissions. A constructive discussion helps us make the most informed decisions.

Thank you for your time and valuable contributions to the review process.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.