- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BIG, a visual instruction data selection method that measures how much the image and instruction each contribute to predicting the same reference answer. Experiments on Vision-Flan and LLaVA-665K across seven benchmarks report improvements over random, single-direction, and existing selection methods.
Critical strengths
On Vision-Flan with Qwen2-VL-2B, BIG slightly exceeds full-data average performance using 50% of the data. Ablations and selected-data analyses examine the effects of percentile calibration and minimum aggregation.
Critical weaknesses
- Experiments only consider 50% multimodal retention. So, evaluating different retention ratios and reporting total scoring and training costs would provide more stronger evidence of practical efficiency.
- Only one training seed is reported, while improvements over the strongest comparators are approximately 0.8–1.1 average RP points. Multiple training runs are needed to establish whether these gains are robust.
- It would be useful to examine whether the retained TextVQA performance is explained by OCR-text or answer coverage. Comparing these statistics with random selection and reporting results on answers unseen in the fine-tuning data would help clarify this.
Questions for authors
See the weakness
Response to Reviewer YZro
W1: Only 50% retention is evaluated; total scoring + training cost is missing.
BIG's advantage is not limited to 50%. At the commonly used budgets of 25%, 50%, and 75%, BIG clearly outperforms same-budget random selection (Random), and it exceeds full-data training at 25% and 50%; at the extremely low budgets of 5% and 10%, BIG and Random are roughly on par. In terms of cost, a single run of BIG (scoring plus training) takes about as long as full-data training while performing better.
Results per retention ratio (Vision-Flan / Qwen2-VL-2B, Avg. RP, Full-data = 100):
The margin grows with the budget (+1.45 → +1.78 → +2.33). At extremely low budgets, random selection is a well-known strong baseline: PreSel (CVPR 2025) notes at a 15% budget that "random selection outperforms many baselines"; COINCIDE (EMNLP 2024) finds on LLaVA-1.5 at a 20% budget that "all baselines perform worse than the random sampling"; and on Vision-Flan at 5%, COINCIDE, EL2N, CLIP-Score, and PRISM also fall below random selection (Sun et al., 2026, Table 7).
Total scoring + training time (8×H20, full-parameter training):
Scoring is done once, whereas training cost is paid again in every run. In workflows that involve repeated training, such as hyperparameter tuning, multiple seeds, comparing budgets, or upgrading the model, the scoring cost is amortized across all runs, and each subsequent run needs only a fraction r of the full-data training time (about 23 min at 50% vs. about 46 min for full data). The cost comparison with gradient-based methods is given in the General Response.
W2: Only one training seed; the margin over the strongest comparator is only ~0.8–1.1 points.
The margin holds across training seeds: BIG outperforms the strongest single-direction comparator, G_Q-only, in all five paired seeds, by about 1.54 points on average.
We added four training seeds for both BIG and G_Q-only (five in total: 40/41/42/43/44). The selected subsets are fixed; only the training seed changes:
BIG also outperforms Random by about 2.65 points on average. At the 25% budget, five seeds give BIG 100.15 ± 0.61 vs. Random 98.14 ± 0.55, with BIG ahead in all five seeds.
W3: Is the TextVQA performance explained by OCR-text or answer coverage?
No. BIG's answer coverage is in fact lower than Random's and its OCR-token coverage is essentially the same, yet BIG still leads consistently on questions whose answers neither subset has seen.
This is consistent with §5.5: BIG recovers examples with low image gain but high instruction gain, which require the model to read specific text in the image as directed by the question.