acceptodds
Withdrawn from ICLR 2027

Gold-LP: Data Selection for Vision-Language Reinforcement Learning with Verifiable Rewards

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards improves vision-language models, and which questions enter training affects what the model learns. Choosing them is itself costly: selectors that estimate each candidate's difficulty from sampled responses must generate over the whole pool before optimization begins, a fifth as many responses as the training run itself, paid whether or not the resulting ranking helps. Existing selectors fall into three categories: sampling responses; reusing an earlier training run's outcomes, which requires one to exist; and substituting the model's own probability, which reduces generation by an amount the published papers do not always make clear. However, these approaches leave a fundamental question unanswered: can a useful subset be identified without generating a single response, and how much accuracy can such a subset deliver? We introduce Gold-LP: one forward pass per candidate scores the initial model's likelihood for the correct option letter under a fixed answer prefix, and a contiguous interval of the resulting ranking is retained, leaving group relative policy optimization unchanged. Training Qwen3-VL-8B-Instruct on 1,000 of 3,023 DOCCI-derived examples at a matched step budget, it reaches the highest mean accuracy among the evaluated fixed-size selections on both holdouts across three runs; the geometry gain over random selection is 1.35 points (unadjusted 95% CI [0.41, 2.30]). Which region is kept matters as much as how candidates are scored: with score and subset size held fixed, accuracy peaks at our interval rather than at the lowest-score tail on both holdouts, exceeding a published mean-centred rule in distribution by roughly four times the across-seed deviation, while three fixed-size comparators adapting published selectors do not reach it. Selection-time generation can therefore be removed entirely without giving up downstream accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data selection, data efficiency, reinforcement learning with verifiable rewards, RLVR, GRPO, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8DBpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GOLD-LP, a generation-free data selection method for reinforcement learning with verifiable rewards (RLVR) on visual multiple-choice questions. For each candidate, the initial VLM performs one forward pass under a fixed answer prefix and scores the log-probability of the correct option letter. The pool is then ranked by this score, and a contiguous interval (ranks [500, 1500) of 3,023 for the main variant, GOLDLP-MATCH) is retained for standard GRPO training. The paper argues that existing outcome-based selectors require costly front-loaded generation (48,368 responses here, about 21.5% of one training run), whereas GOLD-LP requires none. Experiments on Qwen3-VL-8B-Instruct with DOCCI-derived questions show that GOLDLP-MATCH obtains the highest mean accuracy among the evaluated 1,000-example selections, with a 1.35-point gain over random selection on a geometry transfer set. A placement ablation suggests that the retained interval matters as much as the score. However, the in-distribution gain over random is not significant, full-pool training remains clearly stronger in distribution, the geometry gain sits exactly at the adjusted significance threshold, and the second-backbone experiment reverses on geometry. In addition, the score is close to existing confidence-based selection, and the paper does not show that selection brings a practical benefit when the training budget is fixed. Therefore, while the question is relevant and the experimental reporting is careful, the novelty and the strength of the empirical evidence are insufficient.

Critical strengths
  1. Relevant problem. The paper makes the cost of obtaining a selection signal explicit and reports it alongside downstream accuracy, which prior work rarely does.
  2. Simple method. GOLD-LP needs one forward pass per candidate, no generation, and no change to the GRPO objective.
  3. Careful evaluation protocol. Answer-position balancing, rotation-averaged evaluation, paired bootstrap over items and runs, Holm correction, and final-checkpoint evaluation are all well above the norm in this area.
  4. Clean placement ablation. GOLDLP-MATCH and GOLDLP-HARD share the same score, subset size, and training budget, so their comparison isolates the effect of the retained interval.
  5. Transparency. The supplementary material provides code, selected subsets, per-run evaluation files, and statistical reports, which makes the claims checkable.
Critical weaknesses
  1. The novelty is limited. Ranking examples by gold-label confidence is established (e.g., Dataset Cartography, UFO-RL), and retaining an interior band of a ranking is the idea of Moderate Coreset. For single-token answers, the paper itself shows that the ported UFO-RL confidence reduces to the same score, so the only methodological difference is the band location.
  2. The band location is not principled. The offset L = 500 was chosen to match the never-solved coverage of a probe-based arm on this specific pool. The paper gives no reason to expect it to transfer to other pools, models, or budgets, and no sweep over offset or width is reported.
  3. The main empirical evidence is weak. The in-distribution gain over random is not significant (95% CI [−0.21, 5.33], Holm p = 0.635). The headline geometry gain has Holm p = 0.0500, computed within a comparison family assembled after the results were seen. According to the supplementary pre-registration, the primary metric was in-distribution accuracy, and GOLDLP-MATCH was framed as an equivalence test against a probe-based arm, not as superiority over random.
  4. The random baseline is weak. Only one random subset is drawn and reused across all three runs, so variance from subset sampling is not captured. Its geometry standard deviation (0.21) is the smallest among the three-run arms, which partly drives the geometry interval.
  5. Cross-backbone evidence does not support the main claim. On Gemma-3-4B, the geometry ordering reverses. The in-distribution gain (+3.44) is of the same size as the unexplained 2.52-point difference between two evaluations of the same Gemma random checkpoint.
  6. The practical benefit of selection is unclear. Because all arms are trained for the same number of steps, selecting a subset saves no training compute (about 4 hours per run for every static arm). Full-pool training costs nothing to select, is significantly better in distribution, and is statistically indistinguishable on geometry.
  7. The contribution of the score is not isolated from the interval. The score has a Spearman correlation of about 0.81 with the 16-sample pass rate, so it is largely a cheap proxy for it. The paper does not compare against the same retention rule applied to the pass-rate ranking, against coverage-matched random selection, or against a low-cost probe with one or two samples per item.
  8. The scope is narrow. The method applies only to four-option multiple-choice questions with single-token answers. All results come from one DOCCI-derived pool of about 3K items and one geometry transfer set. The untrained model's geometry accuracy is not reported, so it is unclear how much the training moves this benchmark at all.
  9. The writing needs improvement. The key distinction from UFO-RL and Moderate Coreset, namely that only the band location is new, is stated only in the related work (Section 5), near the end of the main text. It should be made explicit in the introduction so that the contribution is not overstated. In addition, the text relies heavily on colons and rhetorical questions to join or introduce clauses (e.g., "Choosing them is itself costly: selectors that ...", "We introduce GOLD-LP: one forward pass per candidate ...", "can a useful subset be identified without generating a single response, and how much accuracy can such a subset deliver?", and the question-style paragraph headings in Section 4.3). This is not conventional academic writing and makes the paper harder to read. Using complete declarative sentences would improve clarity.
Questions for authors
  1. Can the authors report the untrained Qwen3-VL-8B geometry accuracy, and results with several independently drawn random subsets?
  2. What happens if the same rank interval, or a coverage-matched interval, is taken from the 16-sample pass-rate ranking instead of the GOLD-LP ranking? Does GOLD-LP match it at lower cost?
  3. How sensitive are the results to the band offset and width (e.g., L ∈ {250, 500, 750} and B ∈ {500, 1000, 1500})?
  4. Which analyses were pre-specified? Why is the geometry result, which does not appear in the pre-registration, presented as the main evidence?
  5. Is there any setting (e.g., a smaller training budget, a larger pool, or epoch-matched training) in which GOLD-LP selection outperforms full-pool training at equal compute?
Reviewer_C1HjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a method for selecting training examples for reinforcement learning in vision-language models based on a single forward pass to score each candidate based on the model’s likelihood of the correct answer, then selects an interval from the ranking. The selection method is evaluated by training Qwen3-VL-8B-Instruct and Gemma-3-4B-IT on the selected samples and evaluating the resulting models on DOCCI and a Geometry dataset (691-question holdout subset of Geo170K image). The method achieves slightly higher mean accuracy compared to other fixed-size selection methods across two holdout sets.

Critical strengths
  • Sample section for RLVR is a relevant and interesting topic.
  • The paper is clearly written and easy to follow.
Critical weaknesses
  • Comparison to other methods: In direct comparison, the method seems to have very minor gains of 1.1% resp. 0.5% compared to the next best-performing method. Given that this was only tested on (1) a rather unusual benchmark as well as on a small subset of Geo170K, I have to say that I'm not convinced that the incremental performance of the method would justify a publication.
  • Benchmark dataset selection is not representative for current VLM RV methods. Please consider showing your improvement on a broader, especially more representative dataset selection, e.g., MMMU-Pro, MMStar, MathVision, MathVista, MathVerse, DynaMath, WeMath, LogicVista, etc.
  • The method itself also seems to be rather trivial, more an ablation baseline for a really sophisticated selection method than an algorithm worth a publication.
Questions for authors

Given the combination of incremental increase, super small benchmark selection, and very simple method, I don't see how this paper would currently justify a publication.

Reviewer_5n2KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper proposes Gold-LP, an offline data-selection method that first scores visual multiple-choice questions using the initial model’s token likelihood for the correct answer for the purpose of retaining a fixed interval of the ranking during GRPO training.
  • Experiments show that on the Qwen3-VL-8B-Instruct model, using GOLD-GLP to selecting 1,000 of 3,023 training examples achieves the highest mean accuracy among the evaluated fixed-size subsets.
  • The authors conducts ablation studies to separately examine the scoring signal, retained ranking interval, and selection overhead, showing that interval placement itself contributes to improved downstream performance.
Critical strengths
  • The proposed GOLD-GLP method is simple, inexpensive, and effective for select a moderately hard subset training data for more effectively boosting model performance.
  • Experiments show that GOLD-GLP has better numerical performance gain over random data selection and other data selection methods on Qwen.
Critical weaknesses
  • Method improvement over pervious works on likelihood-based data selection is limited. UFO-RL already combines confidence-based scoring with selecting moderately uncertain examples for RL. In my understanding, Gold-LP’s distinction lies mainly in the correct-option scoring setup and interval placement, but limited empirical analysis have been done to show these changes helps improve model performance more.
  • While experiment results show modest improvement on the Qwen model, the authors have admitted in Section 4.2 that the improvement over random selection and other methods is not statistically significant (lines 311-317). This raises concerns about the validity of the method advantage.
Questions for authors
  • Could UFO-RL achieve comparable gains by selecting an offset confidence band rather than examples near the mean confidence? What benefit does the proposed scoring procedure provide beyond adjusting the retained confidence band?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.