acceptodds
Under review as a conference paper at ICLR 2027

Gradient Eligibility Is Not Learning Utility: Hidden Curricula in Vision–Language RL

Abstract

General-purpose vision–language reinforcement learning often trains a single policy across heterogeneous perception and reasoning tasks using group-relative advantages with binary correctness feedback. Only rollout groups containing both correct and incorrect answers yield nonzero relative advantages through this outcome channel, making each query's contribution policy-dependent. Dynamic sampling amplifies this selection by discarding zero-variance groups and refilling the batch, so queries that remain mixed accumulate more updates, creating a hidden curriculum. We ask whether this persistence signals useful learning opportunities or merely reflects continued outcome variability. Before RL, we predict which queries will remain mixed using initial solvability and response demand, measured as the thinking length elicited by a fixed probe VLM. Across Qwen2.5-VL and Qwen3-VL, higher demand predicts longer persistence after accounting for initial solvability and query composition. Under dynamic sampling, the highest-demand decile consequently passes the filter – as often as the lowest-demand decile. Persistence, however, does not imply learning utility. Across matched three-seed interventions at two policy states, high-demand pools pass the filter more often and consume more generated response tokens. Starting from the base policy, both arms improve, but low-demand training gains percentage points more at pass@; all six paired pass@ comparisons favor low-demand training. A past-only inverse-acceptance controller reduces exposure drift by at near-matched cost, with higher observed pass@ and pass@. Thus, controlling which queries receive updates is distinct from deciding which updates are useful.

Primary area:
reinforcement learning
Keywords:
Vision–Language Models, Reinforcement Learning with Verifiable Rewards (RLVR), Group-Relative Policy Optimization (GRPO), Data Filtering, Curriculum Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3VokOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes the general problems locating in the general-purpose RL in LVLM where the rollout groups containing both correct and incorrect answers yield nonzero relative advantages.

It proposes that the hidden curriculum created by the dynamic sampling ( mixed through the discarding and refilling) signals useful learning opportunities. This paper asks the key question: Do queries remain eligible because they still offer useful learning opportunities, or simply because their outcomes remain variable?

This paper aims to answer this question through two steps: 1. eligibility persistence and 2. allocating more updates to these persistent queries produces greater held-out gains.

Critical strengths
  1. The propose of the hidden curriculum, based on the dynamic sampling strategy, it proposes a group-relative reward to decide which samples have the gradient eligibility and apply filter-and-refill for optimization.

  2. This paper also investigates the initial solvability and eligibility persistence as a initial inverstigation/exploration

  3. The core of this paper is exploring the connection of eligibility and utility for a sample through the controlled experiments. This paper proposes thte matched intervention to organize the different training pool with low/high-demand. The experiment result show that the consistent gradient dose not provide a suifficient reason for further training.

  4. This paper proposes inverse-acceptance sampling controller for solving this problem.

Critical weaknesses
  1. This paper proposes a very interesting and useful direction for RL in LVLM. Thus, a comprehensive experiments should be conducted while this paper only focus on a very limited range of backbones.

  2. The effectiveness of the controller is limited becasue of the insuifficient experiments.

Questions for authors

see above

4 comments

Dear Reviewer 3Vok, thank you for recognizing both the central question of our paper and its potential for LVLM RL. Your observation that persistent eligibility alone does not justify further training captures the distinction we hoped to make. We also appreciate your encouragement to support this direction with broader experiments.

Following your suggestions, we added a non-Qwen backbone, a fresh three-seed controller study, and utility experiments with larger pools and longer training. These experiments use GRPO with DAPO-style dynamic sampling. EMA retains the same filter-and-refill procedure and adjusts candidate sampling.

1. Broader backbone coverage: MiMo-VL-SFT-7B

We added experiments with MiMo-VL-SFT-7B to test three aspects of the paper: whether demand predicts persistence across model families, whether the resulting exposure imbalance can be controlled, and how demand relates to learning utility.

Demand and persistence transfer beyond Qwen. We reused demand measured by the Qwen3-VL-4B-Thinking probe. For training efficiency, we excluded queries with all eight MiMo base-policy profiling responses correct, then constructed a 1,064-query pool with 266 queries per demand bin. This balances bin counts rather than initial solvability.

MeasureD1D4
Candidate share, steps 1–10025.06%25.02%
Accepted share, steps 1–10020.66%29.66%
Mixed-group rate, steps 1–2062.03%72.49%
Mixed-group rate, steps 81–10042.43%66.76%

Despite approximately equal candidate shares, D4 receives 1.44 times as much accepted exposure overall. Its mixed-group rate also remains higher late in training. Thus, the allocation pattern appears in a non-Qwen target using demand measured by a Qwen probe.

Utility experiments on MiMo. We also constructed 346 matched query pairs: low demand contains 132 D1 and 214 D2 queries, and high demand contains 346 D4 queries. Pairs match MiMo’s observed initial correct count out of eight rollouts, task category, answer format, profiling horizon, and pool origin. Both arms were trained with LoRA for 80 steps across two seeds.

Both training pools are approximately balanced between perception and reasoning. The held-out evaluation panel contains 400 queries with equal representation of the two task families. The following comparison uses low-demand step 40 and high-demand step 20, at nearby generation budgets:

SeedLow generation tokensHigh generation tokensLow / High tokensOverall Low−High pass@1 (Perception)
112.65M12.47M1.014×+0.20 pp (+1.38 pp)
212.84M12.01M1.069×+0.77 pp (+1.78 pp)

Generation costs include rejected candidates and refill. Low-demand training receives twice as many updates with only 1.4% and 6.9% more generation tokens. Across the two seeds, the mean perception gain is +1.58 pp [query-bootstrap 95% CI: +0.50, +2.77]; the overall gain is +0.48 pp [−0.23, +1.24].

These results provide an example of cheaper rollouts funding useful additional updates on a non-Qwen backbone. They also support treating demand as one axis for task-aware allocation, rather than a universal ranking of training data.

We additionally expanded the Qwen utility experiments beyond the original 80-query pools, using 141 queries per arm for 20 steps and 200 queries per arm for up to 40 steps. In the 200-pair comparison, low-demand training achieves +2.31 pp pass@1 [+0.95, +3.72] at near-matched generation budgets, with one training seed.

2. Stronger evidence for the controller

We addressed the limited controller experiments in two ways: repeating the Qwen comparison across three fresh training seeds and implementing the controller on MiMo.

Three-seed Qwen replication. Within each seed, standard and EMA branch from the same step-40 model and optimizer checkpoint, reset the dataloader, and receive the same 2,560 accepted groups through step 80.

SeedExposure-drift reductionEMA / standard generation tokensEMA−standard pass@1EMA−standard pass@8EMA−standard pass@16
162.03%0.998×−0.14 pp+0.44 pp+1.00 pp
255.34%0.970×+0.39 pp+0.15 pp+0.25 pp
359.95%0.901×+1.23 pp+1.64 pp+2.75 pp
Mean——+0.49 pp+0.74 pp+1.33 pp

Drift and generation costs cover steps 41–80. Performance uses the 16K + 32K evaluation described in our response to Reviewer VRkz.

Exposure control reproduces across all three seeds, and pass@8/16 favors EMA in every seed under this evaluation. The 95% intervals over seed-paired differences are [−1.23, +2.22], [−1.23, +2.71], and [−1.85, +4.52] pp for pass@1, @8, and @16, respectively. These results strengthen the evidence for reliable allocation control, with directionally consistent high-k gains whose average magnitude remains uncertain across training seeds.

Controller transfer to MiMo. We branched standard and EMA from the same MiMo step-40 checkpoint. Both restored the training state, reset the dataloader, and received 1,920 accepted groups over steps 41–100.

MeasureStandardEMA
Accepted-share drift from the uniform target, total variation0.065630.00625
D4 / D1 accepted exposure1.55×1.03×
Training-generation tokens17.30M16.09M
Logged training-step time4.80 h4.57 h
Step-100 overall pass@1 (Perception)77.70% (78.69%)77.78% (79.31%)

In this single-seed experiment, EMA reduces exposure drift by 90.5%, using 7.0% fewer generation tokens and 4.9% less logged training-step time. The overall pass@1 difference is +0.08 pp [query-bootstrap 95% CI: −0.59, +0.72].

Together, the Qwen and MiMo experiments demonstrate reproducible exposure control across training seeds and extend it to a non-Qwen model family.

3. Beyond EMA: using demand to plan training

In this work, we showed that response demand helps predict eligibility persistence, that greater persistence does not necessarily yield greater learning utility, and that EMA can correct exposure imbalance across demand bins. There is still room to improve how those updates are allocated. As training progresses on a fixed pool, low-demand queries can stop producing mixed groups quickly. Maintaining their accepted share can then require more generation and refill, while repeatedly directing updates to the few queries that remain mixed. Balancing demand bins therefore does not by itself ensure useful or broad coverage within each bin.

Our next goal is to use demand to decide how sampling and the training pool should change as the model learns. The first experiment we would like to run is replenishing low-demand queries as their mixed signal declines, testing whether fresh queries provide better learning per token than repeatedly sampling the remaining eligible ones. More broadly, we envision a curriculum organized jointly by demand and initial solvability, with different timing, sampling weights, and replenishment strategies for each quadrant. Current success rates and query-level exposure would help determine when to revisit queries and when to move on.

We also want to make this approach practical for data curation. A lightweight demand predictor could reduce external-probe generation on new queries. In VLMs, combining this view with error analysis could also help identify which parts of the model need further training: if persistent errors on short-response perception tasks arise from visual representations, adapting the vision encoder may be more useful than repeatedly sampling the same queries. These are directions we hope to test beyond the current paper. We will clarify this progression in the revision, and we appreciate your encouragement to develop it further.

Thanks to the rebuttal which answers my questions and I will keep the score.

Thank you for your constructive feedback and for taking the time to discuss our work with us. We enjoyed the discussion and are glad the rebuttal addressed your questions!

Reviewer_BJq1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how dynamic sampling in group-relative reinforcement learning (RL) implicitly alters the training distribution of vision-language models (VLMs). Because the binary-outcome filter only retains rollout groups containing both correct and incorrect responses, gradient-eligible queries receive disproportionately more optimizer updates, giving rise to an implicit curriculum. Empirical evaluations across Qwen2.5-VL and Qwen3-VL demonstrate that high-demand queries remain eligible longer and are accepted more frequently than low-demand queries.

Critical strengths
  • The paper identifies an implicit curriculum mechanism inherent to dynamic sampling, providing a clear mathematical formalization of how binary filtering skews the effective query distribution relative to the nominal dataset.
  • It is interesting that initial policy can reliably predict the informativeness of each individual samples
Critical weaknesses
  • check the writing style of the paper, it now contains a lot of short paragraphs (2-3 sentences) and redundant new concepts: eligibility kernel, refill, mixed occupancy, Accepted exposure, uninterrupted residence ... Many paragraphs read like coding agent outputs (e.g. section 5.1)
  • if the method addresses the problem of GRPO, as authors stated in the conclusion, you should highlight this in the abstract as it doesnt improve over all ˋGeneral-purpose vision-language reinforcement learning' at all
  • limited baselines: I would expect more curriculum RL methods to be compared. if the probe model is to predict the difficulty of each samples than the method should at least compare against hard-mining approaches
  • Missing related work on multi-task VLM RL: https://arxiv.org/abs/2510.21978
Questions for authors
  • The primary utility interventions match the accepted-group budget, under which high-demand training consumes more generated tokens. How would the performance comparison change under a controlled experiment (e.g., wall-clock time)
  • Response demand is measured through the thinking-token length of an external probe model. To what extent does this measure capture intrinsic query difficulty rather than probe-specific reasoning behavior
3 comments

Dear Reviewer BJq1, thank you for recognizing both our mathematical formalization of how binary filtering changes the effective training distribution and the possibility of anticipating these allocation patterns from pre-RL information. Connecting the filtering mechanism to what we can predict before training is central to our paper, and we appreciate your recognition of both contributions.

The next question is whether that predictable participation translates into useful learning. Our pre-RL measurements predict eligibility persistence: a query’s tendency to continue producing mixed groups containing both correct and incorrect answers. Our controlled interventions then test its utility. More persistent pools did not consistently yield greater held-out gains, despite consuming more generation tokens. This distinction is what our title emphasizes. D1 and D4 denote the lowest- and highest-demand quartiles.

Q2. What does response demand add beyond solvability?

Existing data selection uses two kinds of difficulty:

  • Offline: initial solvability, measured before training.
  • Online: success rates during training, as used by the dynamic-sampling filter and variance-based selectors.

These success rates describe where the policy stands. Demand helps predict what follows: which queries will remain mixed, and at what generation cost?

(1) Beyond initial solvability: predicting persistence.

  • Figure 4: two MMK12 geometry queries share initial solvability .188, but their demands are 6.8k vs. 1.2k tokens.
  • Table 7: 86.2% of D1 queries reach the stable all-pass criterion, compared with 38.6% of D4 queries. Among queries reaching that criterion, the median candidate-exposure count is 3 for D1 and 14 for D4.

(2) Does greater persistence translate into greater utility? Our utility interventions match query pairs on category, answer format, and observed solvability at the starting policy. The original early pools and the new Qwen pools use exact correct-count matching; the original late pools match step-40 solvability within a 3/16 caliper. Within these matched conditions, demand separates pools with different persistence and generation costs:

  • D4 used 1.68–2.12× as many generated tokens at the same accepted-group budget in the original Qwen interventions. In MiMo-VL, the corresponding ratio was 1.80–1.85× at equal 80-step budgets.
  • D4 stayed mixed longer without establishing an overall held-out advantage in these comparisons. All six seed-paired pass@1 comparisons across the two original Qwen policy states favored D1.

(3) What demand is for: forecasting the training budget.

  • Low demand: cheap per update, but exhausted quickly. Starting from the base policy, D1 passed the filter at a rate of .224 (D4: .597). It needed 2.67× as many candidate groups for the same accepted-group budget.
  • High demand: lasts longer, but expensive. That persistence did not yield larger held-out gains.

So the question is not which queries are better. It is how much learning signal remains and what that signal costs per token. Dynamic sampling spends its budget on queries that stay mixed, and demand predicts which ones will, before training. Since staying mixed does not mean being useful, budgets should be planned by value per token.

(4) What transfers beyond the probe?

Demand is measured under a fixed probe and decoding setup, but carries information that transfers across targets.

  • Cross-family transfer. In MiMo-VL, Qwen-probe demand is associated with +4.33 pp mixed occupancy per SD of log(1 + demand) [+2.45, +6.22], controlling for observed solvability, category, answer format, and log observation count. Adding source, pool origin, and target-self length gives +2.23 pp [−0.22, +4.69].

  • Target-dependent added value. Beyond a rich pre-RL profile including target-self length, demand adds only .0042 held-out R² in Qwen3-Thinking, versus .0725–.0870 in Qwen2.5. Qwen2.5’s shorter 2,048-token horizon may partly explain this difference.

W3. Curriculum and hard-mining baselines

We agree that a controlled comparison with curriculum and hard-mining methods would strengthen the practical case for demand-aware allocation. Our matched interventions control observed starting solvability, but they do not substitute for a comparison against a selector that adapts to success rates during training.

The results motivate a concrete comparison: the same success-rate-based selector with and without demand, evaluated at comparable generation budgets. This would test whether demand’s information about persistence and cost improves allocation beyond difficulty-based selection alone. We could not complete this baseline within the rebuttal period because of compute limits, having prioritized the additional seeds, non-Qwen experiments, and utility comparisons. We plan to pursue this comparison and would welcome your suggestions on the most relevant baseline.

Q1. What happens under matched compute?

Short answer: At a near-equal token budget, low-demand training still wins at pass@1 in Qwen (one training seed). MiMo provides an additional two-seed comparison at nearby generation budgets.

Qwen

MeasureLow (D1+D2), step 30High (D4), step 20
Generated tokens, incl. rejected candidates and refill80.77M79.09M
Logged training-step time17.99 h18.19 h
pass@1, primary evaluation67.78%65.47%
pass@1, fixed 16K64.58%62.42%
  • Setup: 200 new queries per arm, pair-matched on base correct count (/16), category, and answer format. Low was trained until it first reached High's step-20 budget, and the checkpoint was chosen by cost only.
  • Difference in pass@1: +2.31 pp [+0.95, +3.72] under the primary evaluation and +2.16 pp [+0.78, +3.58] at fixed 16K. The pass@8/16 intervals include zero.
  • Primary evaluation: For any query with a capped 16K response, 16 fresh 32K responses replace the whole group. We adopted this after seeing the 16K results, so we report both.

MiMo-VL

We compared low-demand (D1+D2) step 40 with high-demand (D4) step 20 across two training seeds. Each arm uses 346 queries, approximately balanced between perception and reasoning. The held-out evaluation panel contains 400 queries with equal representation of both task families.

Training seedLow tokensHigh tokensLow / High tokensLow timeHigh timeOverall Low−High pass@1 (Perception)
Seed 112.65M12.47M1.014×4.45 h4.09 h+0.20 pp (+1.38 pp)
Seed 212.84M12.01M1.069×3.42 h2.90 h+0.77 pp (+1.78 pp)

Low receives twice as many updates with only 1.4% and 6.9% more generation tokens. Overall and perception pass@1 differences favor low in both seeds, with larger gains on perception. Together with Qwen, these results illustrate how cheaper rollouts can fund useful additional updates and motivate task-aware allocation. MiMo’s logged training time is 9–18% higher for low, so this is a comparison at nearby generation budgets rather than matched training time.

Measurement scope: Generation tokens include rejected candidates and refill. Times sum the corresponding training steps, including rollout and optimization. Both exclude profiling and evaluation. Qwen uses one training seed and MiMo uses two; the reported confidence intervals resample evaluation queries and condition on the trained seeds.

The last response is about writing and our vision.

W1. Writing

We agree. We will reduce the terminology to three core distinctions (eligibility, accepted exposure, and held-out utility), rewrite Section 5.1 as one continuous account, and reorganize the introduction along the lines of this response.

W2. Abstract scope, and why general-purpose VLM RL

We agree on scope. We will explicitly scope the abstract to binary-correctness group-relative RL with dynamic sampling.

Why this matters for general-purpose VLM RL: filtering determines how a shared training budget is distributed across heterogeneous tasks.

  • Many tasks, one budget. General-purpose VLM RL trains heterogeneous tasks (charts, spatial reasoning, math, science) with one policy and one update budget, as in public reports such as Kimi K2.5 and MiMo-VL. Rewards, curricula, and data mixtures matter greatly, and they are often tuned benchmark by benchmark.
  • The filter is task-agnostic. Queries within one task differ widely in solvability and demand; in Figure 2, one AI2D category spans all four regions. The filter turns these query-level differences into task-level exposure imbalance.

This work started from one question: beyond per-benchmark tuning, what query-level understanding helps plan multi-task training? Response demand came out of that question. Our sampler is a first use of it, and we see demand as a starting point for data curation, task-mixture design, and curricula.

W4. RECAP

Thank you for the pointer. RECAP uses replay and online objective reweighting to preserve general capabilities during reasoning RL, which is directly relevant to multi-task allocation. We see our work as complementary to it and will add RECAP to the related work.

We are investigating our next step, demand as a pre-RL routing signal.

  • Difficulty × demand quadrants. Each quadrant could get its own timing, weight, and replenishment policy. For example, fresh low-demand queries could be added before the low-demand pool runs dry. Our controller would then realize the chosen allocation after filtering. The uniform target in the paper was only a diagnostic.

Your questions helped us articulate the role of response demand: adding a view of persistence and generation cost to the success-rate information already used in training. We believe combining these views can support better data curation and curriculum design, and we would welcome your thoughts.

Reviewer_VRkzOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how correctness-based filtering changes the data used in group-relative vision-language Reinforcement learning. It distinguishes whether a query can contribute a correctness gradient, how often it's accepted during training, and how much training on it actually helps. Filter-and-refill sampling favors queries whose rollout groups are more likely to contain both correct and incorrect answers. Experiments with 3 Qwen policy settings show that queries eliciting longer responses from a fixed probe tend to stay mixed longer and receive more training exposure. In matched low- and high-demand pools, low-demand training gives higher held-out pass@1 in all six seed-paired comparisons, despite using fewer generation tokens. The 45622 paper also test an inverse-acceptance controller that reduces exposure drift, although its performance results are preliminary.

Critical strengths

S1. Matching the low- and high-demand pools and giving them the same accepted-group budget makes the comparison more convincing. These training experiments directly support the paper's distinction between eligibility and learning utility.

S2. The checks for truncation and parsing failures strengthen confidence in the observed mixedness gap. Separating completion reliability from conditional correctness also help explain where the performance gains come from.

Critical weaknesses

W1. I personally think the utility experiments are quite small: one fixed pair of 80-query pools per policy state and just 20 training steps. Three seeds don't tell us whether the result holds for different pools (not enough), and matching accuracy estimated from 16 rollouts can still leave differences in true solve probabilities. The late result is also uncertain across seeds and mostly reflects better completion rather than improved correctness among completed answers.

W2. The controller reduces exposure drift, but there's only one training run per arm. The pass@8 and pass@16 bootstrap intervals do not address variation across training runs, and the pass@1 interval includes zero. The reported token cost covers training generation rather than the external demand profiling, so the full cost remains unclear.

Questions for authors
  1. How's the matched pools selected when multiple queries met the criteria? Do the results hold for other valid pool pairs?

2.Were the standard, oracle, and EMA arms run from the same checkpoint with the same training settings and update budget, apart from sampling? What do the reported costs include, and is demand profiling counted?

  1. Could you evaluate the matched-pool intervention and exposure controller on a VLM from a non-Qwen family to test generalization? (necessary exp. to some extents.)

  2. There's no anonymous code provided (I think I have given enough search), which makes the experiments harder to reproduce. Many paragraphs end with a single word on the last line (many rows actually...), and Table 5 has an awkward side-by-side layout. Could you tidy up these paragraph endings and split Table 5 into two separate tables to improve the presentation?

3 comments

Dear Reviewer VRkz, thank you for your careful reading and for recognizing the importance of separating completion reliability from correctness. We agree that this distinction is essential when interpreting held-out performance. Following up on this issue, we examined completion failures in the added Qwen evaluations and re-evaluated affected queries at a longer 32K response horizon. Each selected query received 16 fresh responses, replacing its entire original group.

Your questions led us to broaden the evidence along each axis you raised: new matched pools, additional training seeds, a non-Qwen target, and a longer evaluation horizon. The results point to a common picture. Initial solvability describes where a query starts. Response demand helps predict its subsequent persistence and generation cost. Our utility experiments then test whether greater persistence translates into greater held-out gains.

The added experiments use GRPO with DAPO-style dynamic sampling, which discards all-correct and all-incorrect rollout groups and refills the training batch. “Standard” denotes this baseline without demand-based sampling correction; EMA retains the same filter-and-refill procedure and adjusts candidate sampling.

1. Generalization across utility pools and targets

We agree that repeating training seeds on one pool does not test sensitivity to pool selection. We therefore constructed two new Qwen pool pairs and added MiMo utility experiments.

New Qwen pools. We added 141 queries per arm for D1 versus D4 and 200 per arm for D1+D2 versus D4. The 141-pair experiment runs for 20 steps; the 200-pair experiment extends to 40 steps and includes a comparison at near-matched generation budgets.

Pairs match the observed initial correct count out of 16 rollouts, category, and answer format. When multiple matches are available, we use minimum-cost matching with fixed-seed random edge costs. Selection uses pre-RL measurements only. Both new pool pairs exclude queries from the original early pools and evaluation panel, checked by query ID and exact image hash.

New poolComparisonLow generation tokensHigh generation tokensLow−High pass@1, pp [95% CI]
141 pairs, D1 vs. D4Low20 / High2045.53M81.49M+0.45 [−0.81, +1.77]
200 pairs, D1+D2 vs. D4Low40 / High40116.40M188.52M+0.44 [−0.97, +1.88]
200 pairs, D1+D2 vs. D4Low30 / High20, near-matched generation budget80.77M79.09M+2.31 [+0.95, +3.72]

The table uses the 16K + 32K evaluation: queries with at least one truncated 16K response received a fresh 32K response group. We added this diagnostic after inspecting the 16K results, replacing selected groups regardless of whether the new answers were better.

For the near-budget comparison, low-demand training continued until it first reached high-demand training’s step-20 generation budget. This selected step 30 by cost rather than performance. Its pass@1 advantage also appears under fixed-16K evaluation: +2.16 pp [+0.78, +3.58].

These comparisons extend the evidence beyond the original 80-query pools. At equal update counts, low-demand training uses substantially fewer generation tokens; at a near-matched generation budget, its additional updates yield a clear pass@1 gain in this run. Each new pool comparison uses one training seed, and the intervals are paired query-bootstrap intervals. The two new pool pairs overlap with each other, so we do not count them as independent pool replications.

MiMo-VL pools. We constructed 346 matched query pairs using MiMo’s pre-RL profiles of eight rollouts per query, with demand bins defined by the fixed Qwen3-VL-4B-Thinking probe. Pairs match the observed correct count, task category, answer format, profiling horizon, and pool origin. The low-demand pool contains 132 D1 and 214 D2 queries; the high-demand pool contains 346 D4 queries. Both training pools are approximately balanced between perception and reasoning; the held-out evaluation panel contains 400 queries with equal representation of the two task families.

We trained MiMo-VL-SFT-7B on each pool using LoRA for 80 steps across two seeds. We compare Low40 with High20 at nearby generation budgets and extend the comparison to Low80 versus High40. Generation costs include rejected candidates and refill; times sum the corresponding training steps. Parentheses show perception results.

ComparisonSeedLow tokensHigh tokensLow / High tokensLow timeHigh timeOverall Low−High pass@1 (Perception)
Low40 / High20112.65M12.47M1.014×4.45 h4.09 h+0.20 pp (+1.38 pp)
Low40 / High20212.84M12.01M1.069×3.42 h2.90 h+0.77 pp (+1.78 pp)
Low80 / High40128.71M25.50M1.126×9.18 h8.13 h+0.64 pp (+0.94 pp)
Low80 / High40228.73M24.81M1.158×7.46 h6.04 h+0.70 pp (+1.44 pp)

At Low40/High20, low-demand training receives twice as many updates with only 1.4% and 6.9% more generation tokens. Positive overall and perception pass@1 differences also appear in both seeds at Low80/High40, where low-demand training uses 12.6% and 15.8% more tokens. These results extend the utility analysis to larger pools and longer training, illustrating how cheaper rollouts can fund useful additional updates.

2. Generalization to MiMo-VL-SFT-7B: persistence and exposure control

We tested both standard filtering and EMA control on MiMo, reusing the Qwen3-VL-4B-Thinking demand profile.

Demand and persistence under standard filtering. For training efficiency, we excluded queries with all eight base-policy profiling responses correct. From the remaining candidates, we constructed a 1,064-query pool with 266 queries per demand bin, balancing bin counts rather than initial solvability. Despite approximately equal candidate shares, the accepted shares become unequal:

MeasureD1D4
Candidate share, steps 1–10025.06%25.02%
Accepted share, steps 1–10020.66%29.66%
Mixed-group rate, steps 1–2062.03%72.49%
Mixed-group rate, steps 81–10042.43%66.76%

D4 receives 1.44 times as much accepted exposure overall and 1.66 times as much over steps 81–100. This extends the observed allocation pattern to another target family.

EMA control. We branched standard and EMA from the same MiMo step-40 checkpoint. Both restored the training state and reset the dataloader, then received 1,920 accepted groups over steps 41–100.

MeasureStandardEMA
Accepted-share drift from the uniform target, total variation0.065630.00625
D4 / D1 accepted exposure1.55×1.03×
Training-generation tokens17.30M16.09M
Logged training-step time4.80 h4.57 h
Step-100 overall pass@1 (Perception)77.70% (78.69%)77.78% (79.31%)

Exposure and cost measurements cover steps 41–100; performance is evaluated at step 100.

EMA reduces exposure drift by 90.5%, using 7.0% fewer generation tokens and 4.9% less logged training-step time. Overall and perception pass@1 differences are +0.08 pp and +0.63 pp, respectively, with query-bootstrap 95% intervals of [−0.59, +0.72] and [−0.34, +1.56].

This single-seed experiment extends allocation control to a non-Qwen target while reducing observed training-generation cost.

3. Robustness: a fresh three-seed controller study

The original standard, oracle, and EMA arms started from the same checkpoint and used the same training settings and accepted-group budget, differing only in sampling. The oracle used retrospective acceptance statistics as a calibration reference.

To address training-run variability, we ran a fresh three-seed Qwen3-Thinking controller study in the new GPU environment—a single RTX PRO 6000 Blackwell GPU, compared with four A100 GPUs for the submission experiments. All three seeds, including seed 1, were newly trained, and we report the step-80 results. For each seed, we trained a standard prefix to step 40, then branched standard and EMA from the same model and optimizer checkpoint. Both branches reset the dataloader and use the same training settings apart from sampling, with the same accepted-group budget: 40 additional updates × 64 groups = 2,560 accepted groups per arm.

SeedExposure-drift reductionEMA / standard generation tokensΔpass@1Δpass@8Δpass@16
162.03%0.998×−0.14 pp+0.44 pp+1.00 pp
255.34%0.970×+0.39 pp+0.15 pp+0.25 pp
359.95%0.901×+1.23 pp+1.64 pp+2.75 pp
Mean——+0.49 pp+0.74 pp+1.33 pp
Training-seed 95% CI, pp——[−1.23, +2.22][−1.23, +2.71][−1.85, +4.52]

Here, Δ denotes EMA minus standard. Drift and generation costs cover steps 41–80; performance is evaluated at step 80 using 16K + 32K evaluation. Queries with at least one parsing failure at 16K received a fresh 32K response group. As in the utility experiments, this diagnostic was added after inspecting the 16K results, and selected groups were replaced regardless of whether performance improved. Under fixed-16K evaluation, the mean differences are −0.46, +0.24, and +0.50 pp for pass@1, @8, and @16, respectively.

Allocation control reproduces across all three seeds, and pass@8/16 favors EMA in every seed under the 16K + 32K evaluation. To address your uncertainty question directly, the table reports 95% t-intervals over the three seed-paired differences, capturing training-run variation rather than only evaluation-query variation. The high-k gains are directionally consistent, while the intervals include zero.

4. Initial solvability and response demand profiling

Initial solvability. Initial rollout accuracy is a practical basis for difficulty-aware data curation in RLVR, and we use it to match the observed starting accuracy of our low- and high-demand pools. We agree that matching the observed correct count out of 16 does not guarantee identical underlying solve probabilities, and we will make this distinction explicit.

Our motivation for introducing response demand is that initial solvability alone leaves an important question open: among queries with similar initial success rates, which will continue to supply mixed groups, and at what generation cost? Our results show that demand provides information about these differences. We therefore see it as a complementary axis for data curation and sampling, alongside initial solvability and online difficulty. We will release the pool-selection procedure and query IDs to make the matching reproducible.

Profiling and cost accounting. Reported training-generation costs include rejected candidates and refill. These token counts measure generated responses; logged training-step times include rollout and optimization.

External demand profiling is separate: the 6,000-query profile, with eight responses per query from the 4B probe, used 177.68M response tokens. We will report this profiling cost separately from training generation.

We reused this profile across Qwen and MiMo experiments, making it a reusable pre-RL measurement. For targets that already produce informative long responses, the target’s own response length may offer a practical alternative: in Qwen3-Thinking, external demand adds only 0.0042 to held-out R² beyond a rich pre-RL profile that includes target response length. This motivates reducing additional profiling when existing target measurements are sufficient.

5. Writing and reproducibility

Thank you for recognizing the paper’s central distinction despite the presentation issues. We will simplify the terminology, connect the experimental descriptions more clearly, fix the paragraph endings, and split Table 5 into two tables. We will also provide an anonymous artifact containing the pool builders and fixed query IDs, training and controller configurations, and evaluation scripts.

Your feedback helped us strengthen both the empirical scope and the reproducibility of the work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

Thank you for the thoughtful reviews and discussion. As the rebuttal period draws to a close, we would like to clarify the central motivation of our paper and summarize the additional evidence.

Compared with SFT, online RL, especially GRPO with dynamic sampling, can spend substantially more tokens and training time obtaining learning signals. Unlike SFT on fixed targets, the policy repeatedly generates rollouts, and correctness-based dynamic sampling discards and replaces groups that provide no relative outcome signal. As RL researchers, we are willing to spend these additional tokens because we hope they enable learning beyond what fixed supervised targets provide. The goal is to turn that additional exploration into real improvements in model performance.

Our paper asks whether queries that keep producing both correct and incorrect answers are worth spending more training budget on. Dynamic sampling repeatedly selects these queries. Our findings show that remaining mixed for longer does not necessarily mean that training on them improves the model more.

We do not claim that low-demand queries are always preferable. Instead, we propose response demand, measured by the thinking length elicited from a fixed probe model, as a complementary axis alongside initial solvability and online success rates. It helps predict which queries will remain mixed and how much generation they will require, providing additional information for planning training.

In response to the reviewers’ questions about compute, generalization, and robustness, we added evidence in three areas:

  1. Utility at comparable generation cost: In a new matched Qwen pool, low-demand training achieves +2.31 pp pass@1 at near-equal generation budgets in a single-seed comparison, with the advantage also present under a fixed 16K-token evaluation.

  2. Transfer beyond Qwen: MiMo-VL-SFT-7B reproduces the demand–persistence pattern and shows positive low-demand training gains at nearby generation budgets, particularly on perception tasks.

  3. Multi-seed robustness and cross-model control: Across three fresh Qwen training seeds, EMA consistently reduces exposure drift by 55–62% at comparable or lower training-generation cost. A separate MiMo experiment extends this result beyond Qwen, reducing drift by 90.5% with 7.0% fewer generated tokens.

The reviewers’ questions helped us test this distinction across broader settings and explain its implications more clearly. We will incorporate the additional results and clarifications into the revised manuscript, and would welcome further discussion of any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.