- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes the general problems locating in the general-purpose RL in LVLM where the rollout groups containing both correct and incorrect answers yield nonzero relative advantages.
It proposes that the hidden curriculum created by the dynamic sampling ( mixed through the discarding and refilling) signals useful learning opportunities. This paper asks the key question: Do queries remain eligible because they still offer useful learning opportunities, or simply because their outcomes remain variable?
This paper aims to answer this question through two steps: 1. eligibility persistence and 2. allocating more updates to these persistent queries produces greater held-out gains.
Critical strengths
-
The propose of the hidden curriculum, based on the dynamic sampling strategy, it proposes a group-relative reward to decide which samples have the gradient eligibility and apply filter-and-refill for optimization.
-
This paper also investigates the initial solvability and eligibility persistence as a initial inverstigation/exploration
-
The core of this paper is exploring the connection of eligibility and utility for a sample through the controlled experiments. This paper proposes thte matched intervention to organize the different training pool with low/high-demand. The experiment result show that the consistent gradient dose not provide a suifficient reason for further training.
-
This paper proposes inverse-acceptance sampling controller for solving this problem.
Critical weaknesses
-
This paper proposes a very interesting and useful direction for RL in LVLM. Thus, a comprehensive experiments should be conducted while this paper only focus on a very limited range of backbones.
-
The effectiveness of the controller is limited becasue of the insuifficient experiments.
Questions for authors
see above
4 comments
2. Stronger evidence for the controller
We addressed the limited controller experiments in two ways: repeating the Qwen comparison across three fresh training seeds and implementing the controller on MiMo.
Three-seed Qwen replication. Within each seed, standard and EMA branch from the same step-40 model and optimizer checkpoint, reset the dataloader, and receive the same 2,560 accepted groups through step 80.
| Seed | Exposure-drift reduction | EMA / standard generation tokens | EMA−standard pass@1 | EMA−standard pass@8 | EMA−standard pass@16 |
|---|---|---|---|---|---|
| 1 | 62.03% | 0.998× | −0.14 pp | +0.44 pp | +1.00 pp |
| 2 | 55.34% | 0.970× | +0.39 pp | +0.15 pp | +0.25 pp |
| 3 | 59.95% | 0.901× | +1.23 pp | +1.64 pp | +2.75 pp |
| Mean | — | — | +0.49 pp | +0.74 pp | +1.33 pp |
Drift and generation costs cover steps 41–80. Performance uses the 16K + 32K evaluation described in our response to Reviewer VRkz.
Exposure control reproduces across all three seeds, and pass@8/16 favors EMA in every seed under this evaluation. The 95% intervals over seed-paired differences are [−1.23, +2.22], [−1.23, +2.71], and [−1.85, +4.52] pp for pass@1, @8, and @16, respectively. These results strengthen the evidence for reliable allocation control, with directionally consistent high-k gains whose average magnitude remains uncertain across training seeds.
Controller transfer to MiMo. We branched standard and EMA from the same MiMo step-40 checkpoint. Both restored the training state, reset the dataloader, and received 1,920 accepted groups over steps 41–100.
| Measure | Standard | EMA |
|---|---|---|
| Accepted-share drift from the uniform target, total variation | 0.06563 | 0.00625 |
| D4 / D1 accepted exposure | 1.55× | 1.03× |
| Training-generation tokens | 17.30M | 16.09M |
| Logged training-step time | 4.80 h | 4.57 h |
| Step-100 overall pass@1 (Perception) | 77.70% (78.69%) | 77.78% (79.31%) |
In this single-seed experiment, EMA reduces exposure drift by 90.5%, using 7.0% fewer generation tokens and 4.9% less logged training-step time. The overall pass@1 difference is +0.08 pp [query-bootstrap 95% CI: −0.59, +0.72].
Together, the Qwen and MiMo experiments demonstrate reproducible exposure control across training seeds and extend it to a non-Qwen model family.
3. Beyond EMA: using demand to plan training
In this work, we showed that response demand helps predict eligibility persistence, that greater persistence does not necessarily yield greater learning utility, and that EMA can correct exposure imbalance across demand bins. There is still room to improve how those updates are allocated. As training progresses on a fixed pool, low-demand queries can stop producing mixed groups quickly. Maintaining their accepted share can then require more generation and refill, while repeatedly directing updates to the few queries that remain mixed. Balancing demand bins therefore does not by itself ensure useful or broad coverage within each bin.
Our next goal is to use demand to decide how sampling and the training pool should change as the model learns. The first experiment we would like to run is replenishing low-demand queries as their mixed signal declines, testing whether fresh queries provide better learning per token than repeatedly sampling the remaining eligible ones. More broadly, we envision a curriculum organized jointly by demand and initial solvability, with different timing, sampling weights, and replenishment strategies for each quadrant. Current success rates and query-level exposure would help determine when to revisit queries and when to move on.
We also want to make this approach practical for data curation. A lightweight demand predictor could reduce external-probe generation on new queries. In VLMs, combining this view with error analysis could also help identify which parts of the model need further training: if persistent errors on short-response perception tasks arise from visual representations, adapting the vision encoder may be more useful than repeatedly sampling the same queries. These are directions we hope to test beyond the current paper. We will clarify this progression in the revision, and we appreciate your encouragement to develop it further.
Thanks to the rebuttal which answers my questions and I will keep the score.
Thank you for your constructive feedback and for taking the time to discuss our work with us. We enjoyed the discussion and are glad the rebuttal addressed your questions!
Dear Reviewer 3Vok, thank you for recognizing both the central question of our paper and its potential for LVLM RL. Your observation that persistent eligibility alone does not justify further training captures the distinction we hoped to make. We also appreciate your encouragement to support this direction with broader experiments.
Following your suggestions, we added a non-Qwen backbone, a fresh three-seed controller study, and utility experiments with larger pools and longer training. These experiments use GRPO with DAPO-style dynamic sampling. EMA retains the same filter-and-refill procedure and adjusts candidate sampling.
1. Broader backbone coverage: MiMo-VL-SFT-7B
We added experiments with MiMo-VL-SFT-7B to test three aspects of the paper: whether demand predicts persistence across model families, whether the resulting exposure imbalance can be controlled, and how demand relates to learning utility.
Demand and persistence transfer beyond Qwen. We reused demand measured by the Qwen3-VL-4B-Thinking probe. For training efficiency, we excluded queries with all eight MiMo base-policy profiling responses correct, then constructed a 1,064-query pool with 266 queries per demand bin. This balances bin counts rather than initial solvability.
Despite approximately equal candidate shares, D4 receives 1.44 times as much accepted exposure overall. Its mixed-group rate also remains higher late in training. Thus, the allocation pattern appears in a non-Qwen target using demand measured by a Qwen probe.
Utility experiments on MiMo. We also constructed 346 matched query pairs: low demand contains 132 D1 and 214 D2 queries, and high demand contains 346 D4 queries. Pairs match MiMo’s observed initial correct count out of eight rollouts, task category, answer format, profiling horizon, and pool origin. Both arms were trained with LoRA for 80 steps across two seeds.
Both training pools are approximately balanced between perception and reasoning. The held-out evaluation panel contains 400 queries with equal representation of the two task families. The following comparison uses low-demand step 40 and high-demand step 20, at nearby generation budgets:
Generation costs include rejected candidates and refill. Low-demand training receives twice as many updates with only 1.4% and 6.9% more generation tokens. Across the two seeds, the mean perception gain is +1.58 pp [query-bootstrap 95% CI: +0.50, +2.77]; the overall gain is +0.48 pp [−0.23, +1.24].
These results provide an example of cheaper rollouts funding useful additional updates on a non-Qwen backbone. They also support treating demand as one axis for task-aware allocation, rather than a universal ranking of training data.
We additionally expanded the Qwen utility experiments beyond the original 80-query pools, using 141 queries per arm for 20 steps and 200 queries per arm for up to 40 steps. In the 200-pair comparison, low-demand training achieves +2.31 pp pass@1 [+0.95, +3.72] at near-matched generation budgets, with one training seed.