- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Data selection can make reinforcement learning with verifiable rewards (RLVR) more effective, but different selection methods have very different computational costs and may not remain equally useful throughout training. In particular, on-policy gradient-alignment methods can identify valuable training examples using the current model, but they require expensive fresh rollouts and gradient computations. Cheaper alternatives such as cached success-rate selection avoid this repeated scoring cost, but may be less effective at some stages of training. The paper is motivated by the observation that the best selection strategy may therefore depend on the model’s current training state, rather than there being a single selector that should be used for the entire run.
The paper’s main idea is to move from static data selection to dynamic strategy switching. It compares on-policy gradient alignment with a cached success-rate selector and finds that on-policy alignment tends to perform better early in GRPO training on MATH, while cached success-rate selection becomes stronger at later checkpoints. Based on this changing ordering, the authors propose starting with on-policy selection and then switching to cached success-rate selection during training. They introduce an auxiliary gradient-based signal, SR-GC, together with temporal confirmation to decide when to switch. In the two evaluated online-switching trajectories, this strategy achieves higher final reward than either continuing on-policy selection or switching to cached SR immediately after the shared prefix, while costing less than continued on-policy selection.
Critical strengths
I had a hard time understanding the paper, so I do not have any specific strengths in mind to highlight.
Critical weaknesses
(Legibility issues)
The paper is very hard to read, and feels strongly LLM written. I had a very hard time understanding what it was trying to say.
Very poorly written introduction. It feels like a weird blend of intro and related work, and I did not understand the work well due to poor writing. Badly made plots. For example, Figure 2 occupies way more space than it should. More seriously, Figure 4 has a y-axis label and y-axis title overlapping, to the point the “-0.4” looks like 0.4, and I was very confused about the plots.
Overall, the presentation should be significantly improved before the paper should be accepted into a venue of ICLR’s caliber.
(Problems with methodology and experiments)
[1] The result that Switch beats both alternatives, is demonstrated on one LLM backbone and one dataset/objective setting. It needs to be more robust, experiments on different models and datasets will make the paper stronger.
[2] Does SR-GC choose a better switch time than a simple fixed schedule? The baselines here are (1) always on-policy, (2) always SR, (3) random. This are not strong enough baselines. Possible alternatives:
-
fixed switch at several plausible times,
-
a tuned fixed switch chosen on development seeds,
-
switch based on cheaper observable statistics,
-
possibly periodic re-evaluation or reverse switching.
Without those, it is difficult to know whether the value comes from adaptive switching or simply from “use expensive selection early, cheap selection later.”
Questions for authors
None, please address the weaknesses mentioned above.
3 comments
I thank the authors for their substantial effort in the rebuttal period.
However, the paper has changed significantly from its early version. While I appreciate the authors' effort, I believe the new paper would benefit from a full review cycle in a future conference, since it would require substantial effort to re-read the new manuscript and judge its merits.
I will keep my current score.
Thank you for your time and for considering our response. We appreciate your feedback, which has helped us improve the manuscript.
Response to Reviewer zUxs
Thank you for the direct comments. Locations refer to the revised PDF.
Legibility and writing.
The manuscript was rewritten for clarity, with a new introduction, a new overview figure (Figure 1, p. 2), a separate related-work section and revised figures.
[1] One backbone and one dataset.
The revision trains four backbones on two datasets: OLMo-3 7B, Qwen3.5-9B, Gemma 4 12B IT and Llama-3.1-8B-Instruct, each on MATH and on MBPP, with five seeds (5–9) per cohort (§5.2, p. 6 l. 284–289; Tables 3–4, p. 6; per-seed results in Tables 24–35, pp. 30–36). All cohorts use the same four methods, refresh interval, SR-GC rule, 275-update horizon and evaluation on 300 held-out questions with eight responses each (§4, p. 5 l. 234–238; App. C, p. 20 l. 1071–1075). The better single selector differs between cohorts: SR minus On-policy is +1.24 [+0.23, +2.25] on Qwen3.5 MATH and −2.20 [−3.35, −1.05] on Qwen3.5 MBPP. The mean reward of Switch stays within 0.88 points of the best method in every cohort and is the highest in five (§5.3, p. 7 l. 362–365). With five seeds, its paired intervals against SR and On-policy include zero. Gemma solves 83.7–84.6% (MATH) and 90.6–91.2% (MBPP) of the training candidates before training, and all of its methods end within 1.00 point of Random (p. 6 l. 322–323).
[2] Does SR-GC choose a better switch time than a fixed schedule?
On OLMo-3 MATH, the gain of Switch comes from moving from gradient alignment to cached SR, and SR-GC supplies the switch time online; the best fixed time is known only in hindsight (p. 8 l. 402–404). The branched OLMo-3 MATH runs compare SR-GC with every fixed switch time on one trajectory per seed (§5.4, p. 8 l. 386–404; App. E, Tables 40–43, pp. 41–42). Against your suggested baselines:
Across all cohorts, SR-GC switches in 39 of the 40 Switch runs on seeds 5–9, at times that vary across cohorts, and with the switch time chosen online by SR-GC, the mean reward of Switch stays within 0.88 points of the best method in all eight cohorts (§5.4, p. 8 l. 383–384; Table 25, p. 31; §8, p. 10 l. 530–533).