- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper examines adaptive prompt sampling in RL, where training prompts are selected based on the pass rate from previous steps. Its central finding is that the gains from these methods disappear when an LLM judge provides the verification signal. The authors argue that judge errors are biased toward particular prompts, and prompt selection based on past judge predictions repeatedly exposes the learning policy to these biases. The empirical results confirm this and the ablations the authors conduct support this explanation, suggesting that the problem lies in repeatedly selecting prompts susceptible to judge bias rather than other factors.
Despite the paper's main contribution being an empirical finding, the range of judge model families and sizes was relatively narrow, which limits the generality of the findings.
Beyond the limited scope of the evaluation, the paper is also poorly presented. For instance, GRESO and MoPPS appear repeatedly before being introduced ( only in the appendix ). The contribution of the theoretical analysis is also unclear: the conclusions are trivial and the authors do not explain what additional insight they provide.
Critical strengths
- The paper studies a relevant problem: Should we be using adaptive sampling methods when we rely on an a llm as a judge?
- The empirical results support the authors' hypothesis.
- The ablations are well targeted. Replaying selection schedules and replacing prompts with others of similar estimated difficulty rollout controls for two potential confounders that could explain the effect observed in the empirical results.
Critical weaknesses
- The paper is difficult to read despite my best efforts. No explanation or even citation of the main baselines.
- Figure legends are not self-contained and provide insufficient context to understand the results. What is Replay and matched replay? These things should be clear from the legend.
- The theoretical analysis formalizes seemingly evident mechanisms without clearly explaining what additional insight it provides beyond the experiments.
- The practical utility of the authors' proposed interventions is unclear; why not just use uniform sampling?
Questions for authors
- The practical utility of the authors' proposed interventions is unclear; why not just use uniform sampling?
- What insights do you have from the theoretical results?
2 comments
Weakness 4 and Question 1 (why not uniform sampling). Uniform sampling is the safe default, and the paper now says when it suffices; the case for selection is cost. Under uniform sampling, many groups in a step carry no learning signal (all responses pass or all fail), and practitioners already run selectors such as DAPO's dynamic sampling, GRESO and MoPPS to avoid paying for them. Our results show that under an LLM judge these selectors buy that saving with exposure to judge errors, and the remedies keep part of the saving without it. Theorem 1 gives the test in advance: where its bound is small, which happens when prompts are rarely revisited, use uniform sampling, since a larger saving needs bias or information from outside the judge. Section 1 now ends: "We recommend the first remedy for an existing selector in Math, the second with the calibrated rule when prompts are revisited and rollouts bind, and otherwise uniform sampling." Section 6 adds: "With rare revisits, harm and savings fade and uniform sampling suffices; with frequent revisits, our remedies keep part of the saving."
For a practitioner already running a selector, the visit bound cuts GRESO's Math events beyond chance from 17.9 ± 2.1 to 5.3 ± 1.3, and triggered verification, at 4.3% more verifier calls, keeps them there (5.8 ± 1.6) and lowers the judged-minus-true gap below uniform sampling's (Table 2b, Section 6).
Where rollouts bind, two arms new in the revision test where Theorem 1 allows a saving: the calibrated keep rule, trained as an arm (the submitted version evaluated it only as a counterfactual), and audit-informed selection. In Science the calibrated rule ends 0.2 ± 0.4 points below uniform sampling with 0.95 of its rollouts (Table 2a). Appendix D.14 adds a comparison with DPPO in separate runs at DPPO's rollout budget per step, without SE and without a uniform arm at that budget; we claim no saving from it, since Theorem 1's cap of about a twentieth holds at uniform sampling's per-step second moment. With an audit by a stronger verifier on unfiltered pools, audit-informed selection ends 0.9 points above uniform sampling with 0.55 of its rollouts in Science and 0.4 above, within one SE, with 0.79 in Math (Table 43). We concede that in Math the calibrated rule needs 0.81 of uniform sampling's rollouts, more than DPPO's 0.60, and that the effect, and hence the case for any remedy, needs a fallible judge and revisits (Section 6, Appendix G).
Range of judges and policies. The training judges come from three model families, and the policies span three sizes of one family, so that the comparison isolates size.
| Judge (Table 2c, Appendices C and F.13) | Family | Size | Role | Agreement with the panel |
|---|---|---|---|---|
| Gemma-3-12B-it | Gemma | 12B | Math training judge | – |
| Gemma-4-12B-it | Gemma | 12B | Science training judge | 0.764 ± 0.011 |
| Majority of Gemma-4-12B-it, Gemma-3-12B-it and Qwen3-32B (new) | Gemma, Qwen | 12B–32B | Science, Table 2c | 0.82 ± 0.01 |
| Gemma-4-26B-A4B-it | Gemma | 25.2B, 3.8B active | Science, Table 2c | 0.941 ± 0.006 |
| DeepSeek-V4-Flash | DeepSeek | – (served through an API) | Science, reference answer in view, Table 2c | 0.964 ± 0.005 |
The policies are Qwen3-1.7B-Base, Qwen3-8B-Base and, new in the revision, Qwen3-14B-Base (Appendix F.14). The Science loss shrinks with policy size but appears in every seed at 8B and 14B:
| Science, MMLU-Pro STEM (Tables 2a, 40, 45) | 1.7B | 8B | 14B |
|---|---|---|---|
| Uniform sampling (%) | 48.8 ± 0.3 | 68.6 ± 0.5 | 73.4 ± 0.7 |
| GRESO − uniform (points) | −8.2 ± 4.0 | −6.9 ± 0.6 | −6.5 ± 0.6 |
| MoPPS − uniform | −10.9 ± 0.9 | −8.2 ± 0.6 | −7.7 ± 0.6 |
| IW-GRESO − uniform | −1.0 ± 0.8 | −0.6 ± 0.7 | −0.2 ± 0.7 |
| DPPO − uniform | −1.8 ± 0.8 | −1.4 ± 0.7 | −0.4 ± 0.7 |
In Math, every size counts events on the same 359 hard prompts. The larger policies produce fewer events under every method, but GRESO's events beyond chance remain several times uniform sampling's, and both remedies stay within one event of uniform sampling at 8B and 14B:
| Math, events beyond chance on the hard set (Tables 2b, 40, 46) | 1.7B | 8B | 14B |
|---|---|---|---|
| Uniform sampling | 1.8 ± 1.2 | 1.2 ± 0.4 | 0.5 ± 0.3 |
| GRESO | 17.9 ± 2.1 | 12.8 ± 1.4 | 8.6 ± 0.9 |
| IW-GRESO | 2.0 ± 0.4 | 1.7 ± 0.3 | 0.6 ± 0.6 |
| GRESO + bound + verification | 5.8 ± 1.6 | 1.7 ± 0.2 | 0.8 ± 0.4 |
Held-out MATH accuracy ends within one point of uniform sampling for every arm at 8B and 14B (Table 46).
These changes are in Sections 1 to 6 and Appendices B, F.14, L.3 and L.7.
Thank you for taking the time and consideration to evaluate our paper and provide comprehensive feedback. As noted above, we sincerely apologize for responding late due to issues with obtaining compute. We hope you are able to take our rebuttals into consideration during your final evaluations.
TL;DR.
Weakness 1 (readability and baselines). We agree. Fitting two domains, a theory section and two remedies into nine pages explains the density but does not excuse it. We made three changes.
Weakness 2 (legends). Main-text captions now name each line, marker and row. In Table 1 and Figure 2b, "replay" and "matched replay" are now the "replayed schedule" and the "matched schedule", and both captions define them. The replayed schedule trains a fresh policy on the prompt sequence of a GRESO run, so its rewards cannot choose its prompts. The matched schedule permutes those prompts among prompts of equal base-policy pass count, keeping visit counts and difficulty but changing which prompts are repeated. The Table 2 caption defines DS, Calibrated, Δ and the asterisk and states its bolding rule, and the Figure 2b caption says that the uniform line is drawn over all prompts and is not a same-prompt comparison.
Weakness 3 and Question 2 (theory). We agree that the mechanisms look evident in hindsight. Appendix L.7 (new; Table 63), to which Section 4 now points, lists what each result supplies beyond the experiments and a check that could have failed.
The theory adds no new mathematics, and we cite the results it restates (Appendix D). Nor does it explain the Science loss, which appears under live selection but not under fixed schedules with the same exposure (Appendix L.3, L.7).