acceptodds
Under review as a conference paper at ICLR 2027

Efficient RL Selects the Judge's Mistakes: Prompt Selection Under LLM-Judged Rewards

Abstract

Group-based reinforcement learning with verifiable rewards increasingly relies on large language models as judges and data selection methods which aim to spend rollouts on mixed prompts (prompts where the group contains at least one accepted and one rejected rollout). We identify a fundamental incompatibility between these two practices: current selection methods do not distinguish between prompts the policy is genuinely learning from and prompts the judge grades incorrectly. We discover that the filtered set of prompts which the policy has not learned tends to concentrate on those which are plagued by the judge’s mistakes, leading to repeated optimization toward answers the judge wrongly accepts or rejects. We show this across selection methods, model and group sizes, learning rates, and the math and science domains. With an exact verifier, these methods deliver their benefits; however, when a judge is used as the verifier, most of their advantage vanishes. In fact, up to a quarter of rollout groups train the policy toward the wrong behavior (versus 5% under uniform sampling), and the number of prompts the policy answers wrongly but the judge accepts increases two to four fold. As these prompts are revisited, each repeated rollout contributes less learning and more exploitation. In mathematics, held-out accuracy shows no consistent change since learned errors are largely problem specific; on the other hand, the policy uses more than double the compute it needs with an exact verifier to achieve the same mastery. On rubric-based science, judge errors appear more stylistic and exploitation transfers to out-of-distribution evaluation, costing 10 to 11 points on MMLU-Pro STEM that are not recovered within training. This loss, too, is selection amplifying the judge’s (stylistic) errors, through the composition of the selected groups. We propose two simple changes to mitigate this issue: bound how much of the rollout budget any prompt may absorb, and use independent verification only on prompts that exceed this limit. Together, they use about 4% additional verifier calls and remove most of the excess exploitation. We also propose importance-weighted selection, which draws prompts in proportion to how likely the selector is to keep them and reweights each by the inverse of that probability, so the expected update equals uniform sampling’s; applied to GRESO, it recovers 9 of 10 lost MMLU-Pro STEM points. Our findings call for reevaluating claims of efficiency for RL data selection under verifiers used in practice and for designing prompt selection and verification in tandem.

Primary area:
reinforcement learning
Keywords:
RLVR, LLM-as-a-judge, prompt selection, dynamic sampling, reward hacking, verifier error, GRPO

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4WTTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines adaptive prompt sampling in RL, where training prompts are selected based on the pass rate from previous steps. Its central finding is that the gains from these methods disappear when an LLM judge provides the verification signal. The authors argue that judge errors are biased toward particular prompts, and prompt selection based on past judge predictions repeatedly exposes the learning policy to these biases. The empirical results confirm this and the ablations the authors conduct support this explanation, suggesting that the problem lies in repeatedly selecting prompts susceptible to judge bias rather than other factors.

Despite the paper's main contribution being an empirical finding, the range of judge model families and sizes was relatively narrow, which limits the generality of the findings.

Beyond the limited scope of the evaluation, the paper is also poorly presented. For instance, GRESO and MoPPS appear repeatedly before being introduced ( only in the appendix ). The contribution of the theoretical analysis is also unclear: the conclusions are trivial and the authors do not explain what additional insight they provide.

Critical strengths
  • The paper studies a relevant problem: Should we be using adaptive sampling methods when we rely on an a llm as a judge?
  • The empirical results support the authors' hypothesis.
  • The ablations are well targeted. Replaying selection schedules and replacing prompts with others of similar estimated difficulty rollout controls for two potential confounders that could explain the effect observed in the empirical results.
Critical weaknesses
  • The paper is difficult to read despite my best efforts. No explanation or even citation of the main baselines.
  • Figure legends are not self-contained and provide insufficient context to understand the results. What is Replay and matched replay? These things should be clear from the legend.
  • The theoretical analysis formalizes seemingly evident mechanisms without clearly explaining what additional insight it provides beyond the experiments.
  • The practical utility of the authors' proposed interventions is unclear; why not just use uniform sampling?
Questions for authors
  • The practical utility of the authors' proposed interventions is unclear; why not just use uniform sampling?
  • What insights do you have from the theoretical results?
2 comments

Thank you for taking the time and consideration to evaluate our paper and provide comprehensive feedback. As noted above, we sincerely apologize for responding late due to issues with obtaining compute. We hope you are able to take our rebuttals into consideration during your final evaluations.

TL;DR.

  • The main text is rewritten for readability: three named measurements, a glossary, baselines explained and cited in Section 2, and self-contained captions.
  • Appendix L.7 lists what the theory adds beyond the experiments, and Section 4 points to it.
  • Why not just uniform sampling: uniform sampling spends rollouts on prompts that carry no learning signal, and selectors (DAPO's dynamic sampling, GRESO, MoPPS) are already used to avoid that. Our remedies keep part of the saving without the selectors' exposure to judge errors; on unfiltered Science, audit-informed selection ends 0.9 points above uniform sampling with 0.55 of its rollouts. Where prompts are rarely revisited, Theorem 1 shows the saving is small and uniform sampling is the right choice.
  • Training judges come from three model families, and policies span 1.7B, 8B and 14B.

Weakness 1 (readability and baselines). We agree. Fitting two domains, a theory section and two remedies into nine pages explains the density but does not excuse it. We made three changes.

  1. Baselines. A new paragraph of Section 2, "Prompt selection methods", explains and cites each method. GRESO skips a prompt before rolling it out, with a probability that grows with its streak of zero-variance groups. MoPPS keeps a Beta posterior over each prompt's pass rate, draws from it, and keeps the prompts whose draws lie closest to 0.5. DPPO prunes prompts whose mean absolute advantage was low at their previous visit and rescales the rest by the inverse of their keep probability, as InfoBatch does in supervised training. Dynamic sampling, introduced with DAPO, discards zero-variance groups after generating them.
  2. Terms. The main text names three measurements (the judged-minus-true gap, events beyond chance and visit amplification), defined in Section 2 before use, and says every other quantity in plain words; a glossary opens Appendix B.
  3. Structure. Sections 3 and 4 open with their conclusion, and Section 2 states the setup before any result.

Weakness 2 (legends). Main-text captions now name each line, marker and row. In Table 1 and Figure 2b, "replay" and "matched replay" are now the "replayed schedule" and the "matched schedule", and both captions define them. The replayed schedule trains a fresh policy on the prompt sequence of a GRESO run, so its rewards cannot choose its prompts. The matched schedule permutes those prompts among prompts of equal base-policy pass count, keeping visit counts and difficulty but changing which prompts are repeated. The Table 2 caption defines DS, Calibrated, Δ and the asterisk and states its bolding rule, and the Figure 2b caption says that the uniform line is drawn over all prompts and is not a same-prompt comparison.

Weakness 3 and Question 2 (theory). We agree that the mechanisms look evident in hindsight. Appendix L.7 (new; Table 63), to which Section 4 now points, lists what each result supplies beyond the experiments and a check that could have failed.

  1. A null for reading counts. Corollary 1 gives the reward-hacking events that a judge with response-independent errors produces by chance. With it, uniform sampling's 11.3 events are 1.8 ± 1.2 beyond chance, and the selectors' excess stays between 13 and 22 events under four alternative nulls.
  2. The judge hides the savings. Proposition 1 gives 0.147 for the probability that a never-solved prompt shows zero variance under our Science judge, against 0.161 ± 0.021 measured, so the judge hides about 84% of these prompts from any selector that reads its rewards.
  3. When uniform sampling is right, before any run. Theorem 1 bounds what an unbiased design can save at uniform sampling's per-step second moment: with the judge's streak alone, about a fifth in Math, a twentieth in Science and 2.4% on an unfiltered pool.

The theory adds no new mathematics, and we cite the results it restates (Appendix D). Nor does it explain the Science loss, which appears under live selection but not under fixed schedules with the same exposure (Appendix L.3, L.7).

Weakness 4 and Question 1 (why not uniform sampling). Uniform sampling is the safe default, and the paper now says when it suffices; the case for selection is cost. Under uniform sampling, many groups in a step carry no learning signal (all responses pass or all fail), and practitioners already run selectors such as DAPO's dynamic sampling, GRESO and MoPPS to avoid paying for them. Our results show that under an LLM judge these selectors buy that saving with exposure to judge errors, and the remedies keep part of the saving without it. Theorem 1 gives the test in advance: where its bound is small, which happens when prompts are rarely revisited, use uniform sampling, since a larger saving needs bias or information from outside the judge. Section 1 now ends: "We recommend the first remedy for an existing selector in Math, the second with the calibrated rule when prompts are revisited and rollouts bind, and otherwise uniform sampling." Section 6 adds: "With rare revisits, harm and savings fade and uniform sampling suffices; with frequent revisits, our remedies keep part of the saving."

For a practitioner already running a selector, the visit bound cuts GRESO's Math events beyond chance from 17.9 ± 2.1 to 5.3 ± 1.3, and triggered verification, at 4.3% more verifier calls, keeps them there (5.8 ± 1.6) and lowers the judged-minus-true gap below uniform sampling's (Table 2b, Section 6).

Where rollouts bind, two arms new in the revision test where Theorem 1 allows a saving: the calibrated keep rule, trained as an arm (the submitted version evaluated it only as a counterfactual), and audit-informed selection. In Science the calibrated rule ends 0.2 ± 0.4 points below uniform sampling with 0.95 of its rollouts (Table 2a). Appendix D.14 adds a comparison with DPPO in separate runs at DPPO's rollout budget per step, without SE and without a uniform arm at that budget; we claim no saving from it, since Theorem 1's cap of about a twentieth holds at uniform sampling's per-step second moment. With an audit by a stronger verifier on unfiltered pools, audit-informed selection ends 0.9 points above uniform sampling with 0.55 of its rollouts in Science and 0.4 above, within one SE, with 0.79 in Math (Table 43). We concede that in Math the calibrated rule needs 0.81 of uniform sampling's rollouts, more than DPPO's 0.60, and that the effect, and hence the case for any remedy, needs a fallible judge and revisits (Section 6, Appendix G).

Range of judges and policies. The training judges come from three model families, and the policies span three sizes of one family, so that the comparison isolates size.

Judge (Table 2c, Appendices C and F.13)FamilySizeRoleAgreement with the panel
Gemma-3-12B-itGemma12BMath training judge–
Gemma-4-12B-itGemma12BScience training judge0.764 ± 0.011
Majority of Gemma-4-12B-it, Gemma-3-12B-it and Qwen3-32B (new)Gemma, Qwen12B–32BScience, Table 2c0.82 ± 0.01
Gemma-4-26B-A4B-itGemma25.2B, 3.8B activeScience, Table 2c0.941 ± 0.006
DeepSeek-V4-FlashDeepSeek– (served through an API)Science, reference answer in view, Table 2c0.964 ± 0.005

The policies are Qwen3-1.7B-Base, Qwen3-8B-Base and, new in the revision, Qwen3-14B-Base (Appendix F.14). The Science loss shrinks with policy size but appears in every seed at 8B and 14B:

Science, MMLU-Pro STEM (Tables 2a, 40, 45)1.7B8B14B
Uniform sampling (%)48.8 ± 0.368.6 ± 0.573.4 ± 0.7
GRESO − uniform (points)−8.2 ± 4.0−6.9 ± 0.6−6.5 ± 0.6
MoPPS − uniform−10.9 ± 0.9−8.2 ± 0.6−7.7 ± 0.6
IW-GRESO − uniform−1.0 ± 0.8−0.6 ± 0.7−0.2 ± 0.7
DPPO − uniform−1.8 ± 0.8−1.4 ± 0.7−0.4 ± 0.7

In Math, every size counts events on the same 359 hard prompts. The larger policies produce fewer events under every method, but GRESO's events beyond chance remain several times uniform sampling's, and both remedies stay within one event of uniform sampling at 8B and 14B:

Math, events beyond chance on the hard set (Tables 2b, 40, 46)1.7B8B14B
Uniform sampling1.8 ± 1.21.2 ± 0.40.5 ± 0.3
GRESO17.9 ± 2.112.8 ± 1.48.6 ± 0.9
IW-GRESO2.0 ± 0.41.7 ± 0.30.6 ± 0.6
GRESO + bound + verification5.8 ± 1.61.7 ± 0.20.8 ± 0.4

Held-out MATH accuracy ends within one point of uniform sampling for every arm at 8B and 14B (Table 46).

These changes are in Sections 1 to 6 and Appendices B, F.14, L.3 and L.7.

Reviewer_fDvhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors show that history-based prompt selectors (GRESO, MoPPS), validated with exact checkers, misbehave under LLM-judged rewards. Indeed, judge errors are response-dependent and persistent, so prompts the policy never solves keep looking "mixed" and are revisited. In Math, the selectors lose most of their mastery lead, need 2.2–3.1 times the steps to reach the same mastery, and show far more reward-hacking events than chance. In Science, they lose 10–11 points on MMLU-Pro STEM. The authors propose two remedies: a visit bound with triggered verification (about 4% extra verifier calls), and an importance-weighted wrapper, which recovers 9 of GRESO's 10 lost points but keeps no rollout savings.

Critical strengths
  • A timely and important finding. The interaction between data selection and imperfect judges is new and practically relevant, and it holds across selectors, domains, scales and hyperparameters.
  • Convincing causal design. The replay control cleanly isolates prompt identity from policy feedback, and the chance-adjusted event counts are well conceived.
  • Rigour. Analysis plans and amendments were written before the runs, and the negative results are reported in detail.
Critical weaknesses
  • The Science mechanism does not match the title. Appendix G shows that the Science selectors do not concentrate on error-prone prompts, so the style-split account rests on correlational evidence, without a replay control.
  • Weak judges. The Math judge has no reference answer, and the Science judge was chosen to be imperfect by design. With stronger or reference-based judges, the Science loss falls to 3 points and then to under 1 point.
  • Small pools. With 406–800 prompts visited 12–16 times each, the regime is far from practice, and the effect already shrinks with a 3,200-prompt pool.
  • Limited remedies. Triggered verification needs the very verifier these domains lack, and the importance-weighted wrapper essentially recovers uniform sampling, close to DPPO and InfoBatch. Besides, the main text is very dense.
Questions for authors
  1. Do the excess events persist in Math with a reference-based judge?
  2. How large is the effect with a realistic pool size (e.g., ~30k prompts)?
  3. Could you run the replay control in Science?
  4. Could strict mastery be evaluated offline at fixed checkpoints, independently of the visit schedule?
2 comments

We would like to thank the reviewer for their time and consideration when reviewing our paper. As noted in the review above, we sincerely apologize for responding so close to the deadline as completing experiments became difficult due to a lack of compute capacity.

TL;DR.

  • The title describes the Math route. In Science the loss follows a feedback route: fixed schedules are unharmed in every seed, while live GRESO is harmed in two of three (Appendix L.3).
  • Stronger judges shrink the effect (Table 2c); with the reference answer shown to the Math judge, about two fifths of the excess remains (Appendix L.8).
  • The effect needs revisits, and larger pools are untested (Table 42). Verification is Math-only; the calibrated and audit-informed arms are new tests of Theorem 1.

Weakness 1 and Question 3 (Science mechanism and title). We agree that the title describes the Math route. We keep it because that mechanism is the paper's central claim, and the abstract and Section 3 now state the Science result as a different route, the feedback between selector and policy (Appendix L.3, new). We ran the replay control in Science: from each seed's GRESO run we trained three fixed schedules, which cannot react to the rewards.

Science, 1.7B, 12B rubric judge (Appendix L.3), MMLU-Pro STEM − same-seed uniform (points)Seed 0Seeds 1–2Seeds harmed (≥ 5 below)
Replayed schedule (GRESO's prompt sequence)+0.3—0 of 3
Difficulty-matched schedule−0.5—0 of 3
Flattened schedule (even visits)+0.5—0 of 3
All three fixed schedules, seeds 1–2 (six runs, mean ± SD)0.0 ± 0.9
Live GRESO (three-seed mean −8.2 ± 4.0, Table 2a)−11.2−0.4 / −13.12 of 3

Under the 12B rubric judge 5 of 6 live selector runs are harmed. In Math, by contrast, the replayed schedule reproduces the harm without feedback (15.3 ± 1.4 events beyond chance against 2.7 ± 1.0 for the matched schedule, Table 1). Forcing a derivation before the answer moves GRESO and MoPPS at seed 0 from 11.2 and 11.4 points below uniform sampling to 0.0 and +0.1: the harmed policies keep their knowledge and change how they answer. We agree that the Science selectors do not concentrate on the judge's most error-prone prompts (Appendix G). Why the judge sustains the terse answer mode under live selection, but not under a fixed schedule, remains open (Section 3).

Weakness 2 and Question 1 (judges). We agree that the effect needs a fallible judge; Section 6 now states this as the paper's scope, and Section 2 states, with citations, that our judges err at rates typical of LLM judges on hard tasks; on JudgeBench (Tan et al., ICLR 2025), strong judges such as GPT-4o do only slightly better than random guessing. Table 2c gives Science under three stronger judges, the majority judge being new.

Training judge, Science 1.7B (Table 2c)Agreement with the panelGRESO − uniform (points)MoPPS − uniform (points)
Gemma-4-12B-it0.764 ± 0.011−8.2 ± 4.0−10.9 ± 0.9
Majority of three 12B–32B judges0.82 ± 0.01−5.2 ± 1.7−6.6 ± 0.4
Gemma-4-26B-A4B-it0.941 ± 0.006−1.6 ± 0.7−0.4 ± 1.0
DeepSeek-V4-Flash, the reference judge (Appendix C)0.964 ± 0.005−0.9 ± 1.0−0.5 ± 0.6

The Gemma-4-26B-A4B-it row has changed since the submitted version, which reported −3.4 ± 0.9 and −2.7 ± 0.8, the 3-point loss quoted in the review; it now uses the measured seed 0 (note W5).

On Question 1, Appendix L.8 (new) trains uniform sampling and GRESO in Math for 300 steps in three seeds with the same weak Math judge, Gemma-3-12B-it, with the reference answer added. Showing the reference does not make it a stronger judge: its base false-positive rate stays at 0.213, unchanged. The strong reference-based case is Science, the DeepSeek-V4-Flash row above (−0.9 ± 1.0 and −0.5 ± 0.6).

Math 1.7B, judge shown the reference (Table 64, Appendix L.8; no-reference rows: Tables 2b, 61)UniformGRESOGRESO − uniform
Events beyond chance, reference shown0.4 ± 0.97.2 ± 1.8+6.8 ± 2.0
Events beyond chance, no reference1.8 ± 1.217.9 ± 2.1+16.1
Held-out false-positive rate (%), reference shown23.8 ± 1.525.2 ± 1.6+1.4 ± 2.2
Held-out false-positive rate (%), no reference34.8 ± 0.255.7 ± 7.1+20.9 ± 7.0

The excess persists at about two fifths of its size without the reference, while held-out exploitation nearly vanishes. Only this judge was shown the reference in Math, and only uniform sampling and GRESO were trained with it.

Weakness 3 and Question 2 (pool size). We have not trained on a 30k pool; Section 8 and Appendix G state larger pools as untested. Table 42 and Appendix F.11 vary the pool with the number of steps fixed.

Pool, Math 1.7B (visits per prompt under uniform sampling) (Table 42; 3,200: Appendix F.11)GRESO, events beyond chance per 359 hard promptsMoPPS, sameGRESO lead, exact verifier (points)
400 (24)23.4 ± 3.527.3 ± 3.6+4.5 ± 1.0
1,600 (6)14.5 ± 1.616.3 ± 1.8+19.5 ± 0.8
3,200 (3)2.5 ± 0.2 (uniform 0.3 ± 0.1); 10.1 ± 0.7 on the pool's 1,422 hard promptsnot run+8.9 ± 0.4
6,400 (1.5)0.5 ± 0.12.5 ± 0.30.0 ± 0.5

At our 300 steps a 30k pool would be revisited even less. Appendix G reads the sweep as revisits, rather than pool size as such, setting the effect, and Section 6 states that "With rare revisits, harm and savings fade and uniform sampling suffices."

Weakness 4 (remedies and density).

  1. Verification. We agree that it needs an independent verifier; Section 5.1 names it (the exact verifier in Math, elsewhere a stronger or reference-based judge). The bound with verification was trained only in Math. There the bound takes GRESO's events beyond chance from 17.9 ± 2.1 to 5.3 ± 1.3, and verification, at 4.3% more verifier calls, keeps them there (5.8 ± 1.6) and lowers the judged-minus-true gap below uniform sampling's (Table 2b, Section 6). In Science we trained only a visit-triggered audit by a stronger judge (unfiltered pool: −4.2 ± 1.6 points against GRESO's −8.5 ± 1.9, Table 55).
  2. Weighting. We agree that with the selector's own propensity it recovers uniform sampling and matches DPPO's and InfoBatch's rescaling up to a constant; IW-GRESO and IW-MoPPS are now presented as that special case (Table 40). The calibrated keep rule, now trained as an arm (the submitted version evaluated it only as a counterfactual on the uniform runs' records), and audit-informed selection, new in the revision (Appendices D.14, D.15 and F.12), test where Theorem 1 allows a saving. In Science the calibrated rule ends 0.2 ± 0.4 points below uniform sampling with 0.95 of the rollouts (Table 2a). On unfiltered pools, with an audit by a stronger verifier, which DPPO does not get, audit-informed selection ends 0.9 points above uniform sampling with 0.55 of its rollouts in Science and 0.4 above, within one SE, with 0.79 in Math; DPPO ends below uniform sampling in Science and within noise in Math (Section 6, Table 43). In Math the calibrated rule needs 0.81 of the rollouts against DPPO's 0.60, so we claim no Math saving over DPPO.
  3. Density. We agree; the page limit explains it but does not excuse it. The main text now names three measurements, defined in Section 2 before use, with a glossary opening Appendix B, and a new paragraph of Section 2 explains and cites each baseline.

Question 4 (mastery at fixed checkpoints). Yes, at saved checkpoints in three seeds: every arm answers each hard prompt eight times afresh. Appendix L.1 (new) adds times to a common target, with two of three seeds estimated from training logs (Table 58).

Math 1.7B, fresh samples (Table 33, Appendix F.5; Table 58, Appendix L.1)Exact verifier as the rewardJudge as the reward
GRESO lead over uniform in strict mastery at step 200 (points)+8.9 ± 0.7+1.0 ± 1.0
GRESO / uniform steps to 50% of the hard set0.95 ± 0.021.23 ± 0.23
Reviewer_PDyuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how history-based prompt selection interacts with fallible LLM rewards. In Math, GRESO and MoPPS concentrate training on judge-error-prone prompts, increasing reward-hacking events and reducing their advantage in a cumulative mastery metric. In rubric-based Science, the selectors lose approximately 10–11 MMLU-Pro STEM points under the weaker judge. The paper analyzes response-dependent errors and sampling bias, then evaluates visit limits, triggered verification, and importance-weighted selection. Its main contribution is a diagnosis of a consequential interaction between selection and reward errors, supported by mitigation experiments.

Critical strengths
  1. A substantial failure mode supported by multiple controls. The Science degradation is large and is examined across three seeds, a larger policy, judge strengths, group sizes, learning rates, and an unfiltered prompt pool. In Math, exact-verifier outcomes expose the gap between observed and true informativeness. Replay versus difficulty-matched replay preserves exposure while changing prompt identities, strengthening the evidence that selection concentrates on problematic prompts. This is useful knowledge even without consistent changes in held-out Math accuracy.

  2. The mitigation study is informative and acknowledges important limits. Comparing floored selection with and without importance weights helps separate softer selection from correction of the sampling distribution. The paper reports that IW-GRESO does not itself save rollouts and includes DPPO as a more economical comparator. The extensive appendix supplies per-seed results and records design amendments. These details support the central empirical contribution and make the work more useful than a standalone negative result.

Critical weaknesses
  1. The main Math efficiency metric is coupled to the training schedule. Strict mastery counts a prompt once any trained visit has eight correct responses. It is cumulative and observed only when that prompt is selected, giving more frequently visited prompts more opportunities to pass; Appendix B.5 acknowledges this. Equal global training steps do not equalize these evaluation opportunities. Thus the reduced mastery lead and the more-than-twofold time-to-mastery ratio do not by themselves establish a comparable loss of independently measured learning efficiency, especially given the absence of a consistent held-out accuracy change. A fixed-set checkpoint evaluation would make this part of the conclusion substantially stronger.

  2. The importance-weighting and efficiency claims need a narrower connection to the actual experiments. Proposition 4 recovers the expected gradient of a uniform target with the same pre-draw fixed normalizer. The experimental uniform arm instead uses epoch permutations and realized token counts, while the weighted arms use different normalization (Appendices B.2 and D.9). The estimated deviation from that exact target is not formally bounded. In addition, Theorem 1 bounds the marginal-probability term of the second moment; the numerical 79.1%/94.8% results rely on a score-homogeneity model and streak-state information along uniform trajectories (Appendix D.8). They are conditional calculations, rather than a general limit on unbiased RL efficiency. These qualifications matter to interpretation, although they do not negate the observed Science failure mode.

Questions for authors
  1. Can saved checkpoints be evaluated on the same entire hard set at common checkpoints with identical fresh-sampling budgets? Current success and time to a common competence target would separate learning progress from exposure-dependent, cumulative mastery.

  2. Does the importance-weighting recovery persist with an independent uniform baseline and the same pre-draw fixed normalizer in all compared arms? Please separate the exact expected-gradient identity from the experimental baseline and optimizer trajectory, and state the assumptions behind the numerical variance-efficiency estimates in the main text.

2 comments

We would like to thank the reviewer for their time and consideration. Your comments were insightful and duly noted. We sincerely apologize for the late response as obtaining compute became an issue.

TL;DR.

  • Mastery is now measured independently of the schedule: on fresh samples at saved checkpoints (Table 33, three seeds), GRESO's step-200 lead is +8.9 ± 0.7 points with the exact verifier and +1.0 ± 1.0 under the judge.
  • "More than double the compute" is removed; the selector-specific part of the slowdown is 1.32 ± 0.27 (Appendix L.1).
  • Appendix L.2 separates the exact identity from the experiments; against an independent-draw uniform baseline with the same normalizer, IW-GRESO ends −0.8 ± 0.9 points.

Weakness 1 and Question 1 (mastery coupled to the schedule). We agree that cumulative strict mastery favours the arm that revisits. Table 33 (Appendix F.5, new) re-measures the saved checkpoints in three seeds: each answers all 359 hard prompts eight times afresh, the exact verifier grades the answers, and every arm gets exactly one fresh group per prompt.

Strict mastery of the hard set on fresh samples, % (Table 33, Appendix F.5)

Arm (reward)Step 100Step 200Step 300Lead over uniform, step 200
Uniform (exact verifier)45.2 ± 0.357.1 ± 0.362.6 ± 0.1–
GRESO (exact verifier)53.6 ± 1.066.0 ± 0.570.5 ± 0.5+8.9 ± 0.7
Uniform (judge)39.8 ± 0.249.0 ± 0.153.7 ± 0.3–
GRESO (judge)40.4 ± 1.150.0 ± 1.154.5 ± 0.7+1.0 ± 1.0

Section 3 now states that on fresh samples the judge removes most of what selection buys.

Appendix L.1 (new) adds the time to a common target and current success. Seed 2 has a checkpoint every 25 steps, scored on sixteen fresh samples per prompt with sampling seeds shared by all arms; in seeds 0 and 1 the crossing step is estimated from the training logs, with a correction fitted on 14 evaluated step-300 checkpoints.

Readouts independent of the visit schedule (Table 58, Appendix L.1)

ReadoutMean ± SE
Steps to 50% of the hard set, GRESO / uniform, exact verifier0.95 ± 0.02
Same, judge1.23 ± 0.23
Same, judge, without the log correction (L.1, Sensitivity)0.94 ± 0.10
Steps to 50%, judge run / exact-verifier run: GRESO2.31 ± 0.12
Same: uniform sampling1.89 ± 0.33
Same: GRESO's ratio / uniform sampling's (selector-specific part)1.32 ± 0.27
Pass@1 on the hard set at step 300, GRESO − uniform, judge: answers at 1,024 / 2,048 tokens−2.4 ± 2.1 / +0.4 ± 0.9
Mastered hard prompts all correct again on return, judge, seeds 0–1: GRESO / uniform (%, mean ± SD)72.6 ± 0.2 / 84.9 ± 1.3

Two of the three seeds are log estimates and the judge ratio depends on the correction, so the time readout does not settle the question; the conclusion rests on Table 33. The 2,048-token row is there because one GRESO run truncates 16.9% of its hard-set answers at 1,024. We agree that the doubling does not by itself show a selector-specific loss: it is mostly shared by all arms. Sections 1 and 3 give the selector-specific part, 1.32 ± 0.27, as context, not as the finding. The abstract no longer says "more than double the compute"; it states the fresh-sample result of Table 33, GRESO's lead at step 200 of +8.9 ± 0.7 points with the exact verifier against +1.0 ± 1.0 under the judge.

We claim no held-out loss in Math (Appendix G). Appendix L.4 (new) shows where the harm appears instead: the judge over-credits GRESO's held-out answers, so a judged held-out evaluation would credit GRESO with 10.1 ± 2.7 points it has not earned, against 4.3 ± 0.1 for uniform sampling.

Judge's acceptance of wrong held-out answers at step 300 (Table 61, Appendix L.4)Base policyUniform, judgeGRESO, judge
Accepted (%)22.434.8 ± 0.255.7 ± 7.1

Weakness 2 and Question 2 (the identity and the experiments). We agree. Appendix L.2 (new) separates what is exact from what is measured.

  1. The exact identity. Proposition 4 holds per step under four conditions: inclusion probabilities fixed by the history and positive; a normalizer fixed before the draw and shared by both sides; group gradients that do not depend on the other groups drawn (GRPO's baseline is the group's own mean); and one optimizer step per batch. Table 59 lists which arm meets which. The weighted arms meet all four, except that their normalizer equals the reference's only in expectation; the epoch-permutation uniform arm meets neither normalizer condition. Adam and the different update variance make the trajectories differ at second order, so the experiments test whether weighting restores uniform sampling's outcomes, not the identity itself.

  2. The independent baseline. We trained uniform sampling with an independent draw at every step and the pre-draw normalizer of the weighted audit-informed arm trained next to it, on the unfiltered Science pool, in seed 0 only.

Seed 0, unfiltered Science pool, MMLU-Pro STEM (Appendix L.2)Points
Independent-draw uniform arm − epoch-permutation uniform arm−0.27
Between two live GRESO runs of the same seed2.4
Audit-informed arm − independent-draw uniform arm+2.4 (SE 0.85)

We also trained this independent-draw arm in the setting of your question (Science, Qwen3-1.7B-Base, the filtered 406-prompt pool, the 12B rubric judge, 200 steps, three seeds): against it, IW-GRESO with the same normalizer ends −0.8 ± 0.9 points, within one SE of zero (Table 59, lower part, new).

Filtered Science pool, MMLU-Pro STEM, three seeds (Table 59, Appendix L.2)STEM (%)− independent-draw uniform (points)
Uniform, epoch permutation, realized token count48.8 ± 0.3+0.3 ± 0.5
Uniform, independent draws, normalizer fixed before the draw48.5 ± 0.6–
IW-GRESO, same normalizer47.8 ± 1.0−0.8 ± 0.9
GRESO40.6 ± 4.3−8.0 ± 4.0

No independent-draw arm was trained in Math; L.2 states this limit.

  1. The numerical bound. We agree that 79.1% and 94.8% are conditional calculations; they now say so where quoted. Section 5 adds: "Both assume score homogeneity and streak-state information along the uniform runs (Appendices D.8, L.2)." It also says that the values bound the rollouts per step that match the marginal-probability term of uniform sampling's second moment on the filtered pools, and Appendix G adds that they "hold for that model and are not a general limit on unbiased selection".
Theorem 1 bound, share of uniform sampling's rollouts (Appendix L.2)MathScience
Three seeds (Table 18)0.791 ± 0.0010.948 ± 0.003
Recomputed from the logs of seeds 0 and 10.7900.952
Design given the exact verifier's streak0.801–

On the unfiltered Science pool the judge's streak allows a saving of 2.4%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.