- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how GRPO's reward-count-dependent advantages can reinforce correct answers that lack a supporting derivation. It analyzes the limitations of zero-mean, two-valued within-group reweighting and proposes Advantage Harness, a prompt selection layer. An offline pass of the initial policy flags prompts whose sampled groups are all correct or all wrong; training then preferentially selects unflagged prompts before generating rollouts. Experiments use Qwen2.5-0.5B-Instruct and Qwen2.5-3B-Instruct, trained on MATH-7.5K and evaluated on eight mathematics benchmarks. The paper reports accuracy improvements with the initial-policy prior, a substantial zero-gradient reduction at 0.5B, and limitations of prior refreshing and of the proposed explanation for the gains.
Critical strengths
S1: The controls help establish that the prompt-specific prior contains useful information.
Table 4 includes a shuffled control with the same over-draw factor and the same multiset of prior values, as well as a difficulty-label filter. At 0.5B, the proposed method reduces the zero-gradient rate from 36.1% under GRPO to 17.8%, while the shuffled control remains at 36.6%. These comparisons provide evidence that the prompt-specific information, rather than over-drawing alone, changes the groups used for training.
S2: The analysis reports meaningful limitations instead of equating accuracy gains with better reasoning.
Tables 5,6 show that the estimated spurious share changes little, and Sec 4.4 reports that refreshing the prior at 3B reduces zero-gradient groups without improving accuracy. These results help distinguish the effects of selection on group composition from its effects on derivation quality and final performance.
Critical weaknesses
W1: The contribution and design choices are not sufficiently justified.
The method is fairly intuitive, but the paper does not show what it adds over simply filtering all-correct and all-wrong prompts using the same offline results. It also does not explain why each update uses only five prompts or whether the benefit depends on this small batch. These gaps make the added value of the proposed design unclear.
W2: The experimental support is limited.
All experiments use two small Qwen2.5-Instruct models, without testing other families or dedicated reasoning models. Accuracy gains are modest and vary across benchmarks. Refreshing the prior also reduces zero-gradient groups without improving accuracy. Given the substantial offline sampling cost, the current results do not convincingly establish a broad practical advantage.
Questions for authors
Q1: Why were five prompts per update and an over-draw factor of three chosen? Were they selected for resource constraints, tuning, or a methodological reason? Are there existing results or a limited sensitivity analysis showing whether the selection benefit remains with a larger prompt batch under a comparable generation budget? This would affect my assessment of whether the result depends on the small-batch setting.
Q2: Is there a comparison against running GRPO on a dataset permanently filtered using exactly the same offline table? If so, what does the proposed per-update selection add under the same prior and training budget? This would directly affect my assessment of the method's additional contribution.
Q3: Are there existing results on another model family or a dedicated thinking/reasoning model? If not, what evidence supports extending the conclusions beyond the two tested Qwen2.5-Instruct models? I am asking about the intended scope and any available evidence, rather than requiring a broad new benchmark suite during rebuttal.
4 comments
Rebuttal by Authors (Part 2)
Response to W2/Q3: We clarified the empirical scope, the prior comparison, and Table 21.
Existing Evidence and Intended Scope: The performance claims refer to Qwen2.5-0.5B-Instruct and Qwen2.5-3B-Instruct. The two scales provide different group-composition regimes: at 0.5B, GRPO draws 31.3% all-wrong and 4.8% all-correct groups; at 3B, these shares are 11.7% and 36.3%. This supports the paper's examination of a prior whose predictive value depends on the policy state. At 3B, refreshing the prior raises coverage of observed zero-gradient groups from 9.4% to 45.3% and changes their rate from 48.0% to 30.9%.
The existing performance evidence is reported separately. At 0.5B, the harness exceeds GRPO, the shuffled prior, and the level-label filter in Avg8 on every seed. At 3B, the stale-prior harness reaches 56.41 Avg8 versus 54.43 for GRPO; the refreshed-prior mean is 53.09. The additional benchmark table extends the task-level evidence using these same policies. The reward-count and expected-count identities apply under their stated binary-reward GRPO assumptions; the empirical conclusions retain the tested-model scope.
Practical Budget Scenarios: For one standalone run, the prior-plus-training counts are 135,936 rollouts at 0.5B and 78,048 for the 3B stale prior. A standalone refreshed-prior run uses 59,200 including the 3,200-rollout source-checkpoint training. When the documented priors are reused, the respective per-run counts are 29,326, 24,864, and 19,600, with the last including the source checkpoint. These explicit scenarios describe the one-time investment and its reuse. The selection layer then preserves the 16,000-rollout training budget while directing that budget toward the groups measured in the control comparisons.
The authors have addressed most of my concerns; accordingly, I lean toward raising my previous score to 3.
Response to Reviewer 9KMd
We appreciate the reviewer's recognition of our response and support for our work!
Best regards,
Authors of Submission 8159
Rebuttal by Authors (Part 1)
We appreciate Reviewer
9KMd's recognition that the shuffled and difficulty-label controls help establish the value of prompt-specific prior information and that the analysis helps distinguish changes in group composition, derivation-to-answer consistency, and final performance. We tried our best to address all the weaknesses and questions raised by the reviewer, as summarized below:W1/Q2: We clarified the comparison with permanent filtering in Appendix F using the selection rule specified in Appendix E.Q1: We clarified the common training protocol and candidate-budget choice, and added the arbitrary-batch identity in Eq. (9).W2/Q3: We clarified the intended empirical and theoretical scope, the stale/refreshed comparison, and the standalone and reuse budgets inTable 21.W1/Q2: Whether a permanent-filter baseline was evaluated using exactly the same offline table. What per-update selection contributes under the same prior and training budget.Response to W1/Q2: Thanks for the valuable comments! We expanded the analytical comparison in Appendix F, using the selection protocol specified in Appendix E.Relationship to Permanent Filtering: We provide an analytical comparison using the same offline table. The harness selects the lowest-scored b prompts from each block of Mb candidates and generates exactly bG rollouts. Permanent filtering constructs a fixed subset by applying a global acceptance rule to the prior table. With a binary prior, if U is the number of unflagged prompts in the current candidate block, the harness selects min(U,b) unflagged prompts and (b−U)+ flagged prompts. With the fractional fallback scores used at 3B, it ranks those scores within each candidate block as well. This specifies the relation using the same prior, without introducing another reward or estimator.
The resulting prompt marginal can be written as DH(q)=w(q)D(q), where w(q)=MPr(q is selected∣q∈Ct). A permanent acceptance set instead gives a conditional distribution on that set. Both are instances of changing the prompt distribution supplied to GRPO. The revised comparison explains their algorithmic relationship; the empirical attribution comes from the existing shuffled-prior, level-label, and estimator controls.
What the Controls Establish: The shuffled control preserves the exact multiset of prior values, the over-draw factor, and the selection mechanism. Its 36.6% zero-gradient rate remains close to GRPO's 36.1%, while the true prior gives 17.8%. The level-label filter gives 23.3% and the harness improves its Avg8 by +0.80 points. These controls support the contribution of prompt-specific policy information. The paper's contribution combines this controlled evidence with the spurious-advantage diagnosis and the analysis connecting selection to advantage mass and trainable-group count.
Q1: Why five prompts per update and an over-draw factor of three were chosen: resources, tuning, or a methodological reason. Existing larger-batch results or sensitivity analysis under a comparable generation budget.Response to Q1: We clarified the training and candidate budgets in Appendix F and the arbitrary-batch analysis in Eq. (9).Choice of Training and Candidate Budgets: The five-prompt batch is a budget-standardization choice. It implements the common protocol of 200 update iterations, 16 rollouts per prompt, and 16,000 generated training rollouts. This protocol standardizes the generation budget and schedules one optimizer step for each nonempty batch across the relevant settings. The over-draw factor M=3 was selected following development-stage offline replay of candidate selection and then fixed for the harness and its shuffled control. It makes 15 table lookups and generates rollouts for five prompts per update. The same values are used across the reported seeds; their role is to specify the shared training and selection protocol.
Dependence on Prompt Batch Size: The zero-gradient event is defined within a group of G rollouts. For a prompt with success probability pq, its probability is z(q)=pqG+(1−pq)G, independently of how many other prompts appear in the update. For any b,
E[Ntgrp]=b(1−zˉt).
Thus, under a fixed policy and prompt marginal, 1−zˉt is the trainable fraction per generated group at every prompt batch size. The 0.5B measurements correspond to 63.9% for GRPO and 82.2% for the harness. The identity explains the group-utilization mechanism separately from the probability that an entire small batch is inactive. The revision presents this as an analytical statement; a different batch size has its own selected marginal and policy trajectory.