acceptodds
Under review as a conference paper at ICLR 2027

Advantage Harness: Regulating GRPO's Advantage Signal by Shaping What It Sees

Abstract

Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each roll- out a magnitude from within-group reward statistics. This magnitude does not tell a reasoned correct answer from a guessed one, and it is largest when few rollouts in the group are correct. We identify this component as the spurious advantage. It arises on bounded-answer tasks and on the bounded sub-cases that cover more than half of MATH-7.5K. On multiple-choice prompts, about half of the correct rollouts in single-correct groups lack a supporting derivation, and training reduces this share but does not remove it. A zero-mean reweighting that gives one value to correct and one to wrong rollouts only rescales the group and cannot act on groups whose rollouts are all correct or all wrong. The lever that remains is the prompt distribution, which GRPO keeps fixed while the pass rate of the policy, and with it the set of groups that carry a gradient, moves during training. To this end, we propose Advantage Harness, a prompt selection layer that over-draws prompts at each update and keeps those least likely to yield a zero-gradient group. The esti- mator and the training rollout budget remain those of GRPO. On Qwen2.5-0.5B, the harness halves the zero-gradient rate and improves the eight-benchmark av- erage over GRPO, a shuffled control and a difficulty-label filter on every seed. On Qwen2.5-3B, a prior rebuilt from a checkpoint after 40 updates restores the zero-gradient reduction. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GRPO, RL, advantage estimation, reasoning, harness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9KMdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how GRPO's reward-count-dependent advantages can reinforce correct answers that lack a supporting derivation. It analyzes the limitations of zero-mean, two-valued within-group reweighting and proposes Advantage Harness, a prompt selection layer. An offline pass of the initial policy flags prompts whose sampled groups are all correct or all wrong; training then preferentially selects unflagged prompts before generating rollouts. Experiments use Qwen2.5-0.5B-Instruct and Qwen2.5-3B-Instruct, trained on MATH-7.5K and evaluated on eight mathematics benchmarks. The paper reports accuracy improvements with the initial-policy prior, a substantial zero-gradient reduction at 0.5B, and limitations of prior refreshing and of the proposed explanation for the gains.

Critical strengths

S1: The controls help establish that the prompt-specific prior contains useful information.

Table 4 includes a shuffled control with the same over-draw factor and the same multiset of prior values, as well as a difficulty-label filter. At 0.5B, the proposed method reduces the zero-gradient rate from 36.1% under GRPO to 17.8%, while the shuffled control remains at 36.6%. These comparisons provide evidence that the prompt-specific information, rather than over-drawing alone, changes the groups used for training.

S2: The analysis reports meaningful limitations instead of equating accuracy gains with better reasoning.

Tables 5,6 show that the estimated spurious share changes little, and Sec 4.4 reports that refreshing the prior at 3B reduces zero-gradient groups without improving accuracy. These results help distinguish the effects of selection on group composition from its effects on derivation quality and final performance.

Critical weaknesses

W1: The contribution and design choices are not sufficiently justified.

The method is fairly intuitive, but the paper does not show what it adds over simply filtering all-correct and all-wrong prompts using the same offline results. It also does not explain why each update uses only five prompts or whether the benefit depends on this small batch. These gaps make the added value of the proposed design unclear.

W2: The experimental support is limited.

All experiments use two small Qwen2.5-Instruct models, without testing other families or dedicated reasoning models. Accuracy gains are modest and vary across benchmarks. Refreshing the prior also reduces zero-gradient groups without improving accuracy. Given the substantial offline sampling cost, the current results do not convincingly establish a broad practical advantage.

Questions for authors

Q1: Why were five prompts per update and an over-draw factor of three chosen? Were they selected for resource constraints, tuning, or a methodological reason? Are there existing results or a limited sensitivity analysis showing whether the selection benefit remains with a larger prompt batch under a comparable generation budget? This would affect my assessment of whether the result depends on the small-batch setting.

Q2: Is there a comparison against running GRPO on a dataset permanently filtered using exactly the same offline table? If so, what does the proposed per-update selection add under the same prior and training budget? This would directly affect my assessment of the method's additional contribution.

Q3: Are there existing results on another model family or a dedicated thinking/reasoning model? If not, what evidence supports extending the conclusions beyond the two tested Qwen2.5-Instruct models? I am asking about the intended scope and any available evidence, rather than requiring a broad new benchmark suite during rebuttal.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

We appreciate Reviewer 9KMd's recognition that the shuffled and difficulty-label controls help establish the value of prompt-specific prior information and that the analysis helps distinguish changes in group composition, derivation-to-answer consistency, and final performance. We tried our best to address all the weaknesses and questions raised by the reviewer, as summarized below:

  • W1/Q2: We clarified the comparison with permanent filtering in Appendix F using the selection rule specified in Appendix E.
  • Q1: We clarified the common training protocol and candidate-budget choice, and added the arbitrary-batch identity in Eq. (9).
  • W2/Q3: We clarified the intended empirical and theoretical scope, the stale/refreshed comparison, and the standalone and reuse budgets in Table 21.

W1/Q2: Whether a permanent-filter baseline was evaluated using exactly the same offline table. What per-update selection contributes under the same prior and training budget.

Response to W1/Q2: Thanks for the valuable comments! We expanded the analytical comparison in Appendix F, using the selection protocol specified in Appendix E.

Relationship to Permanent Filtering: We provide an analytical comparison using the same offline table. The harness selects the lowest-scored bb prompts from each block of MbMb candidates and generates exactly bGbG rollouts. Permanent filtering constructs a fixed subset by applying a global acceptance rule to the prior table. With a binary prior, if UU is the number of unflagged prompts in the current candidate block, the harness selects min⁡(U,b)\min(U,b) unflagged prompts and (b−U)+(b-U)_+ flagged prompts. With the fractional fallback scores used at 3B, it ranks those scores within each candidate block as well. This specifies the relation using the same prior, without introducing another reward or estimator.

The resulting prompt marginal can be written as DH(q)=w(q)D(q)\mathcal D_H(q)=w(q)\mathcal D(q), where w(q)=MPr⁡(q is selected∣q∈Ct)w(q)=M\Pr(q\text{ is selected}\mid q\in\mathcal C_t). A permanent acceptance set instead gives a conditional distribution on that set. Both are instances of changing the prompt distribution supplied to GRPO. The revised comparison explains their algorithmic relationship; the empirical attribution comes from the existing shuffled-prior, level-label, and estimator controls.

What the Controls Establish: The shuffled control preserves the exact multiset of prior values, the over-draw factor, and the selection mechanism. Its 36.6% zero-gradient rate remains close to GRPO's 36.1%, while the true prior gives 17.8%. The level-label filter gives 23.3% and the harness improves its Avg8 by +0.80 points. These controls support the contribution of prompt-specific policy information. The paper's contribution combines this controlled evidence with the spurious-advantage diagnosis and the analysis connecting selection to advantage mass and trainable-group count.

Q1: Why five prompts per update and an over-draw factor of three were chosen: resources, tuning, or a methodological reason. Existing larger-batch results or sensitivity analysis under a comparable generation budget.

Response to Q1: We clarified the training and candidate budgets in Appendix F and the arbitrary-batch analysis in Eq. (9).

Choice of Training and Candidate Budgets: The five-prompt batch is a budget-standardization choice. It implements the common protocol of 200 update iterations, 16 rollouts per prompt, and 16,000 generated training rollouts. This protocol standardizes the generation budget and schedules one optimizer step for each nonempty batch across the relevant settings. The over-draw factor M=3M=3 was selected following development-stage offline replay of candidate selection and then fixed for the harness and its shuffled control. It makes 15 table lookups and generates rollouts for five prompts per update. The same values are used across the reported seeds; their role is to specify the shared training and selection protocol.

Dependence on Prompt Batch Size: The zero-gradient event is defined within a group of GG rollouts. For a prompt with success probability pqp_q, its probability is z(q)=pqG+(1−pq)Gz(q)=p_q^G+(1-p_q)^G, independently of how many other prompts appear in the update. For any bb,

E[Ntgrp]=b(1−zˉt).\mathbb E[N_t^{\mathrm{grp}}]=b(1-\bar z_t).

Thus, under a fixed policy and prompt marginal, 1−zˉt1-\bar z_t is the trainable fraction per generated group at every prompt batch size. The 0.5B measurements correspond to 63.9% for GRPO and 82.2% for the harness. The identity explains the group-utilization mechanism separately from the probability that an entire small batch is inactive. The revision presents this as an analytical statement; a different batch size has its own selected marginal and policy trajectory.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

Response to W2/Q3: We clarified the empirical scope, the prior comparison, and Table 21.

Existing Evidence and Intended Scope: The performance claims refer to Qwen2.5-0.5B-Instruct and Qwen2.5-3B-Instruct. The two scales provide different group-composition regimes: at 0.5B, GRPO draws 31.3% all-wrong and 4.8% all-correct groups; at 3B, these shares are 11.7% and 36.3%. This supports the paper's examination of a prior whose predictive value depends on the policy state. At 3B, refreshing the prior raises coverage of observed zero-gradient groups from 9.4% to 45.3% and changes their rate from 48.0% to 30.9%.

The existing performance evidence is reported separately. At 0.5B, the harness exceeds GRPO, the shuffled prior, and the level-label filter in Avg8 on every seed. At 3B, the stale-prior harness reaches 56.41 Avg8 versus 54.43 for GRPO; the refreshed-prior mean is 53.09. The additional benchmark table extends the task-level evidence using these same policies. The reward-count and expected-count identities apply under their stated binary-reward GRPO assumptions; the empirical conclusions retain the tested-model scope.

Practical Budget Scenarios: For one standalone run, the prior-plus-training counts are 135,936 rollouts at 0.5B and 78,048 for the 3B stale prior. A standalone refreshed-prior run uses 59,200 including the 3,200-rollout source-checkpoint training. When the documented priors are reused, the respective per-run counts are 29,326, 24,864, and 19,600, with the last including the source checkpoint. These explicit scenarios describe the one-time investment and its reuse. The selection layer then preserves the 16,000-rollout training budget while directing that budget toward the groups measured in the control comparisons.

The authors have addressed most of my concerns; accordingly, I lean toward raising my previous score to 3.

AuthorsOpenReview ↗

Response to Reviewer 9KMd

We appreciate the reviewer's recognition of our response and support for our work!

Best regards,

Authors of Submission 8159

Reviewer_KoQHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines a potential issue with advantage assignment in GRPO when rewards are based solely on final-answer correctness: a model may receive a high advantage even when it arrives at the correct answer by chance rather than through sound reasoning. The authors first analyze the limitations of within-group, zero-mean reweighting under binary rewards. They then propose Advantage Harness, which uses an offline prior to select prompts more likely to produce informative gradients before generating training rollouts. The method shows some gains on the smaller model, but its benefits are less consistent on the larger one.

Critical strengths

The paper raises a worthwhile question: when a model gives the correct answer without a sound derivation, what exactly is GRPO rewarding? The analysis of within-group, zero-mean reweighting is clear and helps explain why simply adjusting the advantage weights of correct and incorrect rollouts cannot fully address the issue.

Critical weaknesses
  1. My main concern is the novelty of the method. PCL [1] also expands the candidate pool and selects moderately difficult prompts before rollout generation. For known pass probabilities and a fixed group size, minimizing (p^G + (1-p)^G) ranks prompts in the same order as choosing those with pass probabilities closest to 0.5. Although this paper uses an offline binary prior whereas PCL employs an online value model, their core prompt-selection ideas are quite similar. Moreover, the paper is motivated by spurious advantage arising from flawed reasoning, yet its selection rule does not examine the reasoning traces. In Section 4.3, the spurious share decreases by only 0.5 percentage points, while the spurious positive mass actually increases. As a result, the method seems closer to conventional zero-gradient-group filtering than to a solution specifically targeting spurious advantage.

  2. The measurement of flawed reasoning is not sufficiently validated. The checker mainly compares the last number before the final answer with the value of the selected option, which does not reliably establish whether the full derivation is correct. Appendix F already identifies errors caused by restating option values and misidentifying letters. Although the authors compare the checker against an LLM judge, the stratified sample does not adequately establish its reliability on the overall distribution. Furthermore, Eq. (8) applies flaw rates measured on AQuA with the 0.5B model to analyses involving MATH and the 3B model, without validating this transfer across datasets and model sizes.

  3. The stability and computational efficiency of the gains remain uncertain. Although the 0.5B model outperforms GRPO across all three random seeds, the statistical evidence is not strong enough to rule out random variation. For the 3B model, refreshing the prior reduces the fraction of zero-gradient groups but does not lead to a corresponding accuracy improvement. In addition, constructing the offline prior requires generating a substantial number of extra rollouts, at a cost far exceeding the rollout budget of a single training run. Amortizing this cost over multiple runs may not reflect the practical benefit in a single-run setting.

  4. Some results appear inconsistent. In Table 4, the “Ours−row” difference is nonzero for the Ours row. Table 8 reports a “±” statistic for 3B DAPO despite using only one seed.

[1] Gao, Z., Kim, J., Sun, W., Joachims, T., Wang, S., Pang, R. Y., and Tan, L. Prompt Curriculum Learning for Efficient LLM Post-Training. ICLR 2026.

Questions for authors
  1. Could the authors compare directly against PCL [1] under the same training protocol and use ablations to distinguish the contribution of the spurious-advantage diagnosis from the effect of conventional zero-gradient-group filtering?

  2. Which LLM judge and evaluation criteria were used to assess the derivations? Do the main findings hold after correcting the known parsing errors? Can the flaw rates measured on AQuA be reliably transferred to MATH and the 3B model?

  3. Could the authors provide a comparison in total generated tokens or GPU time that includes the cost of constructing the offline prior, and further examine the stability of accuracy gains across model sizes?

The paper discusses stale priors and reports some negative results. However, it should more clearly distinguish a guessing baseline constructed from answer formats from actual guessing behavior by the model, as well as checker labels from genuine reasoning correctness. In addition, the main experiments use only five prompts per update, making the impact of zero-gradient groups particularly pronounced. It remains unclear whether the observed benefits would carry over to larger-batch training.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

We appreciate Reviewer KoQH's recognition that this work raises a worthwhile question about what GRPO rewards and that the analysis of two-valued, zero-mean reweighting clearly explains the effect of adjusting correct and incorrect rollout weights. We tried our best to address all the weaknesses and questions raised by the reviewer, as summarized below:

  • W1/Q1: We expanded the method comparison in Appendix F and clarified the contribution and mechanism in Sections 2, 3.5, and 4.3.
  • W2/Q2: We clarified the derivation-check definition and judge rubric in Appendix F, added Table 22 in Appendix F using the existing audit labels, and specified the reference-proxy interpretation of Eq. (8).
  • W3/Q3: We clarified the paired seed results and added Table 21, separating standalone and shared-prior costs.
  • W4: We revised the self-comparison in Table 4 and the single-run DAPO reporting in Table 8.
  • Additional comments: We clarified the answer-format reference policies and the arbitrary-batch expected-count identity in Eq. (9).

W1/Q1: Direct comparison with PCL under the same training protocol. PCL's candidate expansion and moderate-difficulty selection, and equivalence of the oracle rankings. Ablations distinguishing the spurious-advantage diagnosis from zero-gradient filtering. The 0.5-point share change and increased positive mass.

Response to W1/Q1: Thanks for the valuable comments! We expanded Appendix F and clarified Sections 2, 3.5, and 4.3.

Relationship to PCL: We provide an analytical and method-level comparison with PCL. For known pass probabilities and a fixed group size, the two oracle rankings coincide. In particular,

z(p)=pG+(1−p)G,z′(p)=G[pG−1−(1−p)G−1],z(p)=p^G+(1-p)^G,\qquad z'(p)=G\bigl[p^{G-1}-(1-p)^{G-1}\bigr],

so z(p)z(p) increases with ∣p−0.5∣|p-0.5| for G>1G>1. We have made this relationship explicit. The implemented harness uses a single offline all-equal-group indicator and fixed-table lookups, while PCL estimates the selection signal with an online value model. Our theoretical and empirical contribution concerns the credit-assignment diagnosis, the reweighting constraint, and the change in the group distribution supplied to GRPO.

What the Existing Controls Isolate: The shuffled prior fixes the over-draw factor and the multiset of scores while changing their prompt assignments. It yields 36.6% zero-gradient groups, compared with 17.8% for the harness. The level-label filter retains the selection procedure and substitutes dataset difficulty, yielding 23.3%. The harness exceeds both controls in Avg8 on every reported seed. The estimator ablations examine the separate route of changing within-group weights. Together, these comparisons distinguish the information in the selection prior, the effect of over-drawing, and the effect of estimator reweighting. They are the controlled comparisons reported in the paper; the expanded PCL discussion specifies the method relationship and its oracle equivalence.

Interpreting the Share and Mass: The diagnosis and the intervention operate at different levels. The selector reads prompts and changes how many generated groups carry an advantage. In Table 9, the 1.33-fold increase in positive mass is the product of a 1.29-fold increase in trainable share and a 1.03-fold change in conditional group mass. The fixed-reference share of that mass changes from 42.3% to 41.8%. Thus the increase in its reference-weighted spurious mass follows directly from the larger total mass. The revision explicitly distinguishes this calculation from trace-level derivation checking and from final benchmark accuracy.

W2/Q2: What the deterministic checker measures. Judge identity and evaluation criteria. Whether the main findings hold after correcting known parsing cases. The 100-example stratified validation and reliability of applying AQuA flaw rates to MATH and 3B analyses.

Response to W2/Q2: We revised Appendix F and the description of Eq. (8), and added the design-weighted calculation in Table 22 using the existing audit.

Checker and Judge Criteria: The deterministic check uses the last-number/selected-option match as an operational check of derivation-to-answer consistency. It records supported, derivation mismatch, no derivation, or uncertain under the stated parsing rules. The existing validation uses a Claude LLM judge to read each full rollout with the checker label hidden. Its rubric asks whether the stated conclusion supports the chosen option or letter, including an explicitly justified nearest-option choice. This is a derivation-to-answer consistency assessment; the revised wording uses that operational definition consistently.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

Table T2. Existing checker validation, with the sample calculation and the design-weighted pool calculation distinguished. Checker-positive means derivation mismatch or no derivation. Precision and recall use the judge's unsupported-conclusion label as the reference.

CalculationAgreementPrecision of checker-positive labelRecall of checker-positive label
Original 100-example label-stratified sample81.00%83.33%87.30%
Design-weighted estimate for the known eligible four-dump pool82.44%88.19%79.57%

Population Weighting of the Existing Audit: The audit sampled 34 supported, 34 mismatch, and all 32 no-derivation examples. Their corresponding counts in the eligible pool are 6,242, 6,455, and 32. We therefore calculate the second row of Table T2 by weighting each audited observation in stratum hh by Nh/nhN_h/n_h. This uses the same archived judgments and the known 12,729-example pool from the untrained dump and the seed-1 GRPO, harness, and shuffled-control dumps. It restores the observed stratum proportions for that pool; the first row retains the original sample calculation.

Known Parsing Cases and the Finding They Test: The full-rollout audit directly examines the parsing cases raised in the review. Among 34 checker-supported examples, the judge identifies eight unsupported conclusions; among 66 checker-positive examples, it identifies 55 unsupported conclusions. Thus the same audited examples contain 63 judge-identified unsupported conclusions overall. The numerical finding that a correct final answer can coexist with an unsupported written conclusion is present under the full-rollout rubric as well as the deterministic labels. This comparison uses the existing full-rollout judgments on the audited examples. The zero-gradient rates, reward-count identities, group-mass decomposition, and benchmark scores are separately computed from verifier rewards and evaluation records and do not use the derivation checker.

Meaning of the AQuA Reference in Eq.8: We clarified that f(k)f(k) in Eq.8 is a fixed AQuA-0.5B reference profile applied to the directly measured group histogram H(k)H(k). Holding this profile fixed allows the diagnostic to examine how a changed group composition would redistribute positive weight under the same reference. The direct derivation-check measurements are the AQuA results in Table 17, Table 1, and Table 5. The MATH and 3B group statistics are directly measured from their own training rewards. For the direct 0.5B check on the same 1,500 AQuA prompts, Table 5 reports advantage-weighted checker-positive shares of 51.1±2.8%51.1\pm2.8\% for the harness, 54.9±4.9%54.9\pm4.9\% for GRPO, and 54.1±3.4%54.1\pm3.4\% for the shuffled control across three seeds. These measurements retain their policy-specific labels; Eq. (8) instead holds its reference profile fixed.

W3/Q3: Accuracy stability at both scales. End-to-end cost including the offline prior. A total-generated-token or GPU-time comparison including the offline prior, with single-run and repeated-run accounting distinguished.

Response to W3/Q3: We clarified the seed-wise comparisons and added Table 21.

Accuracy and Group Composition: At 0.5B, the harness-minus-GRPO Avg8 differences are +1.59, +0.32, and +2.13, with the reported paired p=0.129p=0.129. The corresponding differences against the shuffled control are +1.27, +2.48, and +0.88, and those against the level-label control are +1.02, +0.78, and +0.60. The revision presents the effect sizes, individual seeds, and tests together.

At 3B, the stale-prior harness reaches 56.41 Avg8 compared with 54.43 for GRPO. The refreshed prior reaches 53.09 and changes the zero-gradient rate from 48.0% to 30.9%. This comparison tests prior predictiveness: the refreshed table covers 45.3% of the observed zero-gradient groups, versus 9.4% for the stale table. The group rate describes usable samples; accuracy reflects the complete selected-prompt objective and the resulting policy. Both measurements are reported explicitly.

Tab T3. Generated-rollout accounting for one standalone harness run, calculated from the existing budgets. The refreshed-prior scenario includes constructing its source checkpoint.

SettingTrainingPrior constructionSource checkpointStandalone total
0.5B, initial-policy prior16,000119,9360135,936
3B, initial-policy prior16,00062,048078,048
3B, refreshed prior16,00040,0003,20059,200

Complete Cost Scenarios: Tab. T3 answers the standalone-run question separately from reuse. The 3,200-rollout checkpoint entry is 40×5×1640\times5\times16. Sharing the recorded priors across the evaluated runs gives 29,326 rollouts per 0.5B run and 24,864 per 3B stale-prior run. For the twelve refreshed-prior runs, the average is 19,600 including the source-checkpoint training, or 19,333 when that checkpoint already exists.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3)

The counted quantity is generated rollouts. Prior construction allows at most 4,096 generated tokens per rollout and policy training at most 8,192; these are length caps, not measured sequence lengths. Accordingly, the table identifies the measured rollout budget and the corresponding cap settings with rollout counts as the accounting unit for both prior construction and training. The supported budget claim is that selection adds no training rollouts after its prior is available, with the full prior and checkpoint costs reported separately.

W4: Nonzero Ours−row self-comparison in Table 4. A reported ± statistic for the one-seed 3B DAPO row in Table 8.

Response to W4: We revised Table 4 and Table 8 against the archived run records.

In Table 4, the Ours−row self-comparison and its pp entry are now shown as dashes; the self-difference is identically zero. Every other paired comparison retains its stated reference and seed pairing.

The 3B DAPO row now reports the individual recorded benchmark scores and Avg8 of 54.86 for its single run, without across-seed ± values. We also made its one-seed status and 18,896 generated rollouts explicit. The seed counts for the other methods are retained beside their summaries.

Additional comments: Answer-format guessing baselines versus actual model guessing. Checker labels versus reasoning correctness. Five prompts per update and larger-batch relevance.

Response to Additional comments: We clarified Appendix D, Appendix F, and Eq. (9).

The MATH answer-format analysis constructs explicit reference policies, such as always emitting a particular answer string, and computes their score against the gold answers. These are answer-space baselines. The model-specific evidence is separately supplied by the generated rollouts and the operational derivation-to-answer consistency checks described above.

For any prompt batch size bb, the expected number of trainable groups satisfies

=\sum_{q\in\mathcal S_t}\bigl(1-z_t(q)\bigr),\qquad \mathbb E[N_t^{\mathrm{grp}}]=b(1-\bar z_t).$$ This follows from linearity of expectation and applies to dependent selected prompts. For independent group indicators at a fixed zero-gradient rate, the probability of an entirely inactive update is $\bar z_t^b$. The per-group fraction $1-\bar z_t$ separately determines how much of the generated budget carries nonzero advantage. The revised analysis states this general identity and treats each batch size through its own selected prompt distribution and policy.
Reviewer_m7aaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates spurious advantage signals in GRPO, where correct rollouts without valid reasoning may receive disproportionately large advantages. The authors propose Advantage Harness, a prompt selection strategy that uses offline rollout statistics to identify prompts less likely to produce zero-gradient groups. It filters candidate prompts before generation while keeping the original GRPO advantage estimator and training rollout budget unchanged. Experiments on Qwen2.5-0.5B and 3B models show reduced zero-gradient rates and improved average performance on the 0.5B model, with mixed results at the larger scale.

Critical strengths
  1. The paper investigates an interesting issue in GRPO and proposes a simple prompt selection strategy without modifying the original optimization objective. The experiments also include several useful controls, such as shuffled priors and difficulty-based filtering, to better understand the effect of the proposed method.
Critical weaknesses
  1. The motivation and novelty of the proposed method are not fully convincing. The paper is motivated by spurious advantage signals, but the proposed method mainly reduces zero-gradient groups rather than addressing incorrect reasoning. As shown in Table 6, the estimated spurious advantage share remains almost unchanged. Moreover, similar pre-rollout prompt selection has already been explored in GRESO (Zheng et al., Nuerips 2025) and Prompt Curriculum Learning (Gao et al., ICLR 2026), but neither is included as a direct baseline. The method also requires an additional offline pass with considerable computational cost.

  2. The experimental results are not sufficiently convincing, especially regarding scalability. On the 0.5B model, the improvement over GRPO is relatively small and not statistically significant (p = 0.129, Table 4). The 3B results are also inconsistent across different priors. The refreshed prior substantially reduces zero-gradient groups but fails to improve accuracy, while the stale prior improves accuracy without meaningfully reducing zero-gradient groups. Moreover, the experiments only consider Qwen2.5-Instruct models up to 3B, without evaluating larger or more recent thinking-capable models. It remains unclear whether the method can generalize to modern reasoning models at larger scales.

Questions for authors
  1. Some experimental details appear inconsistent and need clarification. For the 3B stale-prior harness, Table 8 reports three seeds, while Tables 14 and 15 report only two for some statistics. Also, Table 8 describes DAPO as using the same rollout budget, but Table 15 reports 18,896 rollouts for DAPO compared with 16,000 for the other methods. Could the authors clarify these differences?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

We appreciate Reviewer m7aa's recognition that this work investigates an interesting issue in GRPO, introduces a simple prompt-selection strategy while retaining the optimization objective, and includes useful shuffled-prior and difficulty-based controls. We tried our best to address all the weaknesses and questions raised by the reviewer, as summarized below:

  • W1: We revised the Abstract and Sections 2, 3.5, and 4.3, and added the comparison in Appendix F and the cost accounting in Table 21.
  • W2: We clarified the paired results in Table 4, the 3B comparison in Table 8, and the mechanism analysis in Appendix C.
  • Q1: We harmonized the seed reporting in Table 8, Table 14, and Table 15, and revised the DAPO budget descriptions.

W1: Relationship between spurious-advantage diagnosis and zero-gradient selection. Estimated spurious share in Table 6. Direct baselines and methodological differences relative to GRESO and Prompt Curriculum Learning. Offline-pass cost.

Response to W1: Thanks for the valuable comments! We revised the Abstract and Sections 2, 3.5, and 4.3, and added the comparison in Appendix F and Table 21.

From the Diagnosis to Prompt Selection: The diagnosis concerns how GRPO assigns credit: correct rollouts in the same group receive the same reward-count-dependent magnitude, whether their written derivations support the answer or not. Our analysis then clarifies where prompt selection acts. For any two-valued, zero-mean reweighting, n+a+=n−∣a−∣n^+a^+=n^-|a^-| fixes the relative weights within the group, and an all-correct or all-wrong group still has zero advantage. The harness therefore changes the prompt distribution that supplies groups to the unchanged GRPO estimator.

This distinction is reflected in the existing measurements. At 0.5B, the trainable share rises from 63.9% to 82.2%, a factor of 1.29, and the conditional positive advantage mass rises by a factor of 1.03. Their product gives the 1.33-fold total positive mass in Table 9. The fixed-reference spurious-share calculation changes from 42.3% to 41.8%, while its corresponding mass rises by 1.31. These quantities measure different aspects of the same distribution: the count effect describes how much advantage the generated budget supplies, and the fixed-reference share describes its composition. The revised text makes this relationship explicit.

Direct Checks of the Trained Policies: Separately, existing Table 5 checks each final policy on the same 1,500 AQuA prompts. Across three seeds, the advantage-weighted share labelled derivation mismatch or no derivation is 51.1±2.8%51.1\pm2.8\% for the harness, 54.9±4.9%54.9\pm4.9\% for GRPO, and 54.1±3.4%54.1\pm3.4\% for the shuffled control. These are direct policy-specific diagnostics, whereas Table 6 applies one fixed reference profile to the group distributions. The revision distinguishes the two calculations and their denominators.

Relationship to Prompt-Selection Methods: We provide a method-level comparison with PCL and GRESO, both discussed in the submitted Related Work. We have expanded this comparison to distinguish the information used for selection: PCL learns an online value model, GRESO uses reward histories from earlier epochs, and our main harness reads a fixed table constructed by one offline pass. The harness retains the GRPO estimator and generates rollouts only for the selected prompts.

The contribution is supported by the diagnosis, the reweighting analysis, and the controlled examination of the distribution reaching the estimator. In the shuffled control, the candidate budget and multiset of prior values are preserved while their prompt assignments are permuted. Its zero-gradient rate is 36.6%, compared with 17.8% for the harness and 36.1% for GRPO. The level-label control uses the same selection procedure with dataset difficulty labels and reaches 23.3%. These comparisons identify the contribution of prompt-specific policy information under a common training protocol. The PCL/GRESO comparison in the revision is a comparison of methods and information requirements; the empirical controls are the explicitly named runs above.

Offline-Pass and Training Budgets: Table 21 separates constructing the prior from the subsequent training run. For 0.5B, the prior uses 119,936 rollouts and training uses 16,000, giving 135,936 rollouts for a standalone run including its prior. When the same prior serves the nine documented runs, the corresponding average is 29,326 rollouts per run. The exact-budget property of the harness refers to the 16,000 training rollouts; the table separately supplies the one-run and reuse accounting.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

W2: Statistical evidence at 0.5B. Different accuracy and zero-gradient outcomes for stale and refreshed priors at 3B. Scope across model sizes and modern reasoning models.

Response to W2: We clarified Table 4, Table 8, and the analysis in Appendix C.

Table T1. Existing paired Avg8 differences at 0.5B. Each difference is in percentage points and compares runs with the same seed. Differences and tests use the archived unrounded values underlying manuscript Tables 4 and 11; displayed values are rounded.

ComparisonSeed 1Seed 2Seed 3Mean differencePaired-test pp
Harness − GRPO+1.59+0.32+2.13+1.350.129
Harness − shuffled prior+1.27+2.48+0.88+1.540.084
Harness − level-label filter+1.02+0.78+0.60+0.800.022

Paired Evidence at 0.5B: Table T1 presents the full paired pattern and its statistical tests together. The Avg8 differences are positive on each seed for all three comparisons. The p=0.129p=0.129 entry accompanies the +1.35-point comparison with GRPO; the p=0.022p=0.022 entry belongs to the separate comparison with the level-label filter. Our conclusion uses these seed-wise observations together with the prompt-specific controls and measured change in group composition.

Interpreting the 3B Priors: The two priors probe different prompt distributions. The stale-prior harness reaches 56.41 Avg8 versus 54.43 for GRPO and changes mean positive per-rollout advantage from 1.100 to 0.898, with zero-gradient rates of 47.2% and 48.0%. The refreshed prior reaches 53.09 Avg8 and reduces the zero-gradient rate to 30.9%. Its coverage of zero-gradient groups increases from 9.4% under the stale prior to 45.3%. These results distinguish the amount of usable group information from the downstream effect of the full prompt distribution. Accuracy depends on the selected prompts and their conditional gradients; it is not determined by zero-gradient rate alone.

Intended Scope: The empirical statements concern Qwen2.5-0.5B-Instruct and Qwen2.5-3B-Instruct, with the eight core benchmarks and the additional benchmark results already reported in Table 12. These scales expose two different regimes: predominantly all-wrong zero-gradient groups at 0.5B and predominantly all-correct groups at 3B. The reward-count identities and expected-trainable-group analysis apply to binary-reward GRPO independently of the model architecture. The revision keeps this theoretical scope separate from the performance measured for the tested models.

Q1: Three seeds in Table 8 versus two in Tables 14–15 for the 3B stale-prior harness. DAPO's 18,896 generated rollouts versus 16,000 for the other methods.

Response to Q1: We harmonized Table 8, Table 14, and Table 15, and revised the DAPO budget descriptions.

Seed Accounting: The 3B stale-prior harness has three completed runs, with Avg8 values 55.19, 58.32, and 55.74. The main accuracy table already used all three. We have brought the mechanism and compute tables to the same seeds 1–3 using the existing records. Comparisons paired with the two-run shuffled control explicitly use the two common seeds. Including the third harness run, the stale prior is shared by seven documented runs, so its amortized inclusive count is 16,000+62,048/7=24,86416,000+62,048/7=24,864 rollouts per run.

DAPO Budget Accounting: DAPO uses additional generation rounds to fill its trainable batch and is evaluated at update 136. The actual generated counts are 16,144, 15,952, and 16,560 at 0.5B, and 18,896 for the single 3B run. GRPO and the harness each generate 16,000 rollouts in 200 updates. The revised captions now state these actual budgets consistently with Table 15. At 3B, DAPO's generated count is 2,896 rollouts above the GRPO/harness count; the comparison retains that recorded run and makes its budget explicit.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chairs and Reviewers,

We would like to thank the Area Chairs and all reviewers for their time and effort. We appreciate the reviewers’ recognition that this work investigates an interesting issue in GRPO (m7aa) and raises a worthwhile question about what GRPO rewards (KoQH). The reviewers also highlighted the simple prompt-selection strategy that retains the original optimization objective (m7aa) and the clear analysis of within-group, zero-mean reweighting (KoQH). They recognized useful controls for understanding the method’s effects (m7aa), comparisons that help establish the value of prompt-specific prior information (9KMd), and analyses that help distinguish selection’s effects on group composition, derivation quality, and final performance (9KMd).

In response to the reviewers’ comments and suggestions, we further discussed Advantage Harness’s relationship to PCL, GRESO, and permanent filtering with the same prior (Section 2 and Appendix F). We clarified the connection between the spurious-advantage diagnosis and prompt selection (Sections 2, 3.5, and 4.3). We explained the training and selection configurations and provided an analysis for arbitrary prompt batch sizes (Section 3.5 and Appendices C and F). We clarified derivation-check criteria and diagnostic interpretation (Sections 3.3 and 4.3 and Appendix F), as well as the role of answer-format reference policies (Appendix D). We provided a reweighted analysis of the existing checker-validation judgments (Appendix F, Table 22 added). We clarified experimental settings, empirical scope, and result interpretation, and aligned the reported scores, seed counts, and rollout budgets with the archived run records (Sections 4.1–4.4 and Appendices E and F). We reported prior-construction and training rollout costs for standalone and shared-prior use (Appendix F, Table 21 added).

Once again, we thank all the reviewers for their thoughtful and constructive feedback, which has been instrumental in enhancing the quality and impact of this submission. We hope this summary could assist the evaluation of this submission.

Best regards,

Authors of Submission 8159

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.