acceptodds
Under review as a conference paper at ICLR 2027

Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities

Abstract

Reinforcement learning with verifiable rewards (RLVR) can improve reasoning accuracy while producing increasingly concentrated outputs. Under binary rewards, Group Relative Policy Optimization (GRPO) assigns identical positive advantages to correct responses within mixed-reward groups, without accounting for their generation confidence. We propose Probabilistic GRPO (ProGRPO), which introduces a confidence-based Advantage Re-weighting Mechanism (ARM). ARM uses the difference between prompt and response confidence, computed as geometric means of token probabilities over low-probability positions under the old policy. Within each reward class, lower-confidence responses receive larger adjusted advantage coefficients. The correction is disabled for uniform-reward groups and preserves advantage signs under a sufficient bound on its magnitude. Experiments on mathematical reasoning and code generation using Qwen2.5 and DeepSeek-R1-Distill-Qwen models show improvements in accuracy and multi-sample solution coverage. On Qwen2.5-7B, adding ARM to a matched GRPO with Clip-Higher baseline improves average Pass@1 and Pass@32 by 3.3 and 8.3 percentage points, respectively. The complete ProGRPO method improves these metrics over vanilla GRPO by 5.7 and 13.8 percentage points. Training-entropy and output-similarity measurements provide complementary evidence of reduced output concentration in the evaluated settings.

Primary area:
reinforcement learning
Keywords:
LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_siGvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProGRPO, which estimates prompt and response confidence from the geometric mean probability of selected low-probability tokens. It adjusts GRPO advantages using the difference between these confidence scores. Among responses with the same reward, lower-confidence responses receive larger adjusted advantages; groups with all correct or all incorrect answers are left unchanged. Combined with Clip-Higher, the method is evaluated on math and code tasks, including a Qwen2.5-32B model, with additional transfer results on question answering.

Critical strengths
  1. The adjustment is simple. The comparison with GRPO using Clip-Higher under matched training settings on Qwen2.5-7B helps separate the effect of the confidence adjustment from that of clipping.

  2. The experiments cover math, code, and several model sizes, including 32B. They report both pass@1 and pass@k. Analyses of the confidence measure, token selection, and extension to PPO also provide useful information.

Critical weaknesses
  1. The motivation needs stronger support. Equal advantages do not imply equal parameter updates, since the policy gradient already depends on token probabilities. The analysis does not sufficiently explain why the proposed confidence adjustment should encourage useful exploration.

  2. The paper lacks a controlled comparison with Entropy Advantage, a closely related method. Quoting results from different settings does not establish an improvement over it. The 32B experiments also compare only with GRPO, which limits the evidence at larger scales.

Questions for authors

See weakness.

3 comments

We sincerely thank the reviewer for the valuable feedback. Our detailed responses are provided below.


W1.

We agree that equal advantages do not imply identical parameter updates. Motivated by the entropy decline and output concentration observed during GRPO training (p. 8, Figure 3), ARM adjusts the relative learning weights of equally rewarded responses through confidence-dependent advantage corrections. Appendix A.4 (p. 15, Eqs. (35)–(36)) characterizes the additional surrogate-gradient term induced by this reweighting at fixed policy parameters. This analysis concerns coefficient reweighting and does not guarantee larger parameter updates or monotonic entropy growth.

Given prompt confidence cqc_q and response confidence cic_i, ARM uses:

Δi=α(cq−ci),α>0.\Delta_i = \alpha(c_q-c_i), \qquad \alpha > 0.

Thus, lower-confidence correct responses within the same mixed-reward group receive larger relative advantage corrections. We hypothesize that these responses contain successful behaviors the policy does not yet reproduce reliably. Confidence is computed from the old policy, refreshed with new rollouts, and held fixed during the corresponding policy updates (p. 4, Sections 3.1–3.2).

Ablation evidence. Table 10 (p. 18) reports the following unweighted averages across six mathematical reasoning benchmarks:

SettingAverage Pass@1 (%)Average Pass@32 (%)
GRPO + Clip-Higher40.060.2
Constant shift40.761.1
Uniform negative shift40.963.6
Group-centered correction41.763.1
Full ARM43.368.5

The group-centered control supports within-group confidence differentiation, while Full ARM outperforms all listed controls.

Correct-response diversity. On AIME 2024, ProGRPO also produces less similar correct responses (p. 8, Table 5):

MethodSelf-BLEU ↓Semantic cosine similarity ↓
GRPO + Clip-Higher0.75040.9344
ProGRPO0.67460.9233

The accuracy, multi-sample coverage, and diversity results provide behavioral evidence consistent with useful exploration.


W2.

EA comparison at 7B. We implemented Entropy Advantage with Clip-Higher on Qwen2.5-7B. We evaluate ProGRPO at step 360 and EA at steps 350 and 400 under otherwise matched training and evaluation settings. Both EA checkpoints come from one training run with seed 42. Implementation details and full results appear in Appendix B.4, p. 19, Table 12.

All averages below are unweighted means across six mathematical reasoning benchmarks.

MethodAverage Pass@1 (%)Average Pass@32 (%)
Clip-Higher + EA, step 35040.0556.24
Clip-Higher + EA, step 40039.4957.94
ProGRPO, step 36043.3068.50

ProGRPO exceeds both evaluated EA checkpoints on both average metrics.

Matched control at 32B. We added a Qwen2.5-32B Clip-Higher control with α=0\alpha=0, retaining the same training settings while disabling ARM (p. 6, Table 2).

MethodAverage Pass@1 (%)Average Pass@32 (%)
GRPO + Clip-Higher (α=0\alpha=0)50.5570.55
ProGRPO52.7072.20

ARM improves average Pass@1 and Pass@32 by 2.15 and 1.65 percentage points, respectively. Although gains vary across benchmarks, this matched control supports ARM’s incremental benefit beyond Clip-Higher at 32B.


We hope these clarifications and additional results address your concerns. Thank you again for your time and feedback.

Thanks for the authors' response, my concern has been resolved, so I raised my score.

Thank you very much for your constructive feedback! We are delighted that our response resolved your issue, and we would like to express our sincere gratitude for taking the time to write such a detailed review.

Reviewer_dDiROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProGRPO uses generation confidence to adjust GRPO advantages. Its Advantage Re-weighting Mechanism (ARM) compares prompt and response confidence, estimated from low-probability tokens. Less confident correct responses receive greater reinforcement, while uniform-reward groups remain unchanged. The method combines ARM with asymmetric clipping. Experiments on mathematical reasoning and code generation show improvements in accuracy and multi-sample coverage. Matched ablations also support ARM’s contribution beyond clipping alone.

Critical strengths
  1. The method is simple and practical. It uses existing token probabilities to differentiate equally rewarded responses, without an additional model.
  2. The experiments cover several model scales and tasks. The ablations separate ARM’s contribution from clipping and loss aggregation, making the reported improvements more convincing. Critical Weaknesses
Critical weaknesses
  1. The novelty appears incremental. ARM follows the existing idea of using uncertainty to shape advantages. Its main distinction is the prompt–response confidence difference, whose benefit over closely related formulations is not yet fully established.
  2. The confidence formulation is mainly motivated empirically. Prompt confidence can depend on wording, numbers, and notation. The ablations show that the proposed difference helps, but do not fully explain why it should transfer across tasks.
  3. Longer responses may contribute to the gains. ProGRPO generates longer responses later in training. A shared maximum length does not ensure equal token consumption, leaving the contribution of additional reasoning compute unclear.
Questions for authors

Could you clarify what the prompt-confidence term captures? For example, would changing a problem’s wording without changing its meaning substantially affect the correction?

4 comments

We sincerely thank the reviewer for the feedback. Below, we shall address these three weaknesses and provide further clarification on this question.


W1.

We acknowledge prior work on uncertainty-based advantage shaping. ARM specifically combines a prompt-dependent reference across groups with response-dependent weighting within groups, yielding the sequence-level correction cq−cic_q-c_i computed from old-policy probabilities.

Comparison with related formulations. Appendix B.3 (p. 18, Table 10) reports unweighted averages across six mathematical reasoning benchmarks, where cˉ\bar{c} denotes the mean response confidence within a group.

Correction formulationAverage Pass@1 (%)Average Pass@32 (%)
Constant shift40.761.1
Uniform negative shift40.963.6
Group-centered: cˉ−ci\bar{c}-c_i41.763.1
Response-only: 1−ci1-c_i38.560.6
Alternative: 1−cq−ci1-c_q-c_i44.367.1
ARM: cq−cic_q-c_i43.368.5

ARM outperforms the first four controls on both metrics. The alternative achieves higher Pass@1, while ARM achieves higher Pass@32 and produces shorter responses during late training (p. 8, Figure 3). Our default reflects a trade-off among accuracy, multi-sample coverage, and response length.

EA comparison. Under otherwise matched training and evaluation settings, EA at steps 350 and 400 achieves average Pass@1/Pass@32 of 40.05/56.24 and 39.49/57.94, respectively, both below ProGRPO at step 360 (43.30/68.50; Appendix B.4, p. 19, Table 12).


W2 & Q1.

We hypothesize that cqc_q reflects familiarity with the current input formulation and provides an input-dependent shared offset, while response confidence determines relative within-group weighting. We do not assume that cqc_q is strictly invariant to meaning-preserving reformulations.

Reformulation sensitivity. We added evaluations under wording, variable, and notation changes, together with a fixed-response paraphrasing diagnostic (Appendix B.5, pp. 19–21, Tables 13–15). Across 11 mixed-reward groups, the token-mean and sequence-mean checkpoints exhibit normalized correction changes of 3.15% and 1.33%, mean within-group Spearman correlations of 0.970 and 0.994, and no correction-sign flips. These results support the stability of the combined correction under the tested reformulations.

Cross-task performance. The signal uses policy probabilities directly and requires no additional task-specific confidence model. Mathematical reasoning, code generation, and general-domain experiments provide empirical support. Complete ProGRPO improves over GRPO by 2.2 and 3.4 percentage points on MMLU-Pro and GPQA, respectively (p. 7, Table 4). These results concern the complete method and do not separately establish the cross-task benefit of the prompt-confidence component.


W3.

We agree that the same maximum response length does not ensure equal token consumption. We therefore added budget-constrained comparisons (Appendix B.6, pp. 22–23, Tables 16–17 and Figures 5–6).

Inference budget. Under identical cumulative output-token caps, ProGRPO achieves higher conditional response accuracy across all tested budget settings on AIME 2024 and AIME 2025. This metric measures accuracy among completed responses, excluding truncated responses. In majority-voting evaluation, ProGRPO wins 11 of 12 paired comparisons, with one tie. These results support its advantage under the tested budgets, while the contribution of longer individual reasoning trajectories remains unresolved.

Training budget. For training seed 42, the recorded time includes response generation and ARM confidence scoring. At the end of the shared 135.4-hour training window, ProGRPO achieves a smoothed Pass@1 of 11.88% on AIME 2025, compared with 4.15% for GRPO + Clip-Higher (p. 23, Figures 5–6).


We hope these clarifications and additional results address your concerns. Thank you again for your time and feedback.

Thank you for the clarifications and additional experiments. The reformulation and budget-constrained results alleviate several of my concerns. However, the novelty remains incremental, and the prompt-confidence component’s independent benefit is still not fully established. I remain supportive of acceptance and will maintain my rating of 3.

Thank you for your careful reading, thoughtful feedback, and continued support for acceptance. Your feedback has helped us improve the clarity and empirical support of the paper.

Thank you very much for your patience and careful reading of our paper and responses. We would like to highlight the small-scale component ablation in Table 11 (p. 19), conducted on DAPO-17K under a limited training budget. As shown below, adding prompt confidence to the response-only formulation improves average Pass@1 and Pass@4 by 1.22 and 1.34 percentage points, respectively, across six benchmarks. Although limited in scale, this ablation provides empirical support for prompt confidence’s incremental benefit in this setting.

Confidence componentsAverage Pass@1 (%)Average Pass@4 (%)
Prompt only37.8648.73
Response only37.7649.93
Both38.9851.27
Reviewer_K5mqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adds a confidence-based correction to GRPO via an Advantage Re-weighting Mechanism: within a mixed-reward group, each response’s advantage is shifted by the gap between prompt confidence and response confidence, so that less confident correct responses get larger advantages.

Critical strengths

The proposed method needs only old-policy log-probs, which are already computed, meaning cost-effective. The experiments include a properly matched ablation at 7B and show the same direction of gains on 32B. The paper also reports evaluation-seed variance and sign-flip diagnostics.

Critical weaknesses
  1. The ablations do not isolate the claimed mechanism from the shared offset that the correction also introduces.
  2. No training-run repeats are reported, and ARM is isolated from Clip-Higher only at 7B.
Questions for authors
  1. W1: The three formulations in Table 10 share the same within-group ordering and differ only in the offset, however the results differ much, so the offset matters but the effect of ordering is untested. Please run a constant offset matched to the observed mean correction, the group-centered correction alone, and full ARM, and report the between-/within-group decomposition of the correction.
  2. W1: It seems that a uniform negative shift would make ARM a form of negative-sample re-weighting; please compare against it directly.
  3. W2: Please report mean ± std over ≥ 3 training runs for GRPO+Clip-Higher vs ProGRPO on Qwen2.5-7B, and evaluate the baselines on the same evaluation seeds.
  4. W2: Please add the α = 0 Clip-Higher control on Qwen2.5-32B so that ARM's contribution is isolated beyond 7B.
2 comments

We sincerely thank the reviewer for the valuable feedback. Our detailed responses are provided below.


Q1, Q2 & W1

We added three controls: a mean-matched constant shift, a uniform negative shift, and a group-centered correction. All corrections apply only to mixed-reward groups.

Let cˉq=1G∑i=1Gci\bar{c}_q = \frac{1}{G}\sum_{i=1}^{G}c_i. Full ARM decomposes as

Δi=α(cq−cˉq)⏟μq+α(cˉq−ci)⏟εi.\Delta_i = \underbrace{\alpha(c_q-\bar{c}_q)}_{\mu_q} + \underbrace{\alpha(\bar{c}_q-c_i)}_{\varepsilon_i}.

Here, μq\mu_q is the group-shared shift, while εi\varepsilon_i is the within-group correction. The first term is shared within each group; the second has zero within-group mean and differentiates responses by confidence. Zero mean does not imply zero gradient contribution.

Variance decomposition.

Full ARM logs: seeds 42/43/44, steps 1–300 (α=0.3\alpha=0.3, G=8G=8).

SeedGroup-sharedWithin-group
4246.34%53.66%
4345.24%54.76%
4443.74%56.26%

Shares pool within-step variances with equal mixed-group weights, excluding between-step mean drift. They describe correction variability, not performance gains.

For the fixed-shift calibration, we use a pre-existing Full ARM run with seed 42 and α=0.3\alpha=0.3. We aggregate all mixed-group responses from its 300 logged training steps:

Training stepsMixed-group responsesSummed correctionMean correction
1–100200,224−12,639.525602−0.06312693
101–200179,992+6,353.708626+0.03529995
201–300156,544+13,998.731803+0.08942362
Pooled: 1–300536,760+7,712.914826+0.01436939

Totals use unrounded log values. The response-count-weighted mean gives

b+=∑tntmixedΔ‾t∑tntmixed=7712.914826536760=0.01436939.b_{+} = \frac{\sum_t n_t^{\mathrm{mixed}}\overline{\Delta}_t}{\sum_t n_t^{\mathrm{mixed}}} = \frac{7712.914826}{536760} = 0.01436939.

We additionally set b−=−b+b_{-}=-b_{+} as an equal-magnitude negative-shift control. Both constants already include α\alpha and are frozen before training.

The resulting performance comparison is:

SettingCorrectionAverage Pass@1 (%)Average Pass@32 (%)
GRPO + Clip-Higher0040.060.2
Constant shiftb+b_{+}40.761.1
Uniform negative shiftb−b_{-}40.963.6
Group-centered correctionα(cˉq−ci)\alpha(\bar{c}_q-c_i)41.763.1
Full ARMα(cq−ci)\alpha(c_q-c_i)43.368.5

The baseline and Full ARM scores in this table are the main-paper results. Group-centered correction improves Pass@1 and Pass@32 over GRPO + Clip-Higher by 1.7 and 2.9 percentage points, respectively, supporting the usefulness of within-group confidence differentiation without the shared offset. Full ARM exceeds the tested uniform negative shift by 2.4 and 4.9 percentage points, respectively. These results indicate that the tested fixed negative shift alone cannot reproduce Full ARM’s performance (Appendix B.3, p. 18).


Q3 & W2

We conducted three independent ProGRPO training runs on Qwen2.5-7B using seeds 42, 43, and 44. The six-benchmark macro-averages are:

BenchmarkPass@1 Seed 42Pass@1 Seed 43Pass@1 Seed 44Pass@1 Mean ± Std.Pass@32 Seed 42Pass@32 Seed 43Pass@32 Seed 44Pass@32 Mean ± Std.
AIME 202421.3019.7920.0020.36 ± 0.8253.3043.3346.6647.76 ± 5.08
AIME 202515.9015.6212.0814.53 ± 2.1350.0043.3336.6743.33 ± 6.67
AMC 202367.2067.7366.3367.09 ± 0.7192.5092.5087.5090.83 ± 2.89
MATH-50080.5080.7678.7880.01 ± 1.0894.2094.8093.4094.13 ± 0.70
Minerva Math32.0035.1032.2633.12 ± 1.7253.3059.1955.8856.12 ± 2.95
OlympiadBench42.7043.9941.9442.88 ± 1.0467.5071.2268.4069.04 ± 1.94
Average43.2743.8341.9043.00 ± 0.9968.4767.4064.7566.87 ± 1.91

ProGRPO achieves higher macro-average Pass@1 and Pass@32 than GRPO + Clip-Higher in each of the three matched training-seed comparisons. Per-benchmark results appear in Appendix B.8, p. 24, Table 19.


Q4

We added a matched Qwen2.5-32B Clip-Higher control with α=0\alpha=0, retaining the same training settings while disabling ARM (p. 6, Table 2).

MethodAverage Pass@1 (%)Average Pass@32 (%)
GRPO + Clip-Higher (α=0\alpha=0)50.5570.55
ProGRPO52.7072.20

ARM improves the two averages by 2.15 and 1.65 percentage points, respectively, supporting its incremental benefit at 32B.


We hope these clarifications and additional results address your concerns. Thank you again for your time and feedback.

AuthorsOpenReview ↗

Additional Results for Q3 & W2: Comparison across Three Training Seeds (GRPO + Clip Higher)

Q3 & W2

Training-Seed Variability of GRPO + Clip-Higher on Qwen2.5-7B (%)

BenchmarkPass@1 Seed 42Pass@1 Seed 43Pass@1 Seed 44Pass@1 Mean ± Std.Pass@32 Seed 42Pass@32 Seed 43Pass@32 Seed 44Pass@32 Mean ± Std.
AIME2419.4012.9215.8316.05 ± 3.2536.7036.6733.3335.57 ± 1.94
AIME259.104.276.776.71 ± 2.4233.3033.3330.0032.21 ± 1.91
AMC2367.3065.5066.7066.50 ± 0.9287.5095.0095.0092.50 ± 4.33
MATH50073.9076.3977.2375.84 ± 1.7389.4093.8094.0092.40 ± 2.60
Minerva32.9035.9735.5634.81 ± 1.6753.3056.9956.6255.64 ± 2.03
OlympiadBench37.5040.3741.5839.82 ± 2.1061.0061.6063.2061.93 ± 1.14
Average40.0239.2440.6139.96 ± 0.6960.2062.9062.0361.71 ± 1.38

Both methods were trained with seeds 42, 43, and 44. For each training seed, ProGRPO and GRPO + Clip-Higher were evaluated using identical evaluation seed settings. Across the three independent training runs, ProGRPO achieves higher average Pass@1 and Pass@32: 43.00 ± 0.99 / 66.87 ± 1.91, compared with 39.96 ± 0.69 / 61.71 ± 1.38 for GRPO + Clip-Higher. Values are reported as mean ± sample standard deviation across training runs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Results for the Area Chair

Dear Area Chair,

We would like to highlight four results from our rebuttal.

  • Three-seed comparison (p. 24, Table 19). As requested by Reviewer K5mq, we compared ProGRPO with GRPO + Clip-Higher on Qwen2.5-7B using the same three training seeds. ProGRPO achieved higher six-benchmark macro-average Pass@1 and Pass@32 in each of the three matched training-seed comparisons. Across six math benchmarks and three seeds, the mean gains were 3.04 percentage points in Pass@1 and 5.16 in Pass@32.
  • Effect of the shared offset (p. 18, Table 10). In the 7B ablation, the group-centered correction improved both metrics over GRPO + Clip-Higher even after we removed the shared offset. Full ARM also outperformed both fixed-shift controls.
  • ARM at 32B (p. 6, Table 2). On Qwen2.5-32B, adding ARM to the matched GRPO + Clip-Higher baseline improved Pass@1 by 2.15 percentage points and Pass@32 by 1.65 points.
  • Contribution of prompt confidence (p. 19, Table 11). To address Reviewer dDiR’s question, we compared prompt and response confidence together with response confidence alone. In a smaller DAPO-17K experiment, adding prompt confidence improved Pass@1 by 1.22 percentage points and Pass@4 by 1.34 points.

Thank you for considering these additional results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.