- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ProGRPO, which estimates prompt and response confidence from the geometric mean probability of selected low-probability tokens. It adjusts GRPO advantages using the difference between these confidence scores. Among responses with the same reward, lower-confidence responses receive larger adjusted advantages; groups with all correct or all incorrect answers are left unchanged. Combined with Clip-Higher, the method is evaluated on math and code tasks, including a Qwen2.5-32B model, with additional transfer results on question answering.
Critical strengths
-
The adjustment is simple. The comparison with GRPO using Clip-Higher under matched training settings on Qwen2.5-7B helps separate the effect of the confidence adjustment from that of clipping.
-
The experiments cover math, code, and several model sizes, including 32B. They report both pass@1 and pass@k. Analyses of the confidence measure, token selection, and extension to PPO also provide useful information.
Critical weaknesses
-
The motivation needs stronger support. Equal advantages do not imply equal parameter updates, since the policy gradient already depends on token probabilities. The analysis does not sufficiently explain why the proposed confidence adjustment should encourage useful exploration.
-
The paper lacks a controlled comparison with Entropy Advantage, a closely related method. Quoting results from different settings does not establish an improvement over it. The 32B experiments also compare only with GRPO, which limits the evidence at larger scales.
Questions for authors
See weakness.
3 comments
Thanks for the authors' response, my concern has been resolved, so I raised my score.
Thank you very much for your constructive feedback! We are delighted that our response resolved your issue, and we would like to express our sincere gratitude for taking the time to write such a detailed review.
We sincerely thank the reviewer for the valuable feedback. Our detailed responses are provided below.
W1.
We agree that equal advantages do not imply identical parameter updates. Motivated by the entropy decline and output concentration observed during GRPO training (p. 8, Figure 3), ARM adjusts the relative learning weights of equally rewarded responses through confidence-dependent advantage corrections. Appendix A.4 (p. 15, Eqs. (35)–(36)) characterizes the additional surrogate-gradient term induced by this reweighting at fixed policy parameters. This analysis concerns coefficient reweighting and does not guarantee larger parameter updates or monotonic entropy growth.
Given prompt confidence cq and response confidence ci, ARM uses:
Δi=α(cq−ci),α>0.
Thus, lower-confidence correct responses within the same mixed-reward group receive larger relative advantage corrections. We hypothesize that these responses contain successful behaviors the policy does not yet reproduce reliably. Confidence is computed from the old policy, refreshed with new rollouts, and held fixed during the corresponding policy updates (p. 4, Sections 3.1–3.2).
Ablation evidence. Table 10 (p. 18) reports the following unweighted averages across six mathematical reasoning benchmarks:
The group-centered control supports within-group confidence differentiation, while Full ARM outperforms all listed controls.
Correct-response diversity. On AIME 2024, ProGRPO also produces less similar correct responses (p. 8, Table 5):
The accuracy, multi-sample coverage, and diversity results provide behavioral evidence consistent with useful exploration.
W2.
EA comparison at 7B. We implemented Entropy Advantage with Clip-Higher on Qwen2.5-7B. We evaluate ProGRPO at step 360 and EA at steps 350 and 400 under otherwise matched training and evaluation settings. Both EA checkpoints come from one training run with seed 42. Implementation details and full results appear in Appendix B.4, p. 19, Table 12.
All averages below are unweighted means across six mathematical reasoning benchmarks.
ProGRPO exceeds both evaluated EA checkpoints on both average metrics.
Matched control at 32B. We added a Qwen2.5-32B Clip-Higher control with α=0, retaining the same training settings while disabling ARM (p. 6, Table 2).
ARM improves average Pass@1 and Pass@32 by 2.15 and 1.65 percentage points, respectively. Although gains vary across benchmarks, this matched control supports ARM’s incremental benefit beyond Clip-Higher at 32B.
We hope these clarifications and additional results address your concerns. Thank you again for your time and feedback.