- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses GRPO training under noisy supervision. It derives a consensus answer from each rollout group and uses agreement with it as a pseudo-reward, applied only when the group's average generation confidence exceeds a threshold. The corrected reward is interpolated with the verifier reward via a warmup schedule; GRPO is otherwise unchanged. On MATH and Geo3K, it improves over vanilla GRPO and several robustness baselines.
Critical strengths
The method is lightweight and addresses a practical problem without a critic or external reward model. Gains over vanilla GRPO are substantial under high textual noise.
Critical weaknesses
-
The methodological contribution is not clearly established, and the decisive comparisons are missing.
Consensus rewards are central to TTRL, and SCOPE already combines confidence with consensus pseudo-labeling. The noisy-supervision setting and verifier interpolation are real differences, but the paper does not show that this combination is a substantial advance. Controlled comparisons with these reward-construction mechanisms in the same setting are needed.
More importantly, Table 27 shows harmful-override rates of 4%, 19%, and 51% for unanimous, strong-majority, and bare-majority groups, which directly suggests a simple vote-margin gate. Without comparing against such a gate at matched correction coverage and rollout budget, the added value of confidence beyond consensus strength is not shown.
-
The mechanistic analysis has an unresolved normalization issue, and the theory assumes the key property.
The Mean, Bayes, and ConfBayes baselines (Appendix A.3) all take the form with a group-shared offset , which cancels under group normalization:
So for fixed rollouts and the same , they yield identical advantages regardless of the posterior. Yet Table 3 reports different outcomes and interprets them as evidence about denoising strategies. Proposition 3 assumes gated consensus is already more accurate than the original reward and then gives a convex-interpolation bound. It does not show when the confidence statistic guarantees this, or whether lower reward error yields a better normalized update signal.
Questions for authors
- Does confidence gating beat a vote-margin gate at matched correction coverage, rollout budget, and tuning budget?
- How do Mean, Bayes, and ConfBayes differ after group normalization? Please reconcile Eqs. (20), (21), and (23) with Table 3, including any differences in schedules, seeds, or implementation.
- What evidence shows that confidence predicts harmful overrides beyond vote margin?