acceptodds
Withdrawn from ICLR 2027

Denoising-GRPO: Critic-Free Confidence-Gated Consensus Reward Correction for Noisy Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Group Relative Policy Optimization (GRPO) is a practical critic-free method for reasoning reinforcement learning (RL), yet it struggles with noisy supervision, such as inconsistencies between prompts and expected labels. Such noise compromises group-relative advantage estimation, slowing convergence and, in the worst case, steering the model toward degenerate solutions. To address this bottleneck, we propose Denoising-GRPO, a highly practical and scalable drop-in reward-side extension of GRPO. Denoising-GRPO converts within-group answer agreement into a structured correction target. To prevent the policy from being overridden by unstable or incorrect majorities during early training, it uses the model's internal generation confidence to apply this correction only when the group is deemed reliable. Because it operates entirely on the reward construction side, our method avoids heavier auxiliary estimators or external reward models and leaves the efficient critic-free GRPO outer loop unchanged. To thoroughly benchmark this capability, we construct noisy training variants of textual MATH and multimodal Geo3K, simulating realistic corruption scenarios during supervision. All reported test and benchmark results are evaluated on clean held-out data with correct ground-truth targets. We evaluate Denoising-GRPO after both low-noise and high-noise training, and test transfer on GSM8K, held-out same-corpus MATH500, Minerva-Math, MathVista, MathVerse, and MMK12. On clean in-domain tests after high-noise training, Denoising-GRPO improves over GRPO by up to +10.5 on MATH and +3.0 on Geo3K, and also improves clean benchmark performance (e.g., +10.1 on GSM8K). Mechanism diagnostics further show that the confidence gate reduces harmful consensus overrides while recovering most of the gap to clean-supervision GRPO. These results indicate that reasoning models can self-denoise corrupted training supervision and still generalize to clean evaluation data whenever within-group agreement is sufficiently reliable.

Primary area:
reinforcement learning
Keywords:
Self-Denoising RL, Noisy Supervision, Confidence-Aware Correction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KFizOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses GRPO training under noisy supervision. It derives a consensus answer from each rollout group and uses agreement with it as a pseudo-reward, applied only when the group's average generation confidence exceeds a threshold. The corrected reward is interpolated with the verifier reward via a warmup schedule; GRPO is otherwise unchanged. On MATH and Geo3K, it improves over vanilla GRPO and several robustness baselines.

Critical strengths

The method is lightweight and addresses a practical problem without a critic or external reward model. Gains over vanilla GRPO are substantial under high textual noise.

Critical weaknesses
  1. The methodological contribution is not clearly established, and the decisive comparisons are missing.

    Consensus rewards are central to TTRL, and SCOPE already combines confidence with consensus pseudo-labeling. The noisy-supervision setting and verifier interpolation are real differences, but the paper does not show that this combination is a substantial advance. Controlled comparisons with these reward-construction mechanisms in the same setting are needed.

    More importantly, Table 27 shows harmful-override rates of 4%, 19%, and 51% for unanimous, strong-majority, and bare-majority groups, which directly suggests a simple vote-margin gate. Without comparing against such a gate at matched correction coverage and rollout budget, the added value of confidence beyond consensus strength is not shown.

  2. The mechanistic analysis has an unresolved normalization issue, and the theory assumes the key property.

    The Mean, Bayes, and ConfBayes baselines (Appendix A.3) all take the form r~g,i=λtrg,i+bg\widetilde r_{g,i} = \lambda_t r_{g,i} + b_g with a group-shared offset bgb_g, which cancels under group normalization:

    z~g,i=λt(rg,i−μg)λtσg+ϵnorm.\widetilde z_{g,i} = \frac{\lambda_t (r_{g,i} - \mu_g)}{\lambda_t \sigma_g + \epsilon_{\mathrm{norm}}}. So for fixed rollouts and the same λt\lambda_t, they yield identical advantages regardless of the posterior. Yet Table 3 reports different outcomes and interprets them as evidence about denoising strategies. Proposition 3 assumes gated consensus is already more accurate than the original reward and then gives a convex-interpolation bound. It does not show when the confidence statistic guarantees this, or whether lower reward error yields a better normalized update signal.

Questions for authors
  1. Does confidence gating beat a vote-margin gate at matched correction coverage, rollout budget, and tuning budget?
  2. How do Mean, Bayes, and ConfBayes differ after group normalization? Please reconcile Eqs. (20), (21), and (23) with Table 3, including any differences in schedules, seeds, or implementation.
  3. What evidence shows that confidence predicts harmful overrides beyond vote margin?
Reviewer_bEGoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Denoising-GRPO modifies GRPO by combining noisy rewards with answer-consensus rewards when generation confidence passes a threshold. The paper evaluates this approach on textual and multimodal reasoning tasks with artificial training noise.

Critical strengths
  1. The method changes reward construction without adding a critic or external reward model, making it easy to integrate into GRPO.

  2. The controlled noise experiments, component ablations, and gate diagnostics help examine when consensus correction improves or harms the training signal.

Critical weaknesses
  1. My main concern is the method’s value under typical training conditions. For the Full variant, high-noise text gains are 10.5 points on Qwen3-0.6B and 3.8 points on Qwen2.5-Math-7B, while the latter loses 0.3 points under low noise. These backbones differ, so this is not a controlled comparison of model size. The main paper should show the stronger model's low-noise results with multi-seed uncertainty and more clearly explain when the method is useful.

  2. A basic control is missing: Denoising-GRPO trained without artificial corruption. A wrong model answer does not imply noisy supervision, and a confident wrong majority could suppress correct minority answers. Table 22 already reports clean-supervision GRPO. The authors should add Denoising-GRPO under the same conditions, with multiple seeds and measurements of how often consensus correction overrides a correct reward. A systematic experiment with natural annotation errors or verifier mistakes is also needed to support practical applicability.

Questions for authors

Please refer to the weaknesses above.

Reviewer_iW1KOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Denoising-GRPO, which corrects the GRPO reward under noisy supervision through answer consensus within the group, confidence-based gating, and progressive reward interpolation, all without adding extra critics or teacher models.

Critical strengths
  1. The paper focuses on RLVR under noise supervision, which is a practically meaningful problem.
  2. The method is simple and practical, only changing the reward construction without altering the main optimization process of GRPO, with an extra training cost of about 3.1%.
Critical weaknesses
  1. The baseline comparison is seriously incomplete. OLR [1] studies the same noisy-RLVR setting and similarly replaces incorrect labels with policy-generated majority answers, using confidence growth and historical consistency for reliability. The paper neither discusses OLR nor compares with representative unsupervised RLVR methods such as TTRL [2], Co-Reward [3], or Self-Certainty [4], making it unclear whether the gains come from the proposed confidence gate or generic self-consistency learning.
  2. The noise evaluation is insufficiently quantified. Low and high noise describe perturbation severity rather than actual noise ratios. The paper should report the proportion of incorrect reward targets and evaluate controlled noise ratios, while separately analyzing answer corruption, prompt corruption, and image mismatch.

[1] Yang, S., Zhu, G., Song, B., Li, S., Wang, H., Zheng, X., Ma, Y., Chen, Z., Wang, W., & Chen, G. (2026). Can LLMs Learn to Reason Robustly under Noisy Supervision? ArXiv, abs/2604.03993. [2] Zuo, Y., Zhang, K., Qu, S., Sheng, L., Zhu, X., Zhang, Y., Qi, B., Sun, Y., Cui, G., Ding, N., & Zhou, B. (2025). TTRL: Test-Time Reinforcement Learning. ArXiv, abs/2504.16084. [3] Zhang, Z., Zhu, J., Ge, X., Zhao, Z., Zhou, Z., Li, X., Feng, X., Yao, J., & Han, B. (2025). Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models. [4] Zhao, X., Kang, Z., Feng, A., Levine, S., & Song, D.X. (2025). Learning to Reason without External Rewards. ArXiv, abs/2505.19590.

Questions for authors

I would appreciate clarification on the following points:

  1. Could the authors compare against OLR and representative unsupervised RLVR methods such as TTRL, Co-Reward, and Self-Certainty?
  2. Given the 51% harmful-override rate for 5/8 majorities, why does the algorithm accept even weaker pluralities while the theory only analyzes strict majorities?
  3. What happens as the noise ratio increases, and do the conclusions hold separately for answer corruption, prompt corruption, and image mismatch?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.