acceptodds
Withdrawn from ICLR 2027

PC-GRPO: Pairwise Confidence Group Relative Policy Optimization for Post-Training Reasoning Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Reinforcement learning with verifiable rewards (RLVR) is an effective paradigm for post-training reasoning large language models (LLMs), and Group Relative Policy Optimization (GRPO) is a widely used critic-free optimizer in this setting. However, standard GRPO relies on group-wise reward normalization for advantage estimation, which provides coarse credit assignment and is particularly limited under sparse binary rewards. As a result, trajectories with different degrees of partial correctness often receive indistinguishable supervision. We propose Pairwise Confidence Group Relative Policy Optimization (PC-GRPO), which improves GRPO with two components: a pairwise confidence-based advantage estimator and a fine-grained verifiable reward. The first replaces coarse group-normalized advantages with pairwise comparisons that capture relative reward differences and policy confidence. The second provides denser supervision for incorrect trajectories using step-level overlap with reference reasoning traces, without introducing learned reward models. PC-GRPO remains critic-free and preserves the verifiable nature of RLVR. Across mathematical, scientific, knowledge-intensive, and coding benchmarks, PC-GRPO consistently outperforms standard GRPO and other strong baselines across multiple model scales.

Primary area:
reinforcement learning
Keywords:
Large Language Model Reasoning, Reinforcement Learning with Verifiable Rewards, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Dz7NOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PC-GRPO for reasoning model post-training. It combines confidence-weighted pairwise advantage estimation with reference-based partial rewards for incorrect responses. The reward measures semantic overlap between generated and reference reasoning segments. Experiments report improvements across seven benchmarks and several model sizes. Ablations examine both components.

Critical strengths
  1. The experiments cover several reasoning domains and model sizes. The reported improvements over GRPO are consistent.

  2. The component ablations help separate the effects of the advantage estimator and the reference-based reward.

Critical weaknesses
  1. The motivation does not sufficiently establish the need for the proposed method. ICPO [1] already uses intra-group confidence comparisons. CAPO [2] constructs confidence-aware pairwise advantages. Shaping Explanations [3] uses reference-based semantic rewards within GRPO. These closely related methods are neither discussed nor directly compared. The paper does not identify which limitations of these approaches its specific design addresses. This makes the novelty and significance difficult to assess.

  2. The interpretation of the auxiliary reward as partial correctness lacks validation. Semantic similarity to a reference does not establish that a reasoning step is logically correct. The duplicate-suppression property addresses repetition, but not this underlying issue. The reported accuracy gains do not show whether the reward captures valid intermediate progress or mainly encourages reference similarity. This weakens the claimed improvement in credit assignment.

  3. The AIME baseline results of Qwen2.5-instruct need further clarification. The table 18 in Qwen3 technical report [4] gives substantially lower scores for the same original checkpoints (Qwen2.5-instruct). Table 3 omits AIME variability despite the ten evaluation runs stated in Section 5.1.

[1] ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learning

[2] Calibration-Aware Policy Optimization for Reasoning LLMs

[3] Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPO

[4] Qwen3 Technical Report

Questions for authors
  1. What specific limitation of ICPO and CAPO is addressed by the GELU transformation and inverse-sigmoid weighting? Can you provide controlled comparisons with matched rollout budgets and reference supervision?

  2. What evidence supports treating semantic overlap as partial correctness? Please also specify the embedding model and matching threshold used in the experiments.

  3. Can you conduct experiments using more frontier LLMs, such as Qwen3.5, MiniCPM5?

Reviewer_HtHiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PC-GRPO, an extension of GRPO for RLVR-based post-training of reasoning language models. The method consists of two main components. First, it replaces standard group-normalized advantages with a pairwise estimator that combines reward differences with a confidence signal derived from length-normalized behavior-policy log probabilities. Second, it introduces a reference-guided fine-grained reward that assigns partial credit to incorrect trajectories based on their step-level similarity to a reference reasoning trace, while preserving reward 1 for verifier-accepted responses.

Critical strengths

The paper is clearly written and the proposed method is relatively easy to implement on top of an existing GRPO/RLVR pipeline. The empirical evaluation is broad: the authors consider several reasoning domains, multiple model scales, additional model families, and a reasonably strong collection of GRPO-family baselines. The results are consistently positive, and the ablations indicate that both the modified advantage estimator and the fine-grained reward contribute to performance.

Critical weaknesses

The paper addresses an important and well-recognized limitation of GRPO under sparse binary rewards: unsuccessful trajectories with different intermediate reasoning quality may receive indistinguishable supervision. Given that several recent methods already aim to improve credit assignment in this setting, I believe the main contribution should be supported by a clearer motivation for the specific design of PC-GRPO. In particular, it is not sufficiently explained why confidence-weighted pairwise advantage estimation and reference-guided partial rewards are especially appropriate solutions to this problem. The empirical results are consistently positive, but the current formulation appears more as an effective design choice than as a method derived from a clearly articulated underlying principle. More detailed questions are provided below.

Questions for authors
  1. In Eq. (5), the length-normalized log-probability $\ell_i$ under the behavior policy is used as the confidence of response oio_i. What motivates treating $\ell_i$ as a proxy for reasoning confidence or reasoning quality? In particular, a highly probable trajectory may still be confidently wrong, while a correct but less typical reasoning path may have lower likelihood. It would be helpful to report whether $\ell_i$ is empirically correlated with final-answer correctness or partial reasoning quality, and how often high-confidence incorrect trajectories occur.
  2. What is the motivation for introducing the nonlinear reward-gap transformation f(R~i−R~j)f(\tilde R_i-\tilde R_j) in Eq. (6), and in particular for choosing GELU? Proposition 4.4.1 shows that a linear pairwise formulation is closely connected to standard GRPO normalization, so it would be useful to clarify what limitation of the linear reward gap the nonlinear transformation is intended to address.
  3. Since Rfg(q,oi)R_{\mathrm{fg}}(q,o_i) is already bounded in [0,1][0,1], what is the motivation for further scaling it by λfg\lambda_{\mathrm{fg}} in Eq. (9)? This scaling changes both the relative differences among incorrect trajectories and the margin between correct and incorrect responses, which are subsequently transformed by f(R~i−R~j)f(\tilde R_i-\tilde R_j) in Eq. (6). Is there a principled interpretation of λfg\lambda_{\mathrm{fg}} in this reward geometry? In particular, can the authors provide any theoretical justification for choosing λfg=0.5\lambda_{\mathrm{fg}}=0.5, rather than treating it purely as an empirically tuned hyperparameter?
  4. In Eqs. (9)–(10), RfgR_{\mathrm{fg}} is used as partial credit for incorrect trajectories and is computed from similarity to a reference reasoning trace. What evidence supports the assumption that this similarity is a reliable proxy for partial correctness? For example, an incorrect trajectory may have high semantic overlap with the reference while containing a crucial logical error. Could the authors report the relationship between RfgR_{\mathrm{fg}} and actual intermediate reasoning quality, or provide representative high-RfgR_{\mathrm{fg}} and low-RfgR_{\mathrm{fg}} incorrect examples?
Reviewer_8SAwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed PC-GRPO, which includes a "pairwise confidence-based advantage estimator" and a modified verifiable reward. The paper claimed a performance improvement compared to GRPO baselines.

Critical strengths
  1. The paper is well-organised and easy to read. The technology developed is straightforward.
  2. The limitation of GRPO is well stated and, from my point of view, a valid argument.
Critical weaknesses
  1. I am confused by the motivation part of the narrative in the introduction from Line 50 to Line 80. If I understand correctly, the limitations stated by the authors are mainly: (1) the outcome-only reward is sparse; (2) the PRMs can cause various additional problems. However, it's largely unclear why a "reference-guided partial-credit" reward is the solution to these limitations. The explanation from the authors ("distinguish incorrect responses without an additional learned reward model") has not added further information to clarify my confusion.

  2. Secondly, I am not completely sure intuitively why partial rewards derived by distinguishing incorrect responses based on a reference can be an effective way to improve GRPO. To dive into this argument, if the reference is an assumed good ground truth, why even consider GRPO as a learning solution in the first place? I reckon such a learning framework needs proper assumptions to support it, as well as direct theoretical proof instead of simply empirical results.

  3. Regarding the reference cosine similarity, since the purpose is to detect the correct/wrong signals, cosine similarity of semantics does not give any guarantees or direct correlations to the correct/wrong signals. The signals extracted can be largely misclassified by cosine similarity heuristics.

  4. Regarding Proposition 4.4.1. and Proposition 4.4.2, the connection between the significance of these two properties and the potential significance of the proposed method is very vague. It's useful to have these two properties, but more important properties (e.g., better estimation error bound compared to GRPO) are missing.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.