- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes PC-GRPO for reasoning model post-training. It combines confidence-weighted pairwise advantage estimation with reference-based partial rewards for incorrect responses. The reward measures semantic overlap between generated and reference reasoning segments. Experiments report improvements across seven benchmarks and several model sizes. Ablations examine both components.
Critical strengths
-
The experiments cover several reasoning domains and model sizes. The reported improvements over GRPO are consistent.
-
The component ablations help separate the effects of the advantage estimator and the reference-based reward.
Critical weaknesses
-
The motivation does not sufficiently establish the need for the proposed method. ICPO [1] already uses intra-group confidence comparisons. CAPO [2] constructs confidence-aware pairwise advantages. Shaping Explanations [3] uses reference-based semantic rewards within GRPO. These closely related methods are neither discussed nor directly compared. The paper does not identify which limitations of these approaches its specific design addresses. This makes the novelty and significance difficult to assess.
-
The interpretation of the auxiliary reward as partial correctness lacks validation. Semantic similarity to a reference does not establish that a reasoning step is logically correct. The duplicate-suppression property addresses repetition, but not this underlying issue. The reported accuracy gains do not show whether the reward captures valid intermediate progress or mainly encourages reference similarity. This weakens the claimed improvement in credit assignment.
-
The AIME baseline results of Qwen2.5-instruct need further clarification. The table 18 in Qwen3 technical report [4] gives substantially lower scores for the same original checkpoints (Qwen2.5-instruct). Table 3 omits AIME variability despite the ten evaluation runs stated in Section 5.1.
[1] ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learning
[2] Calibration-Aware Policy Optimization for Reasoning LLMs
[3] Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPO
[4] Qwen3 Technical Report
Questions for authors
-
What specific limitation of ICPO and CAPO is addressed by the GELU transformation and inverse-sigmoid weighting? Can you provide controlled comparisons with matched rollout budgets and reference supervision?
-
What evidence supports treating semantic overlap as partial correctness? Please also specify the embedding model and matching threshold used in the experiments.
-
Can you conduct experiments using more frontier LLMs, such as Qwen3.5, MiniCPM5?