- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Geometry-Aware Calibrated Policy Optimization (GCPO), a modification to GRPO that scales each rollout group’s advantages using a combination of semantic disagreement and reward dispersion. Semantic disagreement is measured through either pairwise cosine dispersion or a clustering-based barycentric score, while reward dispersion increases the weight of groups with stronger reward contrast. The paper motivates these components through an analysis of gradient variability and evaluates them on question-answering and mathematical-reasoning benchmarks, with additional statistical analyses, ablations, and experiments across several model backbones.
Critical strengths
The paper addresses a relevant question for group-based policy optimization: rollout groups with different response structures and reward contrasts need not contribute equally useful updates. The proposed modification is straightforward to incorporate into GRPO, and the main QA results show promising improvements over the unmodified method and several uncertainty-based baselines.
The authors investigate the motivation beyond reporting benchmark scores. The statistical analyses distinguish score-gradient variability from reward-weighted update variability, and the paper acknowledges that the relationship with semantic geometry is task dependent. Reporting multiple seeds and ablations of the modulation strength and reward-dispersion component also helps assess the method.
Critical weaknesses
The current experiments do not sufficiently establish that semantic geometry is responsible for the reported gains. The paper removes RD in an ablation, but does not provide an RD-only variant using exactly the same weighting function and normalization. Although the Q-Hawkeye adaptation uses reward variance, it does not replace this controlled comparison. For binary rewards and an even group size, normalized RD reduces to , where is the fraction of correct responses. It therefore emphasizes groups containing both correct and incorrect answers, which also admits a difficulty-based weighting interpretation. It remains unclear how much semantic geometry adds beyond this effect. Relatedly, the theoretical and statistical analyses concern gradient variability, but greater variability does not by itself establish that an update is noisy or uninformative. The semantic–gradient alignment assumption and the weaker evidence on Qasper further limit the interpretation. I would like a clearer separation between what the analysis establishes and the hypothesis that the proposed weighting selectively suppresses harmful variation.
The broader experimental claims need stronger support. The recent baselines are adaptations of their central ideas to a shared GRPO framework, rather than complete reproductions, which should be explicit in the main table and conclusions. Figure 6 reports GCPO results across different backbones without corresponding baseline results for each backbone; this demonstrates applicability but does not establish consistent improvement across models. The practical justification for BoT is also incomplete. It adds NLI-based clustering, yet CD performs better on the reported mathematical-reasoning benchmarks. Without measured training overhead and sensitivity to clustering choices, it is difficult to assess when this additional complexity is worthwhile. These issues leave the scope and practical value of the contribution less clear than the main QA results suggest.
Questions for authors
(1) How does GCPO compare with an RD-only variant using the same reward-dispersion weighting and normalization? Can the authors also control for the average update magnitude to distinguish selective group reweighting from a change in effective optimization strength?
(2) What evidence shows that the groups attenuated by semantic disagreement provide less useful updates, rather than merely more variable ones? Can the authors connect the proposed scores to learning progress or downstream improvement more directly?
(3) Do the gains persist across the backbones in Figure 6 when compared with matched GRPO and uncertainty-based baselines? Please also clarify in the main results which prior methods are evaluated as controlled adaptations.
(4) What is the measured computational overhead of CD and BoT, and when does BoT provide enough additional benefit to justify NLI clustering? How sensitive are its results to the clustering threshold and response order?