acceptodds
Under review as a conference paper at ICLR 2027

Regulating What Policies Learn From: Geometry-aware Calibrated Policy Optimization

Abstract

Critic-free, group-based methods such as Group Relative Policy Optimization (GRPO) enable scalable LLM post-training from sampled rollouts, but typically do not account for differences in learning signal quality across rollout groups. Uncertainty provides a natural internal signal for regulating their contributions, yet existing rollout-level methods largely treat disagreement as a scalar proxy for reliability. We analyze rollout-level uncertainty from an optimization perspective and identify two key limitations through theoretical and statistical analysis: the anisotropic gap, where probability dispersion fails to capture the geometric magnitude of semantic disagreement, and the calibration gap, where uncertainty is decoupled from reward informativeness and therefore cannot distinguish noisy variation from useful reward-differentiated variation. To address these gaps, we propose Geometric-aware Calibrated Policy Optimization (GCPO), which combines geometry-aware uncertainty measures with reward-based calibration to regulate policy updates according to both disagreement structure and learning informativeness. Across QA and mathematical reasoning benchmarks, GCPO delivers stronger GRPO-style optimization than entropy-based uncertainty baselines, while statistical and ablation analyses support the underlying design assumptions. Our results suggest a broader principle for uncertainty-aware policy optimization: uncertainty should characterize not only how much rollouts disagree, but also whether that disagreement provides useful learning signals.

Primary area:
reinforcement learning
Keywords:
Rollout-Level Uncertainty, Reinforcement Learning, Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MEPbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Geometry-Aware Calibrated Policy Optimization (GCPO), a modification to GRPO that scales each rollout group’s advantages using a combination of semantic disagreement and reward dispersion. Semantic disagreement is measured through either pairwise cosine dispersion or a clustering-based barycentric score, while reward dispersion increases the weight of groups with stronger reward contrast. The paper motivates these components through an analysis of gradient variability and evaluates them on question-answering and mathematical-reasoning benchmarks, with additional statistical analyses, ablations, and experiments across several model backbones.

Critical strengths

The paper addresses a relevant question for group-based policy optimization: rollout groups with different response structures and reward contrasts need not contribute equally useful updates. The proposed modification is straightforward to incorporate into GRPO, and the main QA results show promising improvements over the unmodified method and several uncertainty-based baselines.

The authors investigate the motivation beyond reporting benchmark scores. The statistical analyses distinguish score-gradient variability from reward-weighted update variability, and the paper acknowledges that the relationship with semantic geometry is task dependent. Reporting multiple seeds and ablations of the modulation strength and reward-dispersion component also helps assess the method.

Critical weaknesses

The current experiments do not sufficiently establish that semantic geometry is responsible for the reported gains. The paper removes RD in an ablation, but does not provide an RD-only variant using exactly the same weighting function and normalization. Although the Q-Hawkeye adaptation uses reward variance, it does not replace this controlled comparison. For binary rewards and an even group size, normalized RD reduces to (4p(1−p))(4p(1-p)), where (p)(p) is the fraction of correct responses. It therefore emphasizes groups containing both correct and incorrect answers, which also admits a difficulty-based weighting interpretation. It remains unclear how much semantic geometry adds beyond this effect. Relatedly, the theoretical and statistical analyses concern gradient variability, but greater variability does not by itself establish that an update is noisy or uninformative. The semantic–gradient alignment assumption and the weaker evidence on Qasper further limit the interpretation. I would like a clearer separation between what the analysis establishes and the hypothesis that the proposed weighting selectively suppresses harmful variation.

The broader experimental claims need stronger support. The recent baselines are adaptations of their central ideas to a shared GRPO framework, rather than complete reproductions, which should be explicit in the main table and conclusions. Figure 6 reports GCPO results across different backbones without corresponding baseline results for each backbone; this demonstrates applicability but does not establish consistent improvement across models. The practical justification for BoT is also incomplete. It adds NLI-based clustering, yet CD performs better on the reported mathematical-reasoning benchmarks. Without measured training overhead and sensitivity to clustering choices, it is difficult to assess when this additional complexity is worthwhile. These issues leave the scope and practical value of the contribution less clear than the main QA results suggest.

Questions for authors

(1) How does GCPO compare with an RD-only variant using the same reward-dispersion weighting and normalization? Can the authors also control for the average update magnitude to distinguish selective group reweighting from a change in effective optimization strength?

(2) What evidence shows that the groups attenuated by semantic disagreement provide less useful updates, rather than merely more variable ones? Can the authors connect the proposed scores to learning progress or downstream improvement more directly?

(3) Do the gains persist across the backbones in Figure 6 when compared with matched GRPO and uncertainty-based baselines? Please also clarify in the main results which prior methods are evaluated as controlled adaptations.

(4) What is the measured computational overhead of CD and BoT, and when does BoT provide enough additional benefit to justify NLI clustering? How sensitive are its results to the clustering threshold and response order?

Reviewer_FeahOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Geometry-aware Calibrated Policy Optimization (GCPO), a framework for improving GRPO-style policy optimization by regulating the contributions of different rollout groups. The authors identify two limitations of existing uncertainty-based methods: the anisotropic gap, where probability-based uncertainty measures fail to capture the geometric magnitude of semantic disagreement, and the calibration gap, where uncertainty alone cannot distinguish noisy variation from informative reward differences.

To address these limitations, the paper introduces Cosine Dispersion (CD) and Barycentric Transport (BoT) to characterize semantic disagreement, together with Reward Dispersion (RD) to calibrate the resulting uncertainty signals. These quantities are combined to reweight group-level policy updates. Experiments on question answering and mathematical reasoning benchmarks demonstrate promising improvements over several GRPO-based baselines.

Critical strengths
  1. Clear presentation and well-motivated problem formulation The paper is well written and easy to follow. The motivation is clearly presented, and the proposed anisotropic and calibration gaps provide an intuitive explanation of why existing uncertainty-based policy optimization methods may be insufficient. The overall methodology is also straightforward
  2. Promising empirical performance across downstream tasks The paper reports encouraging results on multiple downstream benchmarks. In particular, Table 1 shows that GCPO achieves the best F1 performance on all three evaluated QA datasets: NarrativeQA, Qasper, and HotpotQA. The proposed variants obtain the best results on six out of eight reported metrics overall. The improvements over vanilla GRPO are substantial on several metrics, suggesting that geometry-aware uncertainty modulation combined with reward calibration can be beneficial for policy optimization.
Critical weaknesses
  1. My concern is the theoretical and empirical connection between semantic geometry and gradient geometry, which is central to the motivation of GCPO.

The paper argues that existing entropy-based uncertainty measures fail to capture optimization-relevant gradient variation because they ignore the geometric distances between different responses. However, the proposed CD and BoT metrics are computed using semantic embeddings produced by an external pretrained sentence encoder, whereas the actual policy gradients are determined by the policy model's token-level probabilities and parameterization.

These two representation spaces are fundamentally different. Semantically similar responses may involve substantially different token sequences or likelihood structures, potentially inducing very different policy gradients. Conversely, semantically distant responses do not necessarily lead to proportionally different gradient directions.

In Appendix C, the authors introduce the semantic–gradient alignment assumption that bounds the squared gradient discrepancy by a constant times the semantic distance (Eq. 25). However, this assumption is relatively strong and is not generally guaranteed for pretrained sentence embeddings.

In Figure 1, the correlations are notably dataset-dependent, with substantially weaker associations on Qasper than on NarrativeQA. This raises concerns about how reliably the proposed uncertainty measures capture gradient-relevant information across different tasks.

I would appreciate a more detailed justification of the semantic–gradient alignment assumption and additional analysis of its limitations. In particular, it would be helpful to understand when semantic embedding distances successfully reflect gradient differences, when they fail, and whether the observed alignment is sufficient to explain the downstream optimization gains.

  1. Unclear implementation and potentially unfair comparison with SEED-GRPO

I also have concerns regarding the implementation and fairness of the comparison with SEED-GRPO. In Figure 5, the submission reports a mathematical reasoning baseline labeled "Seed", with performance substantially lower than that reported in the original SEED-GRPO paper (Chen et al., 2025). For example, the original SEED-GRPO paper reports 75.4 on MATH500 using the 1.5B backbone, whereas the corresponding Seed result in Figure 5 appears to be approximately 68.5.

Given that GCPO aims to improve upon semantic-entropy-based uncertainty regulation, a clear and reproducible comparison against SEED-GRPO is important for evaluating the claimed contributions.

  1. The independent contribution of reward dispersion is not sufficiently isolated

Another concern is whether the performance gains can be clearly attributed to the proposed geometric uncertainty measures rather than the reward-based reweighting mechanism.

GCPO combines a geometry-based attenuation term with an RD-based amplification term. However, RD itself is computed from within-group reward differences and directly changes the effective magnitude of the policy update. This raises the question of whether a simpler reward-dispersion-based weighting strategy could already explain a substantial portion of the observed improvements.

Figure 4 of the current submission evaluates the effect of removing RD from CD and BoT, but does not provide a sufficiently complete comparison with RD-only or with existing entropy-based uncertainty measures augmented by the same RD component.

A more controlled ablation, such as GRPO+RD and KLE+RD, would help determine whether semantic geometry provides complementary information beyond reward-based reweighting.

Questions for authors
  1. Semantic–gradient alignment

Could the authors provide stronger theoretical or empirical justification for the relationship between semantic embedding distances and policy gradient differences? In particular, how robust is this relationship to different embedding models, tasks, and response characteristics?

  1. SEED-GRPO implementation

Could the authors clarify whether the SEED baseline is a full reproduction or an adapted implementation of SEED-GRPO? Can a matched comparison with the original method be provided?

  1. Contribution of reward dispersion

How much improvement can be achieved by RD alone? Does combining RD with existing uncertainty metrics, such as semantic entropy or KLE, provide similar improvements to GCPO?

Reviewer_GfraOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GCPO, an extension of uncertainty-aware GRPO that regulates the contribution of each rollout group using semantic geometry and reward dispersion. The motivation identifies two gaps in previous semantic entropy-based GRPO methods. Semantic-cluster probabilities do not capture the magnitude of disagreement, and disagreement alone does not indicate whether a group contains useful reward contrast. GCPO introduces two alternative geometric measures, Cosine Dispersion (CD) and Barycentric Transport (BoT), and combines them with Reward Dispersion (RD) to scale group-level advantages. The experiments on mathematics and QA benchmarks show good performance.

Critical strengths
  1. The motivation is clear and relevant to group-based policy optimization. The distinction between measuring disagreement and deciding how strongly a rollout group should influence learning is useful. Two groups with similar semantic entropy can differ in their semantic geometry, while similar geometric disagreement can coexist with different reward contrasts. These observations provide a coherent motivation for augmenting uncertainty-based regulation with geometric and reward-dependent information.

  2. The empirical results provide meaningful support for the approach. The main QA results report means and standard deviations over five runs and show improvements over GRPO and competitive uncertainty-aware baselines. The RD removal ablation also directly supports including reward-dependent modulation in the complete method.

  3. The method is simple and effective. CD and BoT provide practical ways to incorporate semantic distances, while RD supplies a complementary signal about within-group reward variation. Applying a common scaling factor to the existing group advantages makes the approach straightforward to integrate into GRPO and preserves the existing relative credit assignment within each group.

Critical weaknesses
  1. Adding experiments of RD to other baselines. The existing ablations demonstrate that adding RD to the geometric variants is beneficial. However, they do not establish how much of the improvement over an entropy-based method. A matched SEED-GRPO + RD comparison on a representative QA task would help clarify the source of the gains. This is primarily a limitation in component attribution; the existing results still support the practical effectiveness of the complete GCPO method.

  2. Some calibration claims should be stated more precisely. RD measures the magnitude of reward differences. It does not itself establish that these differences reflect reliable learning signals rather than noisy or otherwise unhelpful reward variation. Likewise, an association with gradient variability does not directly establish downstream learning value. Authors should tone down the claims.

  3. The novelty discussion overgeneralizes the limitations of existing uncertainty measures. The claim that existing methods ignore semantic distances or disagreement structure does not fully apply to KLE, which explicitly incorporates pairwise semantic similarities through a kernel construction. The anisotropic-gap argument should therefore be restricted to purely mass-based measures.

Questions for authors
  1. Could the authors provide a matched SEED-GRPO + RD comparison, ideally on NarrativeQA and AIME, where the RD contribution is substantial? A focused comparison under the same training budget would be sufficient to clarify this issue.

  2. Mathematical-trajectory encoding: Appendix E.3 states that mathematical uncertainty is computed from fully generated solutions using all-MiniLM-L6-v2. What is the maximum input length of the encoding model? As far as I know, the rollout sequence lengths of mathematical problems usually exceed 1,000 tokens, which is far more than the model's input limit.

  3. Line097-099 Section 2.1 cites Chen et al. (2025a) when discussing early token-level entropy methods. However, SEED-GRPO estimates prompt-level uncertainty through semantic entropy across multiple sampled responses, rather than token-level entropy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.