acceptodds
Under review as a conference paper at ICLR 2027

Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLMs

Abstract

Large language models (LLMs) excel at complex tasks with advances in reasoning capabilities. However, existing reward mechanisms remain tightly coupled to final correctness and pay little attention to the underlying reasoning process: trajectories with sound reasoning but wrong answers receive low credit, while lucky guesses with flawed logic may be highly rewarded, affecting reasoning generalization. From a causal perspective, we interpret multi-candidate reasoning for a fixed question as a family of counterfactual experiments. Building on this, we propose Group Causal Counterfactual Policy Optimization (GPO) to explicitly train LLMs to learn generalizable reasoning patterns. It proposes an episodic causal counterfactual reward that jointly captures (i) robustness, encouraging the answer distribution induced by a reasoning step to remain stable under counterfactual perturbations; and (ii) effectiveness, enforcing sufficient variability so that the learned reasoning strategy can transfer across questions. We then construct token-level advantages from this reward and optimize the policy, encouraging LLMs to favor reasoning patterns that are process-valid and counterfactually robust. Extensive experiments on diverse benchmarks demonstrate its advantages.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_prJnOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper proposes a new RL algorithm called GC^2PO. The idea is to automatically generating process reward by adding perturbation to the hidden states of the intermediate tokens. The authors claim that for a reasoning process to be good, it needs to 1. be stable to perturbations 2. preserve norm with perturbations. Following this intuition, they then designed a heuristic score to assign process reward to episodes of reasoning. The episodic reward is then combined with the original outcome reward and distributed to tokens with a surprise-based distribution weight. Experiments demonstrate that this method performs better than state-of-arts baselines.

Critical strengths
  1. The method appears to be empirically improving over state-of-arts baselines.
Critical weaknesses
  1. Presentation issues: The texts in almost all the figures are extremely difficult to read, which makes it very hard to understand the paper. The paper is also difficult to follow with lots of equations but no clear definitions. One example is that I do not know what the perturbations are after reading the full paper, therefore I cannot understand why the norm change of a token after perturbation matters.
  2. Soundness issues: There are more than one math errors in the paper. The most trivial one is in Figure 1. The ground truth to Test question 1 should be 40 instead of 45. The most important one is that in proof of Theorem 3.2, they claim that you can "assume gamma is set sufficiently high to require expansion in all dimensions", but clearly for something like J = (x 0; 0 0) (which does not expand in all dimensions), no matter how high gamma is you can just choose an x to reach that gamma. I also have issues understanding proof for Theorem 3.1.
Questions for authors
  1. In proof of Theorem 3.1, what do you mean by "The term S_{exp} acts as a proxy for maximizing the entropy of the representation distribution" ?
  2. What exactly are the perturbations used in this paper? And why do you think norm preservation under these perturbations can imply expressiveness?
11 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 2

Response to Weakness 1 and Question 2

We sincerely thank the reviewer for the valuable feedback regarding figure legibility and mathematical notation clarity. We fully agree that clear visual presentation and self-consistent notation are essential for accurately conveying our theory and methodology, and we have thoroughly polished and revised the manuscript accordingly:

  • Figure Legibility: We thank the reviewer for pointing out the small font sizes in the figures. Although the original manuscript utilized vector graphics, some annotations and axis labels were indeed insufficiently prominent at standard reading zoom levels. In this revision, we redraw and reformat all figures across both the main text and the appendix, increasing the font sizes of labels, legends, and axes.
  • Refinement of Mathematical Equations and Notation Definitions: We fully appreciate the cognitive load imposed by dense mathematical derivations. In the revised manuscript, in addition to retaining the comprehensive nomenclature table in Appendix A, we review the core equations and added explicit descriptions both in the immediate context surrounding each core equation and wherever key symbols reappear across sections.
  • Concrete Formulation of the Perturbations: As elaborated in Section 3.2 of the main text, the perturbations in this work are not performed via lexical-level substitutions in the discrete text space, but are instead applied within the continuous top-layer hidden state space (Rd\mathbb{R}^d) of the Transformer. Following the counterfactual inference framework (Pearl, 2009; Kusner et al., 2017), for the final boundary token of the ll-th reasoning episode (Ek,lE_{k,l}) in trajectory yky_k, we extract its hidden state vector uk,l(x)∈Rdu_{k,l}(x) \in \mathbb{R}^d after the final normalization layer. The perturbation operator gm:Rd→Rdg_m: \mathbb{R}^d \to \mathbb{R}^d is defined as an adaptive isotropic Gaussian diffusion in the latent space (whose tangential components excite local perturbations on the manifold), i.e., u~k,l(m)(x)=gm(uk,l(x))=uk,l(x)+δm\tilde{u}_{k,l}^{(m)}(x) = g_m(u_{k,l}(x)) = u_{k,l}(x) + \delta_m, where the standard deviation is dynamically scaled according to the norm of the current representation: σ=α⋅∥uk,l(x)∥2d\sigma = \alpha \cdot \frac{\Vert{}u_{k,l}(x)\Vert{}_2}{\sqrt{d}}. This adaptive scaling confines the perturbation within an infinitesimal first-order differential neighborhood of the factual state.
  • Why Norm Preservation (SexpS_{exp}) Implies and Guarantees Expressiveness: The representation norm ratio SexpS_{exp} acts as a critical anti-collapse regularizer whose primary role is to prevent dimensional degeneration, thereby preserving the effective capacity and expressiveness of the high-dimensional representation space. As detailed in Appendix D.3, optimizing solely the stability objective SstaS_{sta}, which measures predictive invariance, easily induces the policy network to slip into a trivial solution where all reasoning states are projected onto a single static constant vector or the zero vector. While the predictive distribution remains invariant under perturbations in such a case, all states collapse onto a zero-dimensional isolated point, stripping the representations of geometric discriminability and expressive capacity. To break this degeneration, we introduce SexpS_{exp} grounded in the optimal decay rate theory of semigroups and dynamical systems. Under a first-order Taylor expansion, the perturbed energy satisfies ∥u~∥22≈∥u∥22+2u⊤Jcδ+δ⊤Jc⊤Jcδ\Vert{}\tilde{u}\Vert{}_2^2 \approx \Vert{}u\Vert{}_2^2 + 2 u^\top J_c \delta + \delta^\top J_c^\top J_c \delta; under isotropic perturbations, maximizing SexpS_{exp} enforces a positive infimum on the quadratic form of the Jacobian matrix JcJ_c across all tangential directions, blocking local volume dissipation and anisotropic collapse in the tangent space. This enables the feature representations to expand non-degenerately and span the compact manifold volume, ensuring that downstream linear projections can express rich causal semantics. The ablation experiments in Figure 8 further confirm that the model retaining SexpS_{exp} achieves markedly higher accuracy across tasks, directly validating the effectiveness and necessity of this mechanism.
AuthorsOpenReview ↗

Response to Weakness 2

Response to Weakness 2

We appreciate the thorough review and have polished the figures and appendix proofs throughout the manuscript accordingly:

  • Correction of the Typo in Figure 1: We sincerely thank the reviewer for catching this error. The annotation in the first row of the illustrative diagram in Figure 1 (Test question 1) indeed contains a typo and should be 40 rather than 45. We will correct this label in the revised manuscript and have cross-checked all illustrative diagrams and numerical examples across both the main text and the appendix.

  • Counterexample Resolution and Proof Reconstruction for Theorem 3.2: We fully concur with the reviewer's critique. Building upon the prior premise in Section 3.2 of the main text, we have reconstructed the proof using the Courant-Fischer min-max theorem and strongly continuous semigroup (C0C_0-semigroup) evolution theory (Pazy, 2012; Li et al., 2010): Let Sdc−1={v∈Rdc:∥v∥2=1}S^{d_c - 1} = \{v \in \mathbb{R}^{d_c} : \Vert{}v\Vert{}_2 = 1\} denote the unit sphere in the tangent space TcCT_c \mathcal{C} of the causal manifold. By the Courant-Fischer theorem, the minimum singular value of the Jacobian matrix is strictly characterized by the infimum of the Rayleigh quotient over the unit sphere σmin⁡2(Jc)=inf⁡v∈Sdc−1v⊤Jc⊤Jcv\sigma_{\min}^2(J_c) = \inf_{v \in S^{d_c - 1}} v^\top J_c^\top J_c v. Under isotropic Gaussian diffusion perturbations, a full-support uniform probability measure is induced on Sdc−1S^{d_c - 1}. If there exists a degenerate direction v0=(0,1)⊤v_0 = (0, 1)^\top such that Jcv0=0J_c v_0 = 0 (i.e., σmin⁡(Jc)=0\sigma_{\min}(J_c) = 0), evolution along this kernel neighborhood will cause the probability measure to undergo irreversible dissipative collapse onto a lower-dimensional singular subspace. According to the optimal decay rate theorem for non-degenerate stationary evolution of strongly continuous semigroups, the infinitesimal generator must satisfy a uniform strict coercivity condition across all directions in the tangent space inf⁡v∈Sdc−1v⊤Jc⊤Jcv≥γ0>0\inf_{v \in S^{d_c - 1}} v^\top J_c^\top J_c v \ge \gamma_0 > 0. Under this revised directional infimum constraint, the counterexample yields v0⊤Jc⊤Jcv0=0<γ0v_0^\top J_c^\top J_c v_0 = 0 < \gamma_0 along the degenerate vector v0=(0,1)⊤v_0 = (0, 1)^\top, and is therefore excluded.

  • Clarification of the Proof Logic for Theorem 3.1: We fully understand that the proof derivations in Appendix B.2 of the original manuscript had overly large logical leaps that may have caused reading difficulties. In Appendix B.2 of the revised manuscript, we have added a structured overview of the proof intuition organized into three clear steps: (1) grounded in the invariance principle of causal mechanisms (Peters et al., 2016; Arjovsky et al., 2019), maximizing the stability term SstaS_{sta} drives the representation mapping in the tangent space to remain predictively invariant exclusively with respect to the true causal semantic variables cc, while eliminating the influence of task-irrelevant spurious variables ss; (2) combining Lyapunov energy preservation conditions with the aforementioned revised directional infimum constraint, we prove that the anti-collapse term SexpS_{exp} blocks local volume contraction on the manifold, forcing the pushforward probability measure of the latent representations on the compact manifold to converge to the uniform measure with maximum differential entropy; and (3) drawing upon classic theoretical frameworks (Khemakhem et al., AISTATS 2020; Zimmermann et al., ICML 2021), under these measure sufficiency and local full-rank bijection conditions in the tangent space, there exists a local diffeomorphism between the learned representation mapping and the true underlying causal variables cc, thereby establishing block-identifiability.

AuthorsOpenReview ↗

Response to Question 1

Response to Question 1

Intuitively, this statement means that by enforcing the norm of latent representations not to decay under local perturbations, features with distinct semantics are prevented from collapsing into a single isolated point or a low-dimensional subspace, thereby forcing the model to spread representations uniformly across the entire geometric manifold. In information theory, a distribution that fills a bounded support as uniformly as possible corresponds precisely to the state that maximizes continuous differential entropy. Specifically, the underlying mechanism operates on two levels:

  • First, SexpS_{exp} is introduced to prevent trivial solutions. The stability term SstaS_{sta} enforces consistency between predictive distributions before and after perturbation. However, if the policy network maps all states to an identical constant vector, the predictive distribution remains invariant, and SstaS_{sta} trivially reaches its maximum. In such a degenerate state, the continuous differential entropy of the representation space collapses, losing feature discriminability. To address this, SexpS_{exp} acts as an opposing regularizer that prevents the representation manifold from contracting under perturbations.

  • Second, the norm-preservation constraint requires the representation mapping to be volume-preserving in the local tangent space. As shown in Section 3.2 and Appendix D.3, the perturbation operators {gm}\{g_m\} satisfy the isotropic full-rank condition in the tangent space. By Liouville's transport theorem, the local evolution of the probability density is governed by the divergence of the displacement field: log⁡p(u~)p(u)=−div(δ(u))+O(∥δ∥2)\log \frac{p(\tilde{u})}{p(u)} = -\text{div}(\delta(u)) + \mathcal{O}(\Vert{}\delta\Vert{}^2). In Eq. (3), the first-order variation of SexpS_{exp} encourages a non-negative divergence (div(δ)≥0\text{div}(\delta) \ge 0). This prevents the probability measure from concentrating onto isolated local points. On a compact manifold, local volume preservation drives the global measure toward a uniform distribution, maximizing continuous differential entropy. As discussed in Appendix D.3, SstaS_{sta} aligns states sharing the same causal mechanism, while SexpS_{exp} penalizes energy dissipation to maintain global uniformity. By the measure extremum theorems in information theory and differential geometry, consider a smooth mapping on a bounded compact support. If this mapping is locally non-contracting everywhere and globally bounded, its pushforward measure converges to the uniform measure on the manifold. Notably, the uniform distribution is the global maximizer of continuous differential entropy over compact sets. We will further emphasize this point in the revised manuscript.

Reviewer_PLoxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Group Causal Counterfactual Policy Optimization, a reinforcement learning framework aimed at improving the generalization of LLM reasoning. The authors argue that existing outcome-based reward methods often fail to distinguish correct reasoning from lucky guesses, since they mainly reward final-answer correctness. To address this, GCPO divides reasoning trajectories into episodes and introduces an episodic causal counterfactual reward with two components: stability, which measures how consistently an episode's predicted answer distribution behaves under perturbations, and expressiveness, which encourages the representation to retain information. These rewards are combined with outcome rewards and distributed to tokens through a GRPO-style optimization objective. Experiments across mathematical reasoning, coding, commonsense reasoning, and logical reasoning benchmarks show improvements over several existing RL methods.

Critical strengths
  1. The paper addresses a very important limitation of outcome-based RL, namely its inability to distinguish valid reasoning from lucky guesses

  2. Combining episodic reward assignment with representation-level perturbations offers an interesting approach to process-aware reinforcement learning

  3. The paper evaluates multiple model scales and reasoning domains, with generally consistent improvements over competitive baselines. The ablation studies, efficiency comparisons, and qualitative examples provide useful insights into the proposed method.

Critical weaknesses
  1. A central claim of the paper is that multiple reasoning trajectories sampled for the same question can be viewed as parallel counterfactual experiments. But in practice, these trajectories are independently sampled from the policy? Theorem 2.1 establishes that an MDP can be represented using an SCM, but this does not mean that independently sampled trajectories constitute valid counterfactual comparisons. This matters because the proposed method relies heavily on this causal interpretation. I also have concerns about the theoretical argument in Appendix B.3 (Equations 36–38). The authors seem to argue that a large expected expansion implies a positive smallest singular value of the representation Jacobian. But for example, consider a Jacobian J = diag(10, 0). We have Tr(JTJ) = 100, but the smallest singular value is still zero, meaning the representation collapses along one direction. I think the trace can be made arbitrarily large while the Jacobian remains rank-deficient? The AM-GM inequality used in Equation 38 only provides an upper bound on the determinant-based quantity and does not establish the claimed implication. I think this step requires a stronger justification for convergence guarantees.

  2. The main motivation of GCPO is to reward correct intermediate reasoning steps even when the final answer is incorrect, and penalize flawed steps in successful trajectories. But in Equations 4–6, the token-level reward is constructed from a nonnegative episodic score and a nonnegative surprise weight. The final token advantage is then obtained by multiplying the trajectory-level normalized advantage by the ratio between the token reward and the trajectory's aggregated reward. As a result, all tokens within the same trajectory share the same advantage sign, although their magnitudes can differ. For example, if a trajectory receives a positive group-level advantage, even an incorrect intermediate step would still receive a positive advantage?

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Weakness 1

Response to Weakness 1

We sincerely appreciate the reviewer's insightful suggestions and fully concur with the high standards required for mathematical rigor. We would like to kindly clarify these concerns from the following two points, and incorporate these supplementary details into the revised manuscript:

(i) Discussion on Multi-Candidate Sampling and Latent-Space Counterfactual Intervention

  • For trajectory sampling (Theorem 2.1), as established by Oberst & Sontag (2019) and Ness et al. (2019) in formulating the mapping from discrete MDPs to Gumbel-Max SCMs, given the identical input prompt xx and policy parameters πθ\pi_\theta, multiple candidate generation paths statistically reflect distinct decision branches under a shared environmental prior, as noted in Appendix B.1. These independently sampled trajectories constitute an empirical Monte Carlo distribution under the same prior, serving as policy exploration controls for group-relative baseline estimation.

  • For reward computation (Section 3.2), rather than directly contrasting these KK independently sampled trajectories as counterfactual pairs, we apply counterfactual interventions directly at the reasoning nodes of a single, fixed trajectory. For any selected intermediate reasoning episode Ek,lE_{k,l}, its preceding causal history (i.e., the context prefix conditioned on the prompt) and the corresponding hidden state uk,l(x)u_{k,l}(x) are held fixed. Applying the local perturbation operator u~(m)=gm(u)\tilde{u}^{(m)} = g_m(u) in the representation tangent space is mathematically equivalent to executing a localized intervention within the SCM. Observing the resulting shifts in the model's subsequent predictive probability distribution to compute the reward strictly follows Pearl's counterfactual framework, i.e., (1) fixing the exogenous environment and prior, (2) performing a localized mechanism intervention, and (3) predicting the potential outcome. As demonstrated by the empirical evaluations in Tables 1 and 3, our method achieves stronger robustness against perturbations and superior OOD generalization, validating the effectiveness of the proposed mechanism.

(ii) Discussion on the Counterexample in Appendix B.3

We are deeply grateful for and sincerely accept the reviewer's critique, recognizing the need for further justification. Building upon the premise of Theorem 3.2 that {gm}\{g_m\} locally spans the tangent space of the manifold with an isotropic, full-rank covariance (Σξ⪰λ0I≻0\Sigma_\xi \succeq \lambda_0 I \succ 0), we have reconstructed the proof using the spherical measure on the manifold's tangent space and the Courant-Fischer min-max theorem. Briefly, let the dimension of the causal manifold's tangent space TcCT_c \mathcal{C} be dcd_c, and let its unit sphere be defined as Sdc−1={v∈Rdc:∥v∥2=1}S^{d_c - 1} = \{v \in \mathbb{R}^{d_c} : \Vert{}v\Vert{}_2 = 1\}. By the Courant-Fischer theorem, the minimum singular value of the Jacobian matrix is strictly governed by the infimum of the Rayleigh quotient over the unit sphere σmin⁡2(Jc)=inf⁡v∈Sdc−1v⊤Jc⊤Jcv=inf⁡v∈Sdc−1∥Jcv∥22\sigma_{\min}^2(J_c) = \inf_{v \in S^{d_c - 1}} v^\top J_c^\top J_c v = \inf_{v \in S^{d_c - 1}} \Vert{}J_c v\Vert{}_2^2. Under isotropic Gaussian diffusion perturbations, the perturbation vectors induce a full-support uniform measure over Sdc−1S^{d_c - 1}. According to strongly continuous semigroup (C0C_0-semigroup) evolution theory (Pazy, 2012; Li et al., 2010), requiring the system to satisfy a dissipation-free energy decay rate constraint under an isotropic perturbation flow is functionally equivalent to requiring the quadratic form of the infinitesimal generator to satisfy strict coercivity inf⁡v∈Sdc−1v⊤Jc⊤Jcv≥δ0>0\inf_{v \in S^{d_c - 1}} v^\top J_c^\top J_c v \ge \delta_0 > 0. Under this revised canonical formulation, the counterexample Jc=diag(10,0)J_c = \text{diag}(10, 0) yields v0⊤Jc⊤Jcv0=0<δ0v_0^\top J_c^\top J_c v_0 = 0 < \delta_0 along the degenerate coordinate direction v0=(0,1)⊤v_0 = (0, 1)^\top, and is therefore excluded.

AuthorsOpenReview ↗

Response to Weakness 2

Response to Weakness 2

We sincerely thank the reviewer for this careful observation. We confirm that when a trajectory receives a positive group-normalized advantage, all tokens within that trajectory indeed share the same sign. We would like to kindly clarify that under critic-free group policy optimization, suppressing flawed intermediate steps does not require assigning localized negative signs; instead, it is achieved through localized gradient compression and relative advantage down-weighting.

First, the core mechanism of policy gradient optimization lies in the relative redistribution of probability mass across the action space, where both the direction and magnitude of parameter updates are governed by the sign and scale of the advantage function. According to the policy gradient theorem, ∇θJ(θ)=E[∑t∇θlog⁡πθ(yk,t∣x,yk,<t)Ak,t(x)]\nabla_\theta \mathcal{J}(\theta) = \mathbb{E}\left[ \sum_{t} \nabla_\theta \log \pi_\theta(y_{k,t} \mid x, y_{k,<t}) A_{k,t}(x) \right], the parameter update is linearly proportional to Ak,t(x)A_{k,t}(x). Following Eqs.(4)–(6), the advantage assigned to an individual token is Ak,t(x)=A^k(x)rk,t(x)r~k(x)A_{k,t}(x) = \hat{A}_k(x) \frac{r_{k,t}(x)}{\tilde{r}_k(x)}. For intermediate steps containing superficial shortcuts, their latent representations fail to preserve predictive consistency and representation energy under manifold perturbations, resulting in extremely low rewards. Consequently, both the episodic score Sk,lS_{k,l} and the token-level reward rk,t(x)r_{k,t}(x) are heavily suppressed, driving the local weighting term rk,t(x)r~k(x)→0\frac{r_{k,t}(x)}{\tilde{r}_k(x)} \to 0. Compared to pivotal reasoning steps within the same trajectory that contribute genuine logical deductions, the advantage allocated to flawed steps is compressed toward zero. Under first-order gradient optimization, a near-zero advantage magnitude stalls the log-probability gradient updates for these tokens, causing their generation likelihood to remain nearly stagnant while high-advantage valid reasoning steps dominate the parameter updates, thereby suppressing these shortcuts in relative probability mass.

Second, the advantage allocation mechanism must be evaluated within the context of intra-group horizontal competition among candidate rollouts. In group-based RL, the group-normalized advantage is computed as A^k(x)=r~k(x)−rˉ(x)sr(x)\hat{A}_k(x) = \frac{\tilde{r}_k(x) - \bar{r}(x)}{s_r(x)}. In outcome-only baselines (such as standard GRPO), as long as the final answer is correct (Rout=1R_{out}=1), the trajectory receives the full reward regardless of intermediate logical gaps, causing "Lucky-Guess" trajectories to receive substantial positive advantages and undergo reinforcement. In contrast, under our GC2PO\text{GC}^2\text{PO} framework, the composite score is defined as a weighted aggregation of the outcome reward and the causal counterfactual reward. Because Lucky-Guess trajectories contain multiple fragile steps with low RcfR_{cf}, their aggregated scalar score r~k(x)\tilde{r}_k(x) is dragged down considerably. Whenever rigorous, "Near-Ideal" trajectories are present within the same group, they elevate the group mean rˉ(x)\bar{r}(x). Empirical statistics from our diagnostic evaluations indicate that in such competitive groups, over 41% of Lucky-Guess trajectories fall below the group mean, causing their trajectory-level advantage to flip entirely to negative, while in the remaining groups, their average advantage is suppressed to a minimal level.

Regarding the alternative of enforcing opposite-sign advantages within a single trajectory (i.e., manually assigning negative values to flawed steps and positive values to valid steps), we conducted theoretical and empirical analyses during early algorithm design. We confirmed that introducing local sign alternations within a single rollout is destructive for long-horizon autoregressive generation. In autoregressive language models, token generation follows a chain of conditional dependencies p(y∣x)=∏tp(yt∣x,y<t)p(y \mid x) = \prod_{t} p(y_t \mid x, y_{<t}). When a successful trajectory contains a flawed step, its subsequent valid derivations are conditioned on the context prefix containing that step. Forcing negative advantages on intermediate tokens while assigning positive advantages to subsequent tokens within the same sequence induces direct gradient conflicts during backpropagation. Without an independent Critic network to provide state-value baselines, such localized sign oscillations trigger severe gradient variance explosion, leading to optimization instability and even mode collapse. Therefore, our current reward calculation and amortization scheme provides a far more stable optimization dynamic.

Empirical results further substantiate the effectiveness of this mechanism. As shown in Table 3 of the main text, on the GSM-Plus benchmark, the accuracy of baseline GRPO drops sharply, whereas GC2PO\text{GC}^2\text{PO} maintains a high win rate with performance degradation reduced to less than half that of the baseline, confirming the efficacy of our approach.

Thank you for your rebuttal. I believe this paper addresses an important research problem, and makes a meaningful step toward improving generalization in reasoning. I do want to keep my positive score, and I do wish this paper to be accepted.

Dear Reviewer PLox

We sincerely appreciate the encouraging feedback, continued support, and kind wishes. Your thoughtful review and constructive comments have been invaluable in helping us strengthen this work.

Best wishes,

The Authors.

Reviewer_MsPhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GC²PO, which augments outcome rewards with an episode-level reward based on stability under latent perturbations and representation norm retention. These rewards are used to construct token-level advantages for GRPO-style training. The paper reports improvements on mathematical and other reasoning benchmarks, and provides theoretical arguments that the reward identifies causal reasoning factors.

Critical strengths
  1. Table 1 reports consistent improvements across three backbones and six mathematical benchmarks. The comparisons include several relevant policy optimization and process-reward methods.
  2. The GSM-Plus results are particularly encouraging. The improvements over GRPO are 8.3 and 7.2 percentage points, compared with 2.0 and 1.5 points on GSM8K. This is the strongest evidence in the paper that the method improves robustness beyond clean accuracy.
Critical weaknesses
  1. I am not convinced by the theoretical guarantees. Appendix B.2 assumes that maximizing SexpS_{\mathrm{exp}} maximizes representation entropy, but Eq. (3) only defines a ratio of representation norms. The connection is not established. There is also a concrete problem in Lemma B.5. Eqs. (36)–(38) infer a positive minimum singular value from a large sum of squared singular values. Taking Jc=diag⁡(L,0)J_c=\operatorname{diag}(L,0) gives an arbitrarily large sum while the minimum singular value remains zero. The arithmetic–geometric mean inequality does not support the claimed conclusion. These gaps undermine the identification and information-preservation guarantees.
  2. I could not reconstruct the reward computation from Section 3.2 or Algorithm 1. The paper does not specify the perturbation operators, their scale, or how the answer distribution q(⋅∣u)q(\cdot\mid u) is obtained from an intermediate hidden state. Is this a next-token distribution, or a distribution over completed answers estimated through additional generation? These choices determine both what the reward measures and its computational cost. Specifying M=8M=8 does not resolve this ambiguity.
Questions for authors
  1. In Eq. (6), rk,t/r~kr_{k,t}/\tilde r_k is nonnegative, so all nonzero token advantages within a trajectory share the same sign. For a trajectory with negative advantage, a larger local reward produces a stronger negative update. Is this the intended behavior, and how does it support learning useful intermediate steps from failures?
  2. Why is the roses-and-flowers example in Figure 17 labeled “Lucky-Guess”? The trajectory gives a valid counterexample showing that the conclusion does not follow. Nevertheless, the counterexample and its conclusion receive rewards of 0.07 and 0.02. Are these measured rewards? Please clarify the process-validity criterion used for this example and the diagnostic evaluation.
5 comments
AuthorsOpenReview ↗

Response to Weakness 1

Response to Weakness 1

We sincerely appreciate the reviewer's insightful evaluation of our theoretical foundations and would like to provide clearer conceptual mappings and operator-theoretic justifications below.

(i) The relationship between the norm decay rate of SexpS_{exp} and the representation entropy

Intuitively, maximizing representation entropy serves to prevent representation collapse, where latent embeddings across diverse reasoning states shrink into an identical locality or a singular degenerate point. Relying solely on the stability objective SstaS_{sta} easily leads to a trivial solution: the policy network can map all inputs into a static constant vector, which naturally achieves maximal stability while reducing representation diversity and differential entropy to zero. The exploration reward SexpS_{exp} in Eq. (3) imposes a direct penalty on the norm dissipation of perturbed embeddings, providing an outward geometric repulsion that prevents local volume shrinkage. In the continuous-space formulation, this creates an alignment-uniformity duality (Wang & Isola, 2020; Zimmermann et al., 2021):

  • Alignment (driven by SstaS_{sta}): clusters latent states sharing identical causal mechanisms cc, thereby stripping away ss;
  • Uniformity (governed by SexpS_{exp}): penalizes the decay of the perturbation energy ratio E[∥u~∥22/∥u∥22]\mathbb{E}[\Vert{}\tilde{u}\Vert{}_2^2 / \Vert{}u\Vert{}_2^2]. Under first-order variation, this enforces a non-negative divergence condition on the displacement field (div(δ)≥0\text{div}(\delta) \ge 0), preventing probability measure from collapsing into isolated singularities.

According to transport theory on continuous dynamical systems, a smooth mapping on a compact manifold that is locally volume-preserving and undergoes isotropic diffusion admits a unique stationary pushforward measure converging weakly to the uniform measure (Haar measure). In information theory (Cover & Thomas, 2006), the uniform distribution over a compact support uniquely achieves the global maximum of differential entropy. Hence, penalizing local norm decay acts as a principled proxy for maximizing representation entropy on the manifold.

(ii) Proof of Theorem 3.2

We acknowledge that the original text's scalar trace bound was insufficient to rule out the counterexample Jc=diag(L,0)J_c = \text{diag}(L, 0). In the revised proof of Theorem 3.2 (Appendix B.3), we formalize the argument using the continuous operator limit and the Courant-Fischer min-max theorem:

As stated in the premise of Theorem 3.2, the perturbation operators locally span the tangent space of the causal manifold with an isotropic, full-rank covariance (Σξ⪰λ0I≻0\Sigma_\xi \succeq \lambda_0 I \succ 0). Let Sdc−1={v∈Rdc:∥v∥2=1}S^{d_c - 1} = \{v \in \mathbb{R}^{d_c} : \|v\|_2 = 1\} denote the unit sphere in the tangent space TcCT_c \mathcal{C}, on which isotropic diffusion induces a rotation-invariant uniform probability measure. By the Courant-Fischer theorem, the minimum singular value is strictly characterized by the infimum of the Rayleigh quotient, i.e., σmin⁡2(Jc)=inf⁡v∈Sdc−1v⊤Jc⊤Jcv=inf⁡v∈Sdc−1∥Jcv∥22\sigma_{\min}^2(J_c) = \inf_{v \in S^{d_c - 1}} v^\top J_c^\top J_c v = \inf_{v \in S^{d_c - 1}} \|J_c v\|_2^2.

Under strongly continuous semigroup (C0C_0-semigroup) theory (Pazy, 2012; Li et al., 2010), requiring optimal, dissipation-free decay rates on an isotropic diffusion field dictates that the infinitesimal generator satisfies coercivity across all tangent directions, inf⁡v∈Sdc−1v⊤Jc⊤Jcv≥γ0>0\inf_{v \in S^{d_c - 1}} v^\top J_c^\top J_c v \ge \gamma_0 > 0. Under this directional infimum condition, the proposed counterexample Jc=diag(L,0)J_c = \text{diag}(L, 0) yields v0⊤Jc⊤Jcv0=0<γ0v_0^\top J_c^\top J_c v_0 = 0 < \gamma_0 along the degenerate coordinate v0=(0,1)⊤v_0 = (0, 1)^\top, and is thus excluded from the admissible class of representations. Consequently, σmin⁡(Jc)≥γ0>0\sigma_{\min}(J_c) \ge \sqrt{\gamma_0} > 0 holds unconditionally. To empirically substantiate this theoretical correction, we are adding numerical evaluations of the representation Jacobian's singular value spectrum across test episodes in Appendix E.3.

AuthorsOpenReview ↗

Response to Weakness 2

Response to Weakness 2

We deeply appreciate the reviewer's thorough scrutiny and would like to provide the following outline to clarify these details:

(i) Computation of q(⋅∣u)q(\cdot \mid u)

  • q(⋅∣uk,l(x))q(\cdot \mid u_{k,l}(x)) is an immediate vocabulary-space projection distribution and does not require executing additional multi-step autoregressive decoding from intermediate states. Let Ek,lE_{k,l} denote the ll-th reasoning episode of the current trajectory yky_k, and let uk,l(x)∈Rdu_{k,l}(x) \in \mathbb{R}^d be the top-layer hidden state of the Transformer corresponding to its last token. This state is mapped to a Softmax distribution over the vocabulary via the frozen backbone LM head (i.e., the unnormalized vocabulary projection matrix Whead∈R∣V∣×dW_{\text{head}} \in \mathbb{R}^{\vert{}\mathcal{V}\vert{} \times d}). When the mm-th latent perturbation is applied to obtain u~k,l(m)(x)\tilde{u}_{k,l}^{(m)}(x), the resulting perturbed predictive distribution is given by q(m)(w∣x)=q(w∣u~k,l(m)(x))=Softmax(Wheadu~k,l(m)(x)τlm)q^{(m)}(w \mid x) = q(w \mid \tilde{u}_{k,l}^{(m)}(x)) = \text{Softmax}\left( \frac{W_{\text{head}} \tilde{u}_{k,l}^{(m)}(x)}{\tau_{\text{lm}}} \right). In practical implementations, to further conserve computation and GPU memory, qq can either be computed over the full vocabulary V\mathcal{V} or approximated as a locally normalized distribution restricted to the top-KvK_v tokens (e.g., Kv=128K_v = 128); in our experiments, we retain the full-vocabulary Softmax by default.

  • Why is this distribution sufficient to represent reasoning validity? Existing research (Meng et al., 2022; Geva et al., 2021; Park et al., 2024) has demonstrated that in CoT reasoning, the final token hidden state at each key reasoning boundary (such as punctuation, line breaks, or step delimiters) acts as a working memory bottleneck. During forward propagation, this hidden vector aggregates the causal beliefs accumulated from preceding derivations, and the model's predictive distribution q(⋅∣u)q(\cdot \mid u) for the next logical keyword or subsequent state directly reflects the causal certainty of that intermediate step on the conceptual manifold. If a step is grounded in sound causal logic, its hidden state maintains stable confidence regarding the subsequent derivation direction under local perturbations (yielding a high SstaS_{sta}); conversely, if it relies on fragile, superficial shortcuts, local perturbations cause its predictive confidence to collapse sharply (yielding a low SstaS_{sta}).

(ii) Formulation and Scale of the gmg_m

Following the Principle of Minimal Change in counterfactual causal inference (Pearl, 2009; Kusner et al., 2017), the perturbation operator gm:Rd→Rdg_m: \mathbb{R}^d \to \mathbb{R}^d is defined as an adaptive local Gaussian diffusion in the latent space, u~k,l(m)(x)=gm(uk,l(x))=uk,l(x)+δm,δm∼N(0,σk,l2Id)\tilde{u}_{k,l}^{(m)}(x) = g_m(u_{k,l}(x)) = u_{k,l}(x) + \delta_m, \quad \delta_m \sim \mathcal{N}\left(0, \sigma_{k,l}^2 I_d\right). To maintain consistent relative geometric scale across hidden vectors of varying depths and norms, the perturbation standard deviation σk,l\sigma_{k,l} is not fixed as a constant; instead, it is dynamically scaled according to the ℓ2\ell_2-norm of the current representation σk,l=α⋅∥uk,l(x)∥2d\sigma_{k,l} = \alpha \cdot \frac{\Vert{}u_{k,l}(x)\Vert{}_2}{\sqrt{d}}, where α\alpha is a dimensionless perturbation intensity coefficient.

(iii) Computational Overhead Analysis

Within each training step, the dominant computational cost lies in (1) Trajectory Generation (Sampling Phase): autoregressively generating KK candidate trajectories, which spanning hundreds to thousands of tokens, requires repeatedly executing full attention computations across all LlayerL_{\text{layer}} Transformer blocks. In contrast, (2) Reward Computation Overhead is highly lightweight: we simply extract the existing hidden vectors uk,lu_{k,l} at each episode boundary of the already-generated token sequence; apply M=8M=8 noise perturbations directly on the GPU via broadcasted addition, completed within milliseconds; and perform a single batched matrix-vector multiplication U~Whead⊤\tilde{U} W_{\text{head}}^\top. Because this bypasses any forward pass through the Transformer blocks, the additional FLOPs required to compute RcfR_{cf} represent only a small fraction of the overall autoregressive sampling and gradient backpropagation costs throughout the RL iterations.

AuthorsOpenReview ↗

Response to Question 1

Response to Question 1

We thank the reviewer for this thoughtful question regarding the gradient behavior of Eq. (6). We would like to clarify that the statement uses standard outcome-only RLVR (e.g., vanilla GRPO) as the reference baseline, rather than intra-trajectory relative values. Furthermore, the sign-preserving scaling design is an intentional mechanism grounded in policy optimization dynamics for critic-free group RL. We elaborate on this from the following perspectives:

(i) Discussion on Advantage Design

The statement "previously negative" specifically refers to the fate of failed trajectories under standard GRPO:

  1. Because standard GRPO relies exclusively on binary outcome rewards (Rout∈{0,1}R_{\text{out}} \in \{0, 1\}), for any prompt with at least one correct rollout, all incorrect trajectories (Rout=0R_{\text{out}} = 0) receive a zero score and are strictly assigned negative group-normalized advantages (A^k,GRPO<0\hat{A}_{k, \text{GRPO}} < 0). As a result, valuable near-miss trajectories, which execute sound intermediate reasoning but fail at the final step, are penalized indiscriminately as a whole.
  2. In GC2PO\text{GC}^2\text{PO}, a near-miss trajectory exhibits high causal invariance and expressiveness across the vast majority of its intermediate steps, accumulating a substantial causal counterfactual reward (RcfR_{\text{cf}}) that far exceeds inferior rollouts in the same group (such as fully-bad paths or fragile guesses). Consequently, in the group-wise comparison across KK candidates, the scalar return r~k(x)\tilde{r}_k(x) of a high-quality near-miss trajectory surpasses the group mean rˉ(x)\bar{r}(x), flipping its trajectory-level advantage to positive (A^k(x)>0\hat{A}_k(x) > 0). Once A^k(x)>0\hat{A}_k(x) > 0, the nonnegative token weight rk,tr~k\frac{r_{k,t}}{\tilde{r}_k} selectively amplifies the positive gradient updates on those high-reward intermediate tokens, thereby allowing the model to learn useful reasoning steps from outcome-failed trajectories.

(ii) Rationale for Sign-Preserving Scaling

The sign-preserving modulation reflects a deliberate trade-off between policy coherence and gradient variance control in critic-free LLM reinforcement learning:

  1. In long-chain mathematical or code reasoning, the validity of a step is heavily conditioned on its preceding context. When a trajectory's overall return remains below the group average (A^k<0\hat{A}_k < 0), most of its state transitions have failed the causal verification, indicating that the global reasoning direction has already derailed. Artificially centering rewards within such a trajectory to force positive token advantages (A^k,t>0\hat{A}_{k,t} > 0) would encourage the model to generate those tokens conditioned on a flawed context, inducing local overconfidence and breaking global trajectory coherence.
  2. rk,t=Sk,lwk,tr_{k,t} = S_{k,l} w_{k,t}, where wk,t∝−log⁡πθold(yk,t)w_{k,t} \propto -\log \pi_{\theta_{\text{old}}}(y_{k,t}) measures token-level information content (surprise). On a trajectory judged to be globally unsuccessful (A^k<0\hat{A}_k < 0), branches explored with very low prior probability (high surprise) that ultimately fail to salvage the solution receive stronger suppression, which aligns with standard conservative exploration principles in RL.

Comments on the paper

2 comments
~Mingxuan_Li1OpenReview ↗

The whole SCM setup is wrong...

I bumped into this paper randomly. Figure 2 is clearly a wrong usage of the SCM framework. How can an endogenous variable have an exgenous variable descendent? And where in the Pearl's book does he ever mention the directed edge as "true correlations" and a new type of dashed directed edge as "spurious correlations"? The whole discussion is built on such a huge mistake...

Thank you for your attention. However, we would appreciate it if you could thoroughly read and reflect on the problem we address before making such emotional and misleading remarks.

First, if strictly expressed following the semiotic conventions of Pearl's classical causal diagrams, Figure 2 indeed does not constitute a textbook-style formal SCM DAG. However, the core issue we argue here is that, in the current post-training of LLMs, how purely outcome-based binary rewards induce models to rely on brittle superficial shortcuts, thereby undermining reasoning generalization. Figure 2 is essentially a motivational conceptual illustration intended to build high-level intuition, which is designed to better articulate our problem. In the legend of Figure 2(a), solid circles represent observable variables, while dashed circles denote unobservable latent variables. In the frameworks of causal inference, "endogenous vs. exogenous" belongs to the dimension of causal generation mechanisms, whereas "observable vs. unobservable" pertains to the measurement dimension. Because the input X simultaneously points to C and S, C and S topologically constitute unobservable endogenous variables elicited by mapping from the external input X, rather than the alleged "endogenous variable pointing to an exogenous variable." The reason why this conceptual diagram can characterize "shortcut learning" is that the input query XX acts as a common cause under the specific training data distribution. In observational data without counterfactual interventions, there exists a confounding backdoor path induced by XX: S←X→C→YS \leftarrow X \rightarrow C \rightarrow Y, which causes S and the answer Y to exhibit strong statistical co-occurrence (P(Y∣S)≈P(Y∣C)P(Y \mid S) \approx P(Y \mid C)). When optimization relies solely on outcome binary rewards, the model cannot distinguish whether this correlation stems from the genuine causal mechanism C→YC \rightarrow Y or the brittle superficial spurious correlation S→YS \rightarrow Y, thereby frequently generating "lucky guesses" and failing under OOD settings. Figures 2(b) and 2(c) concretely contrast robust mechanisms against brittle shortcuts using solid and dashed paths, aiming to help readers grasp this bottleneck through an intuitive and illustrative legend. Furthermore, the formal theory of this work is not built upon the illustration in Figure 2; following the standard Gumbel-Max SCM framework, we formally define the set of endogenous states V={st,at}V=\{s_t, a_t\} and the set of i.i.d. exogenous noise U={ϵ,ν}U=\{\epsilon, \nu\}, and prove the equivalence between multi-candidate sampling and parallel counterfactual interventions under shared priors based on (Oberst & Sontag, 2019; Ness et al., 2019).

We sincerely welcome discussions, but rashly asserting that "the whole discussion is built on such a huge mistake" is also a mistake.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.