- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes a new RL algorithm called GC^2PO. The idea is to automatically generating process reward by adding perturbation to the hidden states of the intermediate tokens. The authors claim that for a reasoning process to be good, it needs to 1. be stable to perturbations 2. preserve norm with perturbations. Following this intuition, they then designed a heuristic score to assign process reward to episodes of reasoning. The episodic reward is then combined with the original outcome reward and distributed to tokens with a surprise-based distribution weight. Experiments demonstrate that this method performs better than state-of-arts baselines.
Critical strengths
- The method appears to be empirically improving over state-of-arts baselines.
Critical weaknesses
- Presentation issues: The texts in almost all the figures are extremely difficult to read, which makes it very hard to understand the paper. The paper is also difficult to follow with lots of equations but no clear definitions. One example is that I do not know what the perturbations are after reading the full paper, therefore I cannot understand why the norm change of a token after perturbation matters.
- Soundness issues: There are more than one math errors in the paper. The most trivial one is in Figure 1. The ground truth to Test question 1 should be 40 instead of 45. The most important one is that in proof of Theorem 3.2, they claim that you can "assume gamma is set sufficiently high to require expansion in all dimensions", but clearly for something like J = (x 0; 0 0) (which does not expand in all dimensions), no matter how high gamma is you can just choose an x to reach that gamma. I also have issues understanding proof for Theorem 3.1.
Questions for authors
- In proof of Theorem 3.1, what do you mean by "The term S_{exp} acts as a proxy for maximizing the entropy of the representation distribution" ?
- What exactly are the perturbations used in this paper? And why do you think norm preservation under these perturbations can imply expressiveness?
11 comments
Response to Weakness 2
Response to Weakness 2
We appreciate the thorough review and have polished the figures and appendix proofs throughout the manuscript accordingly:
-
Correction of the Typo in Figure 1: We sincerely thank the reviewer for catching this error. The annotation in the first row of the illustrative diagram in Figure 1 (Test question 1) indeed contains a typo and should be 40 rather than 45. We will correct this label in the revised manuscript and have cross-checked all illustrative diagrams and numerical examples across both the main text and the appendix.
-
Counterexample Resolution and Proof Reconstruction for Theorem 3.2: We fully concur with the reviewer's critique. Building upon the prior premise in Section 3.2 of the main text, we have reconstructed the proof using the Courant-Fischer min-max theorem and strongly continuous semigroup (-semigroup) evolution theory (Pazy, 2012; Li et al., 2010): Let denote the unit sphere in the tangent space of the causal manifold. By the Courant-Fischer theorem, the minimum singular value of the Jacobian matrix is strictly characterized by the infimum of the Rayleigh quotient over the unit sphere . Under isotropic Gaussian diffusion perturbations, a full-support uniform probability measure is induced on . If there exists a degenerate direction such that (i.e., ), evolution along this kernel neighborhood will cause the probability measure to undergo irreversible dissipative collapse onto a lower-dimensional singular subspace. According to the optimal decay rate theorem for non-degenerate stationary evolution of strongly continuous semigroups, the infinitesimal generator must satisfy a uniform strict coercivity condition across all directions in the tangent space . Under this revised directional infimum constraint, the counterexample yields along the degenerate vector , and is therefore excluded.
-
Clarification of the Proof Logic for Theorem 3.1: We fully understand that the proof derivations in Appendix B.2 of the original manuscript had overly large logical leaps that may have caused reading difficulties. In Appendix B.2 of the revised manuscript, we have added a structured overview of the proof intuition organized into three clear steps: (1) grounded in the invariance principle of causal mechanisms (Peters et al., 2016; Arjovsky et al., 2019), maximizing the stability term drives the representation mapping in the tangent space to remain predictively invariant exclusively with respect to the true causal semantic variables , while eliminating the influence of task-irrelevant spurious variables ; (2) combining Lyapunov energy preservation conditions with the aforementioned revised directional infimum constraint, we prove that the anti-collapse term blocks local volume contraction on the manifold, forcing the pushforward probability measure of the latent representations on the compact manifold to converge to the uniform measure with maximum differential entropy; and (3) drawing upon classic theoretical frameworks (Khemakhem et al., AISTATS 2020; Zimmermann et al., ICML 2021), under these measure sufficiency and local full-rank bijection conditions in the tangent space, there exists a local diffeomorphism between the learned representation mapping and the true underlying causal variables , thereby establishing block-identifiability.
Response to Question 1
Response to Question 1
Intuitively, this statement means that by enforcing the norm of latent representations not to decay under local perturbations, features with distinct semantics are prevented from collapsing into a single isolated point or a low-dimensional subspace, thereby forcing the model to spread representations uniformly across the entire geometric manifold. In information theory, a distribution that fills a bounded support as uniformly as possible corresponds precisely to the state that maximizes continuous differential entropy. Specifically, the underlying mechanism operates on two levels:
-
First, is introduced to prevent trivial solutions. The stability term enforces consistency between predictive distributions before and after perturbation. However, if the policy network maps all states to an identical constant vector, the predictive distribution remains invariant, and trivially reaches its maximum. In such a degenerate state, the continuous differential entropy of the representation space collapses, losing feature discriminability. To address this, acts as an opposing regularizer that prevents the representation manifold from contracting under perturbations.
-
Second, the norm-preservation constraint requires the representation mapping to be volume-preserving in the local tangent space. As shown in Section 3.2 and Appendix D.3, the perturbation operators satisfy the isotropic full-rank condition in the tangent space. By Liouville's transport theorem, the local evolution of the probability density is governed by the divergence of the displacement field: . In Eq. (3), the first-order variation of encourages a non-negative divergence (). This prevents the probability measure from concentrating onto isolated local points. On a compact manifold, local volume preservation drives the global measure toward a uniform distribution, maximizing continuous differential entropy. As discussed in Appendix D.3, aligns states sharing the same causal mechanism, while penalizes energy dissipation to maintain global uniformity. By the measure extremum theorems in information theory and differential geometry, consider a smooth mapping on a bounded compact support. If this mapping is locally non-contracting everywhere and globally bounded, its pushforward measure converges to the uniform measure on the manifold. Notably, the uniform distribution is the global maximizer of continuous differential entropy over compact sets. We will further emphasize this point in the revised manuscript.
Response to Weakness 1 and Question 2
Response to Weakness 1 and Question 2
We sincerely thank the reviewer for the valuable feedback regarding figure legibility and mathematical notation clarity. We fully agree that clear visual presentation and self-consistent notation are essential for accurately conveying our theory and methodology, and we have thoroughly polished and revised the manuscript accordingly: