acceptodds
Under review as a conference paper at ICLR 2027

CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model’s generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating improvements in generalization and robustness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reasoning, Reinforcement Learning, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ug3vOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CLIPO, a contrastive learning framework designed to improve the generalization of reinforcement learning with verifiable rewards (RLVR). Motivated by the limitations of outcome-only supervision, CLIPO introduces a learnable contrastive head and an InfoNCE-based auxiliary reward to encourage consistency among successful reasoning trajectories while distinguishing them from unsuccessful ones. The framework can be integrated into existing policy optimization methods without requiring additional process-level annotations. Experiments across 14 reasoning benchmarks and multiple model architectures demonstrate consistent average improvements over GRPO, GSPO, DAPO, and GMPO, particularly on out-of-distribution and perturbed tasks. Further ablation studies and process-level evaluations support its effectiveness, showing a reduction in flawed-but-correct reasoning trajectories from 31.7% to 25.2%, with minimal computational overhead.

Critical strengths
  1. Clear presentation and well-organized writing. The paper is clearly written and logically structured, with an intuitive motivation, well-explained methodology, and comprehensive experimental analysis. This makes the proposed approach easy to understand and follow.

  2. Consistent improvements in Pass@1 performance. CLIPO achieves consistent Pass@1 improvements across multiple RLVR baselines and reasoning benchmarks, with particularly notable gains on out-of-distribution and perturbed datasets. These results provide empirical evidence of its effectiveness in improving reasoning performance and generalization.

Critical weaknesses
  1. Unjustified Positive Sampling and False Negatives in InfoNCE. CLIPO constructs positive pairs by randomly sampling from correct trajectories, overlooking variations in reasoning quality and distinct solution paths. Moreover, unselected correct trajectories remain in the denominator of the InfoNCE loss—effectively acting as false negatives—which may create undesirable competition among valid alternative solutions. The authors should better justify this sampling design and provide evidence that it actually encourages the model to capture shared features of sound reasoning.

  2. Missing Simple Heuristic Baselines for Reward Weighting. CLIPO relies on contrastive learning to reweight rewards for correct trajectories, yet it omits comparisons against simpler, training-free weighting heuristics. For example, directly scaling rewards or advantages using the trajectory's average policy entropy or log-likelihood would avoid the overhead of training an auxiliary contrastive head. Without showing clear gains over these simpler alternatives, the extra complexity and algorithmic novelty of CLIPO remain insufficiently motivated.

Questions for authors

Potential Diversity Collapse of Reasoning Trajectories. By pulling representations of correct trajectories closer together in embedding space, CLIPO risks inducing mode collapse toward canonical solution patterns, potentially penalizing equally correct but non-standard reasoning paths. The Pass@16 metrics provided in Appendix C.9 do not adequately rule out diversity degradation under higher sampling budgets. Could the authors extend their experiments to Pass@128 and provide a targeted diversity analysis among distinct correct solutions to determine whether CLIPO leads to policy shrinkage? Demonstrating that reasoning diversity is preserved at scale is critical to validating the method's true generalization capacity.

Overall, I find this an interesting attempt to integrate contrastive learning with RLVR. If the authors can adequately address my concerns regarding reasoning diversity and potential noise in the InfoNCE objective, I would be open to raising my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ug3v [1/2]

Thank you for the reviews.

To Q: Three properties of the method and one measurement argue against diversity collapse: the objective is inactive on all-correct groups, the penalty cannot reorder correctness, and Pass@16 is retained while Pass@1 rises.

  1. The objective is off where collapse would do most damage. It applies only to groups with 1 < |P| < G, and an anchor with no positive or no negative in its group is masked out of the loss and receives r^CL = 0, so all-correct groups are never pulled together (Lines 201–203). That is the common case: the term is active in 32–37% of groups, least of all in-distribution, Math 30.4% with 56.9% of groups already saturated, and most on the perturbed and AMC sets at 42.4%, 42.6% and 53.0% (Lines 1227–1231, Table 24).

  2. The penalty cannot reorder correctness. r^CL ∈ [−0.5, 0] applies to correct rollouts only (Lines 230–231), so the spread inside the positive set is at most 0.5 against the 1.0 gap between correct and incorrect, and a non-standard correct path is never pushed below an incorrect one.

  3. The measured signature is the opposite of collapse. On Track II, Pass@16 is retained, 59.95% → 60.53%, while Pass@1 improves 2.3× faster, +1.35 against +0.58, and the share of solvable problems solved on the first attempt rises on 5 of 6 benchmarks, 64.3% → 65.9% (Lines 345–348, 1187–1221, Table 23). Collapse predicts Pass@16 falling as Pass@1 rises.

On Pass@128: k = G = 16 matches the training rollout budget and keeps the comparison on-policy. A larger k probes further into the tail, but it cannot reverse the direction of the evidence above, since collapse severe enough to matter would already appear at k = 16 as coverage falling while greedy accuracy rises. The paper reads Pass@16 as a negative control rather than as a claimed gain, and the benchmark sizes are the reason (Lines 1219–1221): the two negative cells amount to 0.7 and 0.3 problems, inside resolution, while a collapse of practical size would be plainly visible. At AIME's 30 problems Pass@128 moves in 3.3 pp increments, so the metric is coarsest exactly where a large budget is cheapest.

To W1: Your reading of Equation (9) is correct: the denominator ranges over every other rollout of the group, so unselected correct rollouts do appear there. The paper already ablates that property, and the two variants that remove it perform worse, which is why we keep it. SupCon and SoftNN place all correct rollouts in the numerator rather than one sampled positive, with the denominator unchanged. For SoftNN this provably removes the false-negative effect: with the positive set contained in the denominator set, ∂L/∂f(y, y⁺) = exp(f)·(1/Σ_all − 1/Σ_pos) ≤ 0, so no correct rollout is ever pushed away from another. SupCon only attenuates it, since within each of its |P| terms the remaining positives stay in the denominator. Both weaken competition among correct rollouts, and both cost accuracy (Tables 15–16):

LossPositive aggregationTrack I Avg. ΔTrack II Avg. Δ
InfoNCE (ours)single sampled positive+1.12+1.35
SupConall positives, term by term (attenuated)+0.56+0.43
SoftNNall positives, summed in the numerator (removed)+0.48+1.35

SupCon is worse on both tracks. SoftNN loses half the Track I gain and loses the general-reasoning average there, G-Avg. −0.04 against InfoNCE's +0.83, a drop the paper finds for SoftNN across all four base optimizers (Table 17); on Track II it matches our overall average with the gain displaced from math to the competition sets, M-Avg. +0.94 against +1.50 and C-Avg. +1.77 against +1.19. So removing the false negatives does not preserve the effect; at best it relocates it.

The mechanism explains the ordering. Because r^CL applies to correct rollouts only and is clipped at −0.5, the sole component that survives group-relative normalization is the ranking among correct rollouts (Lines 1042–1046), and competition among correct rollouts is therefore the operative signal.

On surface form versus sound reasoning, the step-level evaluation settles the question at the level at which we state the claim. Appendix C.8 conditions on outcome-correct rollouts only, 10,573 rollouts from 2,686 prompt-matched pairs under a method-blind judge, so the answer string, the prompt and the judge are identical across arms and only the derivations differ. Flawed-but-correct reasoning falls 31.7% → 25.2%, with the largest relative reduction in answer forcing, a correct answer asserted without a derivation, 4.4% → 2.0% (Lines 318–322, 341–342, Table 22). The features the head captures are therefore ones that track derivation quality, measured on the process rather than on the embedding geometry.

AuthorsOpenReview ↗

Response to Reviewer ug3v [2/2]

To W2: The τ sweep already rules out the explanation these baselines would test: CLIPO's gain depends on which trajectory receives which penalty, not on the magnitude of the penalties. If CLIPO acted as a bounded reward reweighting of roughly constant magnitude, the sweep would be flat. At τ = 0.2, where the clip binds most often and the penalties sit closest to a common value, CLIPO is on par with GRPO, Track II Avg. 42.13 against 42.25; at τ = 0.02, where the penalties separate, the gain appears at 43.60 (Table 18). What CLIPO supplies is the assignment of penalties to particular trajectories, which entropy or log-likelihood weighting does not.

The signals also differ structurally. Entropy and mean log-likelihood are per-trajectory statistics computable from a single rollout and measure the policy's own confidence, while f(y, ȳ) is relational and cannot be computed from one rollout. Appendix A discusses this confidence-based family and its dependence on model capacity (Lines 817–821). Confidence weighting up-weights what the model already finds probable, which is the direction of the failure mode we target, where our measured effect is a 55% relative reduction in answer forcing.

The head itself costs 7.34 M parameters on a 7B backbone, ≈0.024% additional memory at T = 4096, and +1.03 s on GRPO's 84.83 s per step, about 1.2% (Lines 459–469, Table 9). A training-free heuristic would therefore save nothing measurable, while supplying a different signal.

We welcome further discussion if any concerns remain.

Dear Reviewer ug3v,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Thanks for additional experiments. I have raised my score to three.

Reviewer_uAAJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Since RLVR only focuses on the correctness of the results and ignores the impact of the correctness of the process on the training effect, the author argues that contrastive learning can strengthen the invariant logical flow and suppress reasoning errors. Therefore, this paper proposes CLIPO, which introduces contrastive learning based on RLVR. Specifically, CLIPO maximizes the similarity between positive samples and then converts it into reward signals. In terms of experiments, CLIPO is better than baselines such as GRPO and DAPO in both general reasoning and mathematical reasoning scenarios.

Critical strengths
  1. The paper is clearly and concisely written, and the method is intuitive and easy to understand. It uses sample-level information to estimate the underlying mutual information and incorporates contrastive learning into RLVR by maximizing mutual information between positive samples.
  2. The paper addresses an important research problem: cases in RLVR where the final answer is correct but the reasoning process is flawed. This issue encourages models to learn shortcuts for producing correct answers rather than develop genuine reasoning capabilities. CLIPO seeks to address this problem by introducing an additional reward signal through contrastive learning.
  3. The experimental evaluation is extensive, covering 14 benchmarks in mathematical reasoning, along with evaluations involving symbolic perturbations, difficulty perturbations, and general reasoning tasks. The results demonstrate that CLIPO can improve the performance of all four baseline algorithms.
Critical weaknesses
  1. The authors focus their statistical significance analysis on GRPO+CLIPO and demonstrate statistically significant improvements over GRPO across mathematical and general reasoning tasks. However, Tables 1 and 2 show relatively modest M-Avg. gains for GSPO and DAPO, while subsequent experiments, such as those in Table 7, also evaluate only GRPO+CLIPO. This raises questions about how broadly CLIPO’s effectiveness extends across algorithms. Could the authors provide statistical significance analyses for the other baselines, together with confidence intervals corresponding to Figure 3 and analyses of the flawed-but-correct rate, to substantiate their conclusions? Further experiments explaining why the gains are more pronounced for GRPO than for the other algorithms would also be helpful.
  2. Section 4.3 investigates the effects of contrastive head configurations, contrastive loss variants, and GRPO settings on CLIPO. However, Section 3.2 describes using an additional sentence-level representation and a contrastive head to transform token-level information into sentence-level representations for reward computation. The experiments do not include an ablation that isolates the contribution of the contrastive head itself. Moreover, computing rewards directly from token-level information would be a more straightforward alternative that warrants comparison.
  3. Several prior studies have introduced contrastive learning into policy optimization or RLVR [1–4]. For example, GRACE encodes model-generated reasoning explanations into vector representations, converts similarities between positive and negative samples into rewards, and optimizes the generative model through policy gradients. DisCO establishes a connection between GRPO and discriminative learning under binary rewards and improves training through a discriminative objective and KL-constrained optimization. However, the authors neither discuss how CLIPO differs from these methods nor include them as experimental baselines. In my view, CLIPO is conceptually very similar to these approaches, and introducing contrastive learning into policy optimization or RLVR alone is insufficient to establish its novelty.
  4. A contrastive loss is computed within each rollout group with correct responses as positives and incorrect ones as negatives, and this loss becomes an auxiliary reward added to the verifiable reward. Consequently, it does not increase the reward of any incorrect trajectory. Its main effect is to reduce the rewards of some correct trajectories, thereby differentiating among trajectories that all produce correct answers. The authors appear to provide little explanation or analysis of this specific mechanism, focusing instead on describing the new reward signal as distinguishing correct from incorrect trajectories.

[1] Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective. EMNLP 2026

[2] DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization. NeurIPS 2025

[3] GRACE: Generative Representation Learning via Contrastive Policy Optimization

[4] Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning

Questions for authors

Mroueh [1] shows that calibrating the mean and variance of binary rewards itself induces a weighted contrastive loss. Given that GRPO already incorporates an implicit contrastive mechanism, where exactly do CLIPO’s additional gains come from? Could the authors clarify what GRPO implicitly contrasts, distinguish this from CLIPO’s explicit alignment of successful trajectories in the hidden representation space, and explain how CLIPO changes the resulting policy update? For the remaining concerns, please see Critical Weaknesses.

[1] Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification. ICML 2026

4 comments
AuthorsOpenReview ↗

Response to Reviewer uAAJ [1/2]

We appreciate your valuable feedback and support of our work.

To Q: Mroueh's term and ours act on different objects: GRPO's implicit contrast separates the correct set from the incorrect set, where CLIPO discriminates inside the correct set. The contrastive term induced by mean-and-variance calibration is a function of the binary reward and the prompt-level reward moments alone, so it takes the same value for any two correct rollouts of a prompt: GRPO contrasts the correct set against the incorrect set. CLIPO's term is a function of the relation between two rollouts in a learned embedding space, so it discriminates inside the correct set.

Our saturation analysis makes the distinction exact: a constant r^CL gives r′ = 0.5 r, and group-relative normalization removes a positive affine rescaling exactly, recovering GRPO's advantages (Lines 1042–1046). Everything CLIPO contributes lives in within-correct-set variation, which is what the binary reward cannot see: flawed-but-correct reasoning falls 31.7% → 25.2%, Δ = −6.52 pp with 95% CI [−8.21, −4.80] (Lines 318–322). We cite Mroueh for the implicit-contrast result in Appendix D.6 of the revised PDF.

To W1: What the paper claims is consistency of sign across optimizers, not significance within any one benchmark, and Tables 1 and 2 already establish it. CLIPO improves all 24 aggregate cells, 4 optimizers × 2 tracks × 3 averages:

Δ (M-Avg. / G- or C-Avg. / Avg.)Track ITrack II
GRPO+1.41 / +0.83 / +1.13+1.50 / +1.19 / +1.35
GSPO+0.91 / +0.38 / +0.64+0.33 / +1.26 / +0.80
DAPO+0.22 / +0.04 / +0.13+0.59 / +1.81 / +1.20
GMPO+1.27 / +0.04 / +0.65+1.28 / +0.38 / +0.83

The eight overall-Avg. comparisons are the independent units here, the other two columns being their components. All eight are positive, one-sided sign test p = 2⁻⁸ = 3.9 × 10⁻³, which is not what run-to-run noise produces. Single cells are a separate question, and we draw no conclusion from any one of them: Table 21's three seeds put the spread at roughly ±0.8 on Track I Avg., +1.33 with 95% CI [+0.53, +2.13], so a cell as small as DAPO's +0.13 carries no information by itself (Lines 314–317). Seeds for the other three optimizers would sharpen the individual cells without changing the aggregate statement, which is the claim the paper makes.

On why GRPO benefits most, we report the pattern and decline to explain it. The ordering of the other three is unstable, DAPO being smallest on Track I at +0.13 and second largest on Track II at +1.20, so a post-hoc mechanism would not be worth much. The one hypothesis with independent support is trust-region width: GSPO's clipping interval of 3 × 10⁻⁴ / 4 × 10⁻⁴ mechanically attenuates any reward reshaping (Line 863), and GSPO has the smallest Track II M-Avg. gain at +0.33.

To W2: The head ablation is already in the paper, and the mechanism predicts what the identity-projection variant would show. Table 3 with Tables 13–14 freezes the head: −0.77 Track I Avg. and −0.97 Track II Avg., concentrated on the perturbed sets at P1 −1.12 and P2 −2.48. Table 6 ablates the pooling, Mean / Max / EOS at +1.35 / +1.25 / +1.08 over GRPO.

Your no-head variant differs from that only in using an identity projection in place of a random frozen one, and the paper's mechanism predicts its outcome. With λ = 0.2 and the clip at −0.5, any projection that fails to separate rollouts drives r^CL toward a common value, and a saturated r^CL is removed exactly by group normalization, so the variant converges to GRPO by construction (Lines 1042–1046). The head is also the only trained component in that path: pooled states come from the rollout engine, the contrastive loss trains the head's own parameters, and the per-rollout loss is detached before entering the reward, so one scalar per correct rollout is the entire channel to the policy.

The one alternative we resist on grounds beyond compute is a fully token-level reward. Token × token similarity is O(T²) per pair and O(G²T²) per group at T = 4096, which gives up what makes CLIPO practical: 7.34 M parameters on a 7B backbone, ≈0.024% additional memory and +1.03 s on GRPO's 84.83 s per step (Lines 459–469, Table 9).

AuthorsOpenReview ↗

Response to Reviewer uAAJ [2/2]

To W3: None of the four occupies CLIPO's setting: each changes the policy objective or the reward source, where CLIPO augments the reward on top of an unmodified optimizer. We cite and distinguish all four in Appendix D.6 of the revised PDF. ConSPO contrasts length-normalized sequence log-probabilities with a curriculum margin, so two equally likely correct rollouts are indistinguishable to it. DisCO replaces the group-relative objective with a discriminative one, but the discriminated quantity is still the binary outcome and no trajectory representation appears. GRACE is the closest reward design, yet its task is representation quality itself and its positives and negatives are labelled query–document pairs, where ours come from the verifier inside the rollout group. CoVo builds a label-free intrinsic reward from the consistency and volatility of intermediate likelihoods, replacing the verifiable reward and adding O(NTK) scoring passes, where CLIPO keeps the verifiable reward and adds none.

That difference is also why DisCO and ConSPO are natural additional base optimizers for CLIPO rather than substitutes for it; GRACE's setting requires labelled pairs we do not have.

To W4: Your reading is correct. The clip guarantees by construction that no correct rollout is ranked below an incorrect one, and what CLIPO adds is a second-order ranking inside the positive set. The penalty is multiplied by the correctness mask and then clipped, so incorrect rollouts receive exactly r^CL = 0 and correct ones r^CL ∈ [−0.5, 0], the value being 0 wherever the objective is inactive (Lines 230–231). Correct rollouts therefore receive r′ ∈ [0.5, 1] against r′ = 0, and 0.5 > 0 gives the guarantee. This is the mechanism we intend, and it is why the loss variants that suppress competition among correct rollouts gain less (Tables 15–16, and our response to Reviewer ug3v W1). Appendix D.3 of the revised PDF restates Section 3.2 in these terms, as within-positive-set ranking.

We welcome further discussion if any concerns remain.

Dear Reviewer uAAJ,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

AuthorsOpenReview ↗

Have we addressed your concerns?

Dear Reviewer uAAJ,

Appendix D of the revised PDF is new and marked in blue. D.6 distinguishes CLIPO from Mroueh (2026), ConSPO, DisCO, GRACE and CoVo (Q, W3). D.5 gives all 24 aggregate cells, the sign test and the seed-to-seed spread (W1). D.3 restates the mechanism as within-positive-set ranking, your reading in W4, and the same argument predicts what an identity projection would do (W2).

Does this address your concerns? We would be glad to clarify anything further before the discussion period closes.

Reviewer_W8mQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CLIPO adds a contrastive reward to group based RLVR. It pools each rollout's hidden states and projects them through a learned head. Successful rollouts form positive pairs, and the remaining rollouts enter the contrastive denominator. A clipped penalty derived from this loss adjusts the rewards of successful rollouts before policy optimization. Experiments cover several RLVR optimizers and language models on mathematical and general reasoning tasks. The paper reports improved accuracy and fewer judged process errors, attributing these gains to learning shared reasoning structure.

Critical strengths
  1. The method reuses existing training information. Section 3 builds the auxiliary reward from rollout representations without process annotations or additional rollout generation. This makes the approach practical to integrate into RLVR training.

  2. The design accommodates several policy optimizers. Sections 3 and 4 integrate the same reward mechanism with different RLVR objectives. This gives the method potential utility beyond a single optimization algorithm.

  3. The evaluation addresses generalization and reasoning quality. Section 4 evaluates perturbed problems, and Appendix C.8 examines process errors using matched prompts and judging without method labels. These analyses address the paper's motivation beyond final answer accuracy.

Critical weaknesses
  1. Agreement among successful rollouts does not establish logical validity. Section 1 assumes that correct reasoning is shared while errors are uncorrelated. Responses with correct final answers can share the same invalid shortcut, while valid solutions can follow different reasoning paths. The contrastive head receives only outcome labels, so shared answer patterns can also satisfy its objective. Appendix C.8 reports improved reasoning quality, but does not establish that the learned similarity distinguishes valid derivations from shared shortcuts. This leaves the central denoising mechanism unverified.

  2. The mutual information rationale does not justify the implemented loss. Section 3.1 defines mutual information after marginalizing over prompts, yet computes the loss using responses from the same prompt, including failed responses. This does not establish the sampling conditions of the cited InfoNCE bound. With a single fixed prompt, independent successful responses have zero mutual information, yet the proposed contrastive expression can be positive by separating successes from failures. A derivation for the actual sampling scheme is needed to support the stated objective.

  3. The experiments do not isolate the contribution of learned reasoning geometry. Appendix C.2 acknowledges that clipping can turn the frozen head into a constant rescaling of the binary reward, whose effect group normalization removes. This control therefore changes both the representation learning and the effective reward variation. It cannot distinguish the proposed semantic mechanism from generic reward reweighting. Shuffling the same auxiliary penalties among successful rollouts within each prompt would preserve the reward distribution while testing whether their assignment to particular reasoning paths matters.

Questions for authors
  1. What evidence shows that the learned similarity favors valid reasoning over shared shortcuts when the final answer is held fixed?

  2. How does the actual sampling scheme support the mutual information interpretation in Section 3.1?

  3. Does the benefit remain when the same auxiliary penalties are shuffled among successful rollouts within each prompt?

6 comments
AuthorsOpenReview ↗

Response to Reviewer W8mQ

Thank you for the reviews.

To Q1 and W1: Our step-level evaluation already holds the final answer fixed, so the comparison is about derivations only. All 10,573 judged rollouts are outcome-correct, come from 2,686 prompt-matched pairs, and the judge is blind to the method, so only the derivation differs. The category closest to your concern is answer forcing, a correct answer asserted without a derivation, and it falls most, 4.4% → 2.0%. A shared surface shortcut predicts the opposite, since answer-forced rollouts are the most similar at the surface level. The effect also survives length, formatting and math-density controls with checkpoint and benchmark fixed effects: −5.50 pp, no surface control individually significant, and response length identical across arms at 1,401 vs. 1,404 characters (Lines 318–322, 341–342, 1174–1181).

We do not claim that similarity certifies validity, and the paper declines to read the embedding geometry as evidence of reasoning quality (Line 1015). The claim is the empirical one above: outcome-conditioned consensus reduces measured process errors, most in the shortcut-like category. The revised PDF states that logical status explicitly.

To Q2 and W2: Equation (5) is the MI with the prompt marginalized, not I(y; ȳ | x), and the zero-MI argument applies to the conditional quantity. The text after the equation states this (Line 153). The marginalized MI is the prompt-level structure that two successes share, and it is positive whenever one success is informative about the prompt beyond what the other reveals.

On the bound you are right. Equation (7) draws its negatives from the same prompt and includes failures, so it does not inherit the log(N) − L bound of Equation (4), which assumes negatives from the marginal p(y). The paper already calls it a per-prompt surrogate and notes the bound is loose at a median of 6 negatives per active group (Lines 187, 831–834). Your further point is also correct: the expression can be positive by separating successes from failures alone. That component does not reach the policy. At saturation r′ = 0.5 r, a positive affine rescaling, which group-relative normalization removes exactly (Lines 1042–1046). What survives is the ranking inside the successful set.

To Q3 and W3: You are right that CLIPO-fixed holds the head fixed rather than the penalty assignment, and the paper says as much at Line 365. The τ sweep is the graded form of the shuffle you propose, and it already separates the two hypotheses. At τ = 0.2 the penalties saturate against the clip most often, since r^CL saturates once L^CL ≥ 2.5 and a high temperature drives the loss toward log(G − 1) ≈ 2.71 (Lines 1042–1043); that is the limit in which the assignment of penalties to trajectories carries no information, and there the gain vanishes, Track II Avg. 42.13 against GRPO's 42.25. At τ = 0.02, where the penalties separate, the gain appears, 43.60 (Table 18). A generic reward-reweighting account that is indifferent to which trajectory receives which penalty predicts no difference between these two settings. The caveat we state is that τ also shifts the head's learning dynamics (Figure 7), so this is strong evidence rather than a single-factor isolation.

Two further existing results point the same way. The gain tracks activation, Spearman ρ = +0.60, and not headroom, ρ = −0.03, while GRPO's own gain over the base model shows the reverse pattern, −0.09 and +0.26 (Lines 1232–1238). And the three loss variants share λ and the clip yet are not interchangeable: Track I Avg. +1.12 / +0.56 / +0.48 for InfoNCE / SupCon / SoftNN (Tables 15–16). Reward reweighting of a fixed magnitude predicts neither pattern.

We welcome further discussion if any concerns remain.

Dear Reviewer W8mQ,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

AuthorsOpenReview ↗

Have we addressed your concerns?

Dear Reviewer W8mQ,

Appendix D of the revised PDF is new and marked in blue. D.1 states the step-level result as a claim about judged derivations, with the answer, prompt and judge held fixed (Q1/W1). D.2 separates Equation (5), the MI with the prompt marginalized, from the per-prompt surrogate in Equation (7), which does not inherit the log(N) − L bound (Q2/W2). D.4 gives what the τ sweep separates: at τ = 0.2 the penalties sit near a common value and the gain vanishes, at τ = 0.02 they separate and it appears (Q3/W3).

Does this address your concerns? We would be glad to clarify anything further before the discussion period closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief introduction to the revised PDF

We thank the reviewers. All revisions are marked in blue. New content is in a new Appendix D (Lines 1360–1531).

Appendix D adds:

  1. D.1 Scope of the process-level claim: what the step-level evaluation establishes with the answer, prompt and judge held fixed, stated as a claim about judged derivations rather than about embedding geometry (W8mQ, ug3v).
  2. D.2 Status of the MI motivation: Equation (5) is the MI with the prompt marginalized, Equation (7) is a per-prompt surrogate whose negatives include failures and does not inherit the log(N) − L bound of Equation (4), and Section 3.1 is to be read as motivation (W8mQ).
  3. D.3 The mechanism as within-positive-set ranking: r^CL ∈ [−0.5, 0] on correct rollouts and 0 on incorrect ones, so the clip preserves correct > incorrect by construction (uAAJ, ug3v).
  4. D.4 What isolates the learned geometry: why the τ sweep separates trajectory-specific assignment from constant reward reweighting, what the activation correlations add, and how a confidence-based heuristic differs (W8mQ, uAAJ, ug3v).
  5. D.5 Consistency across base optimizers: all 24 aggregate cells, the sign test, and the seed-to-seed scale (uAAJ).
  6. D.6 Relation to contrastive formulations of policy optimization: Mroueh (2026), ConSPO, DisCO, GRACE and CoVo (uAAJ).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.