- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CLIPO, a contrastive learning framework designed to improve the generalization of reinforcement learning with verifiable rewards (RLVR). Motivated by the limitations of outcome-only supervision, CLIPO introduces a learnable contrastive head and an InfoNCE-based auxiliary reward to encourage consistency among successful reasoning trajectories while distinguishing them from unsuccessful ones. The framework can be integrated into existing policy optimization methods without requiring additional process-level annotations. Experiments across 14 reasoning benchmarks and multiple model architectures demonstrate consistent average improvements over GRPO, GSPO, DAPO, and GMPO, particularly on out-of-distribution and perturbed tasks. Further ablation studies and process-level evaluations support its effectiveness, showing a reduction in flawed-but-correct reasoning trajectories from 31.7% to 25.2%, with minimal computational overhead.
Critical strengths
-
Clear presentation and well-organized writing. The paper is clearly written and logically structured, with an intuitive motivation, well-explained methodology, and comprehensive experimental analysis. This makes the proposed approach easy to understand and follow.
-
Consistent improvements in Pass@1 performance. CLIPO achieves consistent Pass@1 improvements across multiple RLVR baselines and reasoning benchmarks, with particularly notable gains on out-of-distribution and perturbed datasets. These results provide empirical evidence of its effectiveness in improving reasoning performance and generalization.
Critical weaknesses
-
Unjustified Positive Sampling and False Negatives in InfoNCE. CLIPO constructs positive pairs by randomly sampling from correct trajectories, overlooking variations in reasoning quality and distinct solution paths. Moreover, unselected correct trajectories remain in the denominator of the InfoNCE loss—effectively acting as false negatives—which may create undesirable competition among valid alternative solutions. The authors should better justify this sampling design and provide evidence that it actually encourages the model to capture shared features of sound reasoning.
-
Missing Simple Heuristic Baselines for Reward Weighting. CLIPO relies on contrastive learning to reweight rewards for correct trajectories, yet it omits comparisons against simpler, training-free weighting heuristics. For example, directly scaling rewards or advantages using the trajectory's average policy entropy or log-likelihood would avoid the overhead of training an auxiliary contrastive head. Without showing clear gains over these simpler alternatives, the extra complexity and algorithmic novelty of CLIPO remain insufficiently motivated.
Questions for authors
Potential Diversity Collapse of Reasoning Trajectories. By pulling representations of correct trajectories closer together in embedding space, CLIPO risks inducing mode collapse toward canonical solution patterns, potentially penalizing equally correct but non-standard reasoning paths. The Pass@16 metrics provided in Appendix C.9 do not adequately rule out diversity degradation under higher sampling budgets. Could the authors extend their experiments to Pass@128 and provide a targeted diversity analysis among distinct correct solutions to determine whether CLIPO leads to policy shrinkage? Demonstrating that reasoning diversity is preserved at scale is critical to validating the method's true generalization capacity.
Overall, I find this an interesting attempt to integrate contrastive learning with RLVR. If the authors can adequately address my concerns regarding reasoning diversity and potential noise in the InfoNCE objective, I would be open to raising my score.
4 comments
Response to Reviewer ug3v [2/2]
To W2: The τ sweep already rules out the explanation these baselines would test: CLIPO's gain depends on which trajectory receives which penalty, not on the magnitude of the penalties. If CLIPO acted as a bounded reward reweighting of roughly constant magnitude, the sweep would be flat. At τ = 0.2, where the clip binds most often and the penalties sit closest to a common value, CLIPO is on par with GRPO, Track II Avg. 42.13 against 42.25; at τ = 0.02, where the penalties separate, the gain appears at 43.60 (Table 18). What CLIPO supplies is the assignment of penalties to particular trajectories, which entropy or log-likelihood weighting does not.
The signals also differ structurally. Entropy and mean log-likelihood are per-trajectory statistics computable from a single rollout and measure the policy's own confidence, while f(y, ȳ) is relational and cannot be computed from one rollout. Appendix A discusses this confidence-based family and its dependence on model capacity (Lines 817–821). Confidence weighting up-weights what the model already finds probable, which is the direction of the failure mode we target, where our measured effect is a 55% relative reduction in answer forcing.
The head itself costs 7.34 M parameters on a 7B backbone, ≈0.024% additional memory at T = 4096, and +1.03 s on GRPO's 84.83 s per step, about 1.2% (Lines 459–469, Table 9). A training-free heuristic would therefore save nothing measurable, while supplying a different signal.
We welcome further discussion if any concerns remain.
Dear Reviewer ug3v,
We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.
Thanks for additional experiments. I have raised my score to three.
Response to Reviewer ug3v [1/2]
Thank you for the reviews.
To Q: Three properties of the method and one measurement argue against diversity collapse: the objective is inactive on all-correct groups, the penalty cannot reorder correctness, and Pass@16 is retained while Pass@1 rises.
The objective is off where collapse would do most damage. It applies only to groups with 1 < |P| < G, and an anchor with no positive or no negative in its group is masked out of the loss and receives r^CL = 0, so all-correct groups are never pulled together (Lines 201–203). That is the common case: the term is active in 32–37% of groups, least of all in-distribution, Math 30.4% with 56.9% of groups already saturated, and most on the perturbed and AMC sets at 42.4%, 42.6% and 53.0% (Lines 1227–1231, Table 24).
The penalty cannot reorder correctness. r^CL ∈ [−0.5, 0] applies to correct rollouts only (Lines 230–231), so the spread inside the positive set is at most 0.5 against the 1.0 gap between correct and incorrect, and a non-standard correct path is never pushed below an incorrect one.
The measured signature is the opposite of collapse. On Track II, Pass@16 is retained, 59.95% → 60.53%, while Pass@1 improves 2.3× faster, +1.35 against +0.58, and the share of solvable problems solved on the first attempt rises on 5 of 6 benchmarks, 64.3% → 65.9% (Lines 345–348, 1187–1221, Table 23). Collapse predicts Pass@16 falling as Pass@1 rises.
On Pass@128: k = G = 16 matches the training rollout budget and keeps the comparison on-policy. A larger k probes further into the tail, but it cannot reverse the direction of the evidence above, since collapse severe enough to matter would already appear at k = 16 as coverage falling while greedy accuracy rises. The paper reads Pass@16 as a negative control rather than as a claimed gain, and the benchmark sizes are the reason (Lines 1219–1221): the two negative cells amount to 0.7 and 0.3 problems, inside resolution, while a collapse of practical size would be plainly visible. At AIME's 30 problems Pass@128 moves in 3.3 pp increments, so the metric is coarsest exactly where a large budget is cheapest.
To W1: Your reading of Equation (9) is correct: the denominator ranges over every other rollout of the group, so unselected correct rollouts do appear there. The paper already ablates that property, and the two variants that remove it perform worse, which is why we keep it. SupCon and SoftNN place all correct rollouts in the numerator rather than one sampled positive, with the denominator unchanged. For SoftNN this provably removes the false-negative effect: with the positive set contained in the denominator set, ∂L/∂f(y, y⁺) = exp(f)·(1/Σ_all − 1/Σ_pos) ≤ 0, so no correct rollout is ever pushed away from another. SupCon only attenuates it, since within each of its |P| terms the remaining positives stay in the denominator. Both weaken competition among correct rollouts, and both cost accuracy (Tables 15–16):
SupCon is worse on both tracks. SoftNN loses half the Track I gain and loses the general-reasoning average there, G-Avg. −0.04 against InfoNCE's +0.83, a drop the paper finds for SoftNN across all four base optimizers (Table 17); on Track II it matches our overall average with the gain displaced from math to the competition sets, M-Avg. +0.94 against +1.50 and C-Avg. +1.77 against +1.19. So removing the false negatives does not preserve the effect; at best it relocates it.
The mechanism explains the ordering. Because r^CL applies to correct rollouts only and is clipped at −0.5, the sole component that survives group-relative normalization is the ranking among correct rollouts (Lines 1042–1046), and competition among correct rollouts is therefore the operative signal.
On surface form versus sound reasoning, the step-level evaluation settles the question at the level at which we state the claim. Appendix C.8 conditions on outcome-correct rollouts only, 10,573 rollouts from 2,686 prompt-matched pairs under a method-blind judge, so the answer string, the prompt and the judge are identical across arms and only the derivations differ. Flawed-but-correct reasoning falls 31.7% → 25.2%, with the largest relative reduction in answer forcing, a correct answer asserted without a derivation, 4.4% → 2.0% (Lines 318–322, 341–342, Table 22). The features the head captures are therefore ones that track derivation quality, measured on the process rather than on the embedding geometry.