- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CIPS, a prefix-dependent baseline for reinforcement learning with verifiable rewards. It estimates reasoning progress by contrasting the likelihoods of correct and incorrect answers from the same rollout group, providing finer-grained credit assignment without additional reward models or rollouts. Experiments on mathematical and combinatorial reasoning tasks with two model scales show that CIPS consistently improves over Dr. GRPO and prior dense-credit methods, while maintaining similar training-step costs.
Critical strengths
- The paper introduces a simple and well-motivated contrastive prefix score that leverages correct and incorrect answers to mitigate spurious effects in answer likelihood, providing a more informative signal for credit assignment.
- CIPS can be easily integrated into existing RLVR frameworks without requiring additional reward models or rollouts. The paper also provides a theoretical justification for using CIPS as an unbiased baseline under on-policy sampling.
- The experiments are relatively comprehensive, covering two model scales, multiple reasoning benchmarks, and detailed ablations. The results demonstrate consistent improvements over strong baselines with minimal computational overhead.
Critical weaknesses
- Although the authors provide ablations on advantage rescaling, it remains unclear how much of the improvement comes from better prefix-level credit assignment rather than changes in the advantage distribution. More controlled experiments are needed to isolate these effects.
- The theoretical analysis establishes that CIPS is an unbiased baseline under idealized on-policy sampling, but this guarantee does not directly extend to the practical PPO-style optimization setting. Moreover, the paper lacks direct analysis of whether CIPS reduces policy gradient variance.
- The experiments are limited to relatively small models (1.7B and 4B) and short-answer reasoning tasks. It remains unclear whether CIPS generalizes to larger models and more complex tasks, especially those without explicit reference answers.
Questions for authors
- Could the authors further isolate the contribution of prefix-level credit assignment from advantage rescaling, for example by comparing against a prefix-independent baseline with a matched advantage distribution?
- Could the authors provide empirical analysis of policy gradient variance or optimization stability to better demonstrate the effectiveness of CIPS as a baseline under practical PPO-style training?
- Could the authors evaluate CIPS on larger models or more diverse reasoning tasks, and discuss how the method could be extended to settings without explicit reference answers?
2 comments
I appreciate the authors’ detailed response and the new experiments. The additional controls and larger-model results have largely addressed my concerns. I consider this a valuable contribution and will maintain my positive rating.
We thank the reviewer for their constructive feedback, which helped us improve the paper. All changes in the revised manuscript are highlighted in blue.
Question 1 / Weakness 1
We have added this control. We compute the CIPS baseline as usual, but before applying it, we average it over each rollout and apply the average as a uniform baseline to all tokens of that rollout. This is a prefix-independent baseline that keeps the rollout-level advantage distribution of CIPS and removes only the dependence on the prefix. CIPS with a uniform baseline performs between Dr. GRPO and full CIPS, so both effects contribute, and the gap between the last two rows is attributable to prefix-level credit assignment.
This complements the existing controls in Appendix C.5: Dr. GRPO with advantages matched to those of CIPS, and CIPS rescaled to the advantage scale of Dr. GRPO, which still improves over Dr. GRPO on all four metrics.
Question 2 / Weakness 2
We agree with the first point and state it in Section 3 (lines 207–209) and in the limitations. The bias comes from PPO-style clipping and from training on mixed-outcome groups only. Both are shared by all compared methods, including Dr. GRPO, and are not unique to CIPS.
Regarding the second point, we have added an empirical variance analysis as Appendix C.7. We measure the variance of the policy-gradient estimate of CIPS and TIPS relative to that of Dr. GRPO on identical rollouts, at the final checkpoints of policies trained with each of the three methods. Values below 1 indicate lower variance than Dr. GRPO.
On the policy that CIPS training actually produces, the CIPS baseline reduces gradient variance by 21% relative to the group-mean baseline. The effect depends on the policy: on a TIPS-trained policy the variance is roughly on par with Dr. GRPO, and on a Dr. GRPO-trained policy, which CIPS training never visits, it is higher. This supports the claim that CIPS does reduce gradient variance along its own training trajectory, which is the regime that matters for optimization. This is added to the revision in Appendix C.7 and Table 10
Question 3 / Weakness 3
To test whether the improvements carry over to a larger model, we ran CIPS and the two main comparison methods, Dr. GRPO and TIPS, on Qwen3-8B. The results are included in Appendix C.2, Table 6 and shown below. CIPS retains its improvement at this scale.
Regarding task diversity, the paper already evaluates a second, non-mathematical task, Countdown (Appendix C.4), where CIPS is the strongest method on all metrics for both model sizes. We agree that CIPS in its current form is limited to short-answer tasks, as stated in the limitations (Section 7, line 531). Extending it to open-ended outputs is non-trivial and we leave it for future work.
Regarding reference answers: a reference answer is required only for all-incorrect groups. In a mixed-outcome group, which is what all methods train on, the correct bank can be built from the verified-correct peer answers alone, so a verifier without a reference answer would in principle suffice. We have not evaluated this variant and now mention it as possible extension of our work.
We hope these revisions address your concerns, and we would be happy to clarify any remaining points.