acceptodds
Under review as a conference paper at ICLR 2027

CIPS: Correct-Incorrect Prefix Score as a Baseline for RL with verifiable rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) is a simple and effective recipe for improving the reasoning capabilities of large language models, but its sequence-level rewards provide only a coarse signal for credit assignment. Cheaper alternatives to process reward models and Monte Carlo sampling use the likelihood of the correct answer as a prefix-level signal, but this likelihood also reflects spurious context and answer syntax rather than reasoning progress alone. We introduce the correct-incorrect prefix score (CIPS), which uses the likelihood of plausible incorrect answers from the same rollout group as a control for these effects. A prefix counts as progress only if it raises the likelihood of correct answers relative to incorrect ones. We show that CIPS is robust to spurious prefix content, retains the prompt-difficulty information of the group-mean baseline, and additionally distinguishes rollouts within a group and along the reasoning trace. Used as a prefix-dependent baseline in place of the group mean, CIPS requires neither a learned reward model nor additional rollouts. For two model scales and both mathematical and combinatorial reasoning tasks, CIPS consistently outperforms outcome-only training and prior dense-credit methods. Additionally, with essentially unchanged step time, it reaches the final performance of Dr. GRPO with about 25% fewer training steps. The code is publicly available (https://anonymous.4open.science/r/cips-65EB).

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, LLM post-training, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_m1EHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CIPS, a prefix-dependent baseline for reinforcement learning with verifiable rewards. It estimates reasoning progress by contrasting the likelihoods of correct and incorrect answers from the same rollout group, providing finer-grained credit assignment without additional reward models or rollouts. Experiments on mathematical and combinatorial reasoning tasks with two model scales show that CIPS consistently improves over Dr. GRPO and prior dense-credit methods, while maintaining similar training-step costs.

Critical strengths
  1. The paper introduces a simple and well-motivated contrastive prefix score that leverages correct and incorrect answers to mitigate spurious effects in answer likelihood, providing a more informative signal for credit assignment.
  2. CIPS can be easily integrated into existing RLVR frameworks without requiring additional reward models or rollouts. The paper also provides a theoretical justification for using CIPS as an unbiased baseline under on-policy sampling.
  3. The experiments are relatively comprehensive, covering two model scales, multiple reasoning benchmarks, and detailed ablations. The results demonstrate consistent improvements over strong baselines with minimal computational overhead.
Critical weaknesses
  1. Although the authors provide ablations on advantage rescaling, it remains unclear how much of the improvement comes from better prefix-level credit assignment rather than changes in the advantage distribution. More controlled experiments are needed to isolate these effects.
  2. The theoretical analysis establishes that CIPS is an unbiased baseline under idealized on-policy sampling, but this guarantee does not directly extend to the practical PPO-style optimization setting. Moreover, the paper lacks direct analysis of whether CIPS reduces policy gradient variance.
  3. The experiments are limited to relatively small models (1.7B and 4B) and short-answer reasoning tasks. It remains unclear whether CIPS generalizes to larger models and more complex tasks, especially those without explicit reference answers.
Questions for authors
  1. Could the authors further isolate the contribution of prefix-level credit assignment from advantage rescaling, for example by comparing against a prefix-independent baseline with a matched advantage distribution?
  2. Could the authors provide empirical analysis of policy gradient variance or optimization stability to better demonstrate the effectiveness of CIPS as a baseline under practical PPO-style training?
  3. Could the authors evaluate CIPS on larger models or more diverse reasoning tasks, and discuss how the method could be extended to settings without explicit reference answers?
2 comments

We thank the reviewer for their constructive feedback, which helped us improve the paper. All changes in the revised manuscript are highlighted in blue.

Question 1 / Weakness 1

Could the authors further isolate the contribution of prefix-level credit assignment from advantage rescaling, for example by comparing against a prefix-independent baseline with a matched advantage distribution?

We have added this control. We compute the CIPS baseline as usual, but before applying it, we average it over each rollout and apply the average as a uniform baseline to all tokens of that rollout. This is a prefix-independent baseline that keeps the rollout-level advantage distribution of CIPS and removes only the dependence on the prefix. CIPS with a uniform baseline performs between Dr. GRPO and full CIPS, so both effects contribute, and the gap between the last two rows is attributable to prefix-level credit assignment.

Qwen3-1.7B, 120 steps, 3 seedsMATH-500 p@1MATH-500 p@4AIME p@1AIME p@8
Dr. GRPO54.876.23.110.3
Dr. GRPO, advantages matched to CIPS (dynamic)56.476.93.414.0
CIPS, uniform per rollout56.076.63.313.8
CIPS (prefix-dependent)57.477.43.513.8

This complements the existing controls in Appendix C.5: Dr. GRPO with advantages matched to those of CIPS, and CIPS rescaled to the advantage scale of Dr. GRPO, which still improves over Dr. GRPO on all four metrics.

Question 2 / Weakness 2

Could the authors provide empirical analysis of policy gradient variance or optimization stability to better demonstrate the effectiveness of CIPS as a baseline under practical PPO-style training?

We agree with the first point and state it in Section 3 (lines 207–209) and in the limitations. The bias comes from PPO-style clipping and from training on mixed-outcome groups only. Both are shared by all compared methods, including Dr. GRPO, and are not unique to CIPS.

Regarding the second point, we have added an empirical variance analysis as Appendix C.7. We measure the variance of the policy-gradient estimate of CIPS and TIPS relative to that of Dr. GRPO on identical rollouts, at the final checkpoints of policies trained with each of the three methods. Values below 1 indicate lower variance than Dr. GRPO.

Policy trained withTIPS estimatorCIPS estimator
CIPS0.820.79
TIPS1.041.10
Dr. GRPO1.571.68

On the policy that CIPS training actually produces, the CIPS baseline reduces gradient variance by 21% relative to the group-mean baseline. The effect depends on the policy: on a TIPS-trained policy the variance is roughly on par with Dr. GRPO, and on a Dr. GRPO-trained policy, which CIPS training never visits, it is higher. This supports the claim that CIPS does reduce gradient variance along its own training trajectory, which is the regime that matters for optimization. This is added to the revision in Appendix C.7 and Table 10

Question 3 / Weakness 3

Could the authors evaluate CIPS on larger models or more diverse reasoning tasks, and discuss how the method could be extended to settings without explicit reference answers?

To test whether the improvements carry over to a larger model, we ran CIPS and the two main comparison methods, Dr. GRPO and TIPS, on Qwen3-8B. The results are included in Appendix C.2, Table 6 and shown below. CIPS retains its improvement at this scale.

Qwen3-8BMATH-500 p@1MATH-500 p@4AIME p@1AIME p@8Minerva p@1Minerva p@8Avg
Base model53.181.33.016.821.852.038.0
Dr. GRPO74.589.211.326.035.255.448.6
TIPS75.288.611.828.335.655.849.2
CIPS76.389.212.328.036.855.449.7

Regarding task diversity, the paper already evaluates a second, non-mathematical task, Countdown (Appendix C.4), where CIPS is the strongest method on all metrics for both model sizes. We agree that CIPS in its current form is limited to short-answer tasks, as stated in the limitations (Section 7, line 531). Extending it to open-ended outputs is non-trivial and we leave it for future work.

Regarding reference answers: a reference answer is required only for all-incorrect groups. In a mixed-outcome group, which is what all methods train on, the correct bank can be built from the verified-correct peer answers alone, so a verifier without a reference answer would in principle suffice. We have not evaluated this variant and now mention it as possible extension of our work.

We hope these revisions address your concerns, and we would be happy to clarify any remaining points.

I appreciate the authors’ detailed response and the new experiments. The additional controls and larger-model results have largely addressed my concerns. I consider this a valuable contribution and will maintain my positive rating.

Reviewer_BGj1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CIPS, a prefix-dependent baseline for RL with verifiable rewards. It contrasts the likelihood of correct answers with plausible incorrect answers from peer rollouts, aiming to discount changes caused by answer formatting or readiness rather than reasoning progress.

Critical strengths
  1. The idea is simple and well motivated. Contrasting correct and incorrect answers helps distinguish reasoning progress from a general increase in answer likelihood.
  2. CIPS fits into an existing RLVR pipeline.
  3. The evaluation includes three seeds, two model sizes, and useful ablations on advantage scaling, scorer choice, etc. The reported training-time savings make the results practically appealing.
Critical weaknesses
  1. The benefit of updating the CIPS baseline along the reasoning trace could be isolated more clearly. The advantage-matching ablations are helpful, but I would also like a comparison where CIPS is held constant throughout each rollout. This would show how much the prefix-dependent updates add beyond sequence-level reweighting.

  2. The robustness results support reduced sensitivity. However, CIPS changes noticeably after the sentence insertions, and those insertions change continuation outcomes in some cases.

  3. The efficiency claim would be clearer. Asynchronous scoring can explain why elapsed step time barely increases, but it still consumes resources. Reporting scorer GPU time and total GPU-hours would help readers judge how the overhead changes.

  4. Some variants, particularly within-template correct-only scoring, perform close to full CIPS on several metrics. Standard errors or paired differences across seeds would make it easier to judge which components provide a consistent benefit.

Questions for authors

See the weaknesses above.

2 comments

We thank the reviewer for their constructive feedback, which helped us improve the paper. All changes in the revised manuscript are highlighted in blue.

Weakness 1

The benefit of updating the CIPS baseline along the reasoning trace could be isolated more clearly. The advantage-matching ablations are helpful, but I would also like a comparison where CIPS is held constant throughout each rollout. This would show how much the prefix-dependent updates add beyond sequence-level reweighting.

We have run the requested experiment . We compute the CIPS baseline as usual, average it over all boundaries of a rollout, and apply this average as a constant baseline to every token of that rollout. This performance of this variant falls between Dr. GRPO and CIPS with prefix-dependent values, which shows that the step-level nature of CIPS contributes meaningfully to the overall improvement, beyond sequence-level reweighting.

Qwen3-1.7B, 120 steps, 3 seedsMATH-500 p@1MATH-500 p@4AIME p@1AIME p@8
Dr. GRPO54.876.23.110.3
CIPS, constant per rollout56.076.63.313.8
CIPS (prefix-dependent)57.477.43.513.8

This experiment has been added as Appendix C.5 and Table 9a.

Weakness 2

The robustness results support reduced sensitivity. However, CIPS changes noticeably after the sentence insertions, and those insertions change continuation outcomes in some cases.

We agree, and this is what we report in Figure 2a and Section 4 (lines 258–259): CIPS is not invariant to these insertions, but it is clearly more robust than the correct-only or incorrect-only likelihood (average shift of 57% of a natural reasoning step, versus 82% and 76%).

Weakness 3

The efficiency claim would be clearer. Asynchronous scoring can explain why elapsed step time barely increases, but it still consumes resources. Reporting scorer GPU time and total GPU-hours would help readers judge how the overhead changes.

This is a fair point. The scorer does not require additional hardware in our setup: it runs as a vLLM server on the same GPU as the trainer, so all methods use the same number of GPUs and equal step time translates into equal GPU-hours. It does, however, consume compute on that shared GPU. We now report the total GPU-hours used for rollout generation, trainer updates and scoring. Note that the GPU activity as provided by nvidia-smi shows the time any kernel is running, not if it is fully utilized. Since CIPS scoring requests are rather lightweight, the results should be viewed as an upper bound measurement.

Qwen3-1.7B, full runRollout GPU-hTrainer GPU-hScorer GPU-htotal
CIPS7.73.40.611.7

CIPS scoring accounts for around 5% of used GPU hours. That amount of additional compute can overlap with trainer-rollout configurations in most cases. These results have been added to Appendix C.6.

Weakness 4

Some variants, particularly within-template correct-only scoring, perform close to full CIPS on several metrics. Standard errors or paired differences across seeds would make it easier to judge which components provide a consistent benefit.

We have added standard errors over the three seeds for all ablation configurations, directly to Table 9, as well as in Appendix C.8, Table 11 for Table 2. We report them in a separate table to keep the ablation tables in the main text compact.

We hope these revisions address your questions, and we would be happy to clarify any remaining points.

Reviewer_BGj1OpenReview ↗

My concerns have been solved

Thank you for your response. The explanations cleared most of my questions, especially about the efficiency and the robustness of CIPS. I keep the positive score.

Reviewer_EhAdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CIPS as a baseline for fine-grained credit assignment in RLVR. Compared with PRM or MC, it can provides a prefix-dependent baseline in a simple but effective way: for each prefix, contrast the aggregated likelihood of correct answers with those of incorrect answers from peer rollouts. It does not require additional learned reward models or additional rollouts. Experimental results demonstrate the effectiveness of their method.

Critical strengths
  1. The authors propose CIPS, a simple yet effective method as a prefix-dependent baseline in RLVR. It does not need additional learned reward models or rollout sampling.
  2. CIPS is more robust to answer readiness and other spurious factors, and thus correlates better with the final rollout outcome than single-answer likelihood.
  3. Experimental results show CIPS improve math and combinatorial reasoning tasks across two Qwen model scales.
Critical weaknesses
  1. The experiment is based on Qwen3-1.7B and Qwen3-4B only. Experiments on other model families should be tried to demonstrate the generalizability of CIPS.
  2. If the whole rollouts are all wrong answers (i.e., without correct answers), how can CIPS work? Moreover, these cases are filtered as in DAPO, so the effectiveness relies on the model's success rate.
  3. Can CIPS be extended as a reward rather than just a baseline?
Questions for authors

Imagine a rollout process: LLMs might sequentially explore several approaches (A → B → C → D) where each subsequent approach is an improvement upon the previous one. Approach A might finally be identified by the model as incorrect, yet it could still play a useful role for the development of B, C, and D. Can CIPS distinguish such situation with a useless exploration step?

2 comments

We thank the reviewer for their constructive feedback, which helped us improve the paper. All changes in the revised manuscript are highlighted in blue.

Weakness 1

The experiment is based on Qwen3-1.7B and Qwen3-4B only. Experiments on other model families should be tried to demonstrate the generalizability of CIPS.

We have added experiments comparing CIPS, Dr. GRPO, and TIPS on Llama-3.2-1B and Llama-3.2-3B. We find that CIPS performs best on all evaluations except AIME pass@8, which is the one with the highest variance due to small benchmark size (n=60). However, CIPS achieves still the highest averaged score.

Llama-3.2-1BMATH-500 p@1MATH-500 p@4AIME p@1AIME p@8Minerva p@1Minerva p@8Avg
Base model16.135.40.32.33.613.911.9
Dr. GRPO26.647.00.74.26.719.817.5
TIPS27.447.60.74.17.020.417.9
CIPS28.950.20.74.17.421.718.8
Llama-3.2-3BMATH-500 p@1MATH-500 p@4AIME p@1AIME p@8Minerva p@1Minerva p@8Avg
Base model43.064.42.19.016.236.528.5
Dr. GRPO48.466.03.814.518.937.831.6
TIPS48.265.62.910.119.237.830.6
CIPS49.066.94.012.719.537.931.7

These results were added to the manuscript in Table 1 and Section 5.

Weakness 2

If the whole rollouts are all wrong answers (i.e., without correct answers), how can CIPS work? Moreover, these cases are filtered as in DAPO, so the effectiveness relies on the model's success rate.

CIPS remains well defined when every rollout in a group is incorrect, as long as a reference answer is available (as is commonly the case for math and reasoning datasets). The correct bank then contains the reference answer only, and the incorrect bank contains the peers' submitted answers (Eq. 4), so the contrast in Eq. 5 still yields a prefix-dependent baseline. Outcome-only methods such as GRPO, Dr. GRPO, and DAPO cannot produce a learning signal from these groups, since all advantages are zero, which is why dropping them is the established setup. Even though we adopted this filtering for all methods in the main experiments to keep the comparison fair, it is not a requirement of CIPS.

To test this directly, we trained a policy only on all-incorrect groups, simulating a dataset that is too hard for the policy. Figure 4 shows the performance over training steps: CIPS improves the policy from 31.0 to 45.0 pass@1 on MATH-500 without a single correct rollout, whereas Dr. GRPO does not produce a learning signal in this setting.

Weakness 3

Can CIPS be extended as a reward rather than just a baseline?

Yes, and the two views are closely related. Following the potential-based shaping argument of Ng et al. (1999), define the potential Ψ(sk)=c⋅Φ(k)\Psi(s_k) = c \cdot \Phi(k) with terminal potential zero and the shaping reward Fk=Ψ(sk+1)−Ψ(sk)F_k = \Psi(s_{k+1}) - \Psi(s_k) for the step from boundary kk to k+1k+1. The undiscounted return-to-go from boundary kk then telescopes:

R+∑j≥kFj=R−Ψ(sk)=R−c⋅Φ(k),R + \sum_{j \ge k} F_j = R - \Psi(s_k) = R - c \cdot \Phi(k),

which is exactly the CIPS advantage in Eq. 6. Subtracting CIPS as a baseline is therefore equivalent to rewarding every step by the change in CIPS it causes. In this view, a step that raises the likelihood of correct answers relative to incorrect ones receives a positive reward, and by the standard result the optimal policy is unchanged.

Question 1

Imagine a rollout process: LLMs might sequentially explore several approaches (A → B → C → D) where each subsequent approach is an improvement upon the previous one. Approach A might finally be identified by the model as incorrect, yet it could still play a useful role for the development of B, C, and D. Can CIPS distinguish such situation with a useless exploration step?

To the extent that the scorer recognizes it, yes. CIPS does not judge whether a step is correct in itself, but whether the prefix including that step makes correct answers more likely relative to incorrect ones. If the failed approach A contributes something that B, C, and D build on, for instance by ruling out a candidate answer or exposing a constraint, the incorrect-bank likelihood drops or the correct-bank likelihood rises, and CIPS increases after A. A useless exploration step leaves the contrast unchanged.

There is one limitation to this: the signal can only be as good as the scorer. If the value of A becomes apparent only once B builds on it, CIPS remains constant at A and increases at B. We added this as a note in the Limitations section.

We hope these revisions address your questions, and we would be happy to clarify any remaining points.

Thanks for your response, and I will keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.