acceptodds
Under review as a conference paper at ICLR 2027

Cliff: Learning Process Rewards from the First Mistake

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose **Cliff**, a process-level credit assignment strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning higher advantages for the correct prefix and lower scores afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of "ground truth" in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reinforcement Learning, LLM, Process Reward

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3uDNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a process-level credit allocation strategy named "Cliff", aiming to improve the verifiable reward reinforcement learning of large language models. The core insight of this method is that once the reasoning process makes an initial error, the information provided by subsequent reasoning evaluations is extremely limited. Based on this, Cliff uses the readily available teacher model to identify the first erroneous step in the student's reasoning trajectory, clearly dividing the trajectory into the correct prefix and the erroneous suffix, and then allocating word-level advantage values accordingly.

Critical strengths
  1. The paper ingeniously draws on the concept of substantive implication in formal logic, pointing out that deriving under incorrect premises lacks practical significance. Thus, it simplifies the complex process of reward modeling into finding the first erroneous step, significantly reducing the costs of manual annotation and additional modeling.

  2. This strategy does not require training an additional process reward model, nor does it mandate that the teacher model and the student model have the same reasoning mode or tokenizer. It only relies on the teacher model to locate the first error, demonstrating excellent cross-model and cross-task transfer capabilities.

Critical weaknesses
  1. Although the paper indicates that making judgments is easier than solving problems from scratch, the accuracy of the teacher model still directly determines the quality of the training signals. If the teacher model performs poorly in highly complex reasoning tasks, it is prone to introducing noise and even transmitting incorrect supervision signals.

  2. The paper divides the reasoning process by sentences, but the error boundaries in natural language reasoning are often rather vague. Different annotators or models may have differences in determining the first substantive error, and this subjectivity may lead to an imprecise allocation of advantage values.

  3. The paper mentions that for extremely long trajectories, they are directly regarded as completely incorrect and assigned negative advantage values. This hard-cutting strategy is effective in preventing length cracking, but it may mistakenly harm those high-quality trajectories whose reasoning directions are correct but were not completed due to length limitations, resulting in a certain degree of waste of effective signals.

Questions for authors

Lack of discussion of other fine-grained RLVR works:

  1. Unlocking Token Rewards via Training-Free Reward Attribution

  2. Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization in LLM Reasoning

I will raise my score if my concerns were well-addressed.

1 comment
AuthorsOpenReview ↗

Author Response

We're very grateful for your constructive comments, and are glad for your acknowledgement of our paper’s clear motivation and lightweighted design. Below are responses to your suggestions and concerns.


W1: What happens if the teacher model performs poorly on highly complex reasoning tasks?

Cliff requires a teacher with at least some reasoning capability, but it doesn’t need to solve every problem or as strong as GPT-5.6. As shown in Table 2, moderately capable teachers, such as Qwen3-43B and Gemma-3 27B, also yield strong performance.

When the teacher struggles with a problem, the verification filter serves as a safeguard: Cliff applies teacher-provided process signals only to problems for which the teacher produces a verified correct answer. Otherwise, training falls back to the standard GRPO objective for those problems. Thus, a weaker teacher primarily reduces the coverage of Cliff supervision, but it won’t cause the student to learn from unreliable signals, and it won’t be worse than vanilla GRPO.


W2: Error boundaries in natural-language reasoning are often vague and cannot easily be divided by sentence.

We agree that errors in natural-language reasoning do not always align neatly with sentence boundaries. Sentence-level localization is a practical approximation that allows Cliff to translate the teacher's assessment into actionable training signals. Token-level localization would require an excessively fine-grained judgment, whereas paragraph-level localization could group substantial correct reasoning together with an error.

To reduce ambiguity, we design the judging prompt to focus on identifiable factual or logical errors, rather than stylistic preferences or speculative concerns. In particular, the teacher is instructed not to treat an exploratory attempt that is subsequently corrected as the first reasoning error. These criteria aim to make boundary identification more consistent and objective, while acknowledging that some ambiguity is inherent in natural-language reasoning.


W3: Why do the authors treat overlength responses as completely wrong?

This design serves two purposes.

First, it helps prevent uncontrolled growth in response length. A lengthy but inefficient rollout may remain locally correct until it reaches the generation limit, without ever producing a final answer. If its entire observed prefix were credited as correct, it could receive the same advantage as a completed correct rollout, creating an incentive to prolong reasoning without completing the task. Treating such truncated rollouts as unsuccessful avoids this incentive; Appendix D also analyzes the role of this treatment in controlling response length through the mathematical formulation.

Second, it aligns the training objective with successful reasoning under a finite resource budget. At inference time, a model must produce an answer within a bounded generation budget, so locally valid reasoning alone is insufficient if the response does not reach a solution before that budget is exhausted. Accordingly, treating overlength responses as unsuccessful reflects task completion under the available budget, rather than asserting that every intermediate reasoning step is logically incorrect. This treatment is also consistent with the GRPO baseline used in our experiments.


Q1: More related work should be discussed.

Thank you for highlighting these related papers.

[1] derives token-level rewards through gradient-based attribution of an existing reward model's scores, while [2] calibrates teacher–student token log-probability differences using supervision signals at different levels. Cliff differs from these methods in two key respects:

  • Cliff explicitly localizes the first reasoning error. It uses an LLM teacher to identify the first reasoning error in the student's trajectory, then translates that boundary into separate advantages for the prefix and suffix within GRPO.
  • Cliff does not require access to the teacher's logits or gradients. It relies on the teacher's textual judgments, making it applicable to black-box LLMs accessed through APIs.

We will add these studies to the related-work section and clarify these distinctions in the revised manuscript.


We value your insightful comments and suggestions, which have significantly contributed to the improvement of our paper. We will update the paper, incorporating revisions and additional results based on the questions you raised. If you find that your concerns have been resolved, we would be very grateful if you could reconsider the review score.

Reviewer_dCuKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Cliff, a modification of GRPO that uses a teacher model to locate the first consequential reasoning error in a student rollout. The teacher first generates a reference solution; if this solution fails automatic verification, training falls back to ordinary GRPO. Otherwise, the teacher evaluates student responses and identifies a “Pitfall Step” in unsuccessful trajectories. Cliff assigns different token-level advantages before and after this boundary. Experiments cover two student models, three teachers, and mathematical reasoning and coding tasks.

Critical strengths

A simple and broadly compatible way to use teacher feedback. Cliff converts a teacher’s textual error localization into token-level training signals without requiring teacher logits, matching tokenizers, or a separately trained process reward model. This makes the method applicable across model families and to teachers accessible only through an API.

Critical weaknesses
  1. The method relies too heavily on strong teacher models, such as GPT-5.6 Sol, to identify the first mistake.
  2. The paper lacks controlled comparisons that keep the teacher’s correctness labels and all other training settings fixed while changing only the error boundaries, for example to fixed-proportion or shuffled boundaries. If these simpler alternatives yield comparable gains, this would substantially change the interpretation of the method’s contribution.
Questions for authors

With the teacher-provided overall correctness labels, and token-level centering rule held fixed, does Cliff still show a clear advantage over fixed-proportion or random-position segmentation? How does training performance change when the teacher-identified error positions are perturbed?

2 comments
AuthorsOpenReview ↗

Author Response

We're very grateful for your constructive comments, and are glad for your acknowledgement of our paper’s clear design and wide application. Below are responses to your suggestions and concerns.


W1: The method relies heavily on a strong teacher model.

We believe this concern may stem from a misunderstanding of the role and required capabilities of the teacher model, which we would like to clarify.

First, using a teacher to obtain more informative training signals is a core premise of this work. Our aim is to investigate how teacher-provided process signals can improve credit assignment in reinforcement learning. Empirically, Cliff improves performance over vanilla GRPO by 7%, demonstrating the effectiveness of this approach.

Second, Cliff does not require a teacher as strong as GPT-5.6. When the teacher struggles with a problem, the verification filter serves as a safeguard: Cliff applies teacher-provided process signals only to problems for which the teacher produces a verified correct answer. Otherwise, training falls back to the standard GRPO objective for those problems. Thus, a weaker teacher primarily reduces the coverage of Cliff supervision, but it won’t cause the student to learn from unreliable signals, and it won’t be worse than vanilla GRPO. Empirical results in Table 2 also supports that Cliff applies across teachers with different capability levels. Less capable teachers, such as Qwen3-43B and Gemma-3 27B, can still yield strong performance.


W2/Q1: An ablation on the error boundary is needed to validate the method.

We thank the reviewer for highlighting this important factor. We have added an ablation in which the error boundary for each incorrect rollout is sampled uniformly from all sentence positions, from the first sentence to the last. The first table uses Qwen3 as the student model, and the second table uses phi-4.

SettingMath AverageCoding Average
Base50.6615.05
GRPO61.6824.20
Random boundary55.9117.78
Cliff (GPT-5.6 teacher)65.6625.96
SettingMath AverageCoding Average
Base40.2318.28
GRPO49.7821.72
Random boundary44.2117.95
Cliff (GPT-5.6 teacher)51.7324.93

The random-boundary variant substantially underperforms both GRPO and Cliff, indicating that the benefit does not come simply from splitting incorrect rollouts at an arbitrary position.

The results in Table 3 provide further, indirect evidence for the importance of reliable teacher signals. Removing the reference-verification filter for weaker teachers requires them to judge responses to questions they cannot answer correctly themselves. This introduces less reliable supervision and leads to a decline in performance.

Together, these results support the importance of accurate error-boundary identification and reliable process-level supervision for Cliff.


We value your insightful comments and suggestions, which have significantly contributed to the improvement of our paper. We will update the paper, incorporating revisions and additional results based on the questions you raised. If you find that your concerns have been resolved, we would be very grateful if you could reconsider the review score.

Thank you for the response, which has addressed some of my concerns. However, I remain concerned about critical weaknesses. I will therefore maintain my original score.

Reviewer_fvtDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Cliff, a process-level credit assignment method that uses an off-the-shelf teacher LLM to identify the first substantive, uncorrected mistake in a student rollout. The teacher first generates a reference solution, which is checked by an automatic verifier; questions with unsuccessful teacher solutions fall back to GRPO. For eligible questions, teacher judgments determine correctness and the error boundary, allowing Cliff to assign different advantages to the valid prefix and the erroneous suffix. The default configuration avoids explicit positive reinforcement of failed-rollout prefixes before token-level recentering. Experiments cover mathematical reasoning and algorithmic coding with different student and teacher models. The paper reports generally favorable benchmark results, compares against teacher-based outcome rewards and distillation baselines, and studies reference quality, prefix weighting, and training dynamics.

Critical strengths

Cliff combines prompted error localization, as studied in ProcessBench [1], with a simple policy-update rule. It avoids training a dedicated process reward model or aligning teacher–student token distributions, making integration across model families relatively straightforward.

Results on MATH-500, AIME, CodeContests, and LiveCodeBench are generally favorable across different teachers and students. Comparisons with teacher-based outcome rewards, together with reference-quality and prefix-weighting ablations, provide useful evidence that the proposed credit assignment can improve training.

Critical weaknesses

VPPO [2] already partitions failed trajectories at the first error and assigns different credit to prefixes and suffixes. InT [3] uses reference-guided first-error localization, while process-verified RL [4] incorporates first-error propagation into GRPO. Cliff’s teacher pipeline and advantage formulation differ, but direct comparisons are needed to establish their additional value.

The coding judge explicitly overlooks some boundary-case bugs and complexity violations, although evaluation uses execution tests and time limits. Judge validation is limited to a small mathematical dataset. Moreover, stable training curves and the restricted length analysis do not establish broad resistance to reward hacking, an issue requiring targeted testing in light of Reward Under Attack [6].

Questions for authors

Equation (12) divides by λAcor−b\lambda A_{\mathrm{cor}}-b without separating its possible signs; this quantity is negative when λ=0\lambda=0 and b>0b>0. Can the derivation be corrected and its assumptions stated explicitly? What connects the proposed “length mass” to the actual policy update, and do padding or step-splitting tests support the claimed robustness?

Do the improvements remain under matched end-to-end compute, including teacher reference generation and judgment? Please also report uncertainty for the main comparisons and clarify the coverage of questions receiving Cliff supervision. These results would help establish when the additional training overhead is worthwhile.

[1] ProcessBench: Identifying Process Errors in Mathematical Reasoning.

[2] Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning.

[3] InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning.

[4] Process-Verified Reinforcement Learning for Theorem Proving via Lean.

[5] Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning.

[6] Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models.

2 comments
AuthorsOpenReview ↗

Author's Response

We're very grateful for your constructive comments, and are glad for your acknowledgement of our paper’s strong results and clear design. Below are responses to your suggestions and concerns.


W1: Several related studies use similar ideas, so the novelty and advantages of Cliff are unclear.

Thank you for highlighting these related studies. Process-verified RL uses a Lean verifier in place of the teacher LLM used in our work, but its scope is limited to tasks in which responses must be expressed in Lean's formal syntax. Similar to our work, VPPO and InT splits failed trajectories at the first error and assign different credit to the prefix and suffix. However, Cliff differs from these approaches in two key respects:

  • We show that an off-the-shelf LLM teacher can provide process-level supervision without specialized training.
  • We introduce a reference-verification filter to maintain signal quality when the teacher's capabilities are insufficient for a given problem.

We will add these studies to the related-work section and clarified these distinctions in the revised manuscript.


W2: Is the judging prompt valid and robust to reward hacking?

The design of the coding prompt is deliberate, as we find that existing coding benchmarks often have incomplete test coverage: their test cases do not cover all corner cases or extreme input sizes. Our prompt is therefore designed to align the LLM judge's assessments with the outcomes of the actual verifier. In addition, the permissive instructions are intended to reduce speculative false negatives from the LLM judge, particularly when runtime feasibility is difficult to infer from text alone. This design aims to improve agreement with the verifier; it does not establish correctness beyond the verifier's test coverage.

To further evaluate the judge, we conducted human validation on 50 additional coding problems (as an extension of Table 1). We measured human–LLM judgment agreement and the average p-dis, defined as the absolute difference between the token positions at the start of the first erroneous sentence identified by the human annotator and the LLM judge.

TeacherHuman–LLM AgreementAverage p-dis
GPT-5.690%5.52
Qwen3-32B88%6.14
Gemma3-27B88%5.88

These results suggest that, although coding responses are somewhat more challenging to judge than mathematical responses because of the structure of code blocks, the teachers can still provide useful process-level signals, with reasonably high agreement with human judgments and relatively small discrepancies in error localization.

Regarding reward hacking, we acknowledge that we cannot guarantee robustness against adversarial attacks. Our claim is narrower: under out training setting, we do not observe reward-hacking behaviors based on specific response styles or excessive length. We will clarify this scope in the revised manuscript.

AuthorsOpenReview ↗

Continued Response

Q1: Details of the “Length Mass” Analysis.

We thank the reviewer for pointing this out. Eq. (12) aims to identify a range of λ\lambda that ensures

< L_{\mathrm{cor}}(A_{\mathrm{cor}} - b),$$ so that the total advantage of an incorrect rollout remains below that of a correct rollout. The original derivation does not apply when $\lambda A_{\mathrm{cor}} - b < 0$, but this won’t affect the final conclusion. In this case, the left-hand side is negative, whereas the right-hand side is positive under the assumptions of the analysis, so the inequality holds automatically. **Thus, the bound remains $\lambda < \frac{L_{\mathrm{cor}}}{L_{\max}}.$** **“Length mass” is an aggregate quantity used to characterize the overall tendency of training to change response length.** It reflects the combined training signals rather than the behavior of an individual rollout. A positive length mass indicates a tendency to increase response length, whereas a negative length mass indicates a tendency to decrease it. --- **Q2: What is the additional computational cost of Cliff, and what proportion of questions receive Cliff supervision?** Due to resource constraints, we are unable to repeat all experiments under matched compute budgets. The use of external APIs also makes direct compute comparisons difficult. **Appendix C** provides an approximate estimate of the additional cost: each training step of Cliff takes about 80% longer than a step of vanilla GRPO in our current implementation. Note that his implementation has not been extensively optimized. Techniques such as asynchronous RL, as well as settings with a larger number of rollouts $N$ in GRPO, could help reduce or amortize the relative overhead. Given the reported performance gain of 7%, we consider the current overhead an acceptable trade-off, while acknowledging the need for a more systematic compute-efficiency evaluation. The proportion of questions receiving Cliff supervision depends on the teacher model's accuracy. As shown in Fig. 1, Cliff signals are used only for questions that the teacher answers correctly. Empirically, this proportion is approximately 96% for GPT-5.6 and 60% for each of the other two teachers. --- **We value your insightful comments and suggestions, which have significantly contributed to the improvement of our paper. We will update the paper, incorporating revisions and additional results based on the questions you raised. If you find that your concerns have been resolved, we would be very grateful if you could reconsider the review score.**

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Gentle Reminder of Author's Response

Dear reviewers:

Thank you for your valuable feedback. We have carefully addressed your questions and revised the paper accordingly, with all new content highlighted for clarity. We would greatly appreciate it if you could review our responses and revisions. If you find that your concerns have been adequately addressed, we would be grateful if you could consider increasing your scores.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.