acceptodds
Under review as a conference paper at ICLR 2027

Let's Fail Step by Step: Better Contrastive Distillation with Reliable Worse Teachers

Abstract

Contrastive Distillation, which trains a student policy using a reward defined by the log-probability gap between two teacher policies, has recently gained attention as an effective alternative to On-Policy Distillation. However, prior approaches typically contrast a task-specific, RL-trained teacher with a base policy, limiting scalability to larger teacher models and broader task domains. We instead propose to contrast a strong teacher with a negatively steered version of itself, using a natural-language description encouraging undesirable behaviors. This removes the need for task-specific RL training of the teacher while providing a simple and interpretable way to control contrastive rewards. On mathematical reasoning tasks, we show that steering the negative teacher toward incorrect or non-rigorous reasoning substantially improves the contrastive reward's effectiveness for student training, whereas positively steering the teacher using privileged information or behavioral descriptions yields only marginal gains. We explain this asymmetry through the Steering-Success Rate, defined as the probability that a completion from the steered policy is preferred over one from the base policy. We theoretically show that this quantity directly affects the converged student policy's win rate over its KL-regularized reference policy under reasonable conditions. Through a series of ablations, we demonstrate that the final performance of the student policy closely follows the Steering-Success Rate of its corresponding contrastive reward. In particular, negative steering consistently achieves higher success rates than positive steering, and improving the reliability of negative steering further enhances the resulting reward's training effectiveness. Together, our results establish natural-language steering as a simple and intuitive approach to contrastive distillation, while highlighting Steering-Success Rate as a key factor in designing effective contrastive rewards.

Primary area:
generative models
Keywords:
Contrastive Reward, On-Policy Distillation, Contrastive Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dPbMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To reduce the cost of training and storing separate teachers for contrastive distillation, the authors propose CRISP. It uses the same teacher under a normal prompt and a prompt asking it to reason incorrectly. Student-generated responses are scored under both conditions, and the log-probability difference provides the training reward. Experiments with Qwen3-4B and Qwen3-1.7B students on five math benchmarks report gains over the tested baselines. The authors also examine why negative steering works better than positive steering. They introduce Steering-Success Rate (SSR), give conditions under which higher SSR leads to a better optimal student, and study this relationship by varying the prompts and teacher sizes.

Critical strengths
  1. The reward in CRISP is simple and cheap to implement. It uses the same teacher under two prompts, avoiding additional teacher fine-tuning and keeping only one set of teacher weights in memory. This makes the method straightforward to apply to an existing teacher. The reward can also be adjusted by editing the negative instruction.

  2. The reported gains of CRISP over other baselines are significant. In Table 3, the 4B student model reaches 42.28% average accuracy, compared with 33.35% for Direct OPD. The 1.7B student model reaches 28.26%, compared with 22.04% for OP$^2$D. CRISP performs better than the tested baselines on various benchmarks, providing useful evidence that negative prompting can improve the reward used for distillation.

Critical weaknesses
  1. Section 3.1 and Table 2 describe RLCSD as contrasting a teacher given a correct answer or solution with the same teacher without a hint. Original RLCSD instead contrasts correct and incorrect hints [1]. The incorrect hint is intended to reduce effects of providing a reference that are unrelated to correctness, such as changes in style. Removing it changes the reward construction, even if all methods use a common optimizer. If the experiments follow Table 2, the results do not establish how CRISP compares with RLCSD’s correct/incorrect contrast. The authors should clarify the implementation and provide a comparison using the original RLCSD.

  2. Proposition C.1 requires Eqs. (3)-(6). The authors didn't check them for the experimental comparisons or explain why they should be true. Thus, the theorem gives limited support for using SSR to predict which prompt will produce a better student. Figure 5 also shows a discrepancy: Negative-1Shot improves student performance but slightly increases negative teacher accuracy at 0.6B and 14B. These plotted values correspond to lower estimated SSR, contrary to the claim of consistent improvement. Finally, the paper does not report how to use SSR to choose the one-shot example. Its success therefore leaves open whether SSR can actually guide prompt selection before student training.


Reference

[1] Leyi Pan et al. RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation.

Questions for authors
  1. What exact reward was used for the RLCSD baseline? If the implementation follows Table 2, would CRISP still outperform it? A fair comparison would help establish whether the gain comes from CRISP's reward construction. If Table 2 only misdescribes the implementation, please clarify what the reported results actually evaluate.

  2. How do the authors explain the Figure 5 cases where Negative-1Shot improves the student while increasing negative teacher accuracy? Please provide the underlying values and variance for the 0.6B and 14B cases. Can the authors check whether the one-shot reward better ranks correct answers above incorrect answers to the same question?

  3. Can SSR help choose a negative prompt that leads to better student performance? For example, could the authors measure SSR for several candidate prompts on a development set, select the one with the highest SSR without looking at student training results, and then train a student using that prompt? How would the resulting student compare with students trained using the default Negative and Negative-1Shot prompts?

8 comments

We thank the reviewer for the detailed comments.

[W1, Q1] What exact reward was used for the RLCSD baseline?

The method labeled RLCSD in our initial submission contrasts a single Qwen3-14B with and without privileged information (answer or solution), using the same optimization as all other methods (Section 2.3, Line 183). We renamed it to Teacher-Contrast to prevent confusion (Lines 209–215, Tables 2–3). We adapted the original RLCSD for fair comparison, as it uses multiple self-rollouts and combines token-level rewards with an external verifiable reward, unlike our training setup.

[W2-1] Proposition C.1 requires Eqs. (3)-(6). The authors didn't check them (...) or explain why they should be true.

We revised the proposition linking the Steering-Success Rate (SSR) to student performance (Propositions 4.2 and D.1) so it no longer requires these assumptions, and toned down the related claims.

  • The proposition now holds for arbitrary pairs of steering prefixes. Mismatches in their average probability-ratio changes enter the required SSR margin as an explicit additive term (Lines 357–369).
  • We acknowledge in a new Limitations section (Lines 498–508) that the remaining condition is not verified empirically, since it involves quantities known to be difficult to estimate reliably for LLMs. We instead conduct extensive ablations showing that student performance follows the trend of SSR (see W2-3).

[W2-2, Q2] Negative-1Shot improves student performance but slightly increases negative teacher accuracy at 0.6B and 14B. (...) Please provide the underlying values and variance (...)

To clarify, SSR is defined by reasoning quality and empirically estimated via both final-answer accuracy and error recognition, not accuracy alone (revised Lines 381–383, 494–496). Negative-1Shot's stronger training performance can be explained by its suppression of error recognition:

  • For Qwen3-14B, Negative-1Shot reduces error-recognition phrases from 10.04 to 0.69 per 1,000 words while maintaining the decrease in final-answer accuracy (Table 6).
  • Qwen3-1.7B, 4B and 8B have similar accuracy as Qwen3-14B under Negative steering, but recognize their errors more often and yield weaker students (Figure 5).
  • In a new controlled experiment (Appendix C.1), the Negative-1Shot reward achieves the highest ranking accuracy on trajectory pairs differing only in correctness (Table 5).

We revised the claim (Lines 471–472, 491–496, 520–521) to state that Negative-1Shot decreases teacher accuracy for Qwen3-1.7B, 4B, and 8B, and reduces the error-recognition rate at every tested size. We also included final-answer accuracies with standard errors in Table 4:

Steered PolicyNegativeNegative-1Shot
Qwen3-0.6B12.83 (±0.37)13.93 (±0.38)
Qwen3-14B51.89 (±0.55)53.86 (±0.55)

[W2-3, Q3] Can SSR help choose a negative prompt that leads to better student performance?

Yes. We ran the suggested experiment (Appendix C.4, Table 8), and the negative example with the highest SSR yields the best student. We measured the SSR of Qwen3-14B under three candidate prompts differing only in the negative example. Besides the original Negative-1Shot example which begins with an incorrect approach (Figure 7), we tested two mildly incorrect examples (Figures 13 and 14), and trained a student with each prompt.

Steering ModeTeacher Answer Acc [%]Error-Recognition Phrases / 1000 WordsStudent Math Acc@12 [%]
No Steering75.950.62–
Negative51.8910.0441.82
Negative-1Shot53.860.6942.58
Negative-1Shot (Mildly Bad v1)61.520.7740.18
Negative-1Shot (Mildly Bad v2)59.330.7940.15

All three 1-shot methods suppress error recognition similarly, while the two mildly bad examples yield smaller reductions in final-answer accuracy and weaker students than the original Negative-1Shot example.

As consistently shown from the ablations in Section 4.2 and Appendix C, SSR can indeed help construct better contrastive rewards:

  • Negative vs. positive steering (Section 4.2.1): negative steering reduces final-answer accuracy by 24.06%p and yields a much stronger student (Figure 3), while positive steering changes it by less than 1%p (Table 1).
  • With vs. without a reasoning example (Section 4.2.2): the negative demonstration suppresses error recognition and improves the student at every teacher size (Figure 5).
  • Size of the steered policy (Section 4.2.2): under negative steering, Qwen3-0.6B (lowest final-answer accuracy) and Qwen3-14B (lowest error recognition) yield the most consistent reward rankings (Figure 8) and the best trained students (Figure 5, Left).
  • Retaining vs. suppressing error recognition (Appendix C.2): negative prompts that fail to suppress error recognition yield weaker trained students (Table 6).

We once again thank the reviewer and kindly ask them to consider raising the score if their concerns have been addressed.

Dear Reviewer dPbM,

Thank you again for your detailed review. We would like to gently follow up on our response and the revised manuscript. We have addressed your concerns as follows:

  • [W1, Q1] We renamed RLCSD to Teacher-Contrast (Lines 209–215) for the following reasons:
    • This baseline is not meant to be a comparison against RLCSD. Rather, it tests CRISP against a contrastive reward built from privileged information under the same training setup as all other methods.
    • The original RLCSD relies on multiple self-rollouts and an external verifiable reward, which differ from our setup. Comparing against it would therefore mix two different problem settings and would not show which contrastive reward is more effective, which is the main question our paper studies.
  • [W2-1] Proposition 4.2 / D.1 (formerly C.1) has been revised so that it no longer requires the conditions in Eqs. (3)–(6), and it has been strengthened to hold for any pair of steering prefixes.
  • [W2-2, Q2] We clarified why Negative-1Shot yields better students, which is due to the negative one-shot example effectively suppressing error recognition (Figure 5, right).
    • SSR is defined on reasoning quality, so error recognition should be considered together with final-answer accuracy (Lines 381–383).
    • In the requested ranking analysis, the Negative-1Shot reward also shows the highest ranking accuracy (Appendix C.1).
  • [W2-3, Q3] We conducted the requested experiment and an additional ablation, and both show that SSR indeed helps choose a better negative prompt:
    • Among three negative demonstrations with similar error-recognition suppression, the one with the largest drop in final-answer accuracy leads to the best student (Appendix C.4, Table 8).
    • Negative prompts that lower final-answer accuracy but retain or encourage error recognition lead to weaker students (Appendix C.2, Table 6).

Overall, our paper contributes a novel and simple method with strong empirical performance, accompanied by both theoretical analysis and extensive empirical ablations, including:

  • A simple and interpretable contrastive reward that requires no task-specific RL training of the teacher and no privileged information.
  • Roughly twice the performance gain of the strongest baseline among recent on-policy and contrastive distillation methods (Table 3), with more stable convergence (Figure 2).
  • The finding that negative steering is more effective than positive steering, which gives only marginal gains even with privileged information (Table 3 and Appendix C.3) and, unlike negative steering, does not benefit from a larger steered teacher (Figure 3).
  • A theoretical analysis that links the Steering-Success Rate (SSR) to the converged student's win rate (Proposition D.1), now strengthened to hold for any pair of positive or negative steering prefixes.
  • Extensive ablations (steering direction, teacher size, error recognition, privileged-information hints, choice of demonstration) in which student performance consistently follows the trend of SSR, in line with the theory (Lemma 4.1 and Proposition 4.2) and our intuition (Section 4.1, Lines 351–356), showing that SSR can serve as a principled criterion for designing better contrastive rewards.

We once again thank you for the detailed comments, which have helped strengthen our work. If your concerns have been addressed, we would be really grateful if you could consider raising your score accordingly. Please let us know if any questions remain, so that we can address them before the discussion period ends.

Thanks for your response. I appreciate that you confirm that the baseline you used was different from RLCSD and renamed it as Teacher-Contrast. The theoretical result is now conditional and the authors explicitly state those assumptions are not checked. The inaccurate statement about teacher accuracy in Figure 5 is also corrected. I also appreciate your efforts on conducting the new prompt selection experiment. It shows 1shot is very useful to suppress error recognition.

However, my remaining concern is how to use SSR when the two metrics accuracy and error recognition do not agree. This is what happens in Table 8. Negative has lower teacher accuracy, while Negative-1Shot has lower error recognition. Your experiments cannot show this because there Negative-1Shot is better on both. How should we choose between such prompts before seeing student results?

Reviewer_a3ycOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CRISP, a contrastive distillation method that rewards student outputs using the log-probability difference between an unmodified teacher and a negatively prompted version of the same teacher. Experiments on five mathematical reasoning benchmarks show substantial improvements over the evaluated distillation baselines. The paper introduces Steering-Success Rate (SSR) to explain these improvements through theoretical analysis and ablations.

Critical strengths
  • A simple and practical method. Negative prompting provides an intuitive way to construct contrastive rewards without additional teacher training, while allowing both teacher policies to share model parameters.
  • Substantial empirical gains. For Qwen3-4B, average accuracy increases from 24.99% to 42.28%, compared with 33.35% for the strongest evaluated baseline. Improvements also hold across all five benchmarks for Qwen3-1.7B.
  • Useful ablations. Comparisons across steering directions, teacher sizes, and negative demonstrations provide evidence beyond the main performance results.
Critical weaknesses
  • The construction-cost advantage needs qualification. Answer- and solution-hint methods also construct teacher variants through prompting and can share parameters. When reference information is available, constructing these teachers is straightforward. The paper should distinguish the cost of constructing a teacher from the effectiveness of its induced reward.
  • The theoretical interpretation is stronger than the established result. Proposition C.1 requires several matched conditional moments and equal normalization factors; some downstream moment assumptions are not explicit in the main-text proposition. Their empirical validity is not established, limiting the claim that SSR determines downstream performance.
Questions for authors
  • For answer- and solution-hint teachers specifically, how do teacher accuracy, SSR, and reward discrimination compare with negative steering? Table 1 does not directly evaluate these hint-based teachers, leaving the reasons for their weak downstream gains unclear.
  • Could the authors explain more thoroughly why contrastive rewards constructed using answer or solution hints are less effective?
  • Can the authors clarify the additional assumptions in Proposition C.1, assess whether they hold approximately in the experiments, and provide an explicit derivation of the claimed extension to negative steering? This paper is interesting and well-written, but it does not meet the bar for a strong accept. Therefore, I give it a score of 3 and will not consider raising my score further.
4 comments

We thank the reviewer for the constructive comments.

[W1] Answer- and solution-hint methods also construct teacher variants through prompting and can share parameters. (...) The paper should distinguish the cost of constructing a teacher from the effectiveness of its induced reward.

We revised Lines 155–157 to clarify that prior methods using answer or solution hints share CRISP's construction-cost advantage. Still, we highlight that CRISP differs in two respects (Lines 157–159, 214–215):

  • Ground-truth answers or solutions are not always available (e.g., unlabeled data [1]), while CRISP only requires a general, instance-agnostic behavioral description.
  • CRISP consistently outperforms hint-based methods by a large margin (average math Acc@12 [%], Table 3):
MethodQwen3-1.7B StudentQwen3-4B Student
Self-Contrast (Answer)16.9325.90
Self-Contrast (Solution)16.0125.24
Teacher-Contrast (Answer)16.2625.62
Teacher-Contrast (Solution)16.6126.31
CRISP (Ours)28.2642.28

[Q1] For answer- and solution-hint teachers specifically, how do teacher accuracy, SSR, and reward discrimination compare with negative steering?

Hints raise the raw answer accuracy, but yield a lower Steering-Success Rate (SSR) and weaker reward discrimination than negative steering (Appendix C.3, Table 7, Figures 10–12).

We used Qwen3-14B with answer and solution hints to generate 4 completions per prompt on the same 2,048 problems used in Table 1. While the raw final-answer accuracy increases, the reasoning frequently relies on the hints (Figure 11), which are not available at test time. To quantify this, we used Qwen3.8-27B as a judge (Figure 12) to detect such abnormal reasoning and report the valid answer accuracy, i.e., the percentage of completions whose final answers are correct and not flagged (Table 7).

Steering ModeRaw Answer Acc [%]Abnormal Reasoning [%]Valid Answer Acc [%]
Answer Hint91.3722.6271.94 (-4.01%p)
Solution Hint90.0931.1263.53 (-12.42%p)
  • The valid answer accuracy of both hinted teachers falls below No Steering (75.95%).
  • As SSR reflects reasoning quality and not final answer accuracy alone (Lines 381–383), hinted steering has a low SSR. In contrast, negative steering reliably reduces the teacher accuracy by 24.06%p (Table 1).
  • From Figure 10, hint-induced rewards fail to distinguish trajectories based on the accuracy of the generating model, unlike negative steering.

[Q2] Could the authors explain more thoroughly why contrastive rewards constructed using answer or solution hints are less effective?

Our analysis suggests that hint-based rewards are less effective because hinted steering has a low SSR in terms of reasoning quality and weak reward discrimination (Appendix C.3, Lines 1094–1102 and 1122–1133).

  • The hinted Qwen3-14B teacher is flagged as relying on hints in 22.62% (answer) and 31.12% (solution) of its reasoning traces (Table 7). The reasoning encouraged by the hint-based reward is not applicable at test time (Figure 11), which likely explains the limited performance gains in Table 3.
  • In contrast, the CRISP prompt only describes a general undesirable reasoning behavior and contains no problem-specific information (Figure 1).

[W2,Q3] Proposition C.1 requires several matched conditional moments and equal normalization factors; some downstream moment assumptions are not explicit in the main-text proposition. Their empirical validity is not established (...)

We revised the proposition linking SSR to student performance (Propositions 4.2 and D.1) so that it no longer requires these restrictive assumptions, and toned down the related claims.

  • The proposition now holds for arbitrary pairs of steering prefixes. Mismatches in their average probability-ratio changes enter the required SSR margin as an explicit additive term (Lines 357–369). The remaining assumptions are listed in Lines 1801–1803.
  • Negative steering is now derived explicitly: both propositions and the proof cover positive and negative prefixes (Lines 357–372 and 1782–2079).
  • In the new Limitations section (Lines 498–508), we acknowledge that the remaining sufficient condition is not verified empirically, since it involves quantities known to be difficult to estimate reliably for LLMs [2,3]. We instead conduct extensive ablations showing that student performance follows the trend of SSR (Section 4.2, Appendix C.1–C.4).
  • We toned down the statements linking SSR to student performance, e.g., "determines the converged student's win rate" → "leads to a higher win rate under certain conditions".

  • [1] Zuo, et al. "Ttrl: Test-time reinforcement learning." NeurIPS 2025.
  • [2] Liu, et al. "Breaking the curse of horizon: Infinite-horizon off-policy estimation." NeurIPS 2018.
  • [3] Pillutla, et al. "Mauve: Measuring the gap between neural text and human text using divergence frontiers." NeurIPS 2021.

Dear Reviewer a3yc,

Thank you again for your detailed review. We would like to gently follow up on our response and the revised manuscript. We have addressed your concerns as follows:

  • [W1] We revised Lines 155–159 to separate the cost of constructing a teacher from the effectiveness of its reward.
  • [Q1] We conducted the requested analysis of answer/solution-hint teachers and found that they exhibit lower SSR and weaker reward discrimination than negative steering (Table 7, Figure 10).
  • [Q2] We included the requested analysis on why hint-based rewards are less effective (Appendix C.3) which suggests that the hinted teacher's reasoning frequently relies on the hint itself whose strategy is not applicable at test time (Table 7, Figure 11).
  • [W2, Q3] Proposition 4.2 / D.1 has been revised so that it no longer requires the matched-moment and equal-normalization assumptions, and it has been strengthened to hold for any pair of steering prefixes. The negative-steering case is now derived explicitly (Proposition D.1), the remaining assumptions are explicitly stated in Lines 1801–1803, and the statements linking SSR to student performance have been revised (e.g., Lines 26–29, 79–82, 94–96) to address W2.

Overall, our paper contributes a novel and simple method with strong empirical performance, accompanied by both theoretical analysis and extensive empirical ablations, including:

  • A simple and interpretable contrastive reward that requires no task-specific RL training of the teacher and no privileged information.
  • Roughly twice the performance gain of the strongest baseline among recent on-policy and contrastive distillation methods (Table 3), with more stable convergence (Figure 2).
  • The finding that negative steering is far more effective than positive steering, which gives only marginal gains even with privileged information (Table 3 and Appendix C.3) and, unlike negative steering, does not benefit from a larger steered teacher (Figure 3).
  • A theoretical analysis that links the Steering-Success Rate (SSR) to the converged student's win rate (Proposition D.1), now strengthened to hold for any pair of positive or negative steering prefixes.
  • Extensive ablations (steering direction, teacher size, error recognition, privileged-information hints, choice of demonstration) in which student performance consistently follows the trend of SSR, in line with the theory (Lemma 4.1 and Proposition 4.2) and our intuition (Section 4.1, Lines 351–356), showing that SSR can serve as a principled criterion for designing better contrastive rewards.

We once again thank you for the detailed comments that has helped strengthen our work. We would be very grateful if you could consider reflecting our overall contributions listed above in your final assessment. Please let us know if any questions remain, so that we can address them before the discussion period ends.

Dear Reviewer a3yc,

We would like to send a gentle reminder as the discussion period is nearing its end.

Our response and the revised manuscript include the requested analysis of answer- and solution-hint teachers (Appendix C.3) and a revised Proposition 4.2 which has removed the restrictive assumptions. A brief note on whether these resolve your concerns would be very helpful.

Please let us know if any questions remain, so that we can address them before the discussion period ends. If all concerns have been addressed, we would be very grateful if you could take both the empirical and theoretical contributions of this work into account in your final assessment and consider raising the score.

Thank you again for your constructive comments.

Dear Reviewer a3yc,

As the discussion period is nearing its end, we would like to kindly ask whether our response and the revised manuscript have resolved your questions.

If so, we would appreciate it if you could consider raising your score beyond a weak accept, in light of the paper's overall contributions:

  • a simple and interpretable method for contrastive distillation,
  • strong empirical gains over recent baselines, and
  • a generalized theoretical analysis supported by extensive ablations.

If anything remains unclear, please let us know before the discussion period ends. Thank you again for your time and thoughtful feedback.

Reviewer_Zfe6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CRISP, a contrastive distillation method that constructs a reward by comparing a strong teacher with a negatively steered version of itself. Rather than relying on an RL-trained teacher–base model pair, CRISP uses natural-language instructions to induce undesirable reasoning behavior while sharing the same teacher parameters. The authors further introduce Steering-Success Rate (SSR) to characterize the reliability of steering and relate it to downstream student performance. Experiments on five mathematical reasoning benchmarks show substantial improvements over existing on-policy and contrastive distillation baselines, along with an interesting asymmetry between positive and negative steering.

Critical strengths
  • The paper is well written and easy to follow. The proposed concept (SSR) is theoretically supported.
  • The idea of constructing a worse teacher rather than improving an already strong teacher is appealing and reasonable.
  • Substantial performance gains are well supported by informative steering ablations. The improvements over strong distillation baselines are large for both 1.7B and 4B students.
Critical weaknesses
  • The theoretical characterization through SSR is not sufficiently connected to the empirical evidence. Proposition 4.2 requires restrictive assumptions (e.g., matched average probability-ratio changes on correctly and incorrectly ranked pairs and matched normalization factors for the resulting student policies). These conditions are not verified in the experiments. Moreover, the empirical analysis primarily uses average accuracy differences as a proxy for SSR, rather than directly measuring the pairwise win rate in its definition.

  • It remains unclear whether the contrastive reward captures reasoning quality or other behavioral differences introduced by negative steering. Negative steering may affect response length, writing style, uncertainty, and self-correction behavior. The analysis in Figure 4 basically compares trajectories generated by models of different capabilities, which may introduce additional confounding factors. A more convincing validation would examine whether the reward consistently distinguishes correct and incorrect reasoning trajectories for the same problem under controlled generation characteristics.

Questions for authors
  1. Can SSR be measured directly using pairwise preferences rather than average accuracy differences? In particular, how well does the empirical pairwise SSR predict downstream student performance across steering prompts? Can the assumptions of Proposition 4.2 be approximately validated in the experimental setting?

  2. How well does the contrastive reward distinguish correct and incorrect trajectories for the same problem? It would be useful to report pairwise ranking accuracy, especially on examples with similar response lengths and writing styles.

  3. Figure 5 suggests that suppressing error-recognition behavior is important. Can the authors separate the contribution of suppressing self-correction from that of simply increasing the frequency of incorrect final answers? For example, what happens if the negative teacher is encouraged to produce wrong answers while retaining explicit error-checking behavior?

  4. How robust and generalizable is CRISP to non-mathematical tasks?

4 comments

We thank the reviewer for the detailed feedback.

[W1-1, Q1] The theoretical characterization through SSR is not sufficiently connected to the empirical evidence. Proposition 4.2 requires restrictive assumptions (...)

  • We revised Propositions 4.2 and D.1 to remove these restrictive assumptions. The proposition now holds for arbitrary prefix pairs, with the mismatches in their average probability-ratio changes entering the required SSR margin as an additive term (Lines 357–369).
  • The remaining condition involves quantities known to be very difficult to estimate reliably for LLMs with large output spaces [1,2], so we conduct extensive ablations showing that student performance follows the trend of SSR (Section 4.2, Appendix C).
  • We toned down the statements linking SSR to student performance (e.g., "determines the converged student's win rate" → "leads to a higher win rate under certain conditions") and added a Limitations section (Lines 498–508) that acknowledges this difficulty.

[W1-2, Q1] The empirical analysis primarily uses average accuracy differences as a proxy for SSR, rather than directly measuring the pairwise win rate in its definition.

With final-answer correctness as the reward, the pairwise win rate is a function of the accuracy difference: two policies with accuracies A1A_1 and A0A_0 have a win rate of 12+12(A1−A0)\frac12+\frac12(A_1-A_0), so an accuracy difference of Δ%p corresponds to a SSR of (50 + Δ/2)%. We added Lines 391–396 and 428–430 to clarify this.

We also note that SSR is defined by reasoning quality, estimated by both final-answer accuracy and error recognition behavior (Lines 381–383, 494–496), not accuracy alone. Considering both factors to compare reasoning quality, the SSR values of the steering methods in Section 4.2 align well with the trends in student performance.

[W2,Q2] A more convincing validation would examine whether the reward consistently distinguishes correct and incorrect reasoning trajectories for the same problem (...)

We first clarify that the ranking analyses in Figures 4, 8 and 10 all use trajectories generated on the same fixed random subset of training problems.

We conducted an additional experiment on a 1,024 subset of Math-Step-DPO-10K [3] (Appendix C.1, Table 5), consisting of pairs of single reasoning steps that continue the same problem and partial solution, come from the same model, and differ only in correctness. Negative steering methods achieve high ranking accuracy, with Negative-1Shot highest in both ranking accuracy and student performance.

Steering ModeRanking Acc [%]Student Math Acc@12 [%]
Not-Negative52.6425.11
Exploration39.7533.13
Error-Revision49.5129.10
Exploration & Error-Revision38.1835.10
Negative53.5241.82
Negative-1Shot55.8642.58

Not-Negative has comparable ranking accuracy but yields a much weaker student. We hypothesize this is due to the distribution shift between Math-Step-DPO-10K and the Qwen3-4B student's on-policy trajectories, which the Not-Negative reward separates less effectively (Figure 4).

[Q3] What happens if the negative teacher is encouraged to produce wrong answers while retaining explicit error-checking behavior?

This ablation (Appendix C.2, Table 6) shows that a negative teacher giving wrong answers but retaining error recognition yields a weaker student. We tested two variants of the Negative prompt on Qwen3-14B (Figure 9): one drops the sentence suppressing error recognition, the other asks the teacher to explicitly acknowledge incorrect parts.

Steering ModeTeacher Answer Acc [%]Error-Recognition Phrases / 1000 WordsStudent Math Acc@12 [%]
No Steering75.950.62–
Negative51.8910.0441.82
Negative-1Shot53.860.6942.58
No suppression of error recognition37.5027.9432.67
More error recognition38.4827.6132.99

Both variants lower final-answer accuracy more than Negative but increase error recognition, yielding weaker students. Thus, both producing incorrect reasoning and suppressing error recognition are important for negative steering to induce effective contrastive rewards.

[Q4] How robust and generalizable is CRISP to non-mathematical tasks?

While we focus on mathematical tasks, we agree that extending to other domains is interesting future work, as acknowledged in the Limitations section (Lines 501–503).


We hope the revisions and additional experiments have addressed the concerns. If so, we would appreciate the consideration of raising the score. Please let us know if there are any remaining concerns.


  • [1] Liu, et al. "Breaking the curse of horizon: Infinite-horizon off-policy estimation." NeurIPS 2018.
  • [2] Pillutla, et al. "Mauve: Measuring the gap between neural text and human text using divergence frontiers." NeurIPS 2021.
  • [3] Lai, et al. "Step-dpo: Step-wise preference optimization for long-chain reasoning of llms." arXiv:2406.18629.

Dear Reviewer Zfe6,

Thank you again for your detailed review. We would like to gently follow up on our response and the revised manuscript. We have addressed your concerns as follows:

  • [W1-1, Q1] Proposition 4.2 has been revised so that it no longer requires the restrictive assumptions mentioned in the review, and it has been strengthened to hold for any pair of steering prefixes.
  • [W1-2, Q1] We clarified that, with final-answer correctness as the reward, the accuracy difference directly gives the pairwise win rate (SSR) (Lines 391–396). We also clarified that SSR is defined on reasoning quality, so error-recognition behavior should be considered together with final-answer correctness when comparing SSR (Lines 381–383).
  • [W2, Q2] We conducted the requested experiment and observed that negative-steering rewards show the highest ranking accuracy on step pairs that come from the same problem and the same model and differ only in correctness (Appendix C.1).
  • [Q3] We conducted the requested experiment and found that a negative teacher that gives wrong answers while retaining or encouraging error recognition indeed leads to a clearly weaker student (Appendix C.2).
  • [Q4] We agree that evaluating CRISP on other domains is a natural next step. We have added this as a direction for future work in the Limitations section of the revised manuscript (Lines 501–503).

Overall, our paper contributes a novel and simple method with strong empirical performance, accompanied by both theoretical analysis and extensive empirical ablations, including:

  • A simple and interpretable contrastive reward that requires no task-specific RL training of the teacher and no privileged information.
  • Roughly twice the performance gain of the strongest baseline among recent on-policy and contrastive distillation methods, with the best performance for both tested student model sizes (Table 3) and more stable convergence (Figure 2).
  • The finding that negative steering is far more effective than positive steering, which gives only marginal gains even with privileged information (Table 3 and Appendix C.3) and, unlike negative steering, does not benefit from a larger steered teacher (Figure 3).
  • A theoretical analysis that links the Steering-Success Rate (SSR) to the converged student's win rate (Proposition D.1), now strengthened to hold for any pair of positive or negative steering prefixes.
  • Extensive ablations (steering direction, teacher size, error recognition, privileged-information hints, choice of demonstration) in which student performance consistently follows the trend of SSR, in line with the theory (Lemma 4.1 and Proposition 4.2) and our intuition (Section 4.1, Lines 351–356), which makes SSR a practical criterion for designing contrastive rewards.

Thank you again for your thoughtful feedback. If your concerns have been addressed, we would be grateful if you could consider raising your score in light of the above. If anything remains unclear, please let us know before the discussion period ends.

Dear Reviewer Zfe6,

We would like to send a gentle reminder as the discussion period is nearing its end.

Our response and the revised manuscript include the requested experiments (Appendix C.1 and C.2) and a revised Proposition 4.2 which has removed the restrictive assumptions. A brief note on whether these resolve your concerns would be very helpful.

Please let us know if any questions remain, so that we can address them before the discussion period ends. If all concerns have been addressed, we would be very grateful if you could take both the empirical and theoretical contributions of this work into account in your final assessment and consider raising the score.

Thank you again for your detailed review.

Dear Reviewer Zfe6,

As the discussion period is nearing its end, we would like to kindly ask whether our response and the revised manuscript have resolved your questions.

If so, we would appreciate it if you could consider raising your score beyond a weak accept, in light of the paper's overall contributions:

  • a simple and interpretable method for contrastive distillation,
  • strong empirical gains over recent baselines, and
  • a generalized theoretical analysis supported by extensive ablations.

If anything remains unclear, please let us know before the discussion period ends. Thank you again for your time and thoughtful feedback.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their constructive feedback. We have uploaded a revised manuscript with all changes highlighted in blue\color{blue}\text{blue}. Overall, we have updated the manuscript to:

  • relax the assumptions of our theoretical result,
  • add four new analyses (Appendix C) that strengthen the evidence that student performance follows the trend of the Steering-Success Rate (SSR), and
  • correct minor labels, captions, and wording errors.

1. Theory (Section 4.1, Appendix D)

  • Generalized Propositions 4.2 / D.1 (Lines 357–377, 1782–1835). The matched-moment and equal-normalization assumptions are removed. The result now covers any pair of positive or negative steering prefixes, with the mismatch between their average probability-ratio changes entering the required SSR margin as an explicit additive term. The remaining assumptions are listed in Lines 1801–1803.
  • Full proof of Lemma 4.1 added (Lines 1730–1773).
  • Revised tone in claims linking SSR to student performance in the abstract, introduction, Section 4, and conclusion.
  • New Limitations section (Lines 498–508).

2. Empirical Measurement of SSR (Section 4.2)

  • Accuracy-based SSR (Lines 391–396, Table 1 caption). With final-answer correctness as the reward, an accuracy change of Δ%p in the intended direction corresponds to a pairwise win rate (SSR) of (50 + Δ/2)%.
  • SSR reflects reasoning quality (Lines 381–383, 494–496). We have included a clarification that the ground-truth reward underlying SSR is reasoning quality, estimated via both final-answer accuracy and error recognition behavior.
  • Interpreting SSR when the two measures diverge (Appendix C.5, Figures 15–16). We discuss an empirical guideline to clarify how to compare SSR when final-answer accuracy and error recognition disagree.
  • Corrected description of Negative-1Shot (Figure 5 caption, Lines 491–496, 520–521). It lowers final-answer accuracy for the 1.7B/4B/8B teachers, and reduces error recognition at every teacher size. Table 4 (Appendix B) lists the accuracies behind Figure 5 with standard errors.

3. New experiments (Appendix C)

  • C.1 Ranking accuracy on controlled trajectories (Table 5). On 1,024 step-level completion pairs from Math-Step-DPO-10K (same problem and generating model, differing primarily in correctness), the negative-steering rewards obtain the highest ranking accuracy, and Negative-1Shot is highest in both ranking accuracy and student performance.
  • C.2 Ablation on suppressing error recognition (Table 6, Figure 9). Negative prompts that lower teacher accuracy but retain or encourage error recognition yield weaker students.
  • C.3 SSR and reward discrimination of answer- and solution-hint teachers (Table 7, Figures 10–12). Hints raise raw final-answer accuracy, but an LLM judge flags 22.6% (answer) and 31.1% (solution) of reasoning traces as abnormally relying on the hint, and the induced contrastive rewards separate trajectories less clearly than negative steering.
  • C.4 Selecting the one-shot demonstration with SSR (Table 8, Figures 13–14). Among three candidate negative demonstrations with similar error-recognition suppression, the one that reduces final-answer accuracy the most also yields the best student.

4. Corrections and presentation

  • Baseline labels (Section 3.1, Figure 2, Tables 2–3). Corrected a baseline label from OP²D to OPD². The privileged-information baselines previously labeled OPSD and RLCSD are now Self-Contrast and Teacher-Contrast respectively to avoid potential confusion regarding original implementation.
  • Comparison against privileged-information methods revised (Lines 155–159, 209–215).
  • Figures 2–3 slightly enlarged for readability while plotted values are unchanged.
  • Minor edits: unified notation (subscript 0 for the unsteered teacher), added gradient operator on the KL term in Line 185, fixed typos, and updated the AI Use Statement to reflect the newly included use of LLM-as-a-judge in Appendix C.3.
AuthorsOpenReview ↗

Final Author Response

Paper Summary

Recent works in contrastive distillation construct a reward by contrasting a task-specific, RL-trained teacher against its base policy, which makes it costly to obtain and difficult to interpret or adjust. We instead propose CRISP, which contrasts a strong teacher with a negatively steered version of itself that is explicitly instructed to reason incorrectly. CRISP requires no RL training of the teacher, no privileged information, and only one teacher in memory during student training.

Our main results are as follows:

  • CRISP achieves substantial gains on mathematical reasoning that are roughly twice those of the strongest recent on-policy or contrastive distillation methods.
  • We observe an interesting asymmetry between negative and positive steering. Negative steering yields substantial performance gains that further increase with a larger teacher, whereas positive steering yields only limited gains even when the teacher is given the ground-truth solution.
  • We analyze this asymmetry with the Steering-Success Rate (SSR), which measures how reliably steering changes the teacher's reasoning quality in the intended direction and is observed to be substantially higher for negative steering.
  • We prove that a sufficiently higher SSR leads to a higher win rate of the converged student under certain conditions, and student performance consistently follows the trend of SSR across extensive ablations.

Overall, our results establish natural-language steering as an intuitive method for contrastive distillation and show how SSR can help explain and improve the effectiveness of contrastive rewards.

Broader Implications and Contributions

  • Principled understanding of contrastive rewards. We analyze why some contrastive rewards work better than others through SSR, a population-level quantity linked to student performance by theory and ablations. We note that prior works have relied on word clouds [2] or token-category statistics [4,5], which offer limited guidance on how contrastive rewards can be improved in general. We thus expect future works to adopt a quantitative and principled analysis similar to ours.
  • New finding on steerability. We report that a language model can be steered toward incorrect reasoning far more reliably (i.e., with a higher SSR) than toward better reasoning (noted as interesting by Zfe6). Given the link between SSR and student performance, we expect this finding to direct future works toward designing more reliable negative teachers.
  • Interpretable reward control. CRISP addresses a limitation of recent methods that rely on token-level operations such as advantage restriction [2,5], where it is unclear how to improve their effectiveness. The reward of CRISP can instead be adjusted intuitively by editing a natural-language description of the undesired behavior (noted by a3yc, dPbM).
  • Efficient reward ablation. Ablating a contrastive reward in CRISP only requires editing the steering instruction, whereas prior methods [1-4] require additional resource-intensive RL training runs of the teacher policy. We therefore expect CRISP to open up more opportunities for exploring new contrastive rewards and using larger teachers.

Rebuttal Overview

We thank the reviewers for the feedback that has strengthened our paper. The reviewers acknowledged that CRISP is a simple and practical method (a3yc, dPbM) with substantial performance gains (Zfe6, a3yc, dPbM) supported by informative ablations (Zfe6, a3yc). We outline how we addressed each raised issue below:

  • Restrictive assumptions in the theoretical analysis (Zfe6, a3yc, dPbM) were resolved by generalizing Proposition 4.2 and toning down the related claims.
  • Concerns about whether the reward captures reasoning quality (Zfe6, dPbM) were addressed with additional experiments showing that rewards from negative steering most accurately rank reasoning steps that differ mainly in correctness (Appendix C.1).
  • The contribution of suppressing error recognition (Zfe6) was clarified by testing negative prompts that retain error recognition, which consistently yield weaker students (Appendix C.2).
  • The weak gains of hint-based rewards (a3yc) were explained by analyzing the hinted teachers, whose frequent reliance on the hint itself is inapplicable at test time and lowers their SSR in terms of reasoning quality (Appendix C.3).
  • The use of SSR for prompt selection (dPbM) was supported by showing that the negative demonstration with the highest SSR yields the best student (Appendix C.4). We also added an empirical guideline that prioritizes suppressing error recognition over lowering final-answer accuracy for negative prompts (Appendix C.5).

  • [1] Feng et al., 2026 (arXiv:2607.05394)
  • [2] Heo et al., 2026 (arXiv:2607.15161)
  • [3] Yang et al., 2026 (arXiv:2602.12125)
  • [4] Yu et al., 2026 (arXiv:2607.26246)
  • [5] Pan et al., 2026 (arXiv:2606.11709)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.