- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To reduce the cost of training and storing separate teachers for contrastive distillation, the authors propose CRISP. It uses the same teacher under a normal prompt and a prompt asking it to reason incorrectly. Student-generated responses are scored under both conditions, and the log-probability difference provides the training reward. Experiments with Qwen3-4B and Qwen3-1.7B students on five math benchmarks report gains over the tested baselines. The authors also examine why negative steering works better than positive steering. They introduce Steering-Success Rate (SSR), give conditions under which higher SSR leads to a better optimal student, and study this relationship by varying the prompts and teacher sizes.
Critical strengths
-
The reward in CRISP is simple and cheap to implement. It uses the same teacher under two prompts, avoiding additional teacher fine-tuning and keeping only one set of teacher weights in memory. This makes the method straightforward to apply to an existing teacher. The reward can also be adjusted by editing the negative instruction.
-
The reported gains of CRISP over other baselines are significant. In Table 3, the 4B student model reaches 42.28% average accuracy, compared with 33.35% for Direct OPD. The 1.7B student model reaches 28.26%, compared with 22.04% for OP$^2$D. CRISP performs better than the tested baselines on various benchmarks, providing useful evidence that negative prompting can improve the reward used for distillation.
Critical weaknesses
-
Section 3.1 and Table 2 describe RLCSD as contrasting a teacher given a correct answer or solution with the same teacher without a hint. Original RLCSD instead contrasts correct and incorrect hints [1]. The incorrect hint is intended to reduce effects of providing a reference that are unrelated to correctness, such as changes in style. Removing it changes the reward construction, even if all methods use a common optimizer. If the experiments follow Table 2, the results do not establish how CRISP compares with RLCSD’s correct/incorrect contrast. The authors should clarify the implementation and provide a comparison using the original RLCSD.
-
Proposition C.1 requires Eqs. (3)-(6). The authors didn't check them for the experimental comparisons or explain why they should be true. Thus, the theorem gives limited support for using SSR to predict which prompt will produce a better student. Figure 5 also shows a discrepancy: Negative-1Shot improves student performance but slightly increases negative teacher accuracy at 0.6B and 14B. These plotted values correspond to lower estimated SSR, contrary to the claim of consistent improvement. Finally, the paper does not report how to use SSR to choose the one-shot example. Its success therefore leaves open whether SSR can actually guide prompt selection before student training.
Reference
[1] Leyi Pan et al. RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation.
Questions for authors
-
What exact reward was used for the RLCSD baseline? If the implementation follows Table 2, would CRISP still outperform it? A fair comparison would help establish whether the gain comes from CRISP's reward construction. If Table 2 only misdescribes the implementation, please clarify what the reported results actually evaluate.
-
How do the authors explain the Figure 5 cases where Negative-1Shot improves the student while increasing negative teacher accuracy? Please provide the underlying values and variance for the 0.6B and 14B cases. Can the authors check whether the one-shot reward better ranks correct answers above incorrect answers to the same question?
-
Can SSR help choose a negative prompt that leads to better student performance? For example, could the authors measure SSR for several candidate prompts on a development set, select the one with the highest SSR without looking at student training results, and then train a student using that prompt? How would the resulting student compare with students trained using the default Negative and Negative-1Shot prompts?
8 comments
Dear Reviewer dPbM,
Thank you again for your detailed review. We would like to gently follow up on our response and the revised manuscript. We have addressed your concerns as follows:
- [W1, Q1] We renamed RLCSD to Teacher-Contrast (Lines 209–215) for the following reasons:
- This baseline is not meant to be a comparison against RLCSD. Rather, it tests CRISP against a contrastive reward built from privileged information under the same training setup as all other methods.
- The original RLCSD relies on multiple self-rollouts and an external verifiable reward, which differ from our setup. Comparing against it would therefore mix two different problem settings and would not show which contrastive reward is more effective, which is the main question our paper studies.
- [W2-1] Proposition 4.2 / D.1 (formerly C.1) has been revised so that it no longer requires the conditions in Eqs. (3)–(6), and it has been strengthened to hold for any pair of steering prefixes.
- [W2-2, Q2] We clarified why Negative-1Shot yields better students, which is due to the negative one-shot example effectively suppressing error recognition (Figure 5, right).
- SSR is defined on reasoning quality, so error recognition should be considered together with final-answer accuracy (Lines 381–383).
- In the requested ranking analysis, the Negative-1Shot reward also shows the highest ranking accuracy (Appendix C.1).
- [W2-3, Q3] We conducted the requested experiment and an additional ablation, and both show that SSR indeed helps choose a better negative prompt:
- Among three negative demonstrations with similar error-recognition suppression, the one with the largest drop in final-answer accuracy leads to the best student (Appendix C.4, Table 8).
- Negative prompts that lower final-answer accuracy but retain or encourage error recognition lead to weaker students (Appendix C.2, Table 6).
Overall, our paper contributes a novel and simple method with strong empirical performance, accompanied by both theoretical analysis and extensive empirical ablations, including:
- A simple and interpretable contrastive reward that requires no task-specific RL training of the teacher and no privileged information.
- Roughly twice the performance gain of the strongest baseline among recent on-policy and contrastive distillation methods (Table 3), with more stable convergence (Figure 2).
- The finding that negative steering is more effective than positive steering, which gives only marginal gains even with privileged information (Table 3 and Appendix C.3) and, unlike negative steering, does not benefit from a larger steered teacher (Figure 3).
- A theoretical analysis that links the Steering-Success Rate (SSR) to the converged student's win rate (Proposition D.1), now strengthened to hold for any pair of positive or negative steering prefixes.
- Extensive ablations (steering direction, teacher size, error recognition, privileged-information hints, choice of demonstration) in which student performance consistently follows the trend of SSR, in line with the theory (Lemma 4.1 and Proposition 4.2) and our intuition (Section 4.1, Lines 351–356), showing that SSR can serve as a principled criterion for designing better contrastive rewards.
We once again thank you for the detailed comments, which have helped strengthen our work. If your concerns have been addressed, we would be really grateful if you could consider raising your score accordingly. Please let us know if any questions remain, so that we can address them before the discussion period ends.
Thanks for your response. I appreciate that you confirm that the baseline you used was different from RLCSD and renamed it as Teacher-Contrast. The theoretical result is now conditional and the authors explicitly state those assumptions are not checked. The inaccurate statement about teacher accuracy in Figure 5 is also corrected. I also appreciate your efforts on conducting the new prompt selection experiment. It shows 1shot is very useful to suppress error recognition.
However, my remaining concern is how to use SSR when the two metrics accuracy and error recognition do not agree. This is what happens in Table 8. Negative has lower teacher accuracy, while Negative-1Shot has lower error recognition. Your experiments cannot show this because there Negative-1Shot is better on both. How should we choose between such prompts before seeing student results?
We thank the reviewer for the detailed comments.
The method labeled RLCSD in our initial submission contrasts a single Qwen3-14B with and without privileged information (answer or solution), using the same optimization as all other methods (Section 2.3, Line 183). We renamed it to Teacher-Contrast to prevent confusion (Lines 209–215, Tables 2–3). We adapted the original RLCSD for fair comparison, as it uses multiple self-rollouts and combines token-level rewards with an external verifiable reward, unlike our training setup.
We revised the proposition linking the Steering-Success Rate (SSR) to student performance (Propositions 4.2 and D.1) so it no longer requires these assumptions, and toned down the related claims.
To clarify, SSR is defined by reasoning quality and empirically estimated via both final-answer accuracy and error recognition, not accuracy alone (revised Lines 381–383, 494–496). Negative-1Shot's stronger training performance can be explained by its suppression of error recognition:
We revised the claim (Lines 471–472, 491–496, 520–521) to state that Negative-1Shot decreases teacher accuracy for Qwen3-1.7B, 4B, and 8B, and reduces the error-recognition rate at every tested size. We also included final-answer accuracies with standard errors in Table 4:
Yes. We ran the suggested experiment (Appendix C.4, Table 8), and the negative example with the highest SSR yields the best student. We measured the SSR of Qwen3-14B under three candidate prompts differing only in the negative example. Besides the original Negative-1Shot example which begins with an incorrect approach (Figure 7), we tested two mildly incorrect examples (Figures 13 and 14), and trained a student with each prompt.
All three 1-shot methods suppress error recognition similarly, while the two mildly bad examples yield smaller reductions in final-answer accuracy and weaker students than the original Negative-1Shot example.
As consistently shown from the ablations in Section 4.2 and Appendix C, SSR can indeed help construct better contrastive rewards:
We once again thank the reviewer and kindly ask them to consider raising the score if their concerns have been addressed.