acceptodds
Withdrawn from ICLR 2027

Contrastive On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

On-policy distillation (OPD) trains a student model on trajectories sampled from its own policy, providing dense token-level supervision by minimizing the divergence between the teacher's and student's output distributions at each token. Although existing OPD approaches effectively distill strong reasoning capabilities into student models, they inherently inherit uncurated chain-of-thought traces, exacerbating overthinking and reasoning redundancy. To address this limitation, we introduce COPD, a contrastive OPD framework. Specifically, for each token generated by the student, a frozen teacher evaluates the current student state under two contrasting prompts that induce low and high reasoning effort, respectively. The resulting difference in log-probabilities serves as a token-level advantage signal to guide the policy update. Rather than merely mimicking a single teacher distribution, COPD guides the student model toward acquiring concise and efficient reasoning strategies. We evaluate COPD across 9 multimodal benchmarks covering both reasoning and understanding tasks. Empirical results show that COPD substantially reduces reasoning length without hurting task performance, consistently improving efficiency across different tasks and model scales. In addition, this contrastive paradigm extends naturally to on-policy self-distillation (OPSD), establishing self-contrastive supervisory signals that enable a single model to compress its own reasoning without an external teacher.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Efficient Reasoning, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QormOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Contrastive On Policy Distillation (COPD), a method designed to reduce reasoning length while maintaining or improving task performance. Unlike standard on policy distillation (OPD), which aligns the student model with a single teacher distribution, COPD uses two contrasting prompts that encourage light thinking (LT) and heavy thinking (HT). The teacher evaluates each token generated by the student under both prompts, and the difference between their log probabilities serves as a token level advantage signal to update the student policy. The method aims to encourage concise reasoning by increasing the probability of tokens preferred under LT and suppressing those preferred under HT, without requiring explicit correctness rewards or length penalties. Experiments across multiple model configurations and nine multimodal benchmarks show that COPD reduces output length while improving average accuracy compared to existing OPD methods.

Critical strengths

The method can reduce the generation length effectively and intuitively.

Critical weaknesses

The paper demonstrates that the proposed contrastive advantage improves evaluation accuracy, but it is unclear why the log probability ratio between LT and HT should provide supervision signals correlated with answer correctness. The authors hypothesize that the method suppresses redundant reasoning, but do not systematically analyze whether the contrastive advantage actually identifies unnecessary reasoning steps or explain how this improves correctness despite the absence of explicit correctness supervision.

The comparison between LT only and LT minus HT in Table 6 is conducted using only one teacher and student pair, Qwen3 VL 8B and Qwen3 VL 2B. Although the overall method is evaluated across multiple model configurations, it remains unclear whether the benefits of this specific contrastive formulation consistently transfer to other models. Additional ablations using alternative reference prompts or reasoning effort levels would also help establish the importance of the proposed formulation.

The paper claims that COPD encourages more efficient reasoning by suppressing redundant steps. However, the evaluation primarily focuses on answer accuracy and output length, without systematically examining which reasoning steps are removed. It remains unclear whether the method learns to selectively eliminate unnecessary reasoning or simply develops a stronger preference for shorter outputs.

Although COPD improves average accuracy, certain benchmarks exhibit accuracy degradation after training. The paper provides limited analysis of why aggressive compression hurts these tasks or under what conditions additional reasoning and verification remain necessary.

I believe there are existing baselines specifically designed to reduce generation length. It would be important to include comparisons with these methods as well.

Questions for authors

See weaknesses above.

Reviewer_6sAEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is motivated by the observation that the chain of thought of current thinking models keeps getting longer, and it aims to compress it with a contrastive approach. The teacher, which can be a larger model or the student model itself, scores the student’s on-policy rollout under two prompts: one asking for concise (light) thinking and one asking for verbose (heavy) thinking. The token-level difference between the two log-likelihoods is used as the advantage in an RL-style (PPO-style clipped) policy gradient. The paper shows that this improves performance and outperforms other OPD baselines (OPD, ExOPD, VA-OPD), while producing much shorter traces. So it demonstrates successful compression of reasoning.

Critical strengths

1.Experimentally, the paper shows good compression results, using far fewer tokens than the other baselines while keeping good performance. 2.The paper includes many extensive ablation studies. For example, Appendix A.7 (inTable 11) shows that GRPO with a length penalty reaches a similar response length to their method with slightly lower accuracy, but requires much more compute 3.Another ablation I like is the self-distillation variant (COPSD, Table 5). Here the same model acts as both student and teacher, and the same two-prompt contrast is used for training.

Critical weaknesses
  1. The paper does not show any evaluation curves over training.could you explain how the evaluation scores in the main result tables are selected, e.g., which checkpoint is reported and what the 2000 validation split was used for. This makes it hard to judge whether the results are significant. If the evaluation curves of different methods are close together or overlapping, and differ only at certain steps, then the improvement might not be robust.
  2. The method does not use the student’s own log-likelihood at all, even when there is an external teacher. It only uses the contrast between the teacher’s log-likelihoods under the two prompt styles. I think the student log-likelihood is also important. Consider that any token the verbose-prompted teacher assigns a very low likelihood will receive a large positive advantage, even if it is not something we desire, and this is encouraged by the loss. Adding the student’s own log-probability to the advantage would make it a real distillation objective. Without it, the method looks more like using the teacher’s two prompts as a conciseness reward.
  3. Following from the above, since the student’s own log-probability is not used, I am not sure this can still be called distillation, or whether it is just conciseness-oriented RL-style training where the reward comes from a teacher. The title “Contrastive On-Policy Distillation” might therefore be misleading here.
  4. The self-distillation results suggest Table 5 shows that even when the student scores itself with the two prompts, it reaches similar performance. For Qwen3-VL-2B, COPSD gets 56.7 / 0.76K (accuracy /tokens). For comparison, Table 1 reports 57.0 / 0.91K for COPD with the Qwen3-VL-8B teacher The accuracy gap is within 0.5 pp, and self-scoring produces even shorter outputs. This again suggests and i'm curious that adding the student log-probability to the advantage might be make better use of the teacher?
Questions for authors

please see weakness

Reviewer_LTJBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose a novel contrastive on-policy disillation to achieve high-efficiency reasoning by provide two different teachers using low/hight reasoning effort prompt. Experimental results on multimodal reasoning and understanding benchmarks have shown that the prosed method achives accuracy–length pareto frontier.

Critical strengths
  • A simple yet effective method that empowers OPD with efficient reasoning optimization.

  • Extensive experiments are conducted on two types of multimodal benchmarks, encompassing both text-based and vision-language LLMs of varying scales.

Critical weaknesses
  • The central claim that the LT–HT contrast is essential is not well evaluated: The important baseline of standard OPD with an LT-prompted teacher is missed. The LT-prompted 8B teacher already reaches 62.9% accuracy at 0.55K tokens (Table 7), so this baseline may compress comparably. The l\mathcal{l}_LT-only ablation (Table 6) is not informative: log-probabilities are always ≤ 0, so it penalizes every token.

  • The main baselines target accuracy and have no compression mechanism, so large length reductions relative to them are expected. Compression baselines are limited to a single length-penalty GRPO run (Table 11) and loosely specified SFT/DPO (Table 7).

Questions for authors
  • Could you provide the results where LT-teacher OPD as a main baseline?

  • Could you provide results of several established reasoning-compression (e.g., efficient GRPO) methods under the same back-bone model and data?

  • Could you report the generation cap and truncation rate during evaluation, and provide the curves of sequence length and accuracy under longer training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.