acceptodds
Under review as a conference paper at ICLR 2027

IP-OPD: Learning from Intervention Preferences for On-Policy Distillation

Abstract

On-policy distillation (OPD) reduces student-state mismatch by supervising students on trajectories they actually visit. Yet state alignment does not guarantee that supervision at a visited state is useful: even a stronger source can provide locally plausible reasoning that worsens the student’s downstream outcome. We make this distinction explicit through intervention advantage, defined as the change in expected reward caused by inserting a reasoning block at a student-visited state and then returning control to the student. We introduce Intervention-Preference On-Policy Distillation (IP-OPD), which compares source-proposed and native blocks at sparsely selected on-policy states. Both branches are completed by the same student, and discordant terminal outcomes induce bidirectional block-level preferences. Our analysis connects intervention advantage to the return of an explicit intervention-mixture policy and, for binary verifiable outcomes, shows that the population preference optimum ranks blocks consistently with downstream continuation values. Because supervision is determined by downstream student utility rather than source identity, IP-OPD requires neither privileged source logits nor a learned critic or router and naturally supports external, self-, and heterogeneous proposals. Experiments across five mathematical reasoning benchmarks show consistent performance gains, stronger diagnostic signals of intervention utility than several distributional proxies, and improvements under outcome-information-matched and compute-matched controls.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Knowledge Distillation, Large Language Model Reasoning, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_btZiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
3
Read review
Paper summary

The paper proposes IP-OPD, an OPD method that evaluates teacher-proposed reasoning blocks through student continuations. By comparing the outcomes of teacher interventions and student blocks, the method constructs bidirectional preferences for block-level optimization. Experiments on mathematical reasoning benchmarks show improvements over several baselines.

Critical strengths
  1. The proposed method is clean, intuitive, and clearly motivated.

  2. The paper provides meaningful analysis and well-designed diagnostics.

  3. The experiments include training-round-matched, outcome-information-matched, and compute-matched comparisons, which help assess the effectiveness of the proposed method and objective.

Critical weaknesses
  1. Limited evaluation across model scales and families.

The main experiments are conducted on Qwen3-1.7B-Base, with an additional 0.6B experiment in the appendix. Experiments with at least a 4B student and another model family beyond Qwen3 would strengthen the empirical evidence.

  1. Limited evaluation across task domains.

All experiments focus on mathematical reasoning. Additional experiments on other domains would help assess the cross-domain generalization of IP-OPD.

  1. Missing training dynamics analysis.

The paper trains for three epochs on 7,500 problems and reports endpoint results without evaluation learning curves. At the final checkpoint, GRPO already slightly outperforms Vanilla OPD. However, OPD and GRPO may exhibit different learning dynamics, with OPD potentially benefiting from faster early-stage improvement. It is unclear whether IP-OPD retains this advantage or primarily achieves its improvements after extended reward-based training. Evaluation curves would help clarify this distinction. If IP-OPD mainly benefits from longer training, comparisons with stronger teacher-guided reward-based methods, such as LUFFY and Prefix-RFT, would also be important.

Questions for authors
  1. Could the authors extend the experiments to larger students, such as Qwen3-4B, and ideally another model family?

  2. Could the authors evaluate IP-OPD on other verifiable reasoning domains?

  3. Could the authors provide evaluation learning curves for IP-OPD, Vanilla OPD, EOPD, and GRPO? In particular, does IP-OPD retain the early-stage learning efficiency of OPD? If its improvements mainly emerge after extended training, could the authors include teacher-guided reward-based baselines such as LUFFY and Prefix-RFT?


I appreciate the authors' additional experiments, which address many of my concerns. However, several issues remain.

Cross-domain evaluation. While the K&K experiments demonstrate the applicability of IP-OPD beyond mathematical reasoning, the benchmark appears largely saturated. Logic-RL achieves performance similar to IP-OPD, and comparisons with other OPD baselines are missing. Therefore, it remains unclear whether IP-OPD provides additional benefits over existing OPD methods in this domain.

GRPO training dynamics and stronger baselines. I appreciate the authors providing the training curves. The OPD baselines appear to plateau in later training, which is understandable. However, GRPO also plateaus after approximately 1–2 epochs, while IP-OPD continues to improve. I think further explanation of GRPO's training behavior is needed.

I also remain interested in comparisons with LUFFY and Prefix-RFT, as they share similar motivations and approaches with IP-OPD and would serve as relevant baselines.

Presentation. Some newly added sections are unnecessarily verbose and awkwardly written. For instance, I do not see why the sentence "LUFFY and Prefix-RFT are not evaluated, so no superiority over them is claimed" appears in the revised manuscript's discussion of training dynamics. It seems that the authors did not even carefully review the revised manuscript before submission, which raises concerns about the overall quality of the revision.

Overall, the rebuttal has strengthened the paper, but I believe these concerns remain unresolved.

Reviewer_4gKHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Intervention-Preference On-Policy Distillation (IP-OPD), which evaluates local supervision by its usefulness to the current student. At selected student-generated prefixes, a source-proposed block and a native student block are each followed by the same frozen student. A reference-answer verifier supplies binary terminal rewards. Pairs with different outcomes are used for a logistic preference loss over the two blocks, including reversed preferences when the native branch succeeds and the source branch fails. The analysis relates intervention advantage to the return of an explicit intervention-mixture policy and characterizes a fixed-state population preference optimum that ranks blocks by downstream student success. Experiments primarily use Qwen3-1.7B-Base with Qwen3-8B proposals and MATH training. IP-OPD achieves 35.4% Macro Avg@8 versus 31.4% for EOPD, or 34.0% at the EOPD compute budget. Outcome-information-matched winner-imitation and native-native controls support contributions from both the pairwise learning rule and source proposals.

Critical strengths
  1. The improvement survives controls that address the most important alternative explanations. IP-OPD exceeds winner-only imitation under the same outcome-information and collection setup by a reported 2.0 macro points after full training and 1.7 at common compute. These three-seed comparisons provide evidence beyond the headline gain over distribution-based distillation: access to rewards alone and student-only preference sampling do not explain the improvement. This is my main reason to favor acceptance.

  2. The supervision design is useful and technically well qualified. Evaluating both blocks through the same student makes the preference relevant to that student's ability to use a proposal, and bidirectional labeling avoids assuming that the source is always superior. The text-only source interface requires neither source logits nor a learned critic, and deployment uses the student alone. The analysis and independent-rollout diagnostics distinguish noisy observations from expected utility and explicitly acknowledge the limits of the population theorem. This makes the contribution practically attractive without relying on an overstated guarantee.

Critical weaknesses
  1. The contribution of reversed preferences is less securely established than the overall method. The positive-only versus full comparison in Table 3a uses one seed and evaluates only the MATH500/AMC23 average. Although the pairwise objective and candidate budgets are preserved, the policies visit different states and retain different numbers of useful pairs. Appendix E.1 describes a fixed-buffer exposure-matched control but reports no numerical outcome for it. Because learning from native-winning pairs is a central differentiator, the missing result and limited repeatability evidence weaken attribution of the gain to bidirectionality itself. The winner-imitation and native-native controls do not independently isolate that component.

  2. Competitive efficiency is demonstrated against EOPD, but remains incomplete against the strongest reported adapted baselines. DOPD-adapt and CR-style both reach 32.8% Macro Avg@8, above EOPD's 31.4%, yet the displayed runtime comparison and detailed cost tables do not establish their performance at the same budget as IP-OPD. Full IP-OPD itself costs about 28% more GPU-hours than EOPD. The common-budget EOPD result is valid, but it does not establish that IP-OPD offers the best accuracy-cost tradeoff among the stronger alternatives already evaluated. This limits the strength of the practical superiority claim.

Questions for authors
  1. Clarification of an existing experiment: What was the numerical result of the exposure-matched fixed-buffer control described in Appendix E.1? Please report its sample counts and measured effect. Is there evidence across training seeds for the full versus positive-only comparison?

  2. Clarification of competitive efficiency: What are the measured training costs of DOPD-adapt and CR-style, including auxiliary generation and scoring? If their saved checkpoints permit it, how do they compare with IP-OPD at the same GPU-hour budget?

  3. Clarification of novelty: Which specific operational component distinguishes the shared-prefix, same-student, bidirectional pair construction from the closest cited step-preference or counterfactual-distillation method? A precise algorithmic comparison would help establish the incremental contribution. I have not independently verified priority against the external literature.

1 comment

Thank you for providing a detailed response and conducting additional experiments.

The fixed-buffer and three-seed results for native-winning preferences, as well as the common-budget comparison with DOPD-adapt and CR-style, address my two primary concerns. The clarification regarding novelty is reasonable. I am increasing my score to 3.

Reviewer_sUq5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Intervention-Preference On-Policy Distillation (IP-OPD), a framework that improves language model reasoning by evaluating whether a teacher-proposed reasoning block actually benefits the student's downstream performance. Unlike conventional on-policy distillation, which primarily addresses the mismatch between teacher-generated and student-visited states, IP-OPD focuses on the usefulness of supervision at those states. Experiments on five mathematical reasoning benchmarks demonstrate consistent improvements over existing distillation methods. The proposed approach achieves a Macro Avg@8 of 35.4%, compared with 31.4% for EOPD, and maintains an improvement under matched computational budgets.

Critical strengths
  1. Conceptually simple and flexible framework. The proposed method is straightforward and does not require teacher logits, an additional critic, or a learned routing network. Its compatibility with external teachers, self-generated proposals, and heterogeneous models makes it potentially useful in practical distillation settings.

2 Solid theoretical motivation. The formulation of intervention advantage provides a principled perspective on the usefulness of local reasoning interventions. The analysis of the population preference objective further supports the relationship between preference learning and downstream continuation values.

  1. Comprehensive experimental evaluation. The paper evaluates multiple baselines across five mathematical reasoning benchmarks. Importantly, the authors include compute-matched and outcome-information-matched comparisons, making the empirical findings more convincing.
Critical weaknesses
  1. Limited methodological novelty. Although the motivation is compelling, the core learning objective is closely related to existing pairwise preference optimization methods. The main novelty lies in how preference pairs are constructed using intervention rollouts, rather than in a fundamentally new optimization algorithm. The paper could better clarify its conceptual and technical differences from existing outcome-based preference learning and selective distillation approaches.

  2. Theoretical guarantees do not directly establish practical policy improvement. The theoretical results characterize an intervention-mixture policy and a statewise population preference optimum under specific assumptions. However, these results do not guarantee that finite-sample gradient updates to a parameterized language model will improve downstream performance. The relationship between the theoretical analysis and the actual training dynamics could be discussed more thoroughly.

  3. Additional computational overhead. IP-OPD requires additional student continuations and outcome evaluations to construct preference pairs. According to Table 4, full IP-OPD requires 99.0 GPU-hours, compared with 77.4 GPU-hours for EOPD. Although compute-matched results are provided, the additional cost may limit scalability to substantially larger models or more expensive reasoning tasks.

  4. Limited generalization beyond mathematical reasoning. All five main benchmarks focus on mathematical problem-solving with verifiable final answers. It remains unclear whether the approach generalizes to domains where reward signals are noisy, subjective, or difficult to obtain, such as open-ended reasoning, coding, and instruction following.

Questions for authors

Please refer to the weakness part.

1 comment

The rebuttal addresses several of my concerns with additional experiments and clearer explanations, but the limited methodological novelty and remaining questions about scalability and generalization lead me to maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.