acceptodds
Withdrawn from ICLR 2027

Inductive Feedback for Mixed-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Verbal feedback can identify errors and prescribe corrections, providing rich supervision for language-model post-training even when reliable programmatic verifiers are unavailable. Such feedback, often generated by a capable model, can be used to condition the teacher in on-policy distillation, which trains the student to match the teacher’s predictions on student-generated rollouts. However, this approach can transfer teacher preferences that the feedback did not motivate, while leaving much of the feedback's guidance unused. We find that both problems come from the standard on-policy distillation objective, specifically the divergence it minimizes and the distribution it uses as its target. Our proposed method addresses both limitations. First, to isolate the information conveyed by the feedback from the teacher's inherent preferences, we treat verbal feedback as evidence for or against the hypothesis that a particular token comes next at a given prefix. We then adopt a probabilistic confirmation framework which uniquely determines an ordering over the vocabulary based on the teacher's predictions before and after it receives feedback. Using a confirmation score consistent with this ordering, we construct a target distribution within a trust region of the student. Second, to learn from guidance that student rollouts can leave unused, we derive a simple shared-rollout estimator of a symmetric divergence between the student and target distributions over rollouts, reusing student and feedback-conditioned teacher rollouts in both directions through importance weighting. Empirical evaluations show that our method outperforms the common on-policy distillation recipe and a recent contrastive variant on knowledge-based and agentic benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Reinforcement learning, On-policy distillation, Learning from verbal feedback

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yCkzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that verbal feedback provides a rich learning signal for On-Policy Distillation (OPD). To leverage this feedback, the authors introduce a confirmation score (S(v)), defined as the teacher's log-probability ratio with and without verbal feedback (F). This score is incorporated as an additional weighted objective alongside the standard OPD loss. Their empirical results demonstrate that the proposed approach achieves strong performance compared to SDPO and W2S-OPD.

Critical strengths

None

Critical weaknesses
  • The idea of leveraging verbal feedback is well-established and has already been explored in the context of distillation (e.g., Self-Distillation [1]. The proposed method appears to be a straightforward weighted combination of OPD and OSPD. Moreover, much of the derivation of the corrected teacher target is trivial and follows well-known principles [2], with limited theoretical novelty.

  • The paper is extremely poorly written, with several key methodological choices lacking rigorous explanations. Specifically, the authors do not adequately motivate the introduction of the confirmation score, justify why the four proposed conditions are necessary, or explain the need for a symmetric objective. Several concepts, such as preferences and probabilistic confirmation, are introduced without clear definitions or sufficient explanation. Furthermore, important details of the experimental setup (e.g., those related to Figure 1) are not in the main paper.


 [1] Self-Distillation Enables Continual Learning. ICML 2026 Spotlight.
 [2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 Oral.
Questions for authors
  • Could the author explain the motivation for the confirmation score? Why is the log-ratio score a good confirmation score?
  • What's the real difference between the author's objective and a weighted sum of the standard OPD and Self-Distillation OPD objectives? What's the motivation behind symmetric rollouts?
1 comment

the authors introduce a confirmation score (S(v)), defined as the teacher's log-probability ratio with and without verbal feedback (F)

The log-probability ratio is the score used by our W2S baseline, another contrastive method. Our proposed score is relative distance. Please refer to Section 3.

This score is incorporated as an additional weighted objective alongside the standard OPD loss.

The score is not used as an additional weighted objective. The score is used to construct a target which replaces the raw teacher in the proposed divergence. Please refer to Section 4.

The idea of leveraging verbal feedback is well-established

We are not claiming the concept of learning from verbal feedback is our contribution. It is our motivation, and we propose an improved method for learning from such feedback. Please refer to the contributions in the introduction.

Moreover, much of the derivation of the corrected teacher target is trivial and follows well-known principles [2], with limited theoretical novelty.

We cite DPO between Eqs. (5) and (6), where we construct the target based on our proposed score function. We did not claim our derivation is new. Please refer to the contributions in the introduction.

important details of the experimental setup (e.g., those related to Figure 1) are not in the main paper.

Please refer to Appendix C.

The proposed method appears to be a straightforward weighted combination of OPD and OSPD.

What's the real difference between the author's objective and a weighted sum of the standard OPD and Self-Distillation OPD objectives?

Our objective in Eq. (9) is simply the Jeffreys divergence, an unweighted sum of reverse KL and forward KL, and the weights in Eq. (11) are importance sampling weights to fix the distribution mismatch for each leg of the divergence. OPSD is a special case of OPD in which the same model serves as student and teacher, with privileged context supplied to the teacher. Neither that distinction nor a possible combination of OPD and OPSD is relevant to our contribution. Please refer to the respective papers.

the authors do not adequately motivate the introduction of the confirmation score, justify why the four proposed conditions are necessary

Could the author explain the motivation for the confirmation score? Why is the log-ratio score a good confirmation score?

As noted earlier, we propose relative distance, not the log-ratio score. We focus on property A3 because it is the only property that our relative distance score satisfies and the log-ratio baseline lacks. Please refer to the motivation for A3 in lines 206–208 and the supporting experiment in Figure 1.

Reviewer_6hXyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The method transfers feedback-induced changes in teacher probabilities through a bounded relative-distance score anchored to the current student. Mixed student/teacher trajectories support forward and reverse distillation. Evaluation covers taught facts, embodied planning, science, and tools.

Critical strengths
  • Unchanged teacher probabilities produce no local update, giving the feedback transfer a useful anchoring property.
  • Gains over the stronger baseline are 14.3 points in chemistry and 12.0 in physics.
  • The manuscript discloses approximations and includes paired synthetic seeds and a repeated clipping diagnostic.
Critical weaknesses
  1. Estimator: Appendix B.3 conditions on C=(x,ys,F)C=(x,y_s,F) but assumes fresh mixture samples. Appendix D reuses ysy_s, which is fixed given CC. The same conditional change-of-measure argument therefore does not directly apply. This does not invalidate the ideal theorem or allege an undisclosed semi-gradient.
  2. Attribution: Real-task comparisons jointly change the score, anchor, rollout source, divergence, and weighting. Existing sweeps do not isolate the proposed score or shared estimator.
  3. Uncertainty and auditability: Table 1's intervals use three checkpoint evaluations, not independent training runs, and selection is unclear. Smaller gains have acknowledged overlapping intervals. The supplied materials do not provide embodied-task scenarios, full rubrics, human validation, or measured teacher/judge costs.
Questions for authors
  1. What joint sampling law and weighting denominator cover feedback-dependent reuse? Can you compare with fresh student draws under fixed feedback?
  2. On one existing held-out task, can you compare scores and shared/separate estimation while matching anchor, support, proposals, and trajectory budget?
  3. Please identify Table 1's checkpoints and selection rule, report paired or independent-run uncertainty, and provide auditable embodied-task materials.
  4. Just provide more details for all experiments in general.
2 comments

Estimator: Appendix B.3 conditions on C=(x,ys,F)C=(x,y_s,F) but assumes fresh mixture samples. Appendix D reuses ysy_s, which is fixed given CC. The same conditional change-of-measure argument therefore does not directly apply. This does not invalidate the ideal theorem or allege an undisclosed semi-gradient.

What joint sampling law and weighting denominator cover feedback-dependent reuse? Can you compare with fresh student draws under fixed feedback?

We mention this reuse in the paper. However, we consider it a relatively minor implementation optimization. Drawing a fresh student sample to estimate Eq. (9) would offer a practical benefit only if the new answer differed substantially from the original while at the same time, the feedback on the original answer still applied to it. We believe this case is rare enough that it does not justify doubling the number of student answers sampled. Below are the independent objective (Eq. (9)) and its approximation.

Lindependent(q,p)=EC∼D[Ey∼qx![∑tDKL!(qx(⋅∣y<t),∥,pC(⋅∣y<t))]+Ez∼pC![∑tDKL!(pC(⋅∣z<t),∥,qx(⋅∣z<t))]].\begin{aligned} \mathcal{L}_{\text{independent}}(q,p) = \mathbb{E}_{C \sim \mathcal{D}}\Bigg[ & \mathbb{E}_{y \sim q_x}\\!\left[ \sum_t D_{\mathrm{KL}}\\!\left( q_x(\cdot \mid y_{ < t}) \\,\Vert\\, p_C(\cdot \mid y_{ < t}) \right) \right] \\\\ & + \mathbb{E}_{z \sim p_C}\\!\left[ \sum_t D_{\mathrm{KL}}\\!\left( p_C(\cdot \mid z_{ < t}) \\,\Vert\\, q_x(\cdot \mid z_{ < t}) \right) \right] \Bigg]. \end{aligned}

Ldependent(q,p)=EC∼D[∑tDKL!(qx(⋅∣y<ts),∥,pC(⋅∣y<ts))+Ez∼pC![∑tDKL!(pC(⋅∣z<t),∥,qx(⋅∣z<t))]].\begin{aligned} \mathcal{L}_{\mathrm{dependent}}(q,p) = \mathbb{E}_{C \sim \mathcal{D}}\Bigg[ & \sum_t D_{\mathrm{KL}}\\!\left( q_x(\cdot \mid y^s_{ < t}) \\,\Vert\\, p_C(\cdot \mid y^s_{ < t}) \right) \\\\ & + \mathbb{E}_{z \sim p_C}\\!\left[ \sum_t D_{\mathrm{KL}}\\!\left( p_C(\cdot \mid z_{ < t}) \\,\Vert\\, q_x(\cdot \mid z_{ < t}) \right) \right] \Bigg]. \end{aligned}

Attribution: Real-task comparisons jointly change the score, anchor, rollout source, divergence, and weighting. Existing sweeps do not isolate the proposed score or shared estimator.

On one existing held-out task, can you compare scores and shared/separate estimation while matching anchor, support, proposals, and trajectory budget?

Let us clarify this since we provide evidence for both components. We modify two components. We replace the raw teacher with a corrected teacher in OPD's divergence and use a symmetric divergence instead of OPD's reverse KL divergence. Instead of simply running the end-to-end benchmarks with each component and showing it's better than the baselines, we wanted to show how and under what circumstances each component actually helps. To more clearly show the advantage of the corrected target, we constructed the controlled experiment in Figure 1. It shows that our corrected target outperforms the closest contrastive baseline when the teacher is already inclined towards a token and the feedback further reinforces that preference. When the teacher does not favor the correct token at all, the corrected target is at least as good as the baseline. For the other component, we vary the number of student rollouts in Section 5.3 to see how the influence of each leg of the symmetric divergence changes in the final objective. At the two extremes, we see how training behaves when all rollouts come from the student or all come from the teacher. Specifically, Figure 2 shows that a non-negligible fraction of teacher rollouts in the batch is enough to obtain the benefit of the symmetric objective. We believe this figure offers a more honest and accurate ablation than simply removing the forward leg and showing that performance drops.

Please identify Table 1's checkpoints and selection rule, report paired or independent-run uncertainty, and provide auditable embodied-task materials.

During our experiments, we found OPD training runs to be generally consistent compared with runs of methods based on group rollouts like GRPO. For the main results, we used a single training run in each case. In almost all cases, training plateaued well before 200 steps. We reported the performance of the last 3 checkpoints at the evaluation interval noted in Table 3.

Thank you for the detailed response. The explicit distinction between the independent objective and the dependent approximation is helpful, as is the clarification that Table 1 reports the last three checkpoints from a single training run. Figure 1 provides a useful controlled synthetic comparison, and Figure 2 adds fixed-budget rollout-allocation evidence.

Several decision-relevant points remain open. First, the response does not yet establish which change-of-measure, ESS, and variance results survive feedback-dependent reuse of ysy_s. Could you state precisely which theoretical guarantees apply to the implemented estimator? The fixed-feedback fresh-draw comparison also remains unanswered.

Second, Figure 1 isolates the score only in a synthetic task. Figure 2 changes the rollout allocation but retains shared estimation, so it does not compare shared with separate estimation or isolate the proposed components on a matched real task. Its endpoints also contain 1 and 127 student rollouts out of 128, rather than all-student and all-teacher pools.

Finally, three-checkpoint intervals do not measure independent training-run variability...

Reviewer_wE6oOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses online policy distillation with language feedback and proposes a hybrid policy distillation method. The authors provide an analysis of the properties of the corrected target and the estimator under ideal hybrid sampling, and demonstrate performance improvements over SDPO and W2S-OPD across benchmarks covering knowledge acquisition, embodied behavior, scientific reasoning, and tool use.

Critical strengths

The paper studies a significant problem.

Critical weaknesses

The paper could benefit from improved writing: the introduction is a bit lengthy and lacks a conclusion.

The proposed method simultaneously alters the target distribution, the source of trajectories, and the loss function, making it impossible to isolate the individual contribution of each component. In particular, there is a lack of a control experiment for SDPO that utilizes teacher trajectories generated under an equivalent computational budget.

Compared to SDPO, this method requires both feedback-conditioned and unconditioned teacher scores, as well as the generation of additional teacher trajectories. The paper provides no breakdown of these costs or comparisons based on equal computational budgets; thus, it remains unclear whether the performance gains justify the added overhead.

The theoretical analysis assumes sampling from a corrected target distribution, whereas the actual implementation uses a feedback-conditioned teacher as the proposal distribution complete with weight truncation.

Questions for authors

Please see weakness.

2 comments

The proposed method simultaneously alters the target distribution, the source of trajectories, and the loss function, ...

Let us clarify this since we provide evidence for both components. We modify two components. We replace the raw teacher with a corrected teacher in OPD's divergence and use a symmetric divergence instead of OPD's reverse KL divergence. The source of trajectories is simply a consequence of having a symmetric divergence, since the expectation in the forward KL term of this divergence is over the trajectories generated by the teacher. Instead of simply running the end-to-end benchmarks with each component and showing it's better than the baselines, we wanted to show how and under what circumstances each component actually helps. To more clearly show the advantage of the corrected target, we constructed the controlled experiment in Figure 1. It shows that our corrected target outperforms the closest contrastive baseline when the teacher is already inclined towards a token and the feedback further reinforces that preference. When the teacher does not favor the correct token at all, the corrected target is at least as good as the baseline. For the other component, we vary the number of student rollouts in Section 5.3 to see how the influence of each leg of the symmetric divergence changes in the final objective. At the two extremes, we see how training behaves when all rollouts come from the student or all come from the teacher. Specifically, Figure 2 shows that a non-negligible fraction of teacher rollouts in the batch is enough to obtain the benefit of the symmetric objective. We believe this figure offers a more honest and accurate ablation than simply removing the forward leg and showing that performance drops.

Compared to SDPO, this method requires both feedback-conditioned and unconditioned teacher scores, as well as the generation of additional teacher trajectories. The paper provides no breakdown of these costs or comparisons based on equal computational budgets; thus, it remains unclear whether the performance gains justify the added overhead.

The overhead is two extra teacher prefills on the student rollouts. However, we aren't generating additional trajectories per batch. The batch size is fixed. SDPO samples the entire batch from the student, while we sample a fraction 1−λ1-\lambda from the teacher and the rest from the student. We'll mention the two extra prefills as a limitation of our approach. However, we'd like to note that OPD methods are among the most efficient post-training methods, unlike methods that require a group of rollouts for advantage estimation. We therefore believe a method that is more stable and achieves better performance will be beneficial.

The theoretical analysis assumes sampling from a corrected target distribution, whereas the actual implementation uses a feedback-conditioned teacher as the proposal distribution complete with weight truncation.

Naively sampling from the corrected teacher is expensive. While a custom inference kernel could make it more efficient, we found that only about 1% of the importance sampling ratios used to reweight tokens in the forward KL term are clipped (this approximation only affects the forward KL term) (Appendix D). This suggests that trajectories sampled from the raw teacher can be used, with appropriate weighting, to estimate the forward KL divergence between the corrected teacher and the student.

Thanks for the response. I believe my original score reflect my assessment to this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.