- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues that verbal feedback provides a rich learning signal for On-Policy Distillation (OPD). To leverage this feedback, the authors introduce a confirmation score (S(v)), defined as the teacher's log-probability ratio with and without verbal feedback (F). This score is incorporated as an additional weighted objective alongside the standard OPD loss. Their empirical results demonstrate that the proposed approach achieves strong performance compared to SDPO and W2S-OPD.
Critical strengths
None
Critical weaknesses
-
The idea of leveraging verbal feedback is well-established and has already been explored in the context of distillation (e.g., Self-Distillation [1]. The proposed method appears to be a straightforward weighted combination of OPD and OSPD. Moreover, much of the derivation of the corrected teacher target is trivial and follows well-known principles [2], with limited theoretical novelty.
-
The paper is extremely poorly written, with several key methodological choices lacking rigorous explanations. Specifically, the authors do not adequately motivate the introduction of the confirmation score, justify why the four proposed conditions are necessary, or explain the need for a symmetric objective. Several concepts, such as preferences and probabilistic confirmation, are introduced without clear definitions or sufficient explanation. Furthermore, important details of the experimental setup (e.g., those related to Figure 1) are not in the main paper.
[1] Self-Distillation Enables Continual Learning. ICML 2026 Spotlight.
[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 Oral.
Questions for authors
- Could the author explain the motivation for the confirmation score? Why is the log-ratio score a good confirmation score?
- What's the real difference between the author's objective and a weighted sum of the standard OPD and Self-Distillation OPD objectives? What's the motivation behind symmetric rollouts?
The log-probability ratio is the score used by our W2S baseline, another contrastive method. Our proposed score is relative distance. Please refer to Section 3.
The score is not used as an additional weighted objective. The score is used to construct a target which replaces the raw teacher in the proposed divergence. Please refer to Section 4.
We are not claiming the concept of learning from verbal feedback is our contribution. It is our motivation, and we propose an improved method for learning from such feedback. Please refer to the contributions in the introduction.
We cite DPO between Eqs. (5) and (6), where we construct the target based on our proposed score function. We did not claim our derivation is new. Please refer to the contributions in the introduction.
Please refer to Appendix C.
Our objective in Eq. (9) is simply the Jeffreys divergence, an unweighted sum of reverse KL and forward KL, and the weights in Eq. (11) are importance sampling weights to fix the distribution mismatch for each leg of the divergence. OPSD is a special case of OPD in which the same model serves as student and teacher, with privileged context supplied to the teacher. Neither that distinction nor a possible combination of OPD and OPSD is relevant to our contribution. Please refer to the respective papers.
As noted earlier, we propose relative distance, not the log-ratio score. We focus on property A3 because it is the only property that our relative distance score satisfies and the log-ratio baseline lacks. Please refer to the motivation for A3 in lines 206–208 and the supporting experiment in Figure 1.