- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper compares supervised finetuning (SFT) and on-policy distillation (OPD). A theoretical perspective is taken; the paper connects SFT with forward KL and OPD with reverse KL and study the mathematical consequences of the respective implied target policies. The authors assert that 1. reverse KL may be more robust than forward KL against uninformative teachers, 2. reverse KL may be more unstable with misleading teachers, and 3. OPD may suppress correct prefixes when the horizon is sufficiently long. With these assertions the authors aim to provide insights into why OPD may outperform SFT and why it is sometimes observed unstable. For independent interest, the authors provide regret bounds in approximating the respective target quantities.
Critical strengths
Firstly, the decision to study the difference between SFT and OPD is one which is highly relevant for practical RL and distillation. Moreover, the choice of connecting SFT with forward KL and OPD with reverse KL is also a natural and relevant one respecting the natural modelling choice of many existing works. Within this relevant setting, the authors provided quantitative results characterising scenarios where OPD may work better or worse than SFT, particularly in the long-horizon setting - this provides grounds for better insights into the two practical methods.
Secondly, the paper is well written. I was able to understand the intended contribution of the paper, the setup of the problem, the assumptions and results of the paper clearly.
Critical weaknesses
As is with any theoretical work, demonstrating the connections between the described theoretical scenario and actual empirical observations is difficult. The scenarios described by the authors - propositions 5.1, 5.3 and 5.5 - are mathematical statements and not clear if they actually explains the situations seen in practice. To establish a connection between the theoretical formulation and practical observations, the authors would either need to analyse a family of large empirical experiments and show that the situations outlined in the paper occurs in a significant number of cases.
Questions for authors
- Assumption 4.3 asserts a one sided bound whereas the proof of theorem 4.5 uses a two sided bound. should assumption 4.3 be modified?
Response to Reviewer 2B6L
Q1. As with any theoretical work, demonstrating the connections between the described theoretical scenario and actual empirical observations is difficult. The scenarios described by the authors (Propositions 5.1, 5.3 and 5.5) are mathematical statements, and it is not clear whether they actually explain the situations seen in practice. To establish a connection between the theoretical formulation and practical observations, the authors would need to analyse a family of large empirical experiments and show that the situations outlined in the paper occur in a significant number of cases.
A1. We agree that this connection is the hardest part to establish, and we thank the reviewer for raising it. We respond in two parts.
New experiments. We have added training experiments with language models in the new Appendix "Experiments", which test two of the three scenarios.
Scope of the claims. We do not claim that these scenarios account for a significant fraction of the behavior seen in practice. Our claim is narrower: the two objectives have different optimal targets, and Propositions 5.1, 5.3, and 5.5 identify conditions under which this difference alone produces the retention and the fragility reported in the empirical literature. The new experiments show that two of these mechanisms can be reproduced in training under controlled conditions.
Q2. Assumption 4.3 asserts a one-sided bound whereas the proof of Theorem 4.5 uses a two-sided bound. Should Assumption 4.3 be modified?
A2. Assumption 4.3 already imposes a two-sided bound via absolute-value bars: log[pi(a∣x,u)/πref(a∣x,u)]≤B. We have rewritten it in the more explicit two-sided form −B≤logπref(a∣x,u)pi(a∣x,u)≤B to make this clearer.