- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposed a knowledge distillation (KD) algorithm. The work starts from the sequence-level KD which is supervised fine-tuning on teacher-generated trajectories. To mitigate the issue that the teacher's labels may lie in the space far beyond the reach of the student, they propose a mixture policy which is the convex combination of teacher and student distribution, and use this mixture policy to generate trajectories on which the student is trained. To ensure the quality of these trajectories, further quality filtration designs are applied. Through experiments, they show the effectiveness of their proposed algorithm.
Critical strengths
None
Critical weaknesses
- The main idea is just using the mixture policy of teacher and student as the target in sequence-level KD, which is nothing special. Using the mixture distribution is not new in knowledge distillation, for example, people usually use JS divergence as the objective, where the target is naturally mixture policy. The difference here is just that the prefix is also generated with this mixture policy. From a principled perspective, I didn't see the benefits of conditioning on the prefix generated by the mixture policy, so I would regard this work as a totally empirical work with an intuitive algorithm. It needs extensive experiments to support the claims. However, I didn't see such experiments, instead, the work uses outdated models, outdated and extremely simple benchmarks. The comparisons are unclear either, for example, logins-based KD is not explained and the simple on-policy KD baseline is missing (default GKD uses a combined objective). Only instruct models are used, no reasoning models or base models.
Questions for authors
No