- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
On-policy distillation provides dense token-level feedback to student generations but assigning this feedback across the generation across tokens can be tricky. Uniform allocation is bad because not all tokens carry the same importance for a sequence and not all tokens differ between student and teacher similarly. So making OPD better or more efficient is down to figuring out which tokens to grade and how to grade them. ACT uses a known technique of using a future window for grading tokens but presents a novel grading method that decouples position selection from feedback construction. The position selection preserves disagreement before the sign cancellation and feedback retains it's direction and net effect. ACT reuses the teacher and student logprobs already computed for OPD (by computing shaped directional, unsigned future Mismatch and signed future feedback). Controlled experiments justify the effect of each design choice and the overall methods provides a clear win in the models and settings tested.
Critical strengths
To start with, the core idea is developed with a scientific mindset and the paper is very clearly written which makes it easy to follow.
Explanation of why later feedback can act on earlier actions is well-explained and there are clear ablations on this.
ACT separate two decisions: where to provide feedback and what feedback to provide. ACT develops a statistic for each and motivates what each statistic M_t and F_t measures and how M_t-|F_t|, the cancellation gap, leads to the behaviours observed in practice (teacher alternating between supporting and suppressing student action in a window) which can sometimes be non-intuitive. Unsigned future mismatch M_t defines where to intervene and signed future feedback F_t defines how to intervene (constructs the feedback to apply at that point)
There is evidence presented in the paper in the form of experiments that future information contributes to both position selection and feedback aggregation.
The core idea is well-motivated idea that has been thoughtfully developed and clearly articulated in the paper. Each concept is justified and explained and then ablated cleanly, making the paper a good piece of scientific exploration.
Experiments or justifications for feedback control, gate control and local vs future mismatch are clearly presented.
Design choices are nicely ablated and experimental results are convincing even if not exhaustive.
Critical weaknesses
Some of the visualizations are a bit misleading. Figure 1 done as a table instead of a bar chart (given the axes are clipped) would red differently (results would not look as impressive), however this is a cosmetic issue.
Figure 3 is again not easy to interpret. Would benefit from a longer caption explaining the colouring and blocking.
There isn't substantial evidence that the method scales to larger student/teacher sizes (ack that compute availability could be a blocker)
Questions for authors
Can you comment (not necessarily do new experiments) on how the results will scale to sparser feedback situations and larger student model sizes, especially multi-modal ones where the tokens could be either a few characters or an image patch etc?
I thank the authors for their response and new experiments. Majority of my concerns are addressed with the new experiments. I have updated my rating to 3.