- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
the paper tackles a real pain point in on-policy distillation: the per-token teacher signal on student rollouts stops being useful once the student drifts away from the teacher .
DTDD adds a takeover rule on top of plain OPD: chop the rollout into fixed-length segments, compute an averaged nonnegative student/teacher discrepancy per segment, and the first segment crossing a threshold triggers a rewind — keep the student prefix, let the frozen teacher regenerate the suffix, then train the prefix with OPD and the teacher-written suffix with behavior cloning. no trigger means you get exactly OPD back.
Critical strengths
the mechanism is genuinely simple and it is very effective
Critical weaknesses
Table 3's headline is best-observed checkpoint picked every 20 updates, which is exactly what inflates a +4.8 claim.
the long-horizon contribution is also partly undercut by their own clipping control. and scope is narrow: students 0.5B-3B, teachers to 7B, budgets to 18K, only math plus two agent suites;
Questions for authors
see the weakness above.
We thank the reviewer for the careful review. The ALFWorld LoRA gain persists at the final shared checkpoint, and the clipping comparison on DAPO shows where DTDD helps early in training.
W1. Checkpoint selection. In the original ALFWorld LoRA run, DTDD’s advantage was larger at the shared final checkpoint (+7.53 percentage points) than at the methods’ respective peaks (+4.83 points). Table 3 compares DTDD at update 140 (81.32%) with OPD at update 280 (76.49%). At update 400, DTDD reached 77.59% and OPD reached 70.06% (Table 6 in the submission, Table 18 in the revised manuscript).
W2. If clipping already stabilizes OPD, what does DTDD add?
Clipping stabilizes OPD, and DTDD also accelerates early learning. On the same DAPO seed, DTDD first exceeded 20% accuracy at update 60, compared with update 200 for both vanilla and clipped OPD (Table 17 in the revision). Vanilla and clipped OPD give teacher feedback along the path the student takes. When the student’s reasoning goes off track, DTDD lets the teacher continue from the student’s prefix and show it how to proceed. This matters early in training, when a base student may rarely generate a useful reasoning path on its own.
We tested clipping in the long-horizon stability experiments (Table 10 in the submission, §4.2 and Table 8 in the revision) and added a clipped OPD run to the DAPO comparison using the same base student, teacher, and rollout seed. At update 80, DTDD reached 21.34% accuracy, compared with 13.40% for clipped OPD and 13.99% for vanilla OPD. At update 120, DTDD remained ahead of both (§4.3 in the revision). At update 200, vanilla and clipped OPD scored 21.25% and 22.12% on seed 1, and DTDD scored 21.96% in the longer seed-0 run (Tables 16 and 17 in the revision).
Scope. Our experiments cover math reasoning, ALFWorld, and agentic coding, with students from 0.5B to 3B parameters, teachers up to 7B, and response budgets up to 18K tokens. We agree that testing whether the gains extend to larger students and teachers and to longer response budgets is an important next step. Given our compute budget, we leave that evaluation to future work.