- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper shows that EOPD's renormalized top-k forward loss in on-policy distillation equals KL(q‖p) − log P, so it implicitly targets unit selected mass and teacher matching is not stationary. The proposed approach, ReTaCo, instead uses a (k+1)-way target that keeps the tail as one residual symbol with mass (1−β)(1−m), plus a single-sample reverse-KL estimator. Theorem 3.4 gives a unique fixed-prefix optimum with selected mass in [m, r_β], monotone in β. Three student–teacher pairs show gains over Sampled OPD and EOPD on math and code.
Critical strengths
-
The paper has a sound theoretical investigation. The mass decomposition isolates the implicit unit-mass target. A drop-in O(k) loss with an interpretable knob: β=0 preserves teacher mass, β=1 recovers EOPD's forward target.
-
The empirical results are promising across different model family. Table 4 and Fig. 3a match the predicted optimum to ~1e-8. I appreciate the authors acknowledge the mixed OOD results and the confounded EOPD comparison.
-
The paper is easy to follow.
Critical weaknesses
- The conditional-only variant R + mC matches or beats full ReTaCo (TAMC 54.37 vs 52.56; MATH500 83.55 vs 83.78). Since mC is the within-S-normalized loss EOPD already supports, the paper's central contribution seems somewhat to lack strong empirical support.
- The diagnosed effect seems not what training shows. At k=16, m≈0.998, and ReTaCo and EOPD selected masses differ by only 0.0014. Both students keep more tail mass than the teacher, opposite to the inflation predicted by Prop. 3.3, and ε-projection is active at 31.65% of positions. The theory seems not fully explain the gains.
- 47–50% of ablation responses hit the 8,192-token cap (Table 9), and β=0.5 causes length degeneration, so Fig. 3b may measure termination failure, but probably not mass control.
Questions for authors
- Can the authors report truncation rates for every table row and evaluate at a larger length budget
- Can the authors measure actual communication savings versus full-vocabulary OPD
- EOPD runs in a different stack (OpenRLHF, PPO clipping, entropy gate, different GPU placement). However, the paper reports a single-run results. I think it's more meaningful to report the multi-seed or multi-stack experiments.