- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Learnability-Weighted Distillation (LWD) to address imperfect teacher supervision in offline on-policy distillation. The method trains an auxiliary student on teacher-successful problems and measures token-level likelihood changes on teacher-failed trajectories. These signals are aggregated into trajectory-level weights for the final OPD objective. Experiments on mathematical reasoning and code generation demonstrate consistent improvements over Lightning-OPD across two student scales, with gains of up to 2.7 percentage points. LWD also achieves competitive performance against online OPD methods at lower computational cost.
Critical strengths
- Important and well-motivated problem. The paper addresses a practical limitation of offline OPD: imperfect teacher supervision can be repeatedly reinforced during training. The observation that teacher failure does not necessarily invalidate all supervision along a student trajectory is meaningful.
- Simple and practical methodology. LWD provides an efficient alternative to expensive continuation-based estimation. Its trajectory-level weighting preserves the original token-level OPD objective and does not require additional generation for scoring.
- Consistent empirical improvements. The method improves the offline OPD baseline across mathematical reasoning and code generation at two student scales. The paper also includes relevant ablations and computational cost comparisons, demonstrating competitive efficiency relative to online OPD methods.
Critical weaknesses
- Insufficient validation of the learnability signal. The proposed signal measures how successful-only learning changes the likelihood of observed student tokens, rather than the correctness or utility of teacher supervision. While Eq. (15) provides a gradient-alignment interpretation, it does not establish that higher learnability corresponds to more beneficial supervision. Figure 3 offers only qualitative examples, and the paper lacks systematic validation or controls against simpler proxies. In particular, a shuffled-tier baseline is needed to verify whether trajectory-specific learnability rankings actually contribute to the improvements.
- Limited incremental gains over simpler baselines. Although LWD consistently improves Lightning-OPD, its advantage over constant downweighting is modest. In Table 5, LWD achieves 57.1% average accuracy, compared with 56.6% for the best constant-weight baseline. Meanwhile, its GPU-hour cost approximately doubles relative to Lightning-OPD (21.96 vs. 10.64). Without independent training-seed results, statistical uncertainty estimates, or compute-matched comparisons, it remains unclear whether the additional complexity is justified.
Questions for authors
- Does learnability provide information beyond simpler weighting strategies? Could the authors compare against shuffled learnability tiers and inexpensive difficulty-based proxies while preserving the same weight distribution?
- Are the improvements over constant weighting statistically robust and compute-efficient? Could the authors provide results across multiple training seeds, uncertainty estimates, and compute-matched comparisons?
- How sensitive is LWD to teacher-outcome labels? Appendix E reports different mathematics verifier versions and uses response validity rather than functional correctness for code. Could the authors quantify the impact of these choices and examine whether learnability tiers correlate with independently measured supervision quality?