- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies on-policy distillation (OPD) for reasoning with a much stronger teacher, arguing that useful supervision must be new to the student and learnable by it. HinD keeps the teacher's next-token distributions as targets but weights each reasoning position by the bounded drop in the student's KL to the teacher once the student also sees the verified answer, plus an unweighted teacher term on non-reasoning tokens and a reverse KL to the initial student. With Qwen3-1.7B, Qwen3-4B and Nemotron-Nano-4B students trained for 200 updates on DAPO-Math-17K, HinD has the best math and out-of-distribution averages against GRPO, OPD and OPSD in all three settings, with a 0.3-point math margin on Nemotron.
Critical strengths
-
A cheap change to OPD with no extra cost at inference. The hindsight score needs only one teacher-forced pass of the same student, and the extra models serve only training. The method is therefore easy to add to an existing OPD pipeline.
-
The matched-allocation controls test the placement claim directly. Table 3 fixes the total weight or the set of weights and changes only where they go. Spreading the same total weight uniformly, in effect a scaled-down OPD, stays at the Base OOD level while HinD is clearly higher, so the gain is not from distilling less.
Critical weaknesses
-
It is unclear whether the gains exceed training noise, and the reported checkpoint looks favorable. No repeated training is reported: the Table 1 spread comes from decoding seeds, baselines carry none, and several headline gains amount to a single AIME25 problem (3.33 points). In the longer run of Fig. 5, the step-200 column, which reproduces Table 1, coincides with the highest point of the MATH, MMLU-Pro and GPQA-D curves, earlier checkpoints lose to Base on code and Olympiad problems, and later OOD gains hover near zero, yet the 200-update budget is never justified and no baseline curves are shown. The abstract's "consistently improves" also holds only for the aggregates, since HinD trails the best baseline on some benchmark in every setting. Without repeated training and checkpoints chosen on held-out data, the real margin over OPD, OPSD and GRPO, the paper's main empirical claim, is unknown.
-
The experiments show that placement matters, not that high-weight positions are easier to learn. The weight is a teacher-weighted change in the student's log-probabilities once the answer is revealed, and by Bayes' rule this equals how much the teacher's step raises the student's own estimate of reaching that answer, essentially an implicit process reward (Yuan et al., ICML 2025). High weight may therefore mark outcome credit rather than learnability, and the student-specificity argument is never compared with the teacher's own hindsight gain. Nor is the weighting positioned against its closest uncited antecedents on learnable-point selection (Mindermann et al., ICML 2022), weaker teaching by larger teachers (Zhong et al., ACL 2024) and teacher-token replacement in on-policy distillation (Xu et al., ICLR 2025). Since the title and conclusion sell learnability for this student, a simpler reading that explains Table 3 equally well undercuts the central contribution.
Questions for authors
- Could you report several training runs of all four methods on Qwen3-1.7B, with baselines at the Fig. 5 checkpoints and a checkpoint selected on the held-out DAPO split, and say whether the Table 1 baselines are single decodes (almost all their AIME25 entries are multiples of 1/30)?
- Does the advantage survive if the teacher's hindsight gain replaces the student's? Splitting the gains by rollout correctness and comparing post-training KL drops at high- and low-weight positions would also help.
- The mean reasoning weight is only about 0.5 to 1.0 percent, while the non-reasoning and reference terms keep fixed coefficients. How much of the gain remains with w = 0, and how sensitive is it to λ_C and λ_ref?
- The growth with teacher strength rests on one run per teacher, is nearly flat from 8B to 14B, and OPD with the 32B teacher shows no math gain over Base. Does the trend hold across seeds?