acceptodds
Under review as a conference paper at ICLR 2027

Strong Yet Learnable: Rethinking What Reasoning Distillation Needs

Abstract

Reasoning distillation transfers the reasoning capabilities of large language models to smaller students by supervising their intermediate decisions. However, stronger teachers do not necessarily produce better students, as increasingly sophisticated reasoning can become difficult for weaker students to absorb. On-policy self-distillation reduces this mismatch by using the same student as a privileged teacher with access to the verified outcome, referred to as hindsight. Yet its decisions rely on information unavailable to the deployed student and may therefore be difficult to reproduce causally. This raises a more fundamental question: what makes supervision worth learning? Useful supervision must provide knowledge beyond the student's current behavior while remaining learnable for the student. Crucially, these two requirements need not be satisfied by the same source. Hindsight provides a way to assess learnability: it gives the same student additional task information while keeping the model and reasoning state fixed, revealing whether a teacher signal remains compatible with the student under richer information. We therefore introduce HinD, an on-policy distillation framework that decouples knowledge provision from learnability assessment: a strong external teacher supplies new reasoning, while the student's hindsight response assesses its learnability and weights the corresponding supervision accordingly. Across multiple student scales and model families, HinD consistently improves mathematical reasoning and transfers to coding and knowledge-intensive reasoning, with gains that persist under larger sampling budgets and grow with teacher strength.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, on-policy self-distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cfCEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies on-policy distillation (OPD) for reasoning with a much stronger teacher, arguing that useful supervision must be new to the student and learnable by it. HinD keeps the teacher's next-token distributions as targets but weights each reasoning position by the bounded drop in the student's KL to the teacher once the student also sees the verified answer, plus an unweighted teacher term on non-reasoning tokens and a reverse KL to the initial student. With Qwen3-1.7B, Qwen3-4B and Nemotron-Nano-4B students trained for 200 updates on DAPO-Math-17K, HinD has the best math and out-of-distribution averages against GRPO, OPD and OPSD in all three settings, with a 0.3-point math margin on Nemotron.

Critical strengths
  1. A cheap change to OPD with no extra cost at inference. The hindsight score needs only one teacher-forced pass of the same student, and the extra models serve only training. The method is therefore easy to add to an existing OPD pipeline.

  2. The matched-allocation controls test the placement claim directly. Table 3 fixes the total weight or the set of weights and changes only where they go. Spreading the same total weight uniformly, in effect a scaled-down OPD, stays at the Base OOD level while HinD is clearly higher, so the gain is not from distilling less.

Critical weaknesses
  1. It is unclear whether the gains exceed training noise, and the reported checkpoint looks favorable. No repeated training is reported: the Table 1 spread comes from decoding seeds, baselines carry none, and several headline gains amount to a single AIME25 problem (3.33 points). In the longer run of Fig. 5, the step-200 column, which reproduces Table 1, coincides with the highest point of the MATH, MMLU-Pro and GPQA-D curves, earlier checkpoints lose to Base on code and Olympiad problems, and later OOD gains hover near zero, yet the 200-update budget is never justified and no baseline curves are shown. The abstract's "consistently improves" also holds only for the aggregates, since HinD trails the best baseline on some benchmark in every setting. Without repeated training and checkpoints chosen on held-out data, the real margin over OPD, OPSD and GRPO, the paper's main empirical claim, is unknown.

  2. The experiments show that placement matters, not that high-weight positions are easier to learn. The weight is a teacher-weighted change in the student's log-probabilities once the answer is revealed, and by Bayes' rule this equals how much the teacher's step raises the student's own estimate of reaching that answer, essentially an implicit process reward (Yuan et al., ICML 2025). High weight may therefore mark outcome credit rather than learnability, and the student-specificity argument is never compared with the teacher's own hindsight gain. Nor is the weighting positioned against its closest uncited antecedents on learnable-point selection (Mindermann et al., ICML 2022), weaker teaching by larger teachers (Zhong et al., ACL 2024) and teacher-token replacement in on-policy distillation (Xu et al., ICLR 2025). Since the title and conclusion sell learnability for this student, a simpler reading that explains Table 3 equally well undercuts the central contribution.

Questions for authors
  1. Could you report several training runs of all four methods on Qwen3-1.7B, with baselines at the Fig. 5 checkpoints and a checkpoint selected on the held-out DAPO split, and say whether the Table 1 baselines are single decodes (almost all their AIME25 entries are multiples of 1/30)?
  2. Does the advantage survive if the teacher's hindsight gain replaces the student's? Splitting the gains by rollout correctness and comparing post-training KL drops at high- and low-weight positions would also help.
  3. The mean reasoning weight is only about 0.5 to 1.0 percent, while the non-reasoning and reference terms keep fixed coefficients. How much of the gain remains with w = 0, and how sensitive is it to λ_C and λ_ref?
  4. The growth with teacher strength rests on one run per teacher, is nearly flat from 8B to 14B, and OPD with the 32B teacher shows no math gain over Base. Does the trend hold across seeds?
Reviewer_aAdmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether reasoning supervision from a strong teacher is suitable for a weaker student to learn. It proposes HinD, which separates knowledge provision from the assessment of supervision suitability. During training, the student first generates its own reasoning trajectories. An external teacher provides predictive distributions on the same prefixes, while the student scores those prefixes with and without the correct answer. HinD maps the reduction in KL between the teacher and student after adding the answer to bounded weights that regulate supervision at each reasoning position. The external teacher remains the distillation target; hindsight affects only the weights. No additional model calls are needed at inference. The experiments cover two Qwen3 student sizes and Nemotron, evaluating mathematical reasoning and transfer to coding and knowledge-oriented reasoning. The authors also examine different teacher sizes and larger sampling budgets, and use controls matched in total supervision weight and weight distribution to study the effect of weight allocation.

Critical strengths
  1. The matched controls are well designed. Table 3 controls for total supervision weight and its distribution, and includes an allocation based on teacher-to-student KL under the causal view. These comparisons support the effectiveness of hindsight-based weight allocation.
  2. The evaluation covers two model families and examines transfer to coding and knowledge-oriented reasoning beyond the mathematical tasks. The Pass@k results in Table 2 also support improved solution coverage.
Critical weaknesses
  1. My main concern is why the hindsight intervention works. The allocation controls in Table 3 account for total supervision weight and its distribution, supporting the usefulness of the proposed ranking. However, these experiments primarily change how weights are assigned; they do not directly test the specific role of providing the correct answer. Adding the answer supplies task information and changes the input context, and the contribution of each to the weights and training gains remains unclear. Interpreting hindsight gain as student "learnability" therefore needs further mechanistic support. Comparing correct, incorrect, and unrelated answers under comparable prompt formats, lengths, and weight budgets would help clarify the role of answer correctness.
  2. A secondary concern is that the effect of training randomness has not been quantified. Table 1 reports results over three decoding seeds but does not report variation across independent training runs. It is therefore difficult to assess how sensitive the main comparisons are to training randomness.
Questions for authors
  1. Main question: Could you compare correct, incorrect, and unrelated answers while keeping the prompt format, length, and supervision weight budget comparable, to examine how correct-answer information affects the ranking and training gains? This would help explain why the hindsight intervention works.
  2. How many independent training runs underlie the main results? Could you report training variation for a representative configuration to complement the existing decoding-seed results?
Reviewer_9eyeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper answers the question "what makes supervision worth learning" in the context of online self-distillation using privilege information. More specifically, it is hard to replicate the reasoning causality using simply distillation. The authors propose HIND to identify knowledge beyond the student's current behavior but still learnable. The learnability is assed by giving the student additional task information and checking whether a teacher signal remains compatible for the student. Experiments show how HIND improves mathematical reasoning and transfer to coding.

The hint method freezes during training the current student policy and trajectories without verified answers. For each reasoning token (and its prefix), the authors measure three KL divergences: 1) the teacher next-token prediction given the prefix, 2) same but with the student, and 3) same as 2 but with the privilege information. All the KL divergences are combined into a unified loss.

The experiments evaluate Qwen and Llama models at different scale using the DAPO-Math-17K dataset as training and multiple math and other datasets as evaluation. In terms of baseline, the authors use the base student, OPD-ed, OPSED-ed, and standard GRPO. On average, HIND outperforms the other methods on in- and out-of-domain datasets. The authors also show that that HIND contributes to the student learnability information beyond causal mismatch and changes which teacher signals receive the greatest influence during the distillation. Additional analysis are sound.

Overall, this paper is very well written and propose an elegant solution to improve OPD/OPSD weaknesses. Empirical results are strong. My only criticism would be 1) the size of the training is limited (does the approach scale), and 2) the lack of computational overhead analysis. At what price does the performance improvement come?

Critical strengths

Well-written paper, elegant method.

Critical weaknesses

small training set, computational overhead.

Questions for authors

See summary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.