acceptodds
Under review as a conference paper at ICLR 2027

Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning

15 views

Abstract

On-policy self-distillation (OPSD) is an emerging LLM post-training paradigm in which the model serves as its own teacher: conditioned on privileged information such as a reference trace or hint, the same policy provides dense token-level supervision on its own rollouts. However, recent studies show that OPSD degrades complex reasoning by suppressing predictive uncertainty, which supports exploration and hypothesis revision. Our token-level analysis shows that this failure arises from applying a uniform direction of teacher supervision across tokens with different uncertainty levels: conformity to the privileged self-teacher suppresses exploration at high entropy, while deviation from the teacher degrades step accuracy at low entropy. Accordingly, we propose Direction-Adaptive Self-Distillation (DASD), which reframes privileged self-distillation from uniform teacher imitation into entropy-routed directional supervision: high-entropy tokens are pushed away from the privileged teacher to preserve exploration, while low-entropy tokens are pulled toward the teacher to stabilize step-level execution. Across six mathematical reasoning benchmarks, DASD achieves the best macro Avg@16 over strong RLVR and self-distillation baselines. Pass@, execution-and-exploration, and generalization analyses show that these average gains come from preserving exploration without sacrificing step-level execution.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
On-Policy Self-Distillation, Uncertainty-Aware Learning, Reasoning Policy Optimization, Exploration in Language Model Reasoning, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ttuDOpenReview ↗
Rating
2(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper proposes DASD, an entropy-based reweighting algorithm for OPSD. The authors attempt to motivate DASD through a chain of three probes. However, I find several gaps in this logic. The paper is also difficult to read, and newly introduced terms such as “Conformity” and “Novelty” feel somewhat arbitrary. There is also a gap between the proposed metrics and the capabilities they are claimed to measure. Therefore, I currently lean toward rejection.

Critical strengths
  • The experimental results are comprehensive and relatively stable: DASD achieves strong performance across multiple model scales, mathematical benchmarks, thinking-mode settings, and cross-domain evaluations.
Critical weaknesses

The most significant issue lies in the main body of the paper, especially Sections 3 and 4. The authors introduce their method through three probes and three findings. The narrative appears logically structured at first glance, but I find that it does not hold up well under closer examination.

  • In Probe 1, the authors show two failure cases, both of which perform worse than GRPO. I find it difficult to see how this leads naturally to Probe 2; the result may simply indicate that both directions are worse than GRPO. Connecting these two failure cases to token entropy is quite confusing.
  • The claim that “high entropy = reasoning fork” and “low entropy = scaffolding” is not actually demonstrated, and prior work does not appear to have established this equivalence either.
  • Whether StepAcc and E(y) are valid proxies for execution ability and exploration ability is debatable, yet the conclusions of Probe 3 rely directly on these two metrics.

A second concern is that some benchmark scores appear unusually low. For example, after training, Qwen3-4B would normally be expected to achieve 90+ on MATH500 and 45+ on Minerva Math, while the paper reports only around 75+ and 30+, respectively. These results are closer to the level of Qwen2.5, which raises concerns about the experimental setup. The discrepancy may come from the decoding temperature, so I request additional evaluation at temperature = 0.2.

Questions for authors

See Weaknesses.

Reviewer_zfrNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces DASD, a credit assignment method for self-distillation with a privileged teacher. DASD uses student entropy to route teacher guidance, applying attraction at low-entropy positions and repulsion at high-entropy positions, together with a reliability gate. The paper reports improvements across mathematics, code generation, scientific reasoning, and tool use, supported by component ablations and intervention analyses.

The empirical results are promising. My main concern is that the diagnostic probes do not sufficiently establish the causal explanation used to motivate the routing rule. The later experiments support the utility of the proposed design, but the interpretation of high-entropy repulsion as promoting productive exploration remains less firmly established. Given the central role of this diagnostic account in motivating the proposed method, together with the insufficiently established contribution relative to closely related approaches, my assessment is weak reject.

Critical strengths

The main strength of the paper is its broad empirical evaluation. DASD demonstrates improvements across several domains and model families, with comparisons against GRPO and multiple self-distillation baselines, including RLSD, OPSD, and SDPO. These results provide a convincing empirical case for the effectiveness of the proposed approach.

The paper also provides useful and thorough ablations. Table 5 examines the contributions of the routing signal, direction map, and reliability gate. In particular, the comparison with low-entropy attraction alone supports the importance of the repulsive arm in the final training setup. The reported robustness across the tested hyperparameter ranges further strengthens the empirical findings. I also appreciate the interventions in Figure 6, which go beyond surface-level diversity metrics and provide evidence that selected high-entropy forks and subsequent revisions contribute to final correctness, rather than merely increasing epistemic markers.

Critical weaknesses

1. The diagnostic evidence supports investigating entropy-dependent guidance, but does not fully establish the proposed causal explanation.

The paper presents three probes as yielding its routing prescription: attract at low-entropy positions and repel at high-entropy positions. Although the observations motivate this design, several steps in the argument require clarification or stronger evidence.

Probe 1. It is unclear whether “GRPO backbone” means that the signed teacher signal is added to the verifier advantage AGA_G or replaces it. The objective around Line 130 and Appendix A does not make this explicit. If the verifier advantage is absent, the failure of uniform Novelty does not isolate the effect of teacher direction under verifier-anchored training. Table 5’s constant-negative direction ablation provides relevant evidence under the final training setup, but its relationship to Probe 1 should be clarified.

Probe 2. The association between entropy and teacher pressure makes entropy a reasonable candidate routing variable. However, identifying where teacher pressure is strongest does not itself establish which direction of intervention is beneficial there. In particular, the prescription to repel the student from the teacher at high-entropy positions requires evidence beyond this association.

Probe 3. At high-entropy positions, both Conformity and Novelty reduce StepAcc relative to unmodified rollouts. This does not rule out a beneficial trade-off between local execution accuracy and exploration. However, the probe does not report final-answer accuracy, so it does not establish whether the additional exploration improves reasoning outcomes. The reported benefit of Novelty is a 61.3% increase in E(y)E(y), a raw marker count that can be affected by response length; the corresponding length change is not reported. Probe 1 already demonstrates that marker counts and length can rise while reasoning performance deteriorates. Therefore, the increase in E(y)E(y) alone is insufficient evidence of productive exploration.

The later experiments provide meaningful evidence in favor of the design. In Table 5, low-entropy attraction alone achieves 16.3/11.7 on AIME24/AIME25, compared with 35.0/32.1 for full DASD. Appendix I.3 specifies that this ablation retains AGA_G and applies the attractive teacher term only at low-entropy positions. This is relevant training-level evidence for including the repulsive arm. Figure 5 supports distinct effects of the two routing arms, while Figure 6 connects selected forks and revisions to final correctness.

2. The added value of entropy-based directional routing over existing approaches remains insufficiently established.

Beyond the concerns about the diagnostic evidence discussed above, I am not fully convinced why this particular routing rule is needed relative to existing approaches. Entropy has already been widely explored as a signal for guiding token-level optimization and credit assignment. DASD's distinctive contribution is to use entropy not merely to weight updates, but to determine the direction of privileged teacher guidance. While the empirical improvements and ablations demonstrate the effectiveness of this design, the paper does not sufficiently establish why low-entropy attraction and high-entropy repulsion is the appropriate prescription, or what specific limitation of existing entropy-guided or teacher-guidance approaches this choice addresses. Without a stronger justification, it is difficult to assess whether the proposed routing principle offers a generalizable insight beyond an empirically effective heuristic.

This concern is particularly relevant given RLRT [2], which builds on the findings of [1], as does DASD. RLRT reinforces student-chosen tokens on correct rollouts using reversed teacher signals, whereas DASD uses entropy to determine the direction of teacher guidance. Although the two methods employ meaningfully different mechanisms, both explore the benefits of resisting privileged teacher guidance during reasoning. Given their shared motivation in [1], a direct comparison and a clearer discussion of their relationship would be particularly valuable for establishing what entropy-based direction selection contributes beyond existing teacher-guidance strategies.

References

[1] Kim et al., Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?, 2026.

[2] Kim et al., Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR, 2026.

Questions for authors
  • Could the authors specify the exact objective used in Probe 1, including whether it contains the verifier advantage AGA_G, and explain its relationship to the constant-negative direction ablation in Table 5?
  • For the fixed-model intervention in Probe 3, how do no intervention, Conformity, and Novelty at matched high-entropy prefixes compare in final-answer accuracy, response length, and length-normalized marker frequency?
  • How does DASD affect response length during training and evaluation relative to GRPO and the self-distillation baselines?
Reviewer_Jwv4OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a potential failure mode of on-policy self-distillation for reasoning: uniformly imitating a privileged self-teacher can suppress exploration, whereas uniformly moving away from the teacher can impair step-level execution. It proposes Direction-Adaptive Self-Distillation (DASD), which uses student token entropy to route the direction of teacher supervision. Low-entropy positions receive an attractive teacher signal, while high-entropy positions receive a repulsive signal. A normalized teacher–student log-probability gap, modulated by an entropy router and a magnitude gate, provides an additive token-level correction to the GRPO advantage.

The paper supports this design through token-level diagnostics, directional interventions, and ablations. Across six mathematical reasoning benchmarks and three Qwen3 scales, DASD achieves the highest macro Avg@16 among the evaluated baselines. Additional experiments examine thinking-mode training and evaluation, generation budgets, Pass@k scaling, other model families, and separately trained non-mathematical domains. The central contribution is an entropy-routed approach to balancing exploration and execution within privileged self-distillation.

Critical strengths
  1. A well-motivated, targeted design. The paper articulates a clear tension between preserving exploratory reasoning and maintaining reliable local execution. Using student entropy to choose the direction, rather than only the strength, of teacher supervision provides a concrete response to this tension. The connection between the diagnostic analysis and the proposed training rule is a strong aspect of the work.

  2. Broad and consistent empirical support. DASD improves macro Avg@16 over the strongest evaluated baseline by 2.5, 3.7, and 4.5 percentage points at the 1.7B, 4B, and 8B scales, respectively. The evaluation extends beyond the main non-thinking setting to thinking-mode experiments, larger generation budgets, other model families, and separately trained code, science, and tool-use tasks. This breadth strengthens the evidence that the method is useful beyond a single configuration.

  3. Analysis beyond aggregate accuracy. The paper examines execution and exploration separately and includes teacher-free controls, selective sign reversals, and interventions on revision behavior. In particular, the revision interventions assess final correctness rather than relying solely on the frequency of uncertainty markers. These analyses provide meaningful evidence for the proposed explanation, although the complementary ablation requested below would further strengthen it.

Critical weaknesses
  1. Missing important related works and comparison baselines: RLRT and AntiSD. Kim et al., Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR (RLRT), and Cheng et al., Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information (AntiSD), both reverse privileged-teacher supervision to promote exploration, but neither is discussed or evaluated in the current manuscript. Both are important direct comparators. RLRT applies bounded multiplicative reweighting to the GRPO advantage on correct rollouts. AntiSD instead uses an additive correction derived from JSD ascent, stabilized by a gate based on batch-aggregated teacher entropy. DASD differs by using student token entropy to route attractive versus repulsive supervision. Direct discussion and matched-protocol comparisons with both methods are needed to establish the additional value of this routing. The uniformly repulsive Novelty baseline substitutes for neither: it lacks RLRT’s correctness gating and reweighting mechanism, as well as AntiSD’s JSD-based signal shaping and teacher-entropy gate. The characterization of existing teacher signals as uniformly attractive should also be qualified accordingly.

  2. Missing the complementary high-entropy-repulsion-only ablation. Table 5 includes low-entropy attraction only, but not the complementary configuration that retains GRPO and high-entropy repulsion while setting the low-entropy teacher correction to zero. Without this comparison, it remains unclear whether high-entropy repulsion accounts for most of the gains, and how much low-entropy attraction adds to the full method. Reversing the low-entropy arm demonstrates that an adverse direction can be harmful; it does not establish the benefit of retaining that arm over removing it. Similarly, repelling all tokens is not equivalent to repelling only high-entropy tokens. This concern is about isolating the incremental contributions of the two arms, not an absence of high-entropy analysis in general.

Questions for authors

Please refer to the weakness.

Comments on the paper

1 comment

Thank you for the detailed and constructive response. My main concerns have now been largely addressed. Although the evaluation issue has been resolved, it was a serious error that should have been caught before submission, rather than during peer review. Please also check and update other affected results in the appendices.

My other major concern is readability. The evidence cited in the response largely addresses my concerns about rigor. The experiments support entropy-dependent guidance directions and provide some evidence. However, the stronger claim that repulsion from the teacher at high-entropy positions improves reasoning by promoting effective exploration remains a partially supported mechanistic hypothesis rather than a firmly established conclusion. The argument connecting the diagnostic findings, the need for teacher repulsion at certain tokens, and the use of entropy to route that guidance is complex. I still find the readability of Sections 3 and 4 remains a significant concern. I encourage the authors to improve the organization and presentation of these sections, simplify the chain of reasoning, and distinguish the empirical findings from their mechanistic interpretation more clearly.

Overall, my assessment has shifted from leaning toward rejection to a borderline position where either acceptance or rejection would be reasonable. I have raised my score to 2 and lowered my confidence. My underlying assessment is slightly above 2, but does not reach 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.