acceptodds
Withdrawn from ICLR 2027

Locality Guided Self-Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy self-distillation (OPSD) trains a language model on its own responses using supervision from the same model given privileged, training-only context. However, this privileged context can improve reasoning while also altering response style, so matching the teacher’s full predictive distribution may reinforce changes unrelated to the desired reasoning improvement. We introduce the **Target-Locality Hypothesis**, with two testable predictions. **P1 (Correction Retention):** distillation targets that remain local to the student’s predictive distribution can retain most of the useful corrections in the privileged proposal. **P2 (Preference Advantage):** learning from such local targets can yield responses that align better with human preferences than either the unadapted student or full distillation. Motivated by this hypothesis, we propose **Locality-Guided Self-Distillation (LGSD)**, which constructs a distillation target that is as close as possible to the privileged proposal while remaining within a response-level KL budget around the current student, and then trains the student on this local target. Empirical studies, including human-preference evaluation on Arena, provide evidence for both P1 and P2. Across mathematical and logical reasoning benchmarks, LGSD improves over matched OPSD and other baselines, while additional analyses show improved training stability and reduced style drift.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning, self-distillation, training stability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HfcEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the instability of on-policy self-distillation (OPSD), which uses predictions from the same model conditioned on privileged reasoning instructions as teacher signals. However, these instructions may introduce unwanted changes in response style alongside useful reasoning corrections, potentially destabilizing training. To address this issue, the authors propose Locality-Guided Self-Distillation (LGSD), which constructs a training target by interpolating between the student and teacher distributions. The interpolation coefficient α\alpha is adaptively selected for each response to keep the average KL divergence between the target and student within a predefined threshold. Experiments on mathematical reasoning benchmarks show that LGSD improves accuracy by 10.49 percentage points over OPSD on Qwen3-8B while reducing style drift.

Critical strengths
  1. The proposed method is straightforward to implement and introduces little additional computational overhead, making it easy to integrate into existing self-distillation pipelines.
  2. The method achieves substantial performance gains over OPSD on mathematical reasoning benchmarks while mitigating performance degradation during prolonged training. Although the results are encouraging, stronger evidence across more random seeds, additional models, and longer training runs is needed to establish their robustness. The underlying mechanism also remains insufficiently explained.
Critical weaknesses
  1. KL regularization and geometric interpolation between student and teacher distributions are well-established techniques in post-training. Closely related methods, particularly β\beta-OPSD [1] and Veto [2], already employ similar target constructions. The paper does not sufficiently clarify the methodological novelty of its response-adaptive KL projection beyond these existing approaches.

  2. KL projection uniformly scales teacher–student log-probability differences without distinguishing useful reasoning corrections from undesirable stylistic changes. It remains unclear why partial distillation should outperform full distillation, or whether the latter could achieve comparable performance with better-tuned training. Moreover, Table 9 shows higher training-seed standard deviation for LGSD than OPSD (2.46 versus 1.61), despite higher mean accuracy. Thus, the mechanism behind the improvements and the claimed training stability remain insufficiently explained.

  3. Although the paper includes several ablations, more targeted comparisons are needed to isolate the contribution of response-adaptive projection. In particular, comparisons with properly tuned fixed or scheduled interpolation coefficients under matched target KL would be valuable. Evaluations with privileged teachers of varying quality and additional training seeds would further strengthen the evidence for generality and robustness. A comparison with β\beta-OPSD would also help clarify the relationship to concurrent work.

[1] Xu, Jiawei, et al. "$\beta $-OPSD: Deriving with Policy Optimization, Training with Self-Distillation." arXiv preprint arXiv:2607.28582 (2026).

[2] Jang, Ijun, et al. "Stable on-policy distillation through adaptive target reformulation." Findings of the Association for Computational Linguistics: ACL 2026. 2026.

Questions for authors
  1. Could the authors explain more clearly the mechanisms behind the accuracy and stability improvements, particularly given the higher training-seed variance in Table 9?
  2. Could the authors provide additional ablations, as suggested in the weaknesses, to verify the robustness of the improvements and isolate the contribution of the proposed method?
2 comments

We sincerely thank the reviewer for their valuable feedback. We would greatly appreciate hearing if there are any remaining questions or concerns regarding our rebuttal or submission, and we would be happy to provide further clarification or additional evidence

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards, Area Chair

Reviewer_AieZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Locally-Guided Self-Distillation to address style drift in on-policy self-distillation for reasoning models. By using a KL-projected geometric interpolation between the student and teacher distributions, the approach ensures distillation targets stay close to the student. Experiments show that this prevents late-training collapse and outperform raw distillation and other baselines across reasoning and human-preference benchmarks.

Critical strengths

1 The presentation is clear and easy to follow.

2 To my understanding, the method is novel.

Critical weaknesses

1 It is unclear why human preference takes such a large portion of the paper, especially the empirical study, as the majority of human users will not read the reasoning trace of math and logic queries which this paper studies. This does not seem to be the right question to be asking on these domains.

2 It is quite confusing as to why the main results seem to be showing arbitrary checkpoint and trained episode results and missing baselines on some of the base models. This raises concerns for cherry picked results. What happens if you just let the entire training run keep going until it saturates, and then pick the best one on a validation set across all methods, including baselines?

3 The soundness of the empirical results is highly concerning. The authors evaluation set is reported to be of 83, 30, and 30 queries in the math domains without multi-seed runs on the main table. In their other evaluation sets, the number of queries is not reported. Most of the results could very well be noise, especially considering that they use a temperature of 0.6. For robustness the query set should ideally be in aggregate at least around 500 per domain.

Questions for authors

Are there any compute and/or wall clock overhead of this method?

What are the limitations of this work? Please add it into the paper.

1 comment

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards, Area Chair

Reviewer_7QxZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies privileged on-policy self-distillation, where the same model under an additional training-only reasoning instruction serves as the teacher for the deployment-context student. The authors argue that the privileged teacher distribution may contain both useful reasoning corrections and undesirable style/distributional drift. They propose the Target-Locality Hypothesis and LGSD, which projects the privileged teacher distribution toward the student under a response-level KL budget, yielding a response-adaptive geometric interpolation target. The method is evaluated on mathematical reasoning, formal logic transfer, style drift, and Arena preference-likelihood margins.

Critical strengths
  • Important problem framing. The paper raises a real issue in privileged self-distillation: the privileged teacher is not necessarily an ideal target, since it may mix useful reasoning improvements with unwanted distributional or stylistic shifts.

  • Simple and practical target construction. LGSD is easy to implement, adds essentially no extra model-forward cost, and directly addresses the distillation target rather than modifying only the optimizer or loss.

Critical weaknesses
  • The main claim is substantially stronger than the evidence. The paper claims that locality preserves useful reasoning corrections while removing harmful drift, but the evidence mostly relies on indirect proxies such as reference-token gain, likelihood margins, and style-distance metrics. These do not convincingly establish that LGSD selectively keeps reasoning improvements rather than simply shrinking the teacher distribution toward the student.

  • The preference and theoretical evidence are weak for the stated conclusions. The Arena result is based on teacher-forced likelihood margins on existing preferred/rejected responses, not human evaluation of generated outputs. The theory assumes an unobserved ideal target q* lies inside the KL budget while the raw proposal lies outside, which is exactly the key condition that would need to be justified in practice.

  • The empirical case is limited and mixed. The evaluation covers only three models, two from the same family, and several results remain below the base model or competing baselines, especially on OOD logic tasks and longer training settings. LGSD appears to mitigate some OPSD failures, but the results do not yet demonstrate robust, broadly significant improvement.

Questions for authors
  1. Direct human preference evaluation on model-generated responses would be needed to substantiate the P2 claim, rather than relying only on teacher-forced likelihood margins on existing Arena pairs.

  2. Stronger evidence is needed that reference-token gain measures useful reasoning correction, rather than formatting, answer-pattern, or likelihood effects.

  3. More analysis is needed on the robustness of the locality radius, especially whether the chosen value generalizes across models, prompts, and training lengths.

  4. Broader model-family coverage and stronger statistical evidence would be important, given the small number of tested models and the small size of some benchmarks.

  5. Further explanation is needed for the settings where LGSD still degrades with longer training or underperforms base/competing baselines on OOD logic tasks.

2 comments

We sincerely thank the reviewer for their valuable feedback. We would greatly appreciate hearing if there are any remaining questions or concerns regarding our rebuttal or submission, and we would be happy to provide further clarification or additional evidence

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.