acceptodds
Withdrawn from ICLR 2027

MI-SD: Mutual-Information-Guided Multi-Reference On-Policy Self-Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

On-policy self-distillation improves LLM reasoning by aligning supervision with student-generated trajectories. However, single-reference supervision can reduce trajectory diversity and neglect alternative valid solution paths, particularly in complex reasoning tasks. We introduce MI-SD, a multi-reference on-policy self-distillation framework that supervises student rollouts using multiple valid reference solutions. A shared frozen teacher evaluates each student rollout under distinct privileged reference contexts and provides corresponding supervision. What's more, MI-SD can quantify token-level reference sensitivity using the generalized Jensen–Shannon divergence across teacher distributions. At positions where references provide distinct guidance, MI-SD applies MI-gated max-support supervision. We further introduce a canonical-solution anchor to stabilize training. Across Qwen3 models with 1.7B, 4B, and 8B parameters, MI-SD consistently improves average accuracy on AIME24, AIME25, and HMMT25 over OPSD by 1.3, 0.7 and 1.4 percentage points, respectively. These results demonstrate the benefit of structured multi-reference supervision for on-policy self-distillation across the evaluated model scales. Our code will be made publicly available soon.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, post-training, on-policy self-distillation, multi-reference distillation, mutual information, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vBbgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MI-SD, a multi-reference extension of on-policy self-distillation for LLM reasoning. Given a student rollout generated from the problem, a shared frozen teacher evaluates the same trajectory under multiple privileged reference contexts. MI-SD combines multi-reference coverage with MI-gated max-support supervision and a canonical-solution anchor, using an equal-weight generalized Jensen–Shannon divergence across reference-conditioned teacher distributions as a token-level mutual-information signal. Experiments across Qwen3-1.7B, 4B, and 8B on AIME24, AIME25, and HMMT25 show consistent improvements over OPSD, together with comprehensive ablation studies.

Critical strengths
  1. The framework is clear and addresses a relevant limitation of single-reference supervision. Evaluating the same student prefix under multiple reference contexts provides a straightforward way to incorporate potentially complementary guidance into on-policy distillation. The separation between coverage, selective reinforcement, and canonical anchoring makes the design understandable and amenable to controlled analysis.
  2. The paper provides a relatively comprehensive empirical analysis. Beyond the main results, the authors include component ablations, reference-count and MI-threshold studies, matched-data and exposure-matched comparisons, and analyses of training dynamics. These experiments help clarify the contributions and limitations of different components of MI-SD.
Critical weaknesses
  1. The robustness of the reported improvements is not yet sufficiently established. The main results select the best checkpoint independently for each benchmark from multiple training checkpoints, hence the reported macro-average may combine results from different checkpoints. Moreover, the key ablations are based on a single training seed, while many of the reported improvements are relatively modest. Reporting results from a single checkpoint selected using an independent validation criterion, ideally together with variability across multiple training runs, would substantially strengthen the evidence that the gains are robust.

  2. It remains unclear whether the benefits of multi-reference supervision arise from genuinely distinct reasoning routes. The method is motivated by exploiting multiple valid solution paths, but the reference-construction pipeline varies both reasoning perspective and presentation style, and the authors explicitly note that their diversity check does not establish whether the resulting references use distinct mathematical methods. Consequently, reference-conditioned MI may partly reflect differences in wording or exposition rather than meaningful differences in problem-solving strategies. A controlled comparison using increasingly similar references from the same solution method would help quantify how reduced reference diversity affects MI-SD performance.

Questions for authors

Beyond the weaknesses discussed above, I have the following questions:

  1. Model diversity. Given that preserving diverse reasoning trajectories is a key motivation, could the authors additionally report pass@kk curves for OPSD, MI-SDcov, and MI-SD? This would provide a broader view of whether multi-reference training preserves useful diversity across sampled solutions.

  2. Practical availability of diverse references. How practical is it to obtain multiple valid and sufficiently diverse references for each problem? It would be useful to report the construction cost and discuss how MI-SD handles examples for which only one or two reliable solution routes are available.

  3. Robustness to reference quality. How robust is MI-SD to noisy or partially incorrect references? Since max-support can emphasize a token strongly supported by any one reference, a controlled experiment that corrupts or replaces one reference would help assess the sensitivity of the method to reference quality.

If the authors provide additional experiments that convincingly address these concerns, I would be willing to raise my score.

Reviewer_fKWUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims to improve the recent shortcomings of OPSD methods, such as shortening reasoning chains and taking more direct paths to the final answers. Their claim is that, instead of using a single reference solution, the typical case in standard OPSD approaches, using multiple variants of the solution and applying more weight to the students' tokens strongly supported by at least one teacher can improve training. They introduce MI-SD (Mutual-Information-guided Self-Distillation) and combine a) multi-reference coverage, b) max support supervision, and c) canonical-solution anchor. In MI-SD, the weight is computed based on mutual information. Based on teachers' disagreement (I_t), MI-SD selects positions to apply max support weighting (c_t) to the student-generated tokens. The paper performs experiments across 3 Qwen3 model scales on 3 benchmarks. MI-SD numbers are, however, close to the baselines, and it loses to their ablation method (MI-SD_cov) only on 2 of the 3 models. The authors did provide useful ablations in the Discussion section and Appendix F that justify the choices made. However, I have some questions and concerns regarding the empirical evidence and some aspects of the method.

Critical strengths
  • The paper works on an interesting recent phenomenon shown in recent papers: reducing reasoning chains, and taking direct paths to the final answers. Their mutual information setup, computing I_t to compare with the threshold and c_t to compute the reinforcement weight, is intuitive and easy to follow.
  • The authors also provide interesting ablations such as a) comparing MI-SD_cov vs MI-SD b) comparing their method by removing either max-support supervision or the anchor c) comparing with OPSD-flat.
Critical weaknesses
  • Empirical evidence: The setup is intuitive, however does not translate well to the empirical evidence. Given the improvement is modest, the reliability of the observed differences is uncertain with a single training seed. It would be great if the authors could provide multiple seeds results. Especially, their ablated method, MI-SD_cov – removing the anchor SFt and the max support weighting, which is the core of their formulation – beats the MI-SD on 2 of the 3 models. The authors mention this limitation, however the MI guided component is a central part of their contribution and its benefit could be established more evidently.

  • Reference diversity: I am still not fully satisfied by the reference diversity that the paper performs. Even though the authors prompt the model to generate different solutions and use a verifier, the deterministic check of the final solutions only compares after lowercasing and whitespace normalization. As the text also notes, this does not assess whether the solutions use distinct methods. So it might be possible that the 4 solutions are similar in approach and mainly differ in presentation. It would be good to see some human evaluation or LLM as-a-judge results.

  • Checkpoint evaluation: The paper reports the score on each benchmark across checkpoints saved at different steps during training. This means the reported scores do not represent the performance of a single final model checkpoint. Although the Appendix F provides fixed exposure comparison without this selection, it would be nice if the authors could report the final checkpoint results or maybe use a separate validation set for checkpoint selection. It is not comparable against the other baselines (except SGSD) as they use different evaluation protocols.

Questions for authors
  1. How often is the threshold crossed? I.e. How often is the max support term applied during training?
  2. The paper does not discuss the extra cost incurred by using 3 extra forward passes required by their method. Could the authors report some comparison with OPSD.
  3. In MI-gated reinforcement, they introduce masking, however the benefit of masking is not supported empirically.
  4. Question 2 from the discussion section compares K from 2-4, however I would be curious to see how increasing the number of reference solutions affects the performance.
Reviewer_GPpdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the issue that the single-reference supervision in large language models leads to a reduction in the diversity of generated trajectories in complex reasoning tasks. This method introduces multiple effective reference answers to supervise the generation trajectories of the student model, and uses the shared frozen teacher model to evaluate the output of the student model in different privileged reference contexts. Additionally, the authors quantified the reference sensitivity at the word level using the generalized Jensen-Shannon divergence, and applied the mutual information gated maximum support supervision mechanism at the positions where different guidance is provided by the references, while introducing standard answer anchors to stabilize the training process. Experimental results on models with 1.7 billion, 4.0 billion, and 8.0 billion parameters show that this method achieves stable performance improvements on multiple mathematical reasoning benchmark tests.

Critical strengths
  1. Monolithic reference supervision does indeed tend to limit the exploration space of the model, especially in tasks with multiple solution paths such as mathematical reasoning. Introducing multi-reference supervision is an intuitive and reasonable direction for improvement.

  2. By using the generalized Jensen-Shannon divergence to quantify the reference sensitivity at the word level and designing an information mutual gate-based maximum support supervision mechanism based on this, the author demonstrates a profound understanding of the information flow during the distillation process.

Critical weaknesses
  1. Introducing multiple reference solutions and multiple evaluations of the shared teacher model will inevitably significantly increase the computational cost during the training stage. However, the article does not mention the specific costs in terms of training time or memory usage compared to the single reference baseline, and lacks efficiency-based evaluation.

  2. The article does not clearly explain how these multiple effective reference solutions were obtained, whether they were obtained through manual annotation, multi-model integration, or specific search algorithms. This is crucial for the reproducibility and practical application value of the method.

  3. The scale of the experiments is too small and too simplistic. Many solutions in this field simply do not work in real scenarios. The experiments in this article lack any persuasiveness.

  4. There is a lack of in-depth analysis of failure cases. It is not mentioned which specific types of reasoning problems this method may fail on, or whether the mutual information gating mechanism may be misleading in the presence of certain noise references. This limits the understanding of the boundary conditions of the method.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.