- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MI-SD, a multi-reference extension of on-policy self-distillation for LLM reasoning. Given a student rollout generated from the problem, a shared frozen teacher evaluates the same trajectory under multiple privileged reference contexts. MI-SD combines multi-reference coverage with MI-gated max-support supervision and a canonical-solution anchor, using an equal-weight generalized Jensen–Shannon divergence across reference-conditioned teacher distributions as a token-level mutual-information signal. Experiments across Qwen3-1.7B, 4B, and 8B on AIME24, AIME25, and HMMT25 show consistent improvements over OPSD, together with comprehensive ablation studies.
Critical strengths
- The framework is clear and addresses a relevant limitation of single-reference supervision. Evaluating the same student prefix under multiple reference contexts provides a straightforward way to incorporate potentially complementary guidance into on-policy distillation. The separation between coverage, selective reinforcement, and canonical anchoring makes the design understandable and amenable to controlled analysis.
- The paper provides a relatively comprehensive empirical analysis. Beyond the main results, the authors include component ablations, reference-count and MI-threshold studies, matched-data and exposure-matched comparisons, and analyses of training dynamics. These experiments help clarify the contributions and limitations of different components of MI-SD.
Critical weaknesses
-
The robustness of the reported improvements is not yet sufficiently established. The main results select the best checkpoint independently for each benchmark from multiple training checkpoints, hence the reported macro-average may combine results from different checkpoints. Moreover, the key ablations are based on a single training seed, while many of the reported improvements are relatively modest. Reporting results from a single checkpoint selected using an independent validation criterion, ideally together with variability across multiple training runs, would substantially strengthen the evidence that the gains are robust.
-
It remains unclear whether the benefits of multi-reference supervision arise from genuinely distinct reasoning routes. The method is motivated by exploiting multiple valid solution paths, but the reference-construction pipeline varies both reasoning perspective and presentation style, and the authors explicitly note that their diversity check does not establish whether the resulting references use distinct mathematical methods. Consequently, reference-conditioned MI may partly reflect differences in wording or exposition rather than meaningful differences in problem-solving strategies. A controlled comparison using increasingly similar references from the same solution method would help quantify how reduced reference diversity affects MI-SD performance.
Questions for authors
Beyond the weaknesses discussed above, I have the following questions:
-
Model diversity. Given that preserving diverse reasoning trajectories is a key motivation, could the authors additionally report pass@ curves for OPSD, MI-SDcov, and MI-SD? This would provide a broader view of whether multi-reference training preserves useful diversity across sampled solutions.
-
Practical availability of diverse references. How practical is it to obtain multiple valid and sufficiently diverse references for each problem? It would be useful to report the construction cost and discuss how MI-SD handles examples for which only one or two reliable solution routes are available.
-
Robustness to reference quality. How robust is MI-SD to noisy or partially incorrect references? Since max-support can emphasize a token strongly supported by any one reference, a controlled experiment that corrupts or replaces one reference would help assess the sensitivity of the method to reference quality.
If the authors provide additional experiments that convincingly address these concerns, I would be willing to raise my score.