acceptodds
Withdrawn from ICLR 2027

Teaching What the Student Lacks: Reasoning-Mode Transport for On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) trains a student model on its own reasoning trajectories with feedback from a stronger teacher model. However, OPD does not explicitly address imbalanced coverage of reasoning modes at student-visited states. We introduce Reasoning-Mode Transport On-Policy Distillation (RMT-OPD) to prioritize under-covered teacher modes. At states visited along student trajectories that fail final-answer verification, RMT-OPD samples multiple next steps from the student and an answer-conditioned teacher. It uses shared semantic embeddings and optimal transport to compare their local distributions and assign mode deficits. These deficits weight teacher steps during maximum-likelihood training, while subsequent states remain student-generated. RMT-OPD requires only teacher-generated text, without access to teacher logits. Under an idealized discrete-mode model, the weighted deficit equals the teacher probability mass missing from the student. On four mathematical reasoning benchmarks, RMT-OPD performs best among the compared methods and improves Avg4 by 1.7 percentage points over uniform weighting under matched per-state sampling. Fixed-state analyses show smaller teacher–student mode-coverage gaps for ours than MOPD, while gains persist with a larger teacher and on code reasoning tasks.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillation, Reasoning Distillation, Knowledge Distillation, Optimal Transport, Black-Box Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Y5owOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RMT-OPD uses optimal transport to compare teacher and student reasoning modes during on-policy distillation (OPD), prioritizing teacher steps from modes the student rarely generates at states along unsuccessful student trajectories.

Critical strengths

1.The method requires only teacher-generated text, without access to teacher logits or internal representations, making it suitable for teachers available only through text-generation interfaces. 2.With the same number of samples per reasoning state, RMT-OPD achieves 60.3% average mathematical reasoning accuracy, 1.7 percentage points above uniform weighting (Uniform), showing the benefit of selectively allocating supervision weights.

Critical weaknesses

1.RMT-OPD uses 2.82 times the teacher generation volume of SOPD, but comparisons with SOPD and MOPD do not match total compute budgets, leaving unclear whether the accuracy gains justify the additional cost. 2.The code experiments use the same 131 problems for training and testing, varying only which unit tests are used, so the results do not establish improved performance on unseen code problems. 3.Only the newly generated step is passed to the semantic encoder, and the paper does not test whether reasoning-mode deficits and training outcomes remain stable with a different encoder or with the problem and preceding context included. 4.Table 4 checks whether a teacher continuation eventually reaches the correct answer rather than whether the individual step used for training is correct, leaving unclear what proportion of high-weight teacher steps are incorrect.

Questions for authors

1.Does RMT-OPD still outperform the baselines when evaluated on code problems that were not used for training? 2.Do the teacher steps selected for high weights and the final accuracy change substantially when the semantic encoder is replaced or the problem and preceding context are included? 3.According to an independent check, what proportion of high-weight teacher steps are both correct and based on reasoning modes that the student rarely generates?

Reviewer_tVY3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a new OPD style algorithm called Reasoning-Mode Transport On-Policy Distillation (RMT-OPD). The core idea is to create a mapping of student's reasoning steps with that of a teacher (using optimal transport), and then take loss only at steps that exceed some tuned threshold. Empirical validation shows improvement over OPD and some other variants on 4 math benchmarks.

Critical strengths
  • The general intuition to selectively take loss on a subset of the reasoning trace where it deviates into an incorrect answer via semantic mappings is interesting and novel.
  • Even though the evaluation setups are limited, the ablations within those setups are extensive and show clear benefits of the method.
Critical weaknesses
  • Evaluation is limited to only mathematical benchmarks, which are quite saturated at this point. It is also unclear if this approach makes sense when reasoning steps can be vastly different in structure between student and teacher for open-ended reasoning domains (eg, law, finance)
  • Similarly, the method is evaluated only for the Qwen family of models. Different model families can show vastly different behaviors in reasoning patterns, which would ideally necessitate independent evaluations.
  • "Following the MOPD code-training protocol (Yu et al., 2026), we use the same 131 problem prompts for on-policy training and evaluation. For each problem, 50% of the available unit tests are retained for training-time rollout verification, while evaluation uses the complete test suite". This does not seem to show generalization to new prompts, unclear why this protocol should be even used as a standard.
Questions for authors

See weakness section.

Reviewer_jnTdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that a single teacher target at each state visited by the student gives on-policy distillation (OPD) no way to prioritize reasoning modes that the student rarely uses. At each state, the proposed method, RMT-OPD, uses optimal transport (OT) to compare next steps sampled from the student and teacher in a shared embedding space and assigns greater weight to teacher steps from modes the student rarely samples. The authors report that RMT-OPD outperforms existing OPD methods on mathematical reasoning benchmarks, with gains also reported in a cross-scale setting and on coding tasks.

Critical strengths
  • The motivation is clear. By comparing the local distributions of student and teacher steps, the method directly addresses the limitation of using a single target per state. Because it requires only teacher-generated text, it could in principle be applied to a broad range of teachers without access to their logits.
Critical weaknesses
  • The gains attributed to reasoning-mode distillation may come from eliciting Qwen3’s existing thinking ability. The student is Qwen3-4B, a hybrid model run in non-thinking mode, and the teacher is the same model trained in non-thinking mode for 500 GRPO steps. RMT-OPD’s scores on AIME 2024 and AIME 2025 (74.4 and 70.0) are close to those reported for Qwen3-4B in thinking mode (73.8 and 65.6, respectively) [1]. Several results are consistent with this interpretation. Avg4 doubles within the first ten updates (Figure 4), the student outperforms its teacher by 14.3 points on Avg4, and using the 14B teacher yields a lower Avg4 than using the 4B teacher. Without response lengths, a thinking-mode baseline, or results for a non-hybrid student, the experiments cannot rule out this explanation.

  • Giving the teacher the correct answer departs from standard distillation and confounds the comparison. Conditioning the teacher on the correct answer incorporates label information directly into the training signal. This setting is closer to self-distillation with privileged information [2] or to rationalization [3].

  • The experiments do not substantiate the black-box motivation. Both teachers are publicly available Qwen3 models that share the student’s tokenizer, so their logits are accessible. Models from the same family have similar writing styles and step boundaries, making this a particularly favorable setting for comparing steps in embedding space. The paper includes no experiments with a teacher from another model family, a teacher with a different tokenizer, or an API-only teacher.

  • The evaluation covers a single student, a single model family, and a single training encoder. Qwen3-4B is the only student, and the only teachers are its RL checkpoint and Qwen3-14B. The weighting signal relies entirely on BGE, a general-purpose retrieval encoder, and the authors do not train with an alternative encoder. E5 is used only for evaluation and may well share some of BGE’s biases. It therefore remains unclear whether the gains come from the semantic structure of reasoning modes or from properties specific to this setup.

  • The technical novelty over SOPD is limited. The Single Target variant already performs on par with SOPD, and simply training on multiple teacher steps accounts for part of the improvement. The advantage of optimal transport is more modest when compared with weighting each teacher step by its distance to the nearest student sample. RMT-OPD also requires substantially more generation than SOPD. Given that cost, I would have liked to see a comparison that uses the same sampled steps but gives greater weight to those the student assigns lower probability. This would clarify how much optimal transport adds beyond a simpler way of identifying what the student lacks.

[1] Qwen3 Technical Report. arXiv, 2025. [2] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv:2601.18734, 2026. [3] STaR: Bootstrapping Reasoning With Reasoning. NeurIPS, 2022.

Questions for authors
  • Under the same evaluation protocol, what are Qwen3-4B’s accuracy and mean response length in thinking mode? Could the authors also report the mean response length before and after training for each method?

  • How do the student’s performance and the teacher continuation rates in Table 4 change when the teacher has no access to the correct answer?

  • Do the gains persist when using a teacher from a different model family or when training with a different encoder?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.