acceptodds
Under review as a conference paper at ICLR 2027

DiT-OPD: Diagnose, then Teach in On-Policy Distillation

Abstract

On-policy distillation (OPD) enables a student model to learn from teacher feedback on its own generated trajectories. Both the selection of supervision positions and the construction of teacher feedback affect answer accuracy and the reasoning length required to solve a problem. Existing approaches rely solely on statistics to determine where supervision is needed and apply globally uniform teacher feedback, resulting in weak error-correction capabilities and lengthy reasoning chains. To address this problem, we propose DiT-OPD (Diagnose, then Teach in On-Policy Distillation), which extracts guidance needs from the student's available prefix and uses the same diagnostic action to jointly select subsequent blocks and condition teacher scoring. The teacher identifies four candidate guidance needs: knowledge recall, contradiction resolution, derivation progress, and verification guidance. When no clear need is identified, the subsequent block receives no direct distillation supervision; tokens in selected blocks receive action-conditioned supervision. With Qwen3-8B supervising Qwen3-1.7B, DiT-OPD improves average pass@1 and pass@8 across six competition mathematics benchmarks by -0.25% and 0.48%, respectively, over standard OPD, while reducing reasoning length by 12.95%. On LiveCodeBench, it improves pass@8 over the base model by 6.26% and reduces reasoning length by 11.95% relative to standard OPD. In self-repair evaluation, it also improves repair pass@1 over standard OPD by 1.86%, enhancing the student's error-correction capability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On Policy Distill, Large Reasoning Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pojQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiT-OPD, which incorporates Diagnostic Block Selection and Action-Conditioned Feedback into on-policy distillation. The frozen teacher determines whether guidance is needed based on the prefix preceding the student’s current block; when a skip is not performed, the selected action simultaneously determines the supervision mask and the teacher scoring prompt for the next 256-token block. The method computes feedback for the original student token and does not require the teacher during inference. The paper uses Qwen3-8B and Qwen3-1.7B to evaluate answer quality and output length across six math benchmarks, transfer tasks, and the LiveCodeBench self-repair task, and provides component-wise ablation results.

Critical strengths
  1. DBS uses only the prefix preceding the target block for diagnosis, while ACF performs teacher forcing on the original student token; the information boundary of the supervision process is clear, and no teacher is required during inference. This makes the method valuable for practical deployment.
  2. The length gains are relatively consistent. In Table 2, the average reasoning length across the six tasks decreased from 22,672 to 19,736 tokens compared to Standard OPD; in Table 4, the truncation rates for all six tasks were also lower than those of the three OPD controls.
Critical weaknesses
  1. It is unclear whether matching each diagnosed action to its teaching prompt is necessary. The generic-guidance ablation also changes the prompt content, so it cannot separate the benefit of action matching from the effect of the prompts themselves.
  2. The gains in answer quality over OPD baselines are limited. The self-repair evaluation also uses each model’s own failed programs, so the models receive different repair inputs. This makes it difficult to attribute the repair gains to better error correction.
Questions for authors
  1. Could you shuffle the action–prompt assignments while keeping the selected blocks, prompt frequencies, and training settings fixed, and compare answer quality and output length with DiT-OPD?
  2. Could you evaluate all checkpoints on the same set of failed programs with the same test feedback, including w/o DBS and Generic guidance?
  3. Appendix C reports about 148.6 million input tokens for diagnosing 512 trajectories. What are the total teacher token usage and wall-clock time for training?
Reviewer_uMHNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to improve the reasoning quality and generation length of on-policy distillation from two aspects: (1) where to supervise and (2) how to provide feedback. The proposed method, DiT-OPD, consists of two components: Diagnostic Block Selection (DBS), which identifies the type of guidance needed (knowledge recall, contradiction resolution, derivation progress, or verification guidance) from student reasoning prefixes and selects the block for supervision, and Action-Conditioned Feedback (ACF), which adds the guidance into the prompt, then conditions teacher scoring on the same tokens in the block. Experiments were conducted on Qwen3-8B as the teacher and Qwen3-1.7B as the student, covering six competition mathematics benchmarks, and eight transfer tasks on knowledge, reasoning, and coding tasks. Compared with standard OPD and other OPD baselines, DiT-OPD generates much shorter reasoning lengths, but there is weaker evidence for improved accuracy and diagnosis-specific benefits.

Critical strengths
  • The reduction of generation length is empirically effective on all tasks.
  • The method is clear and easy to follow.
Critical weaknesses
  • The evidence does not support the main claim that DiT-OPD improves student error-correction capabilities. The gains are generally small and inconsistent. Pass@1 is below standard OPD, and pass@8 improves by less than 1 point (Table 2). No statistical testing was conducted to measure the significance. In the ablation studies, w/o ACF and w/o DBS are within 1 point difference from DiT-OPD (Table 5), which seem more like noise instead of actual performance gains.
  • Only one student-teacher pair is tested using Qwen models. It is unclear how other model families perform or how the model scales affect the results.
  • The four guidance categories are introduced, but it is not shown how each helps. Readers might expect some qualitative and quantitative analysis showing the effects of each guidance.
Questions for authors
  • What is the effect of varying the length of the blocks?
  • Since the supervision is chosen based on diagnosis of the specific type of error instead of statistics, would it be more useful if you split the blocks by reasoning steps instead of fixed tokens?
Reviewer_hRXfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes DiT-OPD, a diagnose-then-teach variant of on-policy distillation. A frozen teacher reads the student prefix, picks one of four guidance actions (or skip), then both selects which token block gets OPD loss and re-scores that block under an action-specific prompt. Student is Qwen3-1.7B, teacher is Qwen3-8B. Training is very short: a few dozen updates on a few hundred math problems. They report results on several contest math suites, plus LiveCodeBench and a self-repair setup. Ablations cover where-only, how-only, generic prompts, and KL-based selection.

Critical strengths
  • Length control is the clearest positive signal. DiT-OPD tends to produce shorter reasoning and fewer truncations than standard OPD and the other distillation baselines they compare. Ablations make a decent case that coupling block selection with action-conditioned scoring.

  • The diagnose-then-teach framing is easy to understand at a high level, and the paper does run several controls (where-only, how-only, generic prompt) instead of only reporting the final recipe.

Critical weaknesses
  • Looks like the presentation of the claims is poor, especially the abstract. Saying the method "improves" pass@1 when the number is negative is not acceptable. LiveCodeBench length and self-repair deltas are also labeled against "standard OPD" in the abstract, but those tables do not show that baseline; the numbers look closer to other baselines named in the body. I cant trust the headline framing.

  • Primary math quality vs the natural all-token OPD baseline looks null-to-negative on pass@1, and only marginal on pass@8. Per-task results are noisy. If the intro sells better error correction, this is not a clear win on that problem.

  • Training is tiny: short updates, small problem set, single run, no significance. Small gaps should not be read as solid evidence either way.

  • No direct evaluation of diagnosis quality. Without that, it is hard to know whether the teacher is actually "diagnosing" usefully or just providing a fancy prompt/selection heuristic.

  • Scope is narrow (one student/teacher pair, mostly math). Transfer claims feel thin given how the abstract is written.

Questions for authors
  • Please restate the abstract LiveCodeBench length and self-repair deltas with the exact baselines that appear in those tables, or add standard OPD rows. Was "standard OPD" meant to be Reverse-KL or something else?
  • Given pass@1 below standard OPD on the main suite, what is the intended success criterion: shorter reasoning at matched quality, or better error correction? If the latter, please show diagnosis accuracy or a controlled repair setting on math, not only LiveCodeBench self-repair on model-specific failures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.