acceptodds
Under review as a conference paper at ICLR 2027

DuoOPD: Learning from Joint Teacher–Student Outcomes for Multi-Task On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student on its own responses with token-level feedback from a stronger teacher, yet the teacher can fail on questions the student already answers correctly, and how often each model succeeds varies across tasks. OPD ignores these outcomes and, on average, pushes down even the student's correct responses; gating feedback by student correctness fixes the direction but uses the teacher in the same way whether or not it succeeded. We introduce DuoOPD, in which the student's outcome sets the direction of feedback and the joint teacher–student outcome decides how the teacher supports it: when only the teacher succeeds, its verified answer becomes context for scoring the student's failed response, and when only the student succeeds, a weight shared within the task reinforces the whole response. A single rule covers all four outcome combinations without task-specific settings. Across Qwen3 and Llama, DuoOPD outperforms all five baselines in mean macro accuracy, improving over OPD by 2.58 and 5.98 percentage points, and it also leads on two further task mixtures spanning scientific calculation, instruction following, and code generation. Ablations show that outcome-based direction alone stays near the gated baseline, while the joint-outcome designs supply most of the gain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Multi-Task Learning, Knowledge Distillation, Teacher–Student Learning, Outcome-Guided Feedback, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zaRhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DuoOPD modifies sampled-token on-policy distillation using jointly verified teacher and student responses. Student correctness determines the feedback sign; a cached successful teacher response conditions teacher scoring when only the teacher succeeds, while a task-level shared positive weight reinforces responses when only the student succeeds.

Critical strengths
  1. DuoOPD allows the student to learn from a teacher without treating the teacher as uniformly superior. In particular, it reinforces verified student successes even when the teacher fails, addressing a concrete problem in distillation across heterogeneous tasks.

  2. A successful teacher response can guide scoring of a failed student response without being added to the student's input. This uses teacher solutions while preserving the student's inference setting; the overall method improves on the strongest compared baseline by 1.16 macro points for Qwen3 and 4.58 for Llama (Table 1).

Critical weaknesses
  1. One cached teacher answer determines how the teacher is used for each question throughout training (Section 3). If that answer fails, the method uses a different rule than it would after a successful sample. Another teacher sample could therefore change the training signal for the same question. The three-run results do not establish whether performance is stable across independently generated teacher caches.

  2. A correct final answer can still contain wrong reasoning, but DuoOPD reinforces all tokens of a passing response. Appendix D.2 gives such an example: the answer passes while some method names are explained incorrectly. The same problem can affect cached teacher references. The reported answer accuracy therefore does not show whether the method improves or worsens explanation quality.

Questions for authors
  1. Do the three runs share one teacher cache? Do the gains hold when the cache is generated again with different sampling seeds?

  2. Among answers with correct final options, does DuoOPD produce more or fewer incorrect explanations than OPD and OPDVR? Please also check the cached teacher references.

Reviewer_mKgpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DuoOPD improves multi-task on-policy distillation by conditioning feedback on the verified outcomes of both teacher and student responses. Student correctness determines whether tokens are reinforced or suppressed, while joint outcomes determine how feedback is constructed. When only the teacher succeeds, its verified response provides additional context for scoring the student’s failed trajectory. When only the student succeeds, a task-shared positive weight reinforces the complete response. Experiments cover Qwen3-4B → Qwen3-0.6B and Llama-3.1-8B → Llama-3.2-3B on scientific tasks constructed from SciKnowEval V2, with additional Qwen3 mixtures including IFEval and MBPP. On biology, chemistry, and physics, DuoOPD improves macro accuracy over OPD from 64.39% to 66.97% for Qwen3 and from 67.28% to 73.26% for Llama. It also achieves the highest macro accuracy on the additional science and heterogeneous mixtures, reaching 63.48% and 49.24%. Three-run comparisons and component ablations support the approach, particularly the contribution of reference-conditioned feedback.

Critical strengths

The method addresses a meaningful limitation of teacher supervision with targeted, interpretable mechanisms. A stronger teacher can still fail where its student succeeds. DuoOPD explicitly handles this asymmetry rather than treating teacher preferences as uniformly reliable. The component ablations substantiate the proposed design: removing reference conditioning reduces Qwen3 macro accuracy by 1.49 points, removing weight sharing reduces it by 0.71 points, and removing both reduces it by 2.41 points. This provides evidence that the contribution extends beyond simply gating feedback by student correctness.

The empirical evidence is reasonably consistent and carefully reported. DuoOPD achieves the highest mean macro accuracy against five baselines across both model families and all tested mixtures. The paper reports three training runs, standard deviations, fixed final-checkpoint evaluation, and explicit checks for training–test overlap. The 4.58-point improvement over the strongest Llama baseline is particularly compelling. Together, these details increase confidence that the gains are reproducible within the studied settings.

Critical weaknesses

The comparisons do not fully separate the proposed outcome rules from the benefit of additional supervision and computation. DuoOPD uses verified teacher-generated references, whereas the evaluated baselines do not receive equivalent reference information. The ablations establish that references help, but do not compare against a simpler reference-conditioned OPD/OPDVR method or an OPD-plus-verifiable-reward baseline using the same resources. Including teacher-cache generation increases single-run GPU-time by approximately 29–30%. Consequently, the experiments do not fully establish the advantage of the particular four-outcome design over simpler alternatives with comparable information and compute.

The justification for treating a failed teacher sample as a reason to discard token-level weighting remains incomplete. Teacher correctness is determined by one cached stochastic response per question. A failed sample does not necessarily imply that the teacher’s scoring distribution is unreliable on a correct student response. Moreover, averaging weights could help through variance reduction rather than through the proposed teacher-failure mechanism. The current ablation removes sharing altogether, but does not compare outcome-specific sharing against sharing across all successful student responses. This leaves an important part of the method’s interpretation unresolved.

Questions for authors

Comparison clarification: How does DuoOPD compare with reference-conditioned OPDVR using the same cached teacher responses, and with a simple combination of OPD and verifiable-reward optimization? Do its gains persist when total GPU-time includes cache generation?

Mechanism clarification: Does restricting weight sharing to teacher-failure/student-success cases outperform applying it to all successful student responses? This would help distinguish the proposed outcome-specific mechanism from generic averaging or variance reduction.

Reliability clarification: How sensitive are the results to the teacher-cache sampling seed? The three training runs share one cache; changing it would test whether the outcome partition and resulting gains depend substantially on a particular teacher sample.

Reviewer_THEoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DuoOPD, an on-policy distillation method that uses both teacher and student correctness to determine token-level feedback. Student correctness sets the feedback direction, and combination of teacher and studnet determines the degree of feedback. When only the teacher is correct, its verified answer is added to the teacher’s scoring context, and when only the student is correct, positive weights are shared within the task. Experiments on two model families and three task mixtures show improvements over five baselines. Ablations suggest that reference conditioning and weight sharing account for most of the gains.

Critical strengths
  1. The paper has a clear motivation, which teacher model can still fail on questions the student answers correctly, so teacher feedback should account for both models’ outcomes. Figure 1 illustrates this well and makes the problem easy to understand.

  2. DuoOPD achieves the highest performances in the evaluated settings.

Critical weaknesses
  1. I think the main contrubution of DuoOPD is in the two cases where teacher and student correctness differ. Because compared with OPDVR, the change when both models are correct or both are wrong is from gated weights to Softplus, and also the Softplus-only ablation shows little additional benefit. I have concerns about both cases. (1) Teacher correct, Student wrong & (2) Teacher wrong, Student correct.
  • Teacher correct – student wrong: DuoOPD uses correctness rTr_T from an independently generated teacher response, whereas OPD evaluates token probabilities from student-generated prefix. It is unclear how well the independently generated teacher outcome supports feedback decisions under this different conditioning context, even though the results are good.

  • Teacher wrong – student correct: The paper proposes within-task sharing, but it is unclear why responses to different questions should receive the same weight. For example even within a task, teacher probabilities can vary across questions.

    • A comparison with response-level averaging would help determine whether sharing across different responses is necessary.
    • An ablation using a fixed positive weight, without teacher-derived scores, would clarify whether the teacher’s weighting information is needed in this case. Because if we just use fixed positive weight, it will ignore teacher's probability.
  1. Single teacher sample: The method uses only one teacher response per question, although the teacher may succeed/failed on another attempt (e.g., pass@k). Have the authors tried multiple teacher samples and using averaged score as a signal? I believe it would be more robust to sample teacher's response/answer multiple times than the single
Questions for authors
  • Clarification: What evidence supports using correctness from an independently generated teacher response to select feedback on student-generated prefixes?

  • Additional validation: How does within-task weight sharing compare with response-level averaging and a fixed positive weight?

  • Robustness: How robust are the results from the single teacher response per question? It would be better to utilize multiple teacher responses.

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,
Area Chair

AuthorsOpenReview ↗

Revised Paper Uploaded

Dear Reviewers and Area Chair,

Thank you for your thoughtful feedback and constructive discussion. We have uploaded a revised paper incorporating the additional experiments and clarifications from our rebuttal and follow-up responses. Changes are highlighted in blue. The main updates are:

  • Clarifications of the method and experimental findings, integrated into Sections 3–4, with detailed supplementary protocols and results in Appendix E.
  • Additional baseline and ablation comparisons addressing reference access, verifiable rewards, training budget, and weight sharing (Appendices E.1–E.2, Tables 10–12).
  • Teacher-sampling experiments, including independently generated teacher caches (Appendix E.3.1, Table 13) and multiple teacher responses per question (Appendix E.3.2, Table 14).
  • An explanation-quality audit of student responses with correct final answers and verified teacher references (Appendix E.4, Table 15), with implications for final-answer verification discussed in Section 6.

Our detailed point-by-point responses remain in the individual review threads. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.