acceptodds
Under review as a conference paper at ICLR 2027

Learn from Whoever Is Right: Per-Sample Answer Verification for Multi-Teacher On-Policy Distillation

Abstract

Modern large language models (LLMs) rely on reinforcement learning to build strong capabilities in individual domains, but integrating those capabilities into a single deployable model remains challenging. By routing each sample to the teacher whose domain matches it, existing approaches let a domain label decide which teacher provides supervision. However, domain expertise holds only on average: the matched teacher is not always correct on a given sample, while a teacher from another domain sometimes is. The reliable teacher therefore has to be identified per sample, not per domain. In this paper, we introduce **M**ulti-**T**eacher **S**elf-**D**istillation **P**olicy **O**ptimization (MT-SDPO), an on-policy distillation method that unifies several frozen teachers into one student model. MT-SDPO consists of three components: (1) *self-anchors*, where a rollout is supervised by a correct rollout from its own group; (2) *answer-verified eligibility*, where a teacher may supervise a sample only if its own answer passes a verifier; and (3) *privileged distillation*, which merges the anchor and the feedback of all eligible teachers into one context that an exponential moving average self-teacher reads and the student does not, thereby keeping one policy at deployment. On three scientific domains with verifiable answers, MT-SDPO outperforms domain routing in worst-domain accuracy for all five students from three model families. On Qwen3-8B, it lifts worst-domain accuracy by 14.79 points and narrows its domain gap by 74.7%, a better balance than serving one matched teacher per domain. Verified reliability, not domain membership, should decide who teaches.

Primary area:
reinforcement learning
Keywords:
on-policy distillation, multi-teacher distillation, large language models, reinforcement learning, answer verification, multi-domain learning, knowledge transfer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2Zk6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MT-SDPO, which combine teacher feedback with correct student rollouts to guide OPD by a EMA self-teacher. From my aspects, the main contribution of this paper is to develop a data curation and synthesis pipeline for LLM MOPD. The pipeline select teachers via answer verification and constructs privileged supervision for SDPO.

Critical strengths
  • The research gap is well identified, and the framework follows it logical.
Critical weaknesses
  • This method is limited in practical use. 1) inference cost is a major issue in LLM RL and MOPD. Teacher selection and privileged context construction require teacher inference, which could become expensive with multiple large teachers. The paper provides no analysis of the total computational cost and cost-effectiveness. 2) the method emphasizes rollout accuracy through answer verification. The limits of verification are mentioned, but the paper does not discuss how correct final answers with incorrect reasoning chains affect teacher eligibility and online student supervision. 3) the baseline selection is limited. Since the main contribution is data curation and synthesis, SDPO and Domain Routing do not establish how this pipeline compares with alternative data curation and synthesis methods. This comparison is unfair.
  • The writing is hard to follow in the method and experiment sections. While using AI to drafting is acceptable, the generated texts need more careful academic inspecting and editing. In this paper, 1) some content is misplaced and irrelevant. such as the last two paragraphs of section 4.3. 2) some non-academic expressions are used. 3) equation references are inconsistent, such as "Equation (4)" vs. "Eq. (7)".
Questions for authors
  • What is the total computational cost relative to SDPO and Domain Routing, including offline eligibility checks, online feedback generation, and EMA self-teacher evaluation?
Reviewer_DbYMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Multi-Teacher Self-Distillation Policy Optimization (MT-SDPO) to improve multi-teacher on-policy distillation by selecting teachers based on their correctness on each sample rather than domain labels. MT-SDPO uses per-sample answer verification to select eligible teachers and combines their feedback with successful student rollouts to guide token-level distillation through an exponential moving average (EMA) self-teacher. Experiments across three scientific domains and five student models from three model families show consistent improvements in worst-domain accuracy over domain routing. Ablation studies further investigate the effects of answer verification, cross-domain teacher selection, and multi-teacher aggregation.

Critical strengths
  • Clear motivation supported by quantitative evidence. The paper is well-motivated and provides quantitative evidence showing that domain routing can miss correct teachers, highlighting the need for per-sample teacher selection.
  • An intuitive and easy-to-follow methodology. The proposed MT-SDPO uses per-sample answer verification to identify eligible teachers and integrates their feedback with self-anchors to guide student training.
  • Detailed experimental evaluation and ablation studies. The paper evaluates MT-SDPO across different model sizes and families, with controlled ablation studies to examine the effectiveness of proposed different components.
Critical weaknesses
  • Limited evaluation across datasets and diverse domains. The experiments only cover three closely related scientific domains from one dataset. Testing on more datasets and diverse domains would help better assess the generalizability of the method.
  • Inconsistent performance gains over the multi-domain initialization. While MT-SDPO improves worst-domain accuracy over domain routing, its macro accuracy drops below the multi-domain initialization on several models. More analysis would help explain these drops and show when the method works best.
  • Missing comparisons with simpler verified distillation baselines. Since MT-SDPO already identifies teachers that answer correctly, it would be useful to compare it with directly distilling their verified responses. This would help show whether feedback aggregation and privileged distillation provide additional benefits beyond verified teacher selection.
Questions for authors
  1. Since teacher eligibility is based on answer correctness, could some teachers be selected simply by getting the answer right by chance? If so, how reliable is their feedback?
  2. How does MT-SDPO compare with directly distilling from verified teacher responses? Would a simpler baseline help show the benefit of the proposed feedback-based distillation?
  3. How does MT-SDPO perform on other datasets or less closely related domains? Would the improvements over domain routing still hold in these settings?
Reviewer_WTkwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Multi-Teacher Self-Distillation Policy Optimization (MT-SDPO), an on-policy distillation framework that consolidates multiple domain-specialized teachers into a single student policy. Instead of relying solely on domain labels, the method uses answer verification to identify eligible teachers, combines their feedback with successful student rollouts, and employs an EMA self-teacher for privileged distillation. The paper provides an empirical diagnosis of domain-based teacher routing and evaluates MT-SDPO on five student models across three model families. The results show consistent improvements in worst-domain accuracy over Domain Routing, and ablations highlight the importance of answer verification and multi-teacher aggregation.

Critical strengths
  1. The paper identifies an important limitation of domain-based teacher routing: domain expertise does not guarantee correctness on individual samples. The reported increase in teacher coverage from 52.94% to 65.69% supports the motivation for sample-level teacher selection.

  2. MT-SDPO combines successful student rollouts, answer-verified teacher feedback, and privileged self-distillation in a unified framework, establishing a coherent integration of supervision sources. Deploying a single student policy is also practically appealing.

  3. Comparisons with Domain Routing and ablations of verification, cross-domain reassignment, and aggregation provide evidence that supervision allocation matters. In particular, removing verification substantially degrades performance, supporting the need to filter unreliable teacher feedback.

Critical weaknesses
  1. Limited methodological novelty. MT-SDPO builds upon closely related directions, including MOPD for multi-teacher on-policy distillation and QR-Distill (EMNLP 2025) for quality-filtered reasoning-path selection and conditional routing. While MT-SDPO introduces answer-verified teacher eligibility and privileged self-distillation, the novelty of these components and their combination is not sufficiently established. More targeted comparisons or ablations would help clarify the contribution of sample-level teacher verification beyond domain-based routing and existing quality-aware distillation methods.

  2. Teacher answer correctness is an imperfect proxy for feedback quality. A teacher that answers correctly is not necessarily able to provide valid diagnostic feedback on an incorrect student rollout, while lexical sanitization cannot guarantee semantic correctness or eliminate answer leakage. Directly evaluating retained feedback quality and answer leakage, together with sensitivity analyses using multiple teacher probes or alternative eligibility criteria, would strengthen the method's reliability claims.

  3. Limited evaluation scope. All experiments use multiple-choice scientific questions across three related domains, with a single reference answer and a programmatic verifier. Although suitable for controlled comparisons, this setup does not establish generalization to other verifiable tasks or settings without exact-answer supervision. Evaluation on additional task families or more diverse domains would strengthen the broader claims.

  4. Worst-domain gains do not always translate into overall improvements. Although MT-SDPO consistently improves worst-domain accuracy over Domain Routing, macro accuracy decreases relative to the multi-domain initialization for several students. For example, the Llama-3.1-8B-Instruct model's worst-domain accuracy drops from 51.42% to 44.62% after MT-SDPO. The authors should more clearly distinguish balanced performance from overall capability gains and analyze when the method sacrifices strong-domain performance.

Questions for authors

Refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.