acceptodds
Under review as a conference paper at ICLR 2027

Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning

Abstract

Multi-teacher on-policy distillation (OPD) is becoming the standard way to integrate specialist capabilities into one model: train experts with RL, then distill them into the student on its own rollouts. Existing recipes assign supervision at the sequence level – each prompt goes to one domain teacher and every token receives the same weight – which implicitly assumes that a teacher is uniformly useful across a response. We find instead that useful teacher signal is sparse and heterogeneous along a reasoning trajectory, which raises a finer question: *who should teach which token?* **V**erifier-**G**ated Multi-Expert **O**n-**P**olicy **D**istillation (**VG-OPD**) answers it by verification: the counterfactual gain of an expert on a specific answer criterion licenses that expert to teach, its disagreement with the student localizes the supervision, and criterion importance sets its weight; the gated KL enters GRPO as an additive token-level advantage. Instantiated for scientific reasoning with RL-trained capability experts, VG-OPD attains the best overall performance on seven benchmarks for 4B and 8B students, ranking first on five at both scales, with the largest gains on knowledge-intensive scientific reasoning tasks. Further analysis shows that the gains come from localizing verified supervision rather than from adding teachers or distillation loss: misplacing the same supervision budget is the single most damaging change, and indiscriminate distillation drags RL below its own floor where gated distillation lifts it.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillation, Reinforcement Learning, Multi-Expert Learning, Post Training, Scientific Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ynRqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VG-OPD combines specialist teachers to train a student in scientific reasoning. A verifier admits a teacher when its answer meets a criterion that the student fails; that teacher then supervises selected tokens according to its likelihood of the student's response. The method combines this supervision with GRPO and is evaluated on seven benchmarks at two model sizes.

Critical strengths

S1. A specialist may give correct guidance on one part of an answer and poor guidance on another. The paper takes this limitation seriously in designing how teachers supervise the student.

S2. The science-benchmark gains in Table 1 are consistent at both scales, with general reasoning performance largely preserved.

S3. Table 3 examines gating, localization, and weighting separately. I found the coverage-matched random mask particularly helpful as a control.

Critical weaknesses

W1. I do not see a justification for the distillation gradient. Appendix A.1 proves that the unclamped estimator in Eq. 9 has the correct expected KL value, but this result does not justify detaching the estimator and using it as an advantage in Eq. 10. The update can increase KL even with a fixed teacher, unit weights, and no clipping. For a binary student distribution p=(0.8,0.2) and teacher q=(0.6,0.4), it increases the first token's probability, whereas the reverse-KL gradient decreases it. This counterexample calls the claimed connection to OPD into question.

W2. Eq. 6 assigns the same mask to every criterion associated with a teacher. The mask is therefore teacher-specific, not criterion-specific: passing one criterion permits supervision on tokens that may concern another criterion the teacher fails. This does not enforce the exclusion of refused claims described in Figure 6. Figure 3a shows that selected tokens are suppressed after training, but does not establish that those tokens correspond to the verified criterion.

W3. The baseline comparison changes more than routing and localization. Appendix B.7 replaces MOPD's signed log-ratio advantage with the non-positive estimator in Eq. 9, changing its policy-gradient update. The signed advantage is part of the original MOPD method. For CriPO, the description covers advantage flipping but leaves unclear whether the behavior-injection loss from the original CriPO method was included.

W4. Some reported accuracies are difficult to reconcile with a single greedy evaluation. For example, 46.3% on 198 GPQA questions cannot come from an integer number of correct answers, even after rounding; the same holds for 80.9% on 500 MATH500 questions. Averaging could explain these values, but the paper does not describe an averaging procedure. It also omits training seeds and uncertainty estimates.

Questions for authors

Q1. What is the expected gradient of the detached estimator in Eq. 10, and how does it account for the binary counterexample in W1?

Q2. What prevents a licensed teacher from supervising tokens associated with criteria it fails? Could the authors check this directly on a small annotated sample?

Q3. Were MOPD's original signed advantage and CriPO's full behavior-injection loss evaluated? If so, please report those results separately.

Q4. How were Table 1 scores aggregated? Please provide the number of runs, effective test sizes, and uncertainty estimates.

Reviewer_Y2JxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Verifier-Gated Multi-Expert On-Policy Distillation (VG-OPD), a framework that attempts to address the limitations of sequence-level supervision in existing multi-teacher distillation methods. Instead of assigning a single teacher to an entire response, VG-OPD introduces criterion-level expert selection and token-level supervision. Specifically, the method uses verifiers to identify experts that outperform the student on failed criteria, locates supervision positions based on teacher-student disagreement, and weights the distillation signals according to criterion importance. The resulting token-level distillation advantages are integrated into GRPO training. Experiments on seven benchmarks using Qwen3-4B and Qwen3-8B demonstrate that VG-OPD achieves strong performance compared with existing RL and distillation baselines, particularly on scientific reasoning tasks.

Critical strengths
  1. The paper identifies an important limitation of representative multi-teacher on-policy distillation methods, such as MOPD, which assign teachers at the sequence level and apply uniform supervision across all tokens. The proposed criterion-level teacher selection and token-level supervision provide a more fine-grained approach to integrating specialized capabilities.

  2. The experiments cover seven benchmarks and two student model sizes (4B and 8B). VG-OPD achieves competitive or superior performance compared with several RL and distillation baselines, with particularly notable improvements on scientific reasoning tasks.

Critical weaknesses
  1. VG-OPD identifies disagreement tokens based on the teacher's log probabilities of student generated tokens, selecting the lowest 10% and expanding the selected positions to neighboring tokens. However, low teacher likelihood does not necessarily indicate an incorrect reasoning step or a token relevant to the failed criterion. Such disagreement may instead reflect differences in wording or reasoning style. Moreover, the localization mechanism does not explicitly condition on the specific failed criterion. Although the ablation studies demonstrate the effectiveness of this heuristic, they do not establish whether the selected tokens correspond to actual criterion-specific reasoning errors.

  2. The experiments use four capability experts, with only one expert assigned to each capability axis. Consequently, the softmax routing mechanism in Equation 5 effectively reduces to a binary eligibility gate for each criterion rather than selecting among competing experts. It remains unclear whether the proposed routing strategy would remain effective when multiple experts with overlapping capabilities are available within the same domain.

  3. The generalizability of the framework beyond rubric-driven scientific reasoning remains unclear. The method relies heavily on criterion-level verifiers and predefined capability categories. While this structure is suitable for scientific reasoning tasks with rubric-based evaluation, it may be difficult to extend to domains where fine-grained verification is unavailable, unreliable, or expensive. Furthermore, the main experiments are limited to Qwen3-4B and Qwen3-8B in non-thinking mode, leaving its effectiveness on larger models and other model families insufficiently explored.

I would be open to raising my score if the authors can adequately address these concerns.

Questions for authors

What is the additional computational cost of VG-OPD compared with standard GRPO and conventional MOPD, including expert generation and verification? Do the performance improvements persist under matched total compute budgets rather than matched optimization steps?

Reviewer_fXFYOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to selectively integrate multiple specialized teachers through on-policy distillation. It proposes VG-OPD, which uses criterion-level verifiers to determine whether an expert is qualified to supervise a student rollout, identifies supervision positions based on the expert's token log probabilities, and weights the resulting distillation signal by criterion importance. The selective distillation objective is combined with GRPO to retain outcome-based optimization. Experiments with 4B and 8B students across seven benchmarks show improvements over standard RL and multi-teacher distillation methods, with ablations highlighting the importance of teacher eligibility and token localization.

Critical strengths
  • The paper is well motivated and addresses an important limitation of existing multi-teacher distillation methods.
  • The proposed combination of verifier-based expert eligibility and selective token supervision is intuitive and reasonably well justified.
  • The ablations provide convincing evidence that the placement of supervision matters. In particular, replacing the proposed token mask with a coverage-matched random mask leads to a substantial performance drop, suggesting that the gains cannot be explained merely by reducing the amount of distillation.
Critical weaknesses
  • The connection between criterion-level verification and token-level supervision remains insufficiently justified. A teacher performing better on a specific criterion at the response level does not necessarily imply that its local token preferences provide correct supervision. Furthermore, selecting the teacher's lowest-probability tokens is not equivalent to measuring teacher–student disagreement. The paper would benefit from comparing this heuristic with more direct disagreement measures and validating whether the selected tokens correspond to actual reasoning errors.
  • The method relies heavily on curated criteria, reliable verifiers, and pretrained capability experts, making its practicality and generalizability uncertain.
Questions for authors
  1. Why use the teacher's lowest-probability tokens rather than a direct measure of teacher–student disagreement? Have the authors compared the proposed mask against log-probability ratios, token-level KL divergence, or teacher–student top-token disagreement under matched coverage?
  2. Can the authors evaluate whether the selected positions actually correspond to errors relevant to the failed criterion?
  3. The current setup uses one expert per capability axis, making the routing distribution within an axis effectively degenerate. How would VG-OPD handle multiple experts with different strengths on the same criterion? Is the criterion-level verifier sufficiently discriminative for this setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.