acceptodds
Withdrawn from ICLR 2027

OPDX: Optimal Teacher Selection for Multi-Domain On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-Policy Distillation (OPD) improves LLM post-training by providing token-level supervision on trajectories sampled from the student policy. However, extending OPD to multiple domains introduces two core challenges: (1) the intra-domain teacher-student distribution mismatch causes gradient spikes and unstable convergence; and (2) distributional inconsistency between domain-specific teachers induces optimization oscillations. To address these challenges, we reframe the problem from a teacher-selection perspective and propose OPDX, a two-stage framework for multi-domain OPD. First, a domain teacher selection method uses the Teacher Suitability Index (TSI) to select suitable single-domain teachers, alleviating convergence issues caused by distribution bias. Second, we propose a joint optimization distillation mechanism that evaluates optimal teacher group using aggregate TSI and the cross-domain penalty factor to determine the optimal teacher group for multi-domain OPD. This mechanism alleviates optimization conflicts between domain-specific teachers. Experiments on three domains with Qwen3.5 models of varying sizes show that OPDX consistently outperforms strong baselines in single-domain precision, multi-domain integration, and scaling efficiency. OPDX establishes a scalable, generalizable approach to stable multi-domain OPD. The code will be released upon acceptance.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillation, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_uVTfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OPDX proposes a framework for selecting teachers for multi-domain on-policy distillation. It first constructs domain-specific teacher candidates through supervised fine-tuning followed by successive reinforcement-learning rounds, and augments them with privileged information. A Teacher Suitability Index (TSI) screens candidates according to how much teacher-preferred probability mass is accessible to the initial student. A second stage selects a teacher combination using average TSI and a Cross-domain Penalty Factor (CPF), which measures dispersion in teacher entropy. The selected teachers then supervise a shared student. Experiments across three domains and two model sizes show substantial improvements over the reported baselines.

Critical strengths
  1. Selecting teachers according to their compatibility with a particular student is a well-motivated direction. The paper provides explicit, inexpensive-to-evaluate selection scores rather than relying exclusively on teacher benchmark accuracy.
  2. OPDX shows empirical effectiveness and substantial improvements relative to baselines.
Critical weaknesses
  1. The method introduces a complex, multi-stage pipeline and several coupled design choices without establishing robustness or providing a transferable tuning procedure. The pipeline depends on teacher construction through SFT and multiple RL rounds, privileged conditioning, a calibration set and rollout count, the effective-support threshold, the number of retained candidates, and the trade-off between TSI and CPF. These choices may differ for different model families and be difficult to tune. The paper does not provide sufficient justifications for motivating these designs.
  2. The evidence does not yet justify the additional selection machinery relative to simpler alternatives. OPDX is a multi-stage process, which introduces training overhead. Compared to simpler learning methods which can be easier to tune, it is unclear whether the gains of OPDX can be justified.
Questions for authors
  1. How stable are candidate rankings, selected teacher groups, and final performance across calibration-set draws and reasonable changes to the support threshold, retained candidate count, and CPF weight? Were these settings fixed across domains and model sizes, and what procedure selected them?
  2. What is the total computational and tuning cost, including teacher construction and selection? Is there evidence that compatibility measured against the initial student remains predictive throughout distillation?
Reviewer_AiyzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OPDX, a two-stage framework for multi-domain on-policy distillation (OPD) built around teacher selection rather than distillation-time intervention. Stage 1 constructs per-domain candidate teachers (one epoch of SFT followed by five GRPO replay rounds on domain data), augments each checkpoint with privileged information (reference answers or tool-call sequences), and screens them with the Teacher Suitability Index (TSI), which measures the fraction of teacher-preferred probability mass lying within the student's effective support — motivated by a "black-hole" gradient analysis showing that reverse-KL corrections vanish as the student's probability on a teacher-preferred token approaches zero. Stage 2 selects one teacher per domain by maximizing the mean TSI minus a Cross-domain Penalty Factor (CPF), defined as the dispersion of mean teacher entropy across the group and used as a proxy for gradient-scale imbalance; the selected group then supervises standard multi-domain OPD (uniform domain sampling, token-level reverse KL) on a shared student. Experiments on three domains (scientific reasoning, tool use, telecommunications) with Qwen3.5-4B and Qwen3.5-9B show OPDX outperforming SFT, GRPO, and OPSD in single-domain settings (e.g., +7.5 average over OPSD at 4B) and in multi-domain settings where OPSD degrades substantially, with smoother training dynamics. Ablations cover privileged information, TSI, CPF, teacher checkpoint source, and mini-batch composition.

Critical strengths

1. A timely reframing of multi-domain OPD around teacher selection, with a well-evidenced motivation. The paper's central observation — that instability in multi-domain OPD should be addressed by choosing suitable teachers before distillation rather than by correcting updates during training — is a genuine gap in the literature, and the gradient diagnostics (Fig. 1: cross-domain gradient norms and a mean cross-domain cosine of 0.074) plus the teacher-accuracy-vs-suitability evidence (Table 6) make the case concretely. This is consequential because it gives practitioners a new, actionable axis (teacher screening) for building multi-domain distillation pipelines.

2. Strong empirical results with a clear failure-mode contrast. OPDX improves over OPSD by +7.5/+8.5 average points in single-domain settings, and in multi-domain settings where OPSD collapses (45.38 dual-domain, below SFT), OPDX reaches 70.53 with smoother training dynamics (Figs. 3–4). The framework demonstrably stabilizes multi-domain integration across two model scales. This is consequential because it shows the teacher-construction-and-selection pipeline, as a whole, delivers substantial and reproducible value.

Critical weaknesses

1. The proposed selection mechanisms do not carry the reported gains — the paper's own ablations undermine its central claim. The paper attributes its success to a "joint optimization distillation mechanism" that resolves cross-domain conflicts via TSI and CPF. However, the ablations show CPF contributes almost nothing in the exact regime it targets: removing it changes the dual-domain average by only −0.04 (4B) and −2.28 (9B), while privileged-information conditioning — inherited directly from OPSD (Zhao et al., 2026) — accounts for the largest single-component effect (−2.27/−4.84 single-domain). The dual-domain w/o-Prompt ablation further reveals cross-domain compensation: removing privileged information improves Science by +2.37 while degrading Tooluse by −4.12, so the near-constant average masks large opposing per-domain effects. Moreover, CPF addresses only gradient-scale dispersion through an entropy proxy whose justification relies on unverifiable assumptions (shared sensitivity κ, uniformly bounded residuals δ — Appendix A.3 concedes the analysis "does not guarantee reduced directional conflict or convergence stability"), while the paper's own Fig. 1 shows the dominant problem is directional conflict (mean cross-domain cosine 0.074). This is consequential because the paper's headline mechanism is not what produces the gains: what remains is an engineering pipeline whose benefits largely come from existing components (privileged conditioning, RL teacher construction), and the claimed alleviation of optimization conflicts is unsupported by the paper's own evidence.

2. Missing comparison with the multi-teacher OPD literature, and two points that need clarification. The baselines are SFT, GRPO, and OPSD only; more well-known multi-teacher OPD methods are discussed in Related Work but not compared, even though they are the direct competitors for the paper's teacher-selection claim. The compute budget of the screening stage is also not accounted: OPDX requires per-domain SFT plus five GRPO teacher-training rounds before distillation begins, while the GRPO baseline trains only the student, so it is unclear how the methods compare under equalized compute. Finally, OPSD degrades sharply in multi-domain training (45.38 vs. 62.28 single-domain, below SFT); the paper should explain this degradation to establish that the baseline was implemented fairly. This is consequential because the paper's central claim — superior multi-domain OPD via teacher selection — is not yet validated against the methods it positions against, and the reported margins depend on how these points are resolved.

Questions for authors
  1. Please compare against published multi-teacher OPD methods (e.g., domain-routed MOPD, CaMOPD, Open-MOPD) under the same teacher-construction budget. Since the teacher-construction stage (per-domain SFT + five GRPO rounds) dominates the compute, would these methods improve similarly when given your teacher pool? (Clarification request — material to whether the teacher-selection claim, rather than the teacher construction, drives the gains.)
  2. Removing CPF barely changes the dual-domain average (−0.04 on 4B) in the regime it was designed for. Can you provide per-domain gradient-scale dispersion measurements before and after CPF selection, and empirically verify the κ/δ assumptions of Appendix A.3? What is the actual measured reduction in cross-domain gradient imbalance?
  3. In the dual-domain w/o-Prompt ablation, Science improves (+2.37) while Tooluse degrades (−4.12). How do you interpret this cross-domain compensation, and what does it imply for attributing the reported gains to the proposed components rather than to domain rebalancing?
  4. How was OPSD adapted to the multi-domain setting? Its collapse from 62.28 (single-domain) to 45.38 (dual-domain), below SFT, is surprising — please detail the implementation to rule out a misconfigured baseline. (Clarification request.)
  5. TSI is estimated from only five calibration prompts drawn from the training set. Please report how the selected teacher varies across different calibration subsets and across values of the support threshold ϵ, and discuss the leakage risk of using training-set prompts for teacher selection.
Reviewer_PqcTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OPDX proposes a two-stage teacher-selection framework for multi-domain on-policy distillation. It constructs domain-specific teachers through SFT and iterative GRPO, augments them with privileged information, and screens candidates using the Teacher Suitability Index (TSI). It then selects one teacher per domain by combining average TSI with a Cross-domain Penalty Factor (CPF), defined as the dispersion of mean teacher entropies. The selected teachers supervise a shared student through reverse-KL distillation. Experiments on three domains with Qwen3.5-4B and Qwen3.5-9B report improved average accuracy over SFT, GRPO, and OPSD.

Critical strengths
  1. The paper addresses a critical challenge in consolidating domain experts into a shared model. Considering both teacher–student suitability and cross-domain compatibility provides a useful perspective on teacher selection.
  2. These results support the practical promise of the overall pipeline. OPDX achieves higher average accuracy than SFT, GRPO, and OPSD across both model sizes in the single-domain and multi-domain experiments. .
Critical weaknesses
  1. Direct validation of TSI and CPF is insufficient. The existing ablations do not systematically characterize the relationship between TSI and downstream student performance, leaving its effectiveness as a teacher-selection criterion insufficiently established. CPF is a heuristic that requires further empirical validation; Table 3 alone does not convincingly establish that it reliably selects better teacher combinations. This limits the evidence for the paper's central selection mechanisms.
  2. The final student's performance is not directly compared with that of its corresponding domain teachers. Consequently, it is unclear how much expert capability the method retains or loses during integration, limiting assessment of its central capability-integration goal.
Questions for authors
  1. Across candidate checkpoints within each domain, what is the relationship between TSI and final student accuracy, and does TSI-based selection outperform other potential criteria, such as overlap ratio?
  2. Across multiple candidate teacher combinations, does incorporating CPF into the selection score consistently improve downstream student accuracy and reduce gradient-scale imbalance compared with selection by aggregate TSI alone?
  3. How does the final student compare with each selected domain teacher under the same evaluation conditions, and how does this capability retention vary across several teacher combinations?
  4. What motivates using a single-domain mini-batch at each step, and, beyond the accuracy comparison in Table 7, does mixing domains within each batch reduce gradient spikes and convergence instability?
Reviewer_4GmYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OPDX, a two-stage teacher-selection framework designed for multi-domain on-policy distillation (OPD). To address intra-domain mismatch and cross-domain inconsistency, the method first screens domain-specific candidate teachers—constructed via SFT and iterative reinforcement learning—using a Teacher Suitability Index (TSI) that measures accessible teacher-preferred probability mass. It then selects a final cross-domain teacher combination using an aggregate TSI score penalized by a Cross-domain Penalty Factor (CPF), which measures teacher entropy dispersion as a proxy for gradient-scale imbalance. While experiments on Qwen3.5-4B and Qwen3.5-9B across Science, Tooluse, and Telecommunication domains demonstrate improved average accuracy over SFT, GRPO, and OPSD baselines, the ablation studies indicate that the core proposed selection mechanisms (especially CPF) contribute minimally to the performance gains compared to inherited data augmentation techniques like privileged information

Critical strengths

Actionable Problem Formulation: The paper provides a clear, data-driven motivation for shifting the focus of multi-domain OPD from distillation-time interventions to pre-distillation teacher selection. Figure 1 provides compelling empirical evidence of gradient norm volatility and severe directional misalignment (mean cross-domain cosine similarity of 0.074).

Strong Aggregate Empirical Results: The complete OPDX pipeline consistently outperforms the SFT, GRPO, and OPSD baselines across both model scales (4B and 9B) in single-domain and multi-domain evaluations. The training dynamics also appear smoother and less oscillatory than the baselines in dual-domain scenarios.

Critical weaknesses
  1. The ablation studies undermine the core methodological claims. Table 3 demonstrates that removing the CPF (w/o-CPF-OPDX) reduces the dual-domain average accuracy by a negligible 0.04 points on the Qwen3.5-4B model (from 70.53 to 70.49). In contrast, removing privileged-information conditioning (w/o-Prompt-OPDX) causes the largest performance drops: -2.27 on 4B and -4.84 on 9B in single-domain averages. This indicates that the reported gains stem primarily from privileged prompt augmentation rather than the proposed cross-domain selection mechanisms.

  2. The candidate teacher construction stage requires one epoch of SFT followed by up to 5 rounds of iterative GRPO. It appears the standard GRPO and OPSD baselines do not benefit from this massive pre-computation phase, making the comparison uneven in terms of compute budget.

  3. In the dual-domain w/o-Prompt-OPDX ablation for the 4B model, Science accuracy actually improves by +2.37 (from 71.99 to 74.36) while Tooluse accuracy degrades severely by -4.12 (from 69.07 to 64.95). The near-constant average masks these large opposing effects, suggesting that the method might be relying on domain rebalancing rather than harmoniously resolving optimization conflicts.

  4. Missing Multi-Teacher Baselines: The paper positions itself as a solution to multi-teacher distillation, yet only compares against SFT, GRPO, and single-teacher OPSD. It omits direct empirical comparisons with existing multi-teacher OPD methods discussed in the related work (e.g., MOPD).

Questions for authors

Can you evaluate existing multi-teacher OPD baselines (such as MOPD) using the exact same computational budget (SFT + 5 GRPO rounds) for teacher construction to isolate the effectiveness of your selection mechanism?

Given that removing the CPF changes the 4B dual-domain average by only -0.04 points, how do you justify the inclusion and necessity of this penalty factor in the final selection stage?

Can you empirically verify the assumptions of shared sensitivity κ\kappa and bounded residuals \delta used in the CPF derivation? Please provide direct measurements of cross-domain gradient-scale dispersion before and after applying CPF.

In Table 3, the w/o-Prompt-OPDX ablation in the dual-domain setting drastically improves Science while degrading Tooluse. How do you interpret this cross-domain compensation, and does it indicate that the method relies on domain rebalancing rather than true capability integration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.