acceptodds
Under review as a conference paper at ICLR 2027

Beyond Policy Discrepancy: Capability-Aware Multi-Teacher On-Policy Distillation

Abstract

Multi-teacher on-policy distillation (MOPD) integrates domain specialists into a shared student, but determining how strongly each teacher should supervise remains challenging. Existing discrepancy-based allocation can assign strong supervision even when the student already matches or exceeds the teacher's task performance, since behavioral mismatch does not necessarily imply a remaining capability deficit. We propose Capability-Aware Multi-Teacher On-Policy Distillation (CA-MOPD), which combines specialist merging with capability-guided allocation to account for what the student has already learned. CA-MOPD first merges reinforcement-learning specialists into a strong student initialization, then uses verifier-estimated teacher–student capability gaps to adjust domain-level distillation strength. Supervision decreases as capability gaps close and increases when estimated student capability regresses, while dense token-level guidance remains provided by the teachers. Across six benchmarks spanning mathematics, code generation, and instruction following, CA-MOPD achieves an average score of 32.22, outperforming naive MOPD, Open-MOPD, and the merged initialization by 0.86, 0.83, and 0.74 points, respectively. Controlled comparisons further support the benefits of merged initialization and capability-gap guidance under matched training budgets.

Primary area:
reinforcement learning
Keywords:
multi-teacher on-policy distillation, model merging, reinforcement learning, knowledge distillation, capability integration, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_84J7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Capability-Aware Multi-Teacher On-Policy Distillation (CA-MOPD). It first initializes a student by applying Task Arithmetic to domain-specific RL specialists, then adjusts domain-level distillation weights using verifier-estimated teacher–student capability gaps and a regression guard based on the student’s historical-best capability. The underlying dense token-level distillation objective remains unchanged. Experiments use a SmolLM3-3B model across mathematics, code generation, and instruction following. CA-MOPD outperforms naive MOPD, Open-MOPD, and the merged initialization in six-benchmark average.

Critical strengths
  1. Clear motivation. The distinction between behavioral discrepancy and task-level capability advantage is relevant, particularly when specialist merging has already transferred substantial capability.
  2. Useful methodological detail. The appendices describe the controller, optimization objective, training configuration, and state required for checkpoint resumption.
Critical weaknesses
  1. Incomplete contribution isolation. The main experiments do not sufficiently separate the benefits of initialization, capability-based allocation, regression retention, token balancing, and global loss scaling.
  2. Insufficient validation of the capability signal. The controller compares a fixed teacher reference with student estimates from small, changing training batches, without demonstrating that these estimates reliably distinguish capability changes from sampling variation.
  3. Limited evidence of robustness. The reported margins lack uncertainty estimates or independent training-seed results.
Questions for authors

Based on the review above and some other issues, the reviewer has the following questions or comments.

  1. Were the Table 1 baselines (especially the MOPD ones) initialized from the same merged checkpoint?
  2. Could the authors provide the actual ablation experimental results of different CA-MOPD variants, including CA-MOPD without merged initialization, CA-MOPD without capability-based allocation, CA-MOPD without regression retention, CA-MOPD without token balancing, CA-MOPD without global loss scaling? These can help justify the efficacy of each individual component, especially those not already equipped in Naive MOPD and Open-MOPD.
  3. Equation (3) defines capability under a common domain distribution. The implemented estimator instead compares a teacher score fixed on a held-out calibration set with an EMA of student scores on changing training batches. How comparable are the fixed teacher calibration scores and student estimates from changing training batches? Please specify the calibration-data source and selection procedure, clarify whether calibration decoding matches online training decoding, and report the initial teacher/student calibration scores with uncertainty. A fixed, matched teacher–student probe at selected checkpoints would help validate the online signal.
  4. Are the gains reproducible across training seeds? Please report uncertainty for the main comparisons; repeated decoding alone does not measure training variability.
Reviewer_DHsCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-teacher on-policy distillation (MOPD), where domain-specialized RL teachers supervise a shared student. The authors point out that existing discrepancy-based allocation (e.g., Open-MOPD) conflates behavioral mismatch with capability deficit: after specialist merging, the student may already match or exceed a teacher's task performance while still being distributionally different. They propose CA-MOPD, which (1) initializes the student via Task Arithmetic merging, (2) calibrates a fixed teacher capability reference per domain, (3) tracks student capability online via EMA of verifier scores from existing rollouts, and (4) adjusts domain-level distillation weights based on remaining capability headroom and a regression-aware retention mechanism. On SmolLM3-3B across six benchmarks, CA-MOPD averages 32.22, vs. 31.48 (ParamMerge-TA), 31.39 (Open-MOPD), 31.36 (Naive MOPD), with 3.2% runtime overhead.

Critical strengths

S1. The distinction between policy discrepancy and capability gap is insightful and well-evidenced.

Eq. 5 formalizes that md>0⇏gd>0m_d > 0 \not\Rightarrow g_d > 0. Figure 2 shows this is not just theoretical: after merging, the student frequently matches or exceeds the teacher's capability while token-level discrepancy remains large (math: 170/200 steps). This observation is relevant beyond the proposed method and could inform future allocation strategies in multi-teacher distillation.

S2. Low overhead and modular design.

CA-MOPD reuses verifier scores from existing rollouts (no extra generation), adding only 3.2% runtime. It modifies only domain-level weighting, leaving token-level teacher supervision untouched — making it plug-and-play with other MOPD variants. The regression-aware retention mechanism is a reasonable response to cross-domain forgetting.

Critical weaknesses

W1. Marginal gains with zero statistical significance testing.

Over ParamMerge-TA, CA-MOPD improves by 0.74 pts (32.22 vs 31.48). This is measured with avg@64 on AIME (30 problems) and avg@10 on LCB — protocols with substantial sampling variance. No error bars, no multi-seed, no bootstrap CI. Meanwhile code regresses on both LCBv5 (−1.08) and LCBv6 (−1.09), so the method is not uniformly improving but redistributing across domains. A 0.74-point average gain could easily be noise where math/IF happened to fluctuate up more than code fluctuated down. Without significance testing, the central empirical claim is unsubstantiated.

W2. Method contribution is unclear; key ablation is not under the main configuration.

Table 2 tells the story: merge alone takes the score from 24.92 to 31.48 (+6.56); CA-MOPD's controller adds 0.74 on top. The critical ablation (Table 3, capability-gap vs. discrepancy) is done at step-50 from MixSFT — not step-200 from ParamMerge-TA (the main setting). If capability-gap doesn't beat discrepancy under the main config, the method's core contribution collapses. Furthermore, the authors themselves admit "allocation magnitude does not directly predict downstream gain" (Section 4.5): code gets the largest token share (40.13%) yet drops 1.08 pts. This undermines the causal claim that capability-guided allocation drives the improvement. The controller has 8+ hyperparameters (p,fmin⁡,fmax⁡,β,τ,λ,ϵp, f_{\min}, f_{\max}, \beta, \tau, \lambda, \epsilon, warm-up) with no sensitivity analysis — raising the concern that the method is an assembly of heuristics rather than a principled design.

Questions for authors

Q1. Can you provide 3+ seeds with std/CI for all six benchmarks? With 0.74-pt gains on avg@64/avg@10, significance testing is essential. Bootstrap CI is also acceptable.

Q2. Why is the Table 3 ablation (capability-gap vs. discrepancy) done at step-50 from MixSFT instead of step-200 from ParamMerge-TA? Can you provide the same ablation under the main configuration?

Q3. How sensitive is CA-MOPD to β\beta, pp, fmin⁡f_{\min}, τ\tau, λ\lambda? Any sensitivity analysis?

Q4. Code gets the largest token share (40.13%) and 92/200 guard steps, yet drops 1.08 pts. Is this verifier estimation error, cross-domain interference, or teacher-signal quality? Can you show per-domain capability trajectories annotated with checkpoint-level benchmark scores?

Q5. Can you provide a direct experimental comparison with D3-MOPD (Sun et al., 2026)? It also does adaptive domain scheduling; the related-work discussion alone is insufficient.

Q6. Any results on larger models (7B+), >3 domains, heterogeneous teachers, or learned routing? The current setup (single 3B family, 3 domains, known routing, 200 steps) is narrow.

Reviewer_YE2UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies multi-teacher on-policy distillation (MOPD), where a single student learns from several domain-specific RL specialists. It argues that discrepancy-based allocation, as in Open-MOPD, can over-supervise domains where the student already matches or beats its teacher. The proposed method, CA-MOPD, has three parts: 1) It initializes the student by Task-Arithmetic merging of three SmolLM3-3B RL specialists. 2) It estimates per-domain teacher–student capability gaps with verifiers. 3) It sets domain-level distillation weights from the remaining headroom (with a floor), plus a regression guard that raises the weight when the student’s EMA falls below its historical best. These weights are then combined with Open-MOPD’s token-share balancing. Validation covers six benchmarks with a 200-step training budget.

Critical strengths
  1. Merged initialization is a useful empirical observation. Table 2 shows that Task-Arithmetic merging alone (31.48) nearly matches 200 steps of MOPD from the SFT checkpoint (31.53). It also gives a better starting point for later distillation.

  2. Clear presentation and good reproducibility details.

Critical weaknesses
  • The method is incremental, and related work is missing. It combines Task Arithmetic, Open-MOPD's unchanged OPD signal and token balancing, and progress-based task weighting (GradNorm/DWA-style). It does not discuss confidence or reliability-weighted multi-teacher KD.

  • I think also the resutls shows the gain mostly comes from merging, not from the proposed method.

  • I struggled to find the specific and well alignment between the mechanism and the proposed method. The mechanism doesn’t do what the paper says. In math, the best domain, the headroom factor sits at its floor. The regression guard sets the weight on 172/200 steps. The guard is likely triggered by noise. With about 16 samples per step, the EMA’s standard error (≈0.03) is larger than the threshold τ (0.02). The historical best is a running maximum of noisy values, so it’s biased upward. So the improvement in the best domain isn’t explained by capability-aware allocation at all.

  • CA-MOPD in Table 1 starts from the merged checkpoint. The Naive MOPD and Open-MOPD baselines appear to start from MixSFT, so the comparison is not like-for-like.

  • When CA-MOPD also starts from MixSFT (Table 2), it scores 31.53. That is essentially tied with Open-MOPD (31.39) and Naive MOPD (31.36). So most of the gain appears to come from merging, an existing technique, not from capability-aware allocation, the claimed contribution.

  • The capability vs. discrepancy ablation (Table 3) does not close this gap. It is run from MixSFT, not from the merged initialization where the motivation is said to apply. It is evaluated at step 50 only. The "discrepancy" variant keeps the capability-based guard, so it is not a clean comparison.

  • No seeds, variances, or confidence intervals are reported. Average gains under 1 point, on benchmarks with 30 AIME problems each, are not statistically supported.

  • Key baselines are missing: Open-MOPD and Naive MOPD from the merged initialization, and GRPO/RL on the merged student using the same verifiers.

  • The motivating statistic is close to vacuous. m_d (Eq. 2) is a mean of absolute values, so positive discrepancy holds at almost every step. The 170/80/129-step count therefore just counts steps with a nonpositive gap.

  • The paper's own figure contradicts the narrative. Fig. 2 shows math and code discrepancy decaying to near zero within about 30 steps, so a discrepancy-based allocator would already down-weight these domains.

  • The capability gap is not the gap defined in Eq. (4). It compares the teacher on calibration prompts with the student on changing training prompts, which produces ±50pp swings per batch.

  • The guard is probably triggered by noise. There are about 16 verifier samples per domain per step, giving a per-step standard error around 0.11–0.12. After the EMA (β = 0.9), the standard error is still about 0.03, which exceeds τ = 0.02. The historical best C* is a running maximum of noisy values and is biased upward, so R_d > 0 fires often even when true capability is flat. This matches the 172/200 activations in math.

  • The headroom term barely operates in the best domain. In math, the headroom factor sits at its floor and the guard determines the weight. The paper's central idea therefore does not explain the improvement there.

  • The effective learning rate varies. The global scale F = max_j f_j ranges from 0.05 to 2, so the effective step size differs from the baselines.

Questions for authors
  1. What are the results for Naive MOPD and Open-MOPD initialized from ParamMerge-TA under the same 200-step budget? If CA-MOPD’s advantage over them is small, the main claim does not hold.

  2. Can you report mean ± std over at least 3 seeds for Tables 1–3, and bootstrap confidence intervals for the per-benchmark differences?

  3. What is the guard’s false-positive rate? For example, run the controller on rollouts from a frozen (non-updated) student and report how often R_d > 0. Does the result change if τ is scaled to the EMA’s standard error?

  4. Please report results from actually training (not reconstructing) (a) a guard-off, headroom-only variant and (b) a headroom-off, guard-only variant.

  5. What are the per-benchmark scores of the three specialist teachers and the calibration values T_d? RouteRL (31.10) is below the merged model (31.48), which suggests the student already beats the teachers on aggregate. Why is distilling from these teachers beneficial in that regime?

  6. Does the improvement persist with F fixed at 1, so that allocation is separated from changes in effective learning rate?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.