acceptodds
Under review as a conference paper at ICLR 2027

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

Abstract

Reinforcement learning can turn one language model into several specialists, each excellent at a single skill such as mathematics, coding or following instructions, but users need one model with all of these skills. Multi-teacher on-policy distillation (MOPD) merges them by letting the specialists teach one student: the student answers each prompt, and the specialist for that prompt's domain gives feedback on every token. This routing decides which specialist teaches, but not how strongly its feedback moves the shared student. In Qwen3.5 models at three sizes, we find that MOPD's student does not beat one taught by the best single specialist and gains little of the mathematics specialist's advantage. The feedback is unbalanced: instruction-following feedback is several times more spread out than mathematics feedback and dominates the student's updates. We propose Domain-Normalized MOPD (DN-MOPD), which keeps the routing and rescales each domain's feedback by its measured spread. On six public benchmarks, DN-MOPD improves the average score over MOPD at every size, across three random seeds and under two answer-length limits, and recovers most of the lost mathematics gain. Controls with fixed domain weights show that the gain comes mainly from turning down instruction-following feedback rather than turning up mathematics alone, and that fixed weights close to those DN-MOPD measures perform comparably. Combining specialists therefore requires deciding not only which one teaches, but also how strongly its feedback counts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, multi-teacher distillation, knowledge distillation, large language models, post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xiKNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DN-MOPD to balance domain feedback in MOPD by rescaling existing advantages with a clipped ratio of pooled to domain standard deviation. Experiments on Qwen3.5 family models demonstrate the performance of DN-MOPD.

Critical strengths
  • The method is motivated by empirical observations of uneven feedback scales across domains. This method is intuitive, simple, and cost-effective, which need no additional teacher calls.
Critical weaknesses
  • The method's novelty is limited. From my point of view, The idea of reweighting domain signals to balance MOPD is not new, such as Open-MOPD and DeepSeek-V4 [1]. Although the paper discusses how its scaling rule differs from Open-MOPD, it does not provide a direct empirical comparison with closely related weighting methods. A matched comparison, especially Open-MOPD and DeepSeek-V4, is necessary.
  • The empirical validation remains limited. 1, the paper discloses the origin and intended role of the clipping bounds [0.25, 4], but the selection procedure for these endpoints and sensitivity to alternative bounds is unclear. 2, the main experiments only focus on the Qwen3.5 model family, and the performance of DN-MOPD on other architectures is unclear.

[1] Xu, Anyi, et al. "Deepseek-v4: Towards highly efficient million-token context intelligence." arXiv preprint arXiv:2606.19348 (2026).

Questions for authors
  1. How does DN-MOPD differ from related domain-weighting approaches, such as Open-MOPD and DeepSeek-V4, and how does it perform against a comparable weighting baseline under the same experimental setup?
  2. How sensitive is DN-MOPD to the clipping bounds across different MOPD configurations?
5 comments

Thank you for noting that the method is motivated by empirical observations of uneven feedback scales across domains, and that it is intuitive, simple and cost-effective. This reply is split into three comments: W1/Q1 here, then the clipping bounds, then other architectures. The main comparisons below are also in the table of our comment to all reviewers.

W1 and Q1

"A matched comparison, especially Open-MOPD and DeepSeek-V4, is necessary."

DN-MOPD's weighting rule appears in neither report. Open-MOPD sets per-domain loss weights from a running estimate of each domain's mean reward magnitude, and reports that inverting this rule, so that domains with smaller rewards receive larger weights, creates an unstable feedback loop; DeepSeek-V4 uses importance weights for which its report gives no estimation rule. DN-MOPD weights each domain by the inverse spread of its feedback, measured on the current batch: it turns the dominant domain down, has no per-domain weight to set by hand, and completed training in every run of the main experiments (App. D.1).

Matched comparisons fix experts, initialization, prompts, and update budget. (i) Open-MOPD's allocation rule divides by token share and scales by a clipped relative reward-magnitude factor (App. C.8 of the revision). Its division by token share is stated for a token-mean loss, so that each domain's share of the loss meets a target share. This paper averages within each response first, where that share is the domain's share of the responses in the batch, so the port divides by the response share with the same equal target and keeps the source's own clamp on the magnitude factor. Trained at all three sizes: at 16K this port is behind DN-MOPD in Total at every size, by 1.82 points at 9B, 2.25 at 4B and 2.89 at 2B, each interval wholly above zero, and it is ahead of label routing at no size, the difference being not detectable at 9B and 4B and 0.69 points behind at 2B. At 8K, over all six suites, it is behind DN-MOPD at every size as well: 3.79 points at 9B, 4.60 at 4B and 3.84 at 2B, each interval wholly above zero (App. F.3).

SizeCapSuites in TotalDN-MOPD − Open-MOPD port
9B16K5+1.82
4B16K5+2.25
2B16K5+2.89
9B8K6+3.79
4B8K6+4.60
2B8K6+3.84

The 95% interval of every row above is in Table F.3 of the revision, and the per-suite rows behind its 16K totals are in Table F.5.

One qualification on the 16K figures. One code cell of the 2B port cannot be graded: the official runner returns an empty result for one sampled response, and the grader treats any such sample as an infrastructure failure. That reproduced identically on three hosts. We therefore removed that suite from all nine runs of this read — three arms at each of the three sizes, not only the one that lost the cell. The 16K Total here is therefore an average over five suites rather than the six the paper defines, and its code domain rests on a single suite with a correspondingly wider interval (intervals and provenance in Table F.3 and App. F.3; per-suite rows in Table F.5). (ii) Hand-set weights. DeepSeek-V4's report gives no estimation rule, so there is no rule of its to port; what these runs test is the family that it belongs to, per-domain weights set by hand. The conditions are matched, as above; the rule is not reproduced, so we report these as a stress test of hand-set weighting rather than as a port of DeepSeek-V4. We trained four such weightings at 4B and 2B: mathematics doubled, IF quartered, and IF raised to 2 and to 4. At 16K none is detectably ahead of DN-MOPD in Total and six of the eight are detectably behind, by up to 3.14 points (App. C.6 and C.9; the run with IF weight 2 is in Table C.16); the weights that raise IF to 4 are behind at every size and both caps (App. C.8). DN-MOPD reaches the level of the best of them in one run.

(iii) A second control built from the two quantities that Open-MOPD uses, token share and remaining gap (a share-and-gap weighting; App. C.8 states how it differs from Open-MOPD's rule), over three seeds: DN-MOPD is ahead at 9B at 16K, where the three-seed mean difference is 0.79 points and all three seed-matched differences favour DN-MOPD, and at every size at 8K, by 0.91 to 1.93 points on average (Table C.14). At 4B all three seed-matched differences also favour DN-MOPD, by 0.62 points on average, and at 2B two of three do; neither three-seed difference is detectable at 16K (Sec. 4 and App. C.8 of the revision).

W2 and Q2, clipping bounds

"the selection procedure for these endpoints and sensitivity to alternative bounds"

Selection: the same pair comes from the earlier Qwen3 development instrument and was carried unchanged to all three Qwen3.5 sizes (App. A.4); it was not tuned per size, and no run before submission used any other pair. The two bounds are reciprocals, so the rule can amplify or shrink any one domain's weight by at most a factor of four. Neither alternative floor does detectably better, as the sensitivity reported next shows. Before training these runs, we registered the three-point ladder reported next—floors of 0.10, 0.25 and 0.50 with the upper bound held at 4—together with the reason for each new floor, so the alternative floors were not chosen after seeing the results. Because which bound binds depends on the pool, we report sensitivity by pool. Across the three main Qwen3.5 expert pools at all three sizes and both caps (seed 42), no recorded batch reaches the upper bound of 4; only the floor binds, on IF. In those pools every measured factor stays below the upper bound, so no larger value of that bound would change their results. The upper bound is reached in one of the pools whose IF teacher we replaced: on the pool with the tightest IF feedback, DN-MOPD's measured IF factor reaches that bound and is clipped there. The multiplier the rule applies to IF on that pool is therefore the bound itself. That is the rule doing what it is for: raising the weight of a domain whose feedback is unusually concentrated. How a clipped factor is to be treated was registered before any of these pools was read, so this is a case the registration provided for rather than one we found afterwards. No other replaced-teacher pool approaches either bound, and on Granite 4.2 3B every factor is near one. A floor of 0.10 shows no detectable difference from 0.25 at any size or cap; a floor of 0.50 lowers Total by 1.27 points at 2B at 16K and by 1.41 to 1.96 points at every size at 8K (App. D.6 of the revision).

W2, second item: other architectures

"the performance of DN-MOPD on other architectures is unclear"

DN-MOPD has no architecture-specific component: it rescales the per-token feedback already computed by label-routed OPD, using a scale that can be measured from one rollout batch on any architecture. Beyond the three Qwen3.5 sizes and the earlier Qwen3-4B construction, we trained a new expert pool on another family, Granite 4.2 3B, with three GRPO experts trained for 400 rollout iterations each (1,600 optimizer updates each). Before any student was trained on this pool, we registered a rule that maps the first-batch spread ratios to a prediction. The ratios of this pool are close to balanced (1.01, 0.98 and 1.18), for which the rule predicts no detectable difference in Total between DN-MOPD and Label, and neither difference is detectable (+0.14 points at 16K, −0.09 at 8K; one seed). With two further seeds, the three-seed mean difference remains not detectable at either cap (App. E.3 of the revision). Per domain on this pool the three-seed means at 16K are +1.64 for mathematics, with the interval above zero, and −1.25 for code and −0.38 for IF, neither detectable (Table F.6); on seed 42 alone IF is 2.07 points above Label and code 2.47 below (descriptive). A second family has been started: the three experts of an OLMo-3-7B-Instruct pool are trained. That read was registered as a single read and has been taken: the sealed route-2a read is in App. F.5, where the new mathematics expert is 3.38 points above the O7 base on Math at 8K at 97.5% (the code and IF rows are carried from the first read at 95%). The students on this pool are a separate registered read, not part of this one.

Reviewer_GtxHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies feedback-scale imbalance in multi-teacher OPD. It keeps domain-label routing and rescales each domain's distillation advantages using a bounded ratio of pooled to domain-specific log-ratio standard deviation. Experiments at three Qwen3.5 sizes show improvements over label-routed OPD, largely through recovered mathematics performance. Fixed-weight controls indicate that reducing instruction-following feedback explains much of the gain, with limited additional benefit from batch-wise adaptation at the larger sizes.

Critical strengths
  1. Separating teacher assignment from feedback strength provides a useful diagnostic view. Assigning a prompt to the right specialist does not ensure that a shared student can absorb all specialist capabilities. The paper addresses an optimization issue that router comparisons alone can miss.

  2. The modification is lightweight. The positive bounded multiplier reuses existing log-probabilities, adds no teacher calls, and requires no learned router. Preserving advantage signs makes the intervention straightforward to interpret.

Critical weaknesses
  1. The use case relative to simpler integration recipes remains unclear. SeqKD-SFT and task arithmetic have higher Totals at all three sizes in Tables 1–2. Different compute and token exposure prevent a direct conclusion about cost-effectiveness, but parameter merging also avoids the full OPD procedure.

  2. Feedback dispersion is not yet directly linked to harmful update dominance. A large log-ratio spread may reflect unstable feedback, but it can also reflect useful corrections on states visited by the student.

  3. Scale estimation and loss reduction use different weighting schemes. Standard deviations pool tokens, whereas the loss averages within responses and then across responses. Long responses can therefore influence the pooled reference without receiving equivalent weight in the objective. Changes in response length or domain mixture may alter multipliers even without a change in teacher quality. The paper already acknowledges this distinction. A direct comparison with response-balanced scale estimation would establish when it is harmless and when it affects the intervention.

  4. Recovering mathematics and integrating all specialist skills are different objectives. Mathematics provides the clearest gain, while 9B code performance declines relative to Label. A higher equally weighted six-task Total establishes improvement under one utility, but not necessarily better preservation of every specialist's advantages. The paper should specify whether integration means improving the mean, preventing one domain from being suppressed, or retaining each specialist's key gains. The domain trade-offs should then be interpreted under that criterion.

Questions for authors

See above.

6 comments

Thank you for finding the modification lightweight and the intervention straightforward to interpret. Each of your four weaknesses has a comment of its own: W1 here, then W2, then W3, then W4. The main comparisons below are also in the table of our comment to all reviewers.

W1

"The use case relative to simpler integration recipes remains unclear."

DN-MOPD targets pipelines that already integrate specialists via label-routed OPD and whose feedback is dispersed as in our Qwen3.5 pools (IF 2.3 to 4.4 times the pooled first-batch spread, mathematics about half; App. D.2). There it is a drop-in change worth 1.12, 1.97 and 2.34 points over Label in Total at 9B, 4B and 2B (16K, three seeds; nine of nine seed-matched comparisons) (Table C.10). It adds no teacher call, no learned router and no model copy. Per batch it adds 0.12 to 0.19 s in a single-thread CPU benchmark, against an observed median update cycle of 98 to 180 s. (Table D.9) Relative to merging: at the paper's coefficient the 16K difference between DN-MOPD and task arithmetic is not detectable at 9B or 4B after 80 updates, and the same holds after twice the updates. At 160 updates DN-MOPD's Total is 60.7 and 53.1 at 9B and 4B (Table 5), and a paired test shows no detectable difference from task arithmetic's 60.5 and 53.2 (Tables 1 and 2). Task arithmetic's Total is higher at all three sizes, as the paper states (Sec. 3; Tables 1–2), detectably so at 2B and at 8K at the paper's coefficient. That advantage depends on the coefficient: at 0.3 DN-MOPD is ahead by 2.08 and 1.34 points at 4B and 2B at 16K, by 2.20 and 2.53 at 8K and by 1.74 at 9B at 8K, with the 9B 16K difference, +0.33, not detectable (App. F.7). Merging needs experts that were fine-tuned from the student's own initialization; OPD needs only the teachers' scores on the student's own answers. SeqKD-SFT is an offline recipe that imitates teacher answers generated in advance, and it needs them generated with a 16K cap in our baseline before training. Where they can be generated and imitated, our own Table 4 reports its Total above DN-MOPD's at all three sizes; OPD needs only the teachers' scores on the student's own answers, which is the stage this paper studies. As your first strength puts it, assigning a prompt to the right specialist does not ensure that a shared student can absorb all specialist capabilities; this holds wherever specialists teach a shared student and their feedback is unevenly dispersed, which is the condition the first-batch ratio measures.

W2

"Feedback dispersion is not yet directly linked to harmful update dominance."

We test this link by varying the weight on IF, the most dispersed domain, in both directions against one baseline (a second seed-42 Label run trained with the same code). Lowering the IF weight to 0.25 raises mathematics by 2.50 points at 4B and 2.45 at 2B at 16K; in the same runs the IF score is 1.09 points higher at 4B and not detectably different at 2B. At 2B, raising the weight to 2 lowers mathematics by 1.41 points, and raising it to 4 lowers mathematics by 2.16 and 1.35 in the two runs (App. C.9 and C.8), each interval excluding zero, with no detectable change in the IF score in any of the three runs. Both runs of the weights (1, 1, 4) are detectably below DN-MOPD in Total at every size and cap that they cover, by 1.62 to 3.14 points at 16K, and both are detectably below Label at 8K at 4B and 2B (App. C.8 and C.9 of the revision; the runs with IF weights 0.25 and 2 against this baseline are in App. F.2 and Table C.16).

On which domain supplies the update: under the response-averaged training loss with unit domain weights, IF's projection share of the initial student's combined gradient is 87%, 94% and 97% at 2B, 4B and 9B. On the probe prompts its gradient is also the least consistent with itself: for the initial 4B student, the gradients of two disjoint halves of the IF prompts have a cosine of 0.41 (0.49 on a second probe sample), against 0.89 for mathematics and 0.88 for code. IF has the lowest value at every size at the initial, Label and DN-MOPD checkpoints (App. D.5 of the revision).

W3

"A direct comparison with response-balanced scale estimation would establish when it is harmless and when it affects the intervention."

In a direct comparison with response-balanced scale estimation, only two of the 14 cells differ detectably, and they point in opposite directions. We ran the comparison at all three sizes and then again under two other domain mixtures, since you note that a change in mixture can move the multipliers (seed 42). With the paper's mixture, a response-balanced estimate shows no detectable difference from the token-pooled rule at 9B and 4B at either cap or at 2B at 8K; at 2B at 16K it is 0.66 points higher. (Table D.6)

Size16K8K
9B+0.28 includes 0−0.21 includes 0
4B−0.07 includes 0−0.10 includes 0
2B+0.66 above 0−0.11 includes 0

A positive number means the response-balanced estimate scored higher than DN-MOPD, so the one detectable cell is the +0.66 at 2B at 16K. All six cells of the paper's mixture are in the table above; every 95% interval is in Table D.6 of the revision, and the two other mixtures in Table F.8. With IF at 14% and at 60% of the prompts (2B and 4B), it shows no detectable difference in any of the four size-by-mixture cells at 16K; at 8K the token-pooled rule is 0.75 points ahead at 4B with IF at 14%, and the other three cells show no detectable difference. (Table F.8) The revision reports the comparison at the paper's mixture in Sec. 2 and App. D.6; the cells under the two other mixtures are in Table F.8.

Reviewer_78BQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies feedback-scale imbalance in multi-teacher on-policy distillation. DN-MOPD retains domain-based teacher assignment and rescales token-level advantages using clipped ratios of pooled and per-domain log-ratio standard deviations. Experiments across three Qwen3.5 sizes show consistent improvements over label-routed MOPD, primarily in mathematics, although offline distillation and task arithmetic remain stronger overall.

Critical strengths
  1. Simple, controlled intervention. The method changes feedback scaling while keeping teacher assignment and the main OPD recipe fixed. It uses existing scores without additional teacher forward passes.

  2. Extensive and transparent evaluation. The paper includes multiple student seeds, fixed-weight controls, longer training, length diagnostics, and a negative boundary case. These support the reported improvement over Label while making its limitations visible.

Critical weaknesses
  1. Limited novelty and incremental contribution. The method adds domain-wise feedback scaling to an existing MOPD framework, but frozen first-batch weights recover its gains at 9B and 4B. While dynamic scaling improves over this calibration at 2B, its added value over simple fixed weighting is not consistently demonstrated.

  2. Unclear practical advantage over stronger alternatives. SeqKD-SFT and task arithmetic achieve higher aggregate scores at every tested Qwen3.5 size. Although their budgets differ, the practical performance–cost advantage remains unclear, and the overhead of recomputing scales is not quantified.

  3. Limited evidence for transfer across teacher configurations. The main results use one expert pool per size within Qwen3.5, while the earlier Qwen3-4B experiment shows no clear benefit under a different protocol. It remains unclear which teacher and feedback characteristics predict improvement.

  4. Unclear justification for downweighting high-variance feedback. A large log-ratio standard deviation may reflect useful, uneven teacher–student discrepancies rather than merely excessive feedback scale. The experiments support reweighting in the tested settings, but do not clarify when inverse-standard-deviation scaling might suppress valuable supervision.

Questions for authors
  1. Beyond the frozen-calibration gain at 2B, under which conditions is per-batch normalization preferable to a single shared fixed weighting, and does this distinction persist across seeds?
  2. What runtime and memory overhead does per-batch scaling add relative to frozen calibration, and when does DN-MOPD offer a better performance–cost trade-off than SeqKD-SFT or task arithmetic?
  3. How does the method distinguish excessive feedback variation from meaningful teacher–student discrepancies that indicate learning potential? Under what conditions could inverse-standard-deviation scaling suppress useful supervision?
  4. How sensitive are the gains to the lower clipping bound, domain mixture, and alternative scale statistics?
7 comments

Thank you for describing the intervention as simple and controlled, and the evaluation as extensive and transparent. This reply is split into five comments: W1/Q1 here, then W2/Q2, then W3, then W4/Q3, then Q4. The main comparisons below are also in the table of our comment to all reviewers, and the 95% intervals of all of them are in Tables C.10, C.14, C.15, C.16, D.6, D.8, E.3 and F.2.

W1 and Q1

"Beyond the frozen-calibration gain at 2B, under which conditions is per-batch normalization preferable to a single shared fixed weighting, and does this distinction persist across seeds?"

Across the three main expert pools at 16K, neither a single shared constant nor frozen first-batch weights is detectably ahead of per-batch estimation at any size, while per-batch estimation is detectably ahead of frozen weights at one size, 2B. Two conditions determine when this gap appears. (i) The scale moves after the first batch. At 2B the IF spread ratio is 2.32 on the first batch (2.30 as the median of 13 repeats of that batch) and has a median of 5.85 over training (App. D.3), so a weight frozen at update 0 keeps IF at 0.43, while the rule's applied IF weight has a median of 0.25, the floor. This is the size at which per-batch estimation is ahead of the frozen first-batch weights, and the distinction persists across seeds: at 2B the difference favours per-batch estimation on all three seeds, by 0.60 to 1.10 points at 16K and by 0.70 to 1.42 at 8K, and five of these six seed-and-cap differences are detectable. (Table C.12) At 9B the frozen IF weight is already 0.25, and at 4B it is 0.34; at these two sizes the two show no detectable difference at 16K on any seed. (Table C.12) In none of the 18 comparisons (three sizes, three seeds, two caps) are the frozen weights detectably higher (App. C.6 of the revision).

(ii) No weighting is known in advance. With the three Qwen3.5 expert pools of the main experiments: the shared constant (2, 1, 0.25) was chosen after inspecting DN-MOPD's first-batch estimates and already holds IF at 0.25, and it shows no detectable difference from DN-MOPD in Total at 16K at any size (seed 42) (Table C.11). Of the six fixed weightings we trained at 4B and 2B, none is detectably ahead of DN-MOPD at 16K and seven of the twelve are detectably behind (App. C.6 and C.9; the run with IF weight 2 is in Table C.16). Per-batch estimation is therefore the default: on these three pools at 16K, it matches the best fixed weighting in one run with no search. That match—not a higher score than the matched constant—is our claim, since choosing a constant matched to a pool requires ratios that a new pool does not supply in advance. Two of the six hold IF at 0.25, as the rule itself does (Table C.11) — the shared constant and the weighting that leaves mathematics and code at 1 — and neither differs detectably from DN-MOPD at either size, while every one of the seven cells detectably behind holds IF above 0.25. The six pools in which only the IF teacher is replaced exist to vary the IF feedback ratio, and the gain over Label follows that ratio (W3 below). On those six pools the same constant is ahead of DN-MOPD at 16K in five of the six, by 1.16 to 1.49 points, and not detectably different in the sixth; on Granite 4.2 3B it is ahead by 1.22 points in Total at 8K over three seeds, while at 16K the difference, +0.16, is not detectable (both descriptive). The claim above is about the three pools of the main experiments at 16K, where a constant has to be chosen before the ratios of a new pool are known. As stated in the submitted paper, the main tables use a 16K cap "chosen after inspecting both caps". App. A.4 gives the reason—to match the generation budget of the earlier Qwen3 public comparison—and calls it a presentation amendment rather than a preregistered preference for 16K; App. B of the submission already carries complete 8K results for every method and task. For this discussion period the read was registered before any of these cells existed: 16K primary, because that is the cap the submitted main tables use, and 8K reported beside it. At 8K this constant is 0.85 points ahead of DN-MOPD at 4B, with its interval above zero; 0.64 points behind at 2B, with its interval below zero; and not detectably different at 9B. Fixed-weight comparisons on the three main pools are in Tables C.11, C.12, C.15, C.16 and F.2 (App. C.6, C.8, C.9 and F.2).

"What runtime and memory overhead does per-batch scaling add relative to frozen calibration, and when does DN-MOPD offer a better performance–cost trade-off than SeqKD-SFT or task arithmetic?"

Relative to frozen calibration, per-batch estimation adds a median of 0.12 to 0.19 s per batch against an observed median update cycle of 98 to 180 s, with no teacher call, parameter, or optimizer state. It adds temporary token buffers and four standard-deviation reductions over log-ratios that the recipe already holds, measured in a single-thread CPU benchmark of this component (App. D.7 of the revision); a frozen weight adds nothing per batch. The temporary buffers take a median of 14.3, 17.3 and 23.8 MiB of host memory per batch at 9B, 4B and 2B, at most 46.7 MiB at the configured batch and response limits, and the rule runs in a process with no GPU allocation, so it adds no GPU memory (App. F.1).

9B4B2B
Added work, CPU benchmark (s per batch)0.120.140.19
Observed median update cycle (s), DN-MOPD, seed 42170.0139.098.0
Observed median update cycle (s), DN-MOPD, seed 43168.0141.0100.0
Observed median update cycle (s), DN-MOPD, seed 44169.5138.0101.0
Observed median update cycle (s), Label, seed 42174.0144.0103.0
Observed median update cycle (s), Label, seed 43180.0147.0106.0
Observed median update cycle (s), Label, seed 44176.0148.0105.0

The rows above are Table D.9 of the revision in full; the host-memory figures quoted before them, with their ranges and the known-answer control, are in App. F.1.

When the trade-off favours DN-MOPD. (i) Where label-routed OPD is the recipe in use and the feedback is dispersed as in our Qwen3.5 pools (IF 2.3 to 4.4 times the pooled first-batch spread, mathematics about half; App. D.2): 1.12, 1.97 and 2.34 points over Label at 9B, 4B and 2B (16K, three seeds; nine of nine seed-matched comparisons) for the overhead above. (Table C.10) (ii) Against task arithmetic at the paper's coefficient λ = 1.0, the 16K difference is not detectable at 9B or 4B after 80 updates, and the same holds after twice the updates. At 160 updates DN-MOPD's Total is 60.7 and 53.1 at 9B and 4B (Table 5), and a paired test shows no detectable difference from task arithmetic's 60.5 and 53.2 (Tables 1 and 2). Task arithmetic's Total is higher at all three sizes, as the paper states (Sec. 3; Tables 1–2), detectably so at 2B and at 8K at the paper's coefficient. That advantage depends on the coefficient: at 0.3 DN-MOPD is ahead by 2.08 and 1.34 points at 4B and 2B at 16K, by 2.20 and 2.53 at 8K and by 1.74 at 9B at 8K, with the 9B 16K difference, +0.33, not detectable (App. F.7). Merging also presupposes that every expert was fine-tuned from the student's own initialization, while OPD only needs each teacher to score the student's tokens. (iii) Against SeqKD-SFT the trade-off turns on what the pipeline can already produce; our own Table 4 reports both recipes' Totals. The two recipes need different things to exist first. SeqKD-SFT needs the teachers' answers generated offline with a 16K cap in our baseline and then trained on; OPD needs only the teachers' scores on the student's own answers, which is the stage this paper studies. Where those answers can be generated and imitated, SeqKD-SFT is stronger on these pools. In pipelines already running label-routed on-policy distillation, our weighting sets how strongly each teacher's feedback counts, gaining 1.12, 1.97, and 2.34 points over Label in the three Qwen3.5 pools (Table C.10). In the revision the overhead measurement is App. D.7, and the Totals of the task-arithmetic comparisons are in Tables 1, 2 and 5.

W3

"It remains unclear which teacher and feedback characteristics predict improvement."

One rollout batch before any update gives the first-batch spread ratios σ_d/σ_all, which track where DN-MOPD gains. In a controlled test that replaces only the IF teacher of the 2B pool by weaker ones, the point estimates of the gain over Label are in the same order as the first-batch IF ratio, at both caps and across all six pools we built. At 16K they are +1.47, +1.42, +1.17, +0.35, −0.37 and −0.95 points at ratios 1.54, 1.34, 1.01, 0.84, 0.65 and 0.28 (one seed; each ratio is the median of four first-batch measurements; App. E.4 of the revision) (Table E.3, the four pools in the registered bands). The mathematics and code ratios stay at 0.48 to 0.49 and 1.23 to 1.25 across all six pools, so the test moves the IF ratio alone.

Four pools fall in the two ratio bands we registered for scoring, 0.80 to 1.25 and 1.25 to 2.30: the gain is detectable in the three with the higher ratios, at both caps, and not at 0.84. (Table E.3, the four pools in the registered bands) The two pools below 0.80, where the rule raises the IF weight instead of lowering it, were registered as descriptive; they show no gain, and at 0.28 DN-MOPD is detectably below Label at 16K.

IF teacher of the 2B poolIF ratio16K8KRegistered as
Retrained, 200 rollout iterations1.54+1.47 above 0+2.13 above 0scored
Interpolated, α = 0.751.34+1.42 above 0+1.95 above 0scored
Interpolated, α = 0.651.01+1.17 above 0+0.91 above 0scored
Retrained, 100 rollout iterations0.84+0.35 includes 0+0.43 includes 0scored
Interpolated, α = 0.500.65−0.37 includes 0+0.09 includes 0description only
Interpolated, α = 0.250.28−0.95 below 0−0.58 includes 0description only

All six pools, both caps, DN-MOPD − Label in Total; the 95% intervals of the four pools in the registered bands are in Table E.3 of the revision.

In the three Qwen3.5 pools of the main experiments the ratios are far from balanced: 0.48 to 0.57 for mathematics, and for IF 2.30 at 2B (the median of 13 first-batch repeats), 2.93 at 4B and 4.41 at 9B (single first batches, Table D.1). There DN-MOPD gains 1.12 to 2.34 points over Label in Total at 16K on average over three seeds. (Table C.10) On Granite 4.2 3B, a rule registered before any student was trained on this pool predicts no detectable difference for its balanced ratios (1.01, 0.98 and 1.18), and none is detected on seed 42 or over three seeds (App. E.3). The earlier Qwen3-4B construction you point to fits the same reading: its median mathematics multiplier is 1.00, against 1.54 to 2.06 in the Qwen3.5 runs, and Label already transfers mathematics there (App. E.2). In the revision the ratios and the replaced-teacher pools are App. E.4, marked in blue; the earlier construction is App. E.2.

Comments on the paper

3 comments

We thank the three reviewers and the area chair for your time and comments. The paper makes one claim: in multi-teacher on-policy distillation, label routing decides which specialist teaches, not how strongly its feedback counts. DN-MOPD sets that strength from current-batch feedback spread, adding no teacher call, parameter, or optimizer state. With experts, initialization, prompts and update budget fixed, DN-MOPD is on average 1.12, 1.97 and 2.34 points above Label in Total at 9B, 4B and 2B at 16K over three seeds, and higher in all nine seed-matched comparisons. We tested it along the directions the reviews raised. The four results below, and one more in our next comment, summarize the new experiments; the table gives the main comparisons at both caps, with paired 95% intervals in Tables C.10, C.14, C.15, C.16, D.6, D.8, E.3 and F.2.

Reviewers note the simple, controlled intervention and extensive evaluation (78BQ), the diagnostic separation of teacher assignment from feedback strength (GtxH), and the intuitive, cost-effective design (xiKN).

  1. The IF weight moves mathematics in both directions (78BQ W4/Q3, GtxH W2). Against a second seed-42 Label run, 2B mathematics at 16K is 2.45 points higher at IF weight 0.25, 1.41 lower at weight 2, and 2.16 and 1.35 lower in the two runs at weight 4 (App. C.9 and C.8), and the IF score shows no detectable change in any of these four runs. Both runs at (1, 1, 4) are detectably below DN-MOPD in Total wherever they ran.

  2. Across the main expert pools, no fixed, hand-set, or share-and-gap weighting we trained is detectably ahead of DN-MOPD at 16K, and DN-MOPD needs no weight search (xiKN W1/Q1, 78BQ W1/Q1). Of six fixed weightings trained at 4B and 2B, none is detectably ahead of DN-MOPD in Total at 16K and seven of the twelve are detectably behind; all seven hold the IF weight above the 0.25 the rule applies here: 2 and 4 in the reverse controls, 1 with mathematics alone upweighted, 0.43 in the frozen update-0 weights at 2B. A share-and-gap weighting is not detectably ahead at any size over three seeds. At these pools and this cap it matches the best fixed weighting in one run. The one variant ahead at 16K in the table, by 0.66 points at 2B, is DN-MOPD's own rule under a response-balanced scale statistic, not another weighting. Open-MOPD's own allocation rule, ported, is behind DN-MOPD at all three sizes at 16K (see our reply to xiKN).

  3. The first-batch ratio tracks where the gain appears (78BQ W3, xiKN W2). Replacing only the 2B pool's IF teacher gives six pools whose gains over Label fall in the same order as their first-batch IF ratio at both caps: at 16K, 1.47, 1.42 and 1.17 points at ratios 1.54, 1.34 and 1.01, and no detectable difference at 0.84. There is no gain at the two lower ratios, where the rule raises the IF weight instead of lowering it (−0.37 and −0.95 points; one seed, descriptive). On a new family, Granite 4.2 3B, the ratios are close to 1 (1.01, 0.98, 1.18) and the prediction registered before training holds: no detectable difference from Label.

  4. The added cost is four reductions per batch on the CPU, and against merging DN-MOPD is level at 9B and 4B at 16K (78BQ W2/Q2, GtxH W1). Per-batch estimation adds a median of 0.12 to 0.19 s in a single-thread CPU benchmark, against an observed median update cycle of 98 to 180 s, and its buffers take 14.3 to 23.8 MiB of host memory. Against task arithmetic at the paper's coefficient the 16K difference is not detectable at 9B or 4B after 80 updates. After 160 updates of the same recipe DN-MOPD's Total is 60.7 and 53.1 at 9B and 4B, and a paired test shows no detectable difference from task arithmetic's 60.5 and 53.2.

Differences are Total points, seed 42 unless the row says otherwise; three entries are 9B / 4B / 2B unless the row gives its own order; n.d. means the 95% interval includes 0.

Experimenta − b16K8K
Main result, three seedsDN-MOPD − Label+1.12 / +1.97 / +2.34+2.71 / +2.94 / +2.87
IF weight 0.25 / 2 / 4 (App. C.9 run) at 2B, mathematicsvariant − Label (second seed-42 run)+2.45 / −1.41 / −2.16+3.78 / −1.30 / −2.53
IF weight 4, App. C.8 runDN-MOPD − variant+1.62 / +2.46 / +2.64+3.55 / +4.36 / +3.87
Share-and-gap weighting, three seedsDN-MOPD − variant+0.79 / n.d. / n.d.+1.93 / +1.10 / +0.91
Response-balanced estimatevariant − DN-MOPDn.d. / n.d. / +0.66n.d. / n.d. / n.d.
Replaced IF teacher, 2B (ratio 1.54 / 1.34 / 1.01 / 0.84)DN-MOPD − Label+1.47 / +1.42 / +1.17 / n.d.+2.13 / +1.95 / +0.91 / n.d.
Mixture with IF at 14% / 60%, 2BDN-MOPD − Label+2.27 / +1.26+3.06 / +1.51
Mixture with IF at 14% / 60%, 4B, three seedsDN-MOPD − Labeln.d. / n.d.+2.79 / +1.79

This comment lists the changes in the revised PDF. It and the one before it answer what the three reviews raise in common; the points each reviewer raises alone are answered in the comments addressed to them, five for 78BQ, four for GtxH and three for xiKN. Changes are marked in blue. Main text: the response-balanced estimate (Sec. 2); the integration criterion, the run with the IF weight raised to 4 and the consistency of the IF gradient at three sizes (Sec. 3); the matched weightings (Sec. 4); the first-batch ratio with Granite 4.2 3B, the replaced IF teacher, the floor and the median absolute deviation (Sec. 5). Appendix: seeds 43 and 44 of the frozen-weight control (the update-0 control of App. C.6), App. C.8 and C.9, the 2B and 9B gradient tables (App. D.5), App. D.6, D.7, E.3 and E.4. Corrections: the three-seed mean intervals in App. C.5 now share each draw's resampled questions across seeds, and all six remain above zero; the 80-update intervals of the per-domain table in App. C.3 now use the same bootstrap draws as the Total; code and IF intervals change in the second decimal, and the 4B code interval at 16K now includes zero; App. A.1 counts expert training in rollout iterations (four optimizer updates each). The code and data in the supplementary material follow these three corrections. App. D.1 now states which quantity Open-MOPD clips (its relative reward-magnitude factor, before normalization).

On the estimator, the bounds and the prompt mixture (GtxH W3, 78BQ Q4, xiKN W2/Q2): a response-balanced scale estimate differs detectably from the token-pooled rule in only two of 14 cells, and those two point in opposite directions; a floor of 0.10 shows no detectable difference from 0.25. (Tables D.6, D.7 and F.8) Under two other prompt mixtures, DN-MOPD is 2.27 and 1.26 points above Label at 2B at 16K (one seed), and 2.79 and 1.79 points above at 4B at 8K over three seeds, while the three-seed means at 4B at 16K, +0.68 and +0.59, are not detectable. Where the first comment's numbers come from, in the order they appear there: the nine-row table and the statements around it come from Tables C.10, C.16, C.15, D.6 and D.1; the Granite pool in App. E.3 and Table F.6; then Tables D.9, C.14, E.3, D.8 and D.8. (Table D.8)

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.