acceptodds
Under review as a conference paper at ICLR 2027

Delta-MOPD: Cross-Tokenizer Multi-Teacher On-Policy Distillation

Abstract

Multi-teacher on-policy distillation (MOPD) integrates the capabilities of multiple teachers into a single student model through supervision on student-generated trajectories. However, existing MOPD research has largely focused on models with a shared tokenizer, limiting the integration of powerful teachers with different tokenizers. This raises a broader question: can multiple teachers with different tokenizers jointly guide the same student trajectories through parallel signal fusion? We introduce Delta-MOPD, to the best of our knowledge the first on-policy distillation framework to align and fuse policy shifts, the changes between reference and enhanced teacher checkpoints, across multiple teachers with different tokenizers. Building on prior cross-tokenizer alignment, we extend token matching to the student's Top- candidates, enabling cross-tokenizer on-policy distillation while retaining most of their probability mass. However, we find that policy shifts based on log-probability differences can produce large signals in low-probability regions, where supervision is less reliable than in high-probability regions. We address this with probability smoothing, which, in Qwen3-4B single-teacher experiments, achieves higher average scores with only 11.4–28.7% of the absolute shift magnitude observed without smoothing. For multi-teacher fusion, we separate the directions and magnitudes of these shifts to account for differences in signal scale, and combine them according to teacher reliability. Controlled experiments focusing on signal fusion further demonstrate its effectiveness over established baselines. Across five mathematical reasoning benchmarks, Delta-MOPD improves macro-average accuracy over Qwen3-1.7B, Qwen3-4B, and Qwen3-8B by 6.86, 2.37, and 2.24 percentage points, respectively, outperforming conventional MOPD baselines and alternative signal-fusion methods. The resulting Qwen3-8B student achieves a macro-average accuracy of 61.85%, surpassing the strongest teacher, Phi-4-reasoning-plus, at 61.66%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Multi-Teacher Distillation, Cross-Tokenizer Alignment, Heterogeneous Language Models, Policy-Shift Transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_D4TLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Delta-MOPD, a framework for multi-teacher on-policy distillation where teachers may use different tokenizers. The method has three components: an extension of SimpleOPD’s exact byte-span alignment from the sampled token to the student’s top-16 candidates; probability smoothing, which adds a constant τ\tau to both checkpoint probabilities before the log-ratio to damp large shifts in low-probability regions; and a reliability-weighted fusion rule.

Experiments distill two teacher pairs into Qwen3-1.7B, 4B and 8B students. Across five math benchmarks, Delta-MOPD improves the five-benchmark average over the original students and over the strongest baseline by 0.21, 0.35 and 0.76 points.

Critical strengths
  1. The research problem this paper solves is timely and interesting, as currently the mainstream MOPD only uses same-tokenizer teachers.

  2. The idea of normalizing per-teacher directions is a reasonable way to keep one teacher's scale from dominating the direction of the update

Critical weaknesses
  1. Although the paper targets cross-tokenizer MOPD, its main results rely on one teacher that shares the student’s base tokenizer (JustRL) and one genuinely cross-tokenizer teacher (Phi-4). JustRL differs from the Qwen3 students only in special-token configuration (Appendix E), so aligning it is nearly trivial, and there are no experiments with multiple genuinely cross-tokenizer teachers. Moreover, Phi-4’s contribution appears small relative to JustRL’s. According to Table 6, Phi-4 accounts for only about 16% of the smoothed shift magnitude before fusion. According to Table 3, Phi-4 alone yields only modest gains for the Qwen3-4B and Qwen3-8B students, even with smoothing (57.32 → 57.87 and 59.61 → 60.47). This suggests that Phi-4 acts mainly as a small correction to the JustRL signal rather than as an equal partner in the fusion.

  2. Table 2 reports alignment coverage for Phi-4, Qwen3.5 and GLM tokenizers, but neither Qwen3.5 nor GLM is used as a teacher in any experiment, and the table does not identify which checkpoints were measured. The coverage numbers therefore show only that alignment is feasible for these tokenizers, not that the method trains well with them. The table also omits JustRL and QuestA.

  3. The benefit of probability smoothing is inconsistent across settings. With JustRL as the single teacher, smoothing improves Qwen3-4B substantially (54.88 → 59.34) but lowers Qwen3-1.7B (42.28 → 38.97) and barely changes Qwen3-8B (60.99 → 61.09). Moreover, smoothing is evaluated mainly in single-teacher settings. Delta-MOPD without smoothing is not reported on the main teacher pair.

  4. Several key comparisons are missing. The paper cites cross-tokenizer OPD methods such as SimCT and CompassOPD in its related work but does not compare against them; even a single-teacher comparison on Phi-4 would show whether the proposed top-k alignment transfers supervision better. Also, there is no same-family reference, such as OPD or Direct-OPD from a strong Qwen3 teacher at comparable compute.

  5. The training setup is limited. Training rollouts are capped at 2,048 tokens while evaluation allows 31,744, so supervision likely covers only the beginning of most reasoning traces, and the handling of truncated responses is not described. Also, since the sequential-MOPD and Delta-MOPD contain the same number of forward passes of teachers, the claim that Delta-MOPD beats sequential distillation “using 300 rather than 600 optimizer steps” should be accompanied by wall-clock or GPU-hour comparisons.

  6. Evaluation is limited to a single domain. Training prompts, all teacher pairs, the AMC23 calibration set and all five evaluation benchmarks are mathematical. A setting with teachers specialized in different domains, such as math and code, evaluated on both, together with a few non-math benchmarks to check for regressions, would substantially strengthen the paper.

Questions for authors

Please refer to weaknesses.

6 comments

Thank you for your feedback. We added joint math and code distillation with three distinct tokenizers, expanded the coverage analysis, tested Phi-4 and smoothing, added OPD baselines, measured training costs, and evaluated capabilities outside the training domains.

W1 — Although the paper targets cross-tokenizer MOPD, its main results rely on one teacher that shares the student's base tokenizer (JustRL) and one genuinely cross-tokenizer teacher (Phi-4). JustRL differs from the Qwen3 students only in special-token configuration (Appendix E), so aligning it is nearly trivial, and there are no experiments with multiple genuinely cross-tokenizer teachers. Moreover, Phi-4's contribution appears small relative to JustRL's.

Joint distillation across tokenizers. We originally paired JustRL, whose tokenizer is close to the student's, with Phi-4 to test shared supervision. We now also pair Phi-4-reasoning → Phi-4-reasoning-plus with Qwen3.5-9B → DN-MOPD-Qwen3.5-9B-teacher-code. The two teacher pairs and the Qwen3-8B student each use a distinct tokenizer, and both teachers score the same student rollouts.

We sampled 10,000 prompts each from Skywork-OR1-RL-Data (math) and Eurus-2-RL-Data (code) without replacement and shuffled them into a shared fixed 1:1 pool. Joint training raised the overall mean from 69.15% to 70.04% and outscored both single-teacher variants and domain routing in both domains (R30, revised Table 4).

Table R30. Joint math and code results on Qwen3-8B.

MethodMath (%)Code (%)Overall ± SD (%)
Original Qwen3-8B70.8467.4769.15 ± 0.41
Phi-4 only71.2067.6769.43 ± 0.44
DN 9B only70.4967.9369.21 ± 0.29
Domain routing71.0368.0069.51 ± 0.22
Delta-MOPD71.8668.2270.04 ± 0.30

Phi-4 in mathematics-only distillation. All reported math-only runs used AIME23 for calibration. AMC23 was tested in preliminary experiments and mistakenly named in the submission (R31, revised Table 7).

Table R31. AIME23 calibration.

Teacher pairRef. (%)Post (%)Gain
Phi-465.3170.730.1699
JustRL18.5439.270.2937

The reported 16% is Phi-4's pre-fusion shift magnitude share (revised Table 16). Phi-4 has a lower gain and greater mean student–reference distance than JustRL (0.6223 vs. 0.3475). R32 shows the weights using gains alone and using gains with the mean distances recorded during training (revised Table 8).

Table R32. Mathematics teacher weights.

Weight settingPhi-4 (%)JustRL (%)
Calibration gain only22.4877.52
Calibration gain and mean distances during training18.0581.95

Adding Phi-4 to smoothed JustRL raised the math mean by 0.55 pp at 4B and 0.91 pp at 8B (R33, revised Table 3).

Table R33. Five-benchmark math mean ± SD (%).

StudentOriginalSmoothed Phi-4Smoothed JustRLDelta-MOPDGain over JustRL (pp)
Qwen3-4B57.32 ± 0.2257.87 ± 0.2859.34 ± 0.4459.89 ± 0.35+0.55
Qwen3-8B59.61 ± 0.2960.47 ± 0.2061.09 ± 0.3662.00 ± 0.29+0.91

Phi-4 in joint math and code distillation. We calibrate on HMMT February for math and LiveCodeBench v5 for code, then select gains by input domain (R34, revised Table 7). The calculated weights favor Phi-4 on math and DN 9B on code. Their recorded weights averaged 39.84% and 58.20%, respectively. The sum is below 100% because some student tokens map to neither teacher (R35, revised Table 8 and Figure 5).

Table R34. Cross-domain calibration.

DomainTeacher pairRef. (%)Post (%)Gain
MathematicsPhi-443.3351.150.1483
MathematicsDN 9B40.2141.880.0283
ProgrammingPhi-473.7372.88−0.0320
ProgrammingDN 9B69.1773.730.1602

Table R35. Cross-domain teacher weights.

Weight settingDomainPhi-4 (%)DN 9B (%)
Calibration gain onlyMathematics76.8723.13
Calibration gain and reference distancesMathematics68.0531.95
Calibration gain onlyProgramming12.7687.24
Calibration gain and reference distancesProgramming7.6392.37
Overall mean weightBoth domains39.8458.20

Adding Phi-4 to DN 9B raised the math mean from 70.49% to 71.86% (+1.37 pp), the code mean from 67.93% to 68.22% (+0.29 pp), and the overall mean by 0.83 pp (R30, revised Table 4).

In an AIME25 grid-coloring problem, domain routing missed cases. The Delta-MOPD student solved it correctly and more concisely with Boolean checks and one counting rule for all 16 configurations. Each binary variable represents a shared edge (revised Appendix D.1.1).

For each (S1, S2, S3, S4), compute:
C1 = (S1 + S3 == 1)
C2 = (S2 + S3 == 1)
C3 = (S1 + S4 == 1)
C4 = (S2 + S4 == 1)
Then, count = C1 + C2 + C3 + C4
Then total colorings += 2^{count}

W2 — Table 2 reports alignment coverage for Phi-4, Qwen3.5 and GLM tokenizers, but neither Qwen3.5 nor GLM is used as a teacher in any experiment, and the table does not identify which checkpoints were measured. The coverage numbers therefore show only that alignment is feasible for these tokenizers, not that the method trains well with them. The table also omits JustRL and QuestA.

We repeated the alignment coverage measurements, identified the tokenizer releases, and added results for JustRL and QuestA. We also jointly distilled Qwen3-8B with a Qwen3.5-based code teacher and Phi-4, improving the student in both mathematics and programming.

1. Coverage measurement. After losing the original coverage data, we generated 1,024 new responses to 256 mathematics prompts with Qwen3-1.7B. Using the tokenizer files listed in Table R36, we checked sampled-token boundaries and Top-16 candidate mapping on those rollouts. The results are close to those in Table 2 of the original submission (Table 18 in Appendix B.3.1 of the revised manuscript).

2. JustRL and QuestA. Both showed nearly 100% vocabulary coverage and sampled-token boundary alignment in Table R36. This supports treating them as having approximately the same tokenizer as Qwen3, as in our mathematics experiments. Direct-OPD also uses these models [1]. The original Table 2 focused on tokenizers that differed substantially from the student's.

3. Qwen3.5 in training. The coverage study in Table R36 uses the Qwen3.5-27B tokenizer. The joint training experiment in Table R30 of our response to W1 uses DN 9B, based on Qwen3.5-9B. On the Qwen3-8B cross-domain rollouts, DN 9B mapped 93.338% of the Top-16 candidates on mathematics prompts and 95.594% on programming prompts (Table 19 in Appendix B.3.2 of the revised manuscript).

Table R36. Tokenizer coverage measured on Qwen3-1.7B mathematics rollouts.

Teacher tokenizerVocabulary (%)Sampled (%)Top-16 (%)Top-16 Mass (%)
Phi-4-reasoning-plus65.3990.9087.0990.17
Qwen3.5-27B86.7797.4592.1896.58
GLM-4.7-Flash82.4991.6988.5891.02
JustRL99.998799.9997.1899.25
QuestA99.997499.9497.1599.22

W3 — The benefit of probability smoothing is inconsistent across settings. With JustRL as the single teacher, smoothing improves Qwen3-4B substantially (54.88 → 59.34) but lowers Qwen3-1.7B (42.28 → 38.97) and barely changes Qwen3-8B (60.99 → 61.09). Moreover, smoothing is evaluated mainly in single-teacher settings. Delta-MOPD without smoothing is not reported on the main teacher pair.

To test the effect of probability smoothing in dual-teacher distillation, we compared Delta-MOPD with and without smoothing using the original mathematics teacher pair and the cross-domain teacher pair. Smoothing raised the mean score in both comparisons. With JustRL alone, its effect varied by student size.

1. Single teacher. With JustRL alone, smoothing changed the mean by −3.31 pp at 1.7B, +4.46 pp at 4B, and +0.10 pp at 8B (Table R37 and Table 3 of the revised manuscript). One possible explanation for the drop at 1.7B is that smoothing can attenuate useful supervision for low-probability candidates.

Table R37. Single-teacher smoothing ablation with JustRL at three student sizes.

Student (JustRL only)Without smoothing (%)With smoothing (%)Difference (pp)
Qwen3-1.7B42.2838.97−3.31
Qwen3-4B54.8859.34+4.46
Qwen3-8B60.9961.09+0.10

2. Two teachers. With JustRL and Phi-4 on Qwen3-1.7B, probability smoothing raised the mathematics mean from 42.35% to 43.52% (+1.17 pp), with branch centering off in both runs. With Phi-4 and DN 9B on Qwen3-8B, smoothing raised the cross-domain mean from 69.73% to 70.04% (+0.31 pp) (Table R38).

Table R38. Dual-teacher smoothing ablations in mathematics and cross-domain distillation.

Dual-teacher settingOriginal student (%)Without smoothing (%)With smoothing (%)Difference (pp)
Qwen3-1.7B, JustRL + Phi-4, five math benchmarks36.4542.3543.52+1.17
Qwen3-8B, Phi-4 + DN 9B, four math and code benchmarks69.1569.7370.04+0.31

References

[1] Feng et al. (2026). Weak-to-Strong Generalization via Direct On-Policy Distillation.

W4 — Several key comparisons are missing. The paper cites cross-tokenizer OPD methods such as SimCT and CompassOPD in its related work but does not compare against them; even a single-teacher comparison on Phi-4 would show whether the proposed top-k alignment transfers supervision better. Also, there is no same-family reference, such as OPD or Direct-OPD from a strong Qwen3 teacher at comparable compute.

We added a cross-tokenizer comparison with CompassOPD using Phi-4 and a same-family OPD baseline using a Qwen3-14B teacher. Our single-teacher method outscored CompassOPD at all three student sizes. Delta-MOPD also outscored the same-family OPD baseline on AIME24 and AIME25 at comparable GPU cost.

1. Cross-tokenizer OPD. With the same Phi-4 checkpoints, our single-teacher method scored 1.29, 0.55, and 1.64 pp higher than CompassOPD at 1.7B, 4B, and 8B (Table R39, Table 3 of the revised manuscript).

Table R39. Single-teacher comparison with CompassOPD using Phi-4.

MethodQwen3-1.7BQwen3-4BQwen3-8B
Original student36.45 ± 0.3357.32 ± 0.2259.61 ± 0.29
CompassOPD (Phi-4)38.19 ± 0.2757.32 ± 0.2458.83 ± 0.41
Our single-teacher method (Phi-4)39.48 ± 0.2957.87 ± 0.2860.47 ± 0.20

In the separate JustRL control on Qwen3-1.7B, strict alignment yielded an AIME24/25 mean of 50.05%, above SimpleOPD at 46.41% and close to native Direct-OPD at 51.30% (Table 1 of the revised manuscript).

2. Same-family OPD. Starting from Qwen3-4B's AIME24/25 mean of 68.75%, standard OPD with a Qwen3-14B teacher reached 69.64% (+0.89 pp). Delta-MOPD reached 72.66% (+3.91 pp), 3.02 pp above standard OPD (Table R40).

Table R40. Standard OPD with a Qwen3-14B teacher versus Delta-MOPD.

Model / methodAIME24AIME25Avg.Δ vs. original student
Qwen3-14B teacher80.3172.1976.25—
Original Qwen3-4B70.0067.5068.75—
Qwen3-4B + standard OPD71.5667.7169.64+0.89
Qwen3-4B + Delta-MOPD75.9469.3872.66+3.91

Both methods ran for 300 steps on eight GPUs. Measured step times total 74.65 GPU-hours for OPD (111.97 s/step) and 86.47 GPU-hours for Delta-MOPD (Table 9 of the revised manuscript). The OPD run took 9.40 wall-clock hours, equivalent to 75.24 GPU-hours.

W5 — The training setup is limited. Training rollouts are capped at 2,048 tokens while evaluation allows 31,744, so supervision likely covers only the beginning of most reasoning traces, and the handling of truncated responses is not described. Also, since the sequential-MOPD and Delta-MOPD contain the same number of forward passes of teachers, the claim that Delta-MOPD beats sequential distillation “using 300 rather than 600 optimizer steps” should be accompanied by wall-clock or GPU-hour comparisons.

We clarified how truncated rollouts are handled and measured the training costs of joint and sequential distillation on the same hardware. When a training rollout reaches the token cap, we retain the generated prefix and compute the loss on its valid tokens. Joint distillation used less GPU time than either sequential order.

1. Truncation. At the 2,048-token cap, we keep the generated prefix and compute loss on valid tokens. We do not insert EOS at the cutoff and mask padding after a natural EOS. Evaluation allows 31,744 tokens for complete reasoning and answers (Section 4.1 of the revised manuscript).

2. Training cost. Delta-MOPD scores with four checkpoints per step. Sequential distillation scores with the active teacher's two checkpoints per step. On the same eight GPUs, the 300-step Delta-MOPD run used 86.47 GPU-hours, compared with 126.63 and 123.98 GPU-hours for the two 600-step sequential runs, reductions of 31.7% and 30.3%, respectively (Table R41 and Table 9 in Section 5.4 of the revised manuscript). The GRPO comparison in Table 10 of the revised manuscript uses a separate run.

Table R41. Cumulative step time and GPU-hours for joint and sequential distillation.

MethodOptimizer stepsCumulative step time (hours)GPU-hours
Sequential distillation (JustRL → Phi-4)60015.83126.63
Sequential distillation (Phi-4 → JustRL)60015.50123.98
Delta-MOPD30010.8186.47
Reviewer_28DYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Delta-MOPD combines multiple teachers with different tokenizers through on-policy distillation of teacher checkpoint differences. It aligns student top-k candidates across tokenizers, smooths low-probability policy shifts, and combines shift directions and magnitudes using reliability weights. Experiments across three Qwen3 student sizes and five mathematical reasoning benchmarks show improvements over the original students and evaluated distillation baselines.

Critical strengths
  1. The framework addresses a practical limitation of multi-teacher distillation, which is combining supervision across tokenizers. Alignment retains approximately 90–97% of student Top-16 probability mass in the tested settings, supporting the feasibility of this approach.
  2. The smoothing experiments provide useful evidence that large log-probability shifts can be counterproductive. For Qwen3-4B with JustRL, smoothing improves average accuracy from 54.88% to 59.34%. The paper also acknowledges cases where smoothing hurts performance.
Critical weaknesses
  1. The incremental gains lack uncertainty estimates. Improvements over the strongest baseline are only 0.21, 0.35, and 0.76 percentage points across student sizes, while the strongest teacher is surpassed by just 0.19 points. Although results average three training runs, no standard deviations or confidence intervals establish whether these differences are reliable.
  2. The experiments do not sufficiently isolate the proposed fusion mechanism. Table 4 adds another teacher and the fusion method simultaneously. The controlled experiment removes smoothing but does not separately test different parts such as centering, direction–magnitude separation, and reliability weighting. Therefore, it remains unclear whether the full mechanism is necessary compared with simpler, equally smoothed and weighted fusion.
Questions for authors
  1. Can you provide per-seed results and uncertainty estimates for Table 3, particularly the comparisons with the strongest baselines and teacher?
  2. How does Delta-MOPD compare with reliability-weighted averaging of centered, smoothed shifts, without direction–magnitude separation, under the same training settings?
4 comments

Thank you for asking us to quantify the gains and isolate the effect of the fusion rule. We now report the mean and standard deviation across runs with different seeds. We also compare direction–magnitude fusion with reliability-weighted linear fusion under the same training conditions and ablate branch centering and reliability weighting separately. We respond to each comment below.

W1 — The incremental gains lack uncertainty estimates.

Q1 — Can you provide per-seed results and uncertainty estimates for Table 3, particularly the comparisons with the strongest baselines and teacher?

To quantify variability across runs, we added the mean and standard deviation of the five-benchmark average across different random seeds to Table 3 of the revised manuscript. Delta-MOPD has a higher mean than the strongest distillation baseline at each of the three student sizes, and the distilled Qwen3-8B also has a slightly higher mean than the strongest teacher.

1. Strongest baselines. Delta-MOPD leads the strongest baseline by 0.42, 0.55, and 0.91 pp at 1.7B, 4B, and 8B, respectively. Table R23 gives the means and standard deviations for each comparison (Table 3 of the revised manuscript).

Table R23. Mathematics results against the strongest baseline at each student size.

StudentStrongest baselineBaseline avg. ± SD (%)Delta-MOPD avg. ± SD (%)Difference (pp)
Qwen3-1.7BSequential (Phi-4 → JustRL)43.10 ± 0.3643.52 ± 0.27+0.42
Qwen3-4BJustRL with smoothing59.34 ± 0.4459.89 ± 0.35+0.55
Qwen3-8BJustRL with smoothing61.09 ± 0.3662.00 ± 0.29+0.91

2. Strongest teacher. The original Qwen3-8B was 2.05 pp below Phi-4-reasoning-plus. Delta-MOPD raised the student mean by 2.39 pp to 62.00%, which is 0.34 pp above the strongest teacher (Table R24 and Table 3 of the revised manuscript).

Table R24. Qwen3-8B mathematics results and teacher checkpoint scores.

ModelMath avg. ± SD (%)Δ vs. original Qwen3-8B (pp)
Original Qwen3-8B59.61 ± 0.290.00
Qwen3-8B + Delta-MOPD62.00 ± 0.29+2.39
Phi-4-reasoning-plus (strongest teacher)61.66 ± 0.21+2.05
Phi-4-reasoning56.71 ± 0.18−2.90
JustRL-DeepSeek-1.5B36.45 ± 0.25−23.16
DeepSeek-R1-Distill-Qwen-1.5B22.80 ± 0.31−36.81

3. Cross-domain setting. Delta-MOPD scored 70.04% overall compared with 69.51% for domain routing, the strongest baseline by overall mean. The 0.53 pp gain came mainly from mathematics (+0.83 pp), with a smaller gain in programming (+0.22 pp) (Table R25 and Table 4 of the revised manuscript).

Table R25. Selected cross-domain baseline results on Qwen3-8B.

MethodMath avg. (%)Code avg. (%)Overall avg. ± SD (%)
Original Qwen3-8B70.8467.4769.15 ± 0.41
Phi-only71.2067.6769.43 ± 0.44
CompassOPD (DN 9B)70.9467.9569.44 ± 0.39
Domain routing71.0368.0069.51 ± 0.22
Delta-MOPD71.8668.2270.04 ± 0.30

W2 — The experiments do not sufficiently isolate the proposed fusion mechanism. Table 4 adds another teacher and the fusion method simultaneously. The controlled experiment removes smoothing but does not separately test different parts such as centering, direction–magnitude separation, and reliability weighting. Therefore, it remains unclear whether the full mechanism is necessary compared with simpler, equally smoothed and weighted fusion.

Q2 — How does Delta-MOPD compare with reliability-weighted averaging of centered, smoothed shifts, without direction–magnitude separation, under the same training settings?

To isolate the effect of the fusion rule, we ran controlled comparisons with the same teachers, probability smoothing, and reliability weights. Direction–magnitude fusion achieved higher mean scores than reliability-weighted linear fusion in both mathematics and cross-domain distillation, with and without branch centering.

1. Fusion rule. With centering enabled, direction–magnitude fusion improved the mean by 0.79 pp in mathematics and 0.12 pp in the cross-domain setting. With centering disabled, the gains were 0.52 and 0.65 pp, respectively. See Table R26 and the revised manuscript (Tables 13–14 in Appendix A.5 and Tables 21–22 in Appendix E.2).

Table R26. Matched comparison of reliability-weighted linear and direction–magnitude fusion.

SettingBranch centeringReliability-weighted linear fusion (%)Direction–magnitude (%)Difference (pp)
Qwen3-4B, mathematicsOn58.9059.69+0.79
Qwen3-4B, mathematicsOff59.3759.89+0.52
Qwen3-8B, mathematics and codeOn69.5069.62+0.12
Qwen3-8B, mathematics and codeOff69.3970.04+0.65

2. Branch centering. Removing centering raised the mean by 0.15–0.42 pp across the four settings, so the revised method uses the simpler formulation without branch centering (Table R27 and Tables 11–12 in Appendix A of the revised manuscript).

Table R27. Branch-centering ablations in mathematics and cross-domain distillation.

SettingWith centering (%)Without centering (%)Difference (pp)
Qwen3-1.7B, mathematics43.3143.52+0.21
Qwen3-4B, mathematics59.6959.89+0.20
Qwen3-8B, mathematics61.8562.00+0.15
Qwen3-8B, mathematics and code69.6270.04+0.42

3. Reliability weighting. In the centered Qwen3-4B mathematics experiment, changing from equal weights to gain-only weights raised the mean from 58.34% to 59.56%. Adding the distance term raised it further to 59.69%. The calibration gain term accounts for most of the improvement in this comparison (Table R28 and Table 24 in Appendix E.3.2 of the revised manuscript).

Table R28. Reliability-weight ablations on Qwen3-4B in mathematics.

WeightingβλFive-benchmark math avg. (%)
Equal weights0058.34
Gain only10059.56
Gain and distance10159.69

In the cross-domain experiment without centering, the same three weight settings gave overall means of 69.40%, 69.85%, and 70.04%, respectively. The default weights improved both mathematics and programming over equal weights (Table R29 and Table 23 in Appendix E.3.1 of the revised manuscript).

Table R29. Reliability-weight ablations on Qwen3-8B in mathematics and programming.

WeightingβλMath avg.Code avg.Overall avg.
Equal weights0071.0967.7069.40
Gain only10071.6768.0269.85
Default weights10171.8668.2270.04

We hope the reported standard deviations and matched comparisons address your concerns, and we would appreciate your reassessment.

Dear Reviewer 28DY,

Thank you for taking the time to review our paper. Your feedback has helped us improve the manuscript.

As the author–reviewer discussion period draws to a close, we would appreciate it if you could review our rebuttal and let us know whether our responses address your concerns. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Best regards,

The Authors

Dear Reviewer 28DY,

With only a few hours remaining in the author–reviewer discussion period, we would like to briefly follow up on our rebuttal. We would appreciate it if you could review our responses and let us know whether they address your concerns. We remain available to provide further clarification before the discussion closes.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_EnKWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Delta-MOPD, a multi-teacher on-policy distillation method that lets teachers with different tokenizers supervise the same student rollouts. Following Direct-OPD, each teacher’s supervision is a policy shift: the log-probability difference between its enhanced checkpoint and its reference checkpoint. The method has the components of Top-k alignment, Probability smoothing, and Reliability-weighted fusion. Each teacher’s shift is centered and split into a direction and a magnitude. These are averaged with weights. Validation uses two teachers and three Qwen3 students. Delta-MOPD improves the original students by large margin and beats the strongest baseline at each size. The paper also reports an alignment-coverage analysis, a controlled fusion study with JustRL and QuestA, training-dynamics diagnostics, and sensitivity to the hyperparameters.

Critical strengths
  1. A practically motivated problem with a useful alignment result. Supervising one student with teachers from different model families is a genuine practical need.

  2. The paper shows that strict Top-k one-to-one alignment keeps about 87–92% of student candidates and 90–97% of their probability mass, and it costs only about 1.25 points relative to native Direct-OPD (Tables 1–2). This is a simple, reusable engineering result.

  3. The main results are averaged over three seeds, cover three student sizes and five benchmarks, and include several sensible multi-teacher baselines, plus a hyperparameter sensitivity study.

  4. The low-probability log-ratio issue is a real and well-illustrated observation (Fig. 2).

Critical weaknesses
  1. The method is an incremental combination of existing works. Policy-shift supervision comes from Direct-OPD, alignment extends SimpleOPD, cross-family shift transfer resembles CompassOPD, reliability weighting resembles TrustMOPD, and smoothing is standard additive smoothing inside a log-ratio. The first claim also overlooks offline cross-tokenizer multi-teacher fusion such as FuseLLM.

  2. Only one main teacher is actually cross-tokenizer. Appendix E shows JustRL and Qwen3 share all 151,643 base token entries and the same BPE merges, so the claim rests on Phi-4 alone. Phi-4 contributes only about 16% of the shift magnitude, and the controlled fusion study (JustRL + QuestA) also uses tokenizers that share the Qwen base vocabulary.

  3. Capability integration, the core motivation for MOPD, is never tested. Both teachers are math models evaluated only on math, so the setting is closer to ensemble distillation than to integrating heterogeneous capabilities.

  4. Probability smoothing is confounded with step size. Shrinking shift magnitude to 11.4–28.7% shrinks the effective update, and the unsmoothed 4B run collapses late in a way typical of an overly large step. No magnitude-matched or reduced-learning-rate control separates the two explanations.

  5. The smoothing evidence is inconsistent and the ablation starts from a degraded run. Smoothing hurts 1.7B with JustRL by 3.31 points, yet it is applied at all sizes. Table 4's starting configuration already falls below the original student (54.88 vs 57.32), so its +4.46 mostly reflects recovery from failure.

  6. The reliability weighting is not shown to matter. β = 0 is within 0.47 points of the default, β = 10 lowers OlympiadBench by about 2.4 points, λ is never ablated, and the realized weights ρ are never reported. The weights may concentrate on JustRL, which would be consistent with Delta-MOPD being close to JustRL with smoothing at 4B and 8B.

  7. The gains are not shown to be statistically significant. No standard deviations are reported despite three seeds, the margins over the best baseline are only 0.21–0.76 points, and "surpasses the strongest teacher" rests on a 0.19-point difference. The M⁺ diagnostic changes by construction under centering, so it is not independent evidence.

  8. Key baselines are missing. These include TrustMOPD (cited but not compared), CompassOPD- or SimCT-style cross-tokenizer baselines, and standard multi-teacher OPD. Also missing is GRPO on the same prompts with a matched budget, a natural alternative for verifiable math.

  9. The headline claim is barely tested. JustRL shares Qwen’s base vocabulary, so only Phi-4 is truly cross-tokenizer, and it contributes about 16% of the supervision signal. The paper never shows two genuinely different tokenizers working together, which is what its title promises.

Questions for authors
  1. Please report mean ± std over the three seeds for every row of Table 3. Are the margins over the best baseline at each size statistically significant?

  2. What are the AMC23 gains g_b for both teachers, and what is the realized distribution of ρ during training? If ρ is concentrated on JustRL, what does the fusion add beyond single-teacher JustRL with smoothing?

  3. Does unsmoothed training with magnitude-matched rescaling, or a correspondingly lower learning rate, recover the benefit attributed to smoothing?

  4. Does Table 3’s Linear Fusion use probability smoothing? If not, please report Linear Fusion with smoothing so the direction/magnitude decomposition is isolated.

  5. Please report a uniform-weight variant (β = 0, λ = 0), a λ = 0 variant, a no-centering variant, and Delta-MOPD without smoothing at 1.7B.

  6. How does Delta-MOPD compare with TrustMOPD and with GRPO on the same 105k prompts under a matched budget?

  7. How are rollouts truncated at the 2,048-token training cap handled, given evaluation allows 31,744 tokens? Was Phi-4-reasoning-plus evaluated with its recommended decoding settings?

  8. I would like to suggest to evaluate a genuinely cross-tokenizer teacher pair, such as Phi-4 with GLM or Qwen3.5, which are already analyzed in Table 2.

  9. I recommend to test a multi-domain setting, such as math, code and instruction following.

  10. I think it is better to report the compute cost of the four teacher forward passes per step.

13 comments

Thank you for your detailed feedback and suggestions. We added a joint mathematics and programming distillation experiment in which the two teachers and the student each use a distinct tokenizer, and evaluated instruction following outside the training domains. We also report means and standard deviations for the main mathematics results, add comparisons with CompassOPD and GRPO, and run ablations to address your questions. In the centering ablation, removing branch centering improved mean scores in all four tested settings, so we adopted the simpler formulation in the revised manuscript. We respond to each comment below.

W1 — The method is an incremental combination of existing works. The first claim also overlooks offline cross-tokenizer multi-teacher fusion such as FuseLLM.

Our contribution includes a new MOPD training paradigm in which teachers with different tokenizers jointly supervise the same student rollouts. We align and fuse their policy shifts to make this joint supervision possible.

  1. A new MOPD training paradigm. To our knowledge, Delta-MOPD is the first to study multi-teacher on-policy distillation with different tokenizers. Different tokenizers can split the same student rollout into different tokens, so the teachers' policy shifts need to be aligned before token-level fusion. Delta-MOPD aligns these shifts to the student's candidate tokens and fuses their directions and magnitudes separately, allowing both teachers to contribute to the same update. In joint math and code distillation, Delta-MOPD achieved 70.04% overall versus 69.51% for the conventional MOPD baseline (Ma et al., 2026) with the same cross-tokenizer components (Table 4 of the revised manuscript).
  2. Similar ideas address different problems. For example, Direct-OPD uses a teacher's policy shift for weak-to-strong transfer in a shared token space. We use policy shifts to bring supervision from multiple teachers with different tokenizers into a single student update. We also want to emphasize that although additive smoothing is a standard technique, our contribution lies in identifying a pathology in the log-probability difference signal and showing how to mitigate it with a simple and effective correction.

FuseLLM is a relevant precedent for cross-tokenizer knowledge fusion, and we now discuss it in Section 2 of the revised manuscript. FuseLLM aligns and fuses the next-token distributions of source models on a fixed text corpus [1]. Delta-MOPD samples responses from the current student policy, aligns the policy shifts between each teacher's reference and enhanced checkpoints to the student's top-k candidate space, and fuses these shifts for a student update. This allows teachers with different tokenizers to jointly supervise the reasoning trajectories that the student generates during training.

[1] Wan et al. (2024). Knowledge Fusion of Large Language Models.

W2 — Only one main teacher is actually cross-tokenizer. Appendix E shows JustRL and Qwen3 share all 151,643 base token entries and the same BPE merges, so the claim rests on Phi-4 alone. Phi-4 contributes only about 16% of the shift magnitude, and the controlled fusion study (JustRL + QuestA) also uses tokenizers that share the Qwen base vocabulary.

Q8 — I would like to suggest to evaluate a genuinely cross-tokenizer teacher pair, such as Phi-4 with GLM or Qwen3.5, which are already analyzed in Table 2.

We agree that Phi-4 was the only teacher in our original main experiments whose tokenizer differed substantially from the student's, and this pairing was deliberate. We wanted to test whether a weaker teacher with nearly the same tokenizer as the student (JustRL) and a stronger teacher with a different tokenizer (Phi-4) could jointly supervise the same student rollouts. We consider this pairing more representative of the teacher combinations encountered in practice.

Following your suggestion, we added a joint mathematics and programming experiment in which the two teachers and the student use three distinct tokenizers.

  1. Experimental setup. We use Phi-4-reasoning and Phi-4-reasoning-plus as the mathematics teacher pair, and Qwen3.5-9B and DN-MOPD-Qwen3.5-9B-teacher-code as the programming teacher pair. Both pairs supervise the same Qwen3-8B rollouts. We sampled 10,000 mathematics prompts from Skywork-OR1-RL-Data and 10,000 programming prompts from Eurus-2-RL-Data without replacement, then combined and shuffled them into a fixed 1:1 training pool shared by all distillation methods in Table 4 of the revised manuscript.

  2. Results. Joint distillation with the two teachers improved Qwen3-8B, raising its overall average from 69.15% to 70.04%. Delta-MOPD also outperformed both single-teacher variants and the domain-routing baseline (69.51%). Table R1 reports the full results from Table 4 in Section 4.5 of the revised manuscript. Scores are percentages, and Δ is the gain over the original student in percentage points. The overall average weights all four benchmarks equally.

Table R1. Joint mathematics and programming results on Qwen3-8B.

Model / methodAIME24AIME25Math Avg.Math ΔHumanEval+LCB v6Code Avg.Code ΔOverall Avg. ± SDOverall Δ
Teacher checkpoints
Phi-4-reasoning (math ref)70.7361.2565.99—90.9350.4670.70—68.34 ± 0.31—
Phi-4-reasoning-plus (math post)75.6269.5872.60—90.8949.7570.32—71.46 ± 0.38—
Qwen3.5-9B (code ref)71.2554.7963.02—83.1950.1866.69—64.85 ± 0.28—
DN-MOPD-Qwen3.5-9B (code post)71.9856.7764.38—85.1852.7168.95—66.66 ± 0.47—
Qwen3-8B student
Original student75.0066.6770.84—85.2149.7267.47—69.15 ± 0.41—
Phi-only75.1767.2271.20+0.3685.5849.7567.67+0.2069.43 ± 0.44+0.28
DN-only74.9366.0470.49−0.3586.1349.7267.93+0.4669.21 ± 0.29+0.06
CompassOPD / Phi74.1766.4670.32−0.5284.7650.3667.56+0.1068.94 ± 0.33−0.21
CompassOPD / DN75.2166.6770.94+0.1085.7550.1467.95+0.4869.44 ± 0.39+0.29
Domain routing74.0667.9971.03+0.1986.1949.8068.00+0.5369.51 ± 0.22+0.36
Mean fusion74.2467.8871.06+0.2285.2550.0667.66+0.1969.36 ± 0.35+0.21
Delta-MOPD76.0167.7171.86+1.0285.8450.5968.22+0.7570.04 ± 0.30+0.89

W3 — Capability integration, the core motivation for MOPD, is never tested. Both teachers are math models evaluated only on math, so the setting is closer to ensemble distillation than to integrating heterogeneous capabilities.

Q9 — I recommend to test a multi-domain setting, such as math, code and instruction following.

We added a capability integration experiment covering mathematics and programming, and evaluated instruction following as an out-of-domain task. Joint distillation improved the student's performance in both training domains and also improved its instruction-following accuracy.

Mathematics and programming. As shown in Table R1 in our response to W2 and Q8, Delta-MOPD raised the original student's mathematics average from 70.84% to 71.86% and its programming average from 67.47% to 68.22%. It also outscored both single-teacher variants in both domains and achieved the highest overall average among student variants (70.04%).

Instruction following. We evaluated IFEval as an out-of-domain task after joint mathematics and programming training. Prompt-level strict accuracy rose from 85.77% to 86.88%, and loose accuracy rose from 87.80% to 89.03% (Table R2, Table 26 in Appendix F.2 of the revised manuscript).

Table R2. IFEval after joint mathematics and programming training.

Model / methodStrict (%)Δ (pp)Loose (%)Δ (pp)
Original Qwen3-8B85.77–87.80–
Phi-only86.14 ± 0.98+0.3788.23 ± 0.87+0.43
DN-only85.58 ± 0.18−0.1888.23 ± 0.28+0.43
Domain routing85.71 ± 0.38−0.0688.17 ± 0.64+0.37
Mean fusion85.52 ± 0.38−0.2588.17 ± 0.18+0.37
Delta-MOPD (ours)86.88 ± 0.67+1.1189.03 ± 0.65+1.23

W4 — Probability smoothing is confounded with step size.

Q3 — Does unsmoothed training with magnitude-matched rescaling, or a correspondingly lower learning rate, recover the benefit attributed to smoothing?

We added two controls to separate the effect of probability smoothing from that of step size. Neither fixed signal rescaling nor a lower learning rate fully matched the accuracy achieved with smoothing.

We tested both controls separately for JustRL and Phi-4 on Qwen3-4B. With smoothing disabled, we scaled either the original teacher signal or the learning rate by 0.114 for JustRL and 0.287 for Phi-4, matching the respective relative shift magnitudes of 11.4% and 28.7% (Section 5.2 of the revised manuscript). Table R3 reports the results for both teachers, with equal weight for AIME24 and AIME25 (Table 20 in Appendix E.1.1 of the revised manuscript).

  • JustRL. Fixed signal rescaling recovered part of the gain from smoothing. A lower learning rate reduced accuracy relative to the unsmoothed run.
  • Phi-4. A lower learning rate recovered part of the gain from smoothing (Table 20 in Appendix E.1.1 of the revised manuscript).

Table R3. Probability smoothing, fixed signal rescaling, and learning-rate controls on Qwen3-4B.

ConditionJustRL AIME24JustRL AIME25JustRL Avg.Phi-4 AIME24Phi-4 AIME25Phi-4 Avg.
Original student70.0067.5068.7570.0067.5068.75
Unsmoothed72.8165.2169.0172.2965.9469.12
Fixed signal rescaling74.3865.7370.0573.6865.2169.45
Lower learning rate66.2565.2165.7373.3366.3969.86
Smoothing76.4670.1073.2873.6566.3570.00

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We sincerely thank all reviewers, the Area Chair (AC), the Senior Area Chairs (SACs), and the Program Chairs (PCs) for their time and effort throughout the review process. The revised manuscript addresses the main shared concerns as follows.

  • Joint distillation across tokenizers and domains. The mathematics and programming teachers jointly supervised the same rollouts sampled from Qwen3-8B's current policy. The two teachers and the student each use a distinct tokenizer. The student improved in both domains, with its overall average rising from 69.15% to 70.04% (Table 4 in Section 4.5 of the revised manuscript). We also report instruction following after joint training (Table 26 in Appendix F.2 of the revised manuscript).
  • Matched fusion comparisons. With teachers, probability smoothing, reliability weights, and centering held fixed within each comparison, direction–magnitude fusion scored higher than linear fusion in both the mathematics and cross-domain settings (Tables 13–14 in Appendix A.5 and Tables 21–22 in Appendix E.2 of the revised manuscript).
  • Revised method and uncertainty. Removing branch centering improved mean scores in all four tested settings. The main results in the revised manuscript use this simpler formulation, and the original centered results are in Tables 11–12 in Appendix A. Table 3 of the revised manuscript reports means and standard deviations alongside the mathematics results.
  • New baselines and measured training cost. We added CompassOPD comparisons (Tables 3–4 of the revised manuscript) and compared with GRPO at comparable GPU time and at equal training steps and prompt exposures (Table 10 of the revised manuscript). On the same eight GPUs, the 300-step Delta-MOPD run used 30.3%–31.7% fewer GPU-hours than the two 600-step sequential distillation runs (Table 9 in Section 5.4 of the revised manuscript). We also report a same-family OPD comparison with a Qwen3-14B teacher in our response to D4TL, W4.

Locations of changes in the revised manuscript

ChangeLocation in the revised manuscript
Related work on FuseLLM and cross-tokenizer knowledge fusionSection 2
Fusion without branch centering and comparisons with centered variantsSection 3.3 and Appendix A.1–A.5, Tables 11–14
Rollout truncation during trainingSection 4.1
Updated mathematics results and standard deviations, cross-domain experiments, and CompassOPD baselinesSections 4.4–4.5, Tables 3–4
Teacher calibration, weight calculations, and weight ablationsSection 5.3, Tables 7–8, and Appendix E.3, Tables 23–24 and Figure 5
Teacher scoring time and training cost comparisons with sequential distillation and GRPOSection 5.4, Tables 9–10
Tokenizer mapping coverage on student rolloutsAppendix B.3, Tables 18–19
Qualitative examples from cross-domain distillationAppendix D.1
Probability smoothing controls and matched fusion comparisonsAppendix E.1–E.2, Tables 20–22
Out-of-domain programming and instruction-following evaluationsAppendix F.1–F.2, Tables 25–26

Delta-MOPD introduces a new MOPD paradigm in which teachers with distinct tokenizers jointly supervise the same student-generated rollouts. We hope our manuscript gives all reviewers and readers new ideas to build on. Once again, we are grateful to the reviewers, ACs, SACs, and PCs for their efforts and for the constructive feedback that helped strengthen the manuscript. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.