- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Delta-MOPD, a framework for multi-teacher on-policy distillation where teachers may use different tokenizers. The method has three components: an extension of SimpleOPD’s exact byte-span alignment from the sampled token to the student’s top-16 candidates; probability smoothing, which adds a constant to both checkpoint probabilities before the log-ratio to damp large shifts in low-probability regions; and a reliability-weighted fusion rule.
Experiments distill two teacher pairs into Qwen3-1.7B, 4B and 8B students. Across five math benchmarks, Delta-MOPD improves the five-benchmark average over the original students and over the strongest baseline by 0.21, 0.35 and 0.76 points.
Critical strengths
-
The research problem this paper solves is timely and interesting, as currently the mainstream MOPD only uses same-tokenizer teachers.
-
The idea of normalizing per-teacher directions is a reasonable way to keep one teacher's scale from dominating the direction of the update
Critical weaknesses
-
Although the paper targets cross-tokenizer MOPD, its main results rely on one teacher that shares the student’s base tokenizer (JustRL) and one genuinely cross-tokenizer teacher (Phi-4). JustRL differs from the Qwen3 students only in special-token configuration (Appendix E), so aligning it is nearly trivial, and there are no experiments with multiple genuinely cross-tokenizer teachers. Moreover, Phi-4’s contribution appears small relative to JustRL’s. According to Table 6, Phi-4 accounts for only about 16% of the smoothed shift magnitude before fusion. According to Table 3, Phi-4 alone yields only modest gains for the Qwen3-4B and Qwen3-8B students, even with smoothing (57.32 → 57.87 and 59.61 → 60.47). This suggests that Phi-4 acts mainly as a small correction to the JustRL signal rather than as an equal partner in the fusion.
-
Table 2 reports alignment coverage for Phi-4, Qwen3.5 and GLM tokenizers, but neither Qwen3.5 nor GLM is used as a teacher in any experiment, and the table does not identify which checkpoints were measured. The coverage numbers therefore show only that alignment is feasible for these tokenizers, not that the method trains well with them. The table also omits JustRL and QuestA.
-
The benefit of probability smoothing is inconsistent across settings. With JustRL as the single teacher, smoothing improves Qwen3-4B substantially (54.88 → 59.34) but lowers Qwen3-1.7B (42.28 → 38.97) and barely changes Qwen3-8B (60.99 → 61.09). Moreover, smoothing is evaluated mainly in single-teacher settings. Delta-MOPD without smoothing is not reported on the main teacher pair.
-
Several key comparisons are missing. The paper cites cross-tokenizer OPD methods such as SimCT and CompassOPD in its related work but does not compare against them; even a single-teacher comparison on Phi-4 would show whether the proposed top-k alignment transfers supervision better. Also, there is no same-family reference, such as OPD or Direct-OPD from a strong Qwen3 teacher at comparable compute.
-
The training setup is limited. Training rollouts are capped at 2,048 tokens while evaluation allows 31,744, so supervision likely covers only the beginning of most reasoning traces, and the handling of truncated responses is not described. Also, since the sequential-MOPD and Delta-MOPD contain the same number of forward passes of teachers, the claim that Delta-MOPD beats sequential distillation “using 300 rather than 600 optimizer steps” should be accompanied by wall-clock or GPU-hour comparisons.
-
Evaluation is limited to a single domain. Training prompts, all teacher pairs, the AMC23 calibration set and all five evaluation benchmarks are mathematical. A setting with teachers specialized in different domains, such as math and code, evaluated on both, together with a few non-math benchmarks to check for regressions, would substantially strengthen the paper.
Questions for authors
Please refer to weaknesses.
6 comments
W2 — Table 2 reports alignment coverage for Phi-4, Qwen3.5 and GLM tokenizers, but neither Qwen3.5 nor GLM is used as a teacher in any experiment, and the table does not identify which checkpoints were measured. The coverage numbers therefore show only that alignment is feasible for these tokenizers, not that the method trains well with them. The table also omits JustRL and QuestA.
We repeated the alignment coverage measurements, identified the tokenizer releases, and added results for JustRL and QuestA. We also jointly distilled Qwen3-8B with a Qwen3.5-based code teacher and Phi-4, improving the student in both mathematics and programming.
1. Coverage measurement. After losing the original coverage data, we generated 1,024 new responses to 256 mathematics prompts with Qwen3-1.7B. Using the tokenizer files listed in Table R36, we checked sampled-token boundaries and Top-16 candidate mapping on those rollouts. The results are close to those in Table 2 of the original submission (Table 18 in Appendix B.3.1 of the revised manuscript).
2. JustRL and QuestA. Both showed nearly 100% vocabulary coverage and sampled-token boundary alignment in Table R36. This supports treating them as having approximately the same tokenizer as Qwen3, as in our mathematics experiments. Direct-OPD also uses these models [1]. The original Table 2 focused on tokenizers that differed substantially from the student's.
3. Qwen3.5 in training. The coverage study in Table R36 uses the Qwen3.5-27B tokenizer. The joint training experiment in Table R30 of our response to W1 uses DN 9B, based on Qwen3.5-9B. On the Qwen3-8B cross-domain rollouts, DN 9B mapped 93.338% of the Top-16 candidates on mathematics prompts and 95.594% on programming prompts (Table 19 in Appendix B.3.2 of the revised manuscript).
Table R36. Tokenizer coverage measured on Qwen3-1.7B mathematics rollouts.
| Teacher tokenizer | Vocabulary (%) | Sampled (%) | Top-16 (%) | Top-16 Mass (%) |
|---|---|---|---|---|
| Phi-4-reasoning-plus | 65.39 | 90.90 | 87.09 | 90.17 |
| Qwen3.5-27B | 86.77 | 97.45 | 92.18 | 96.58 |
| GLM-4.7-Flash | 82.49 | 91.69 | 88.58 | 91.02 |
| JustRL | 99.9987 | 99.99 | 97.18 | 99.25 |
| QuestA | 99.9974 | 99.94 | 97.15 | 99.22 |
W3 — The benefit of probability smoothing is inconsistent across settings. With JustRL as the single teacher, smoothing improves Qwen3-4B substantially (54.88 → 59.34) but lowers Qwen3-1.7B (42.28 → 38.97) and barely changes Qwen3-8B (60.99 → 61.09). Moreover, smoothing is evaluated mainly in single-teacher settings. Delta-MOPD without smoothing is not reported on the main teacher pair.
To test the effect of probability smoothing in dual-teacher distillation, we compared Delta-MOPD with and without smoothing using the original mathematics teacher pair and the cross-domain teacher pair. Smoothing raised the mean score in both comparisons. With JustRL alone, its effect varied by student size.
1. Single teacher. With JustRL alone, smoothing changed the mean by −3.31 pp at 1.7B, +4.46 pp at 4B, and +0.10 pp at 8B (Table R37 and Table 3 of the revised manuscript). One possible explanation for the drop at 1.7B is that smoothing can attenuate useful supervision for low-probability candidates.
Table R37. Single-teacher smoothing ablation with JustRL at three student sizes.
| Student (JustRL only) | Without smoothing (%) | With smoothing (%) | Difference (pp) |
|---|---|---|---|
| Qwen3-1.7B | 42.28 | 38.97 | −3.31 |
| Qwen3-4B | 54.88 | 59.34 | +4.46 |
| Qwen3-8B | 60.99 | 61.09 | +0.10 |
2. Two teachers. With JustRL and Phi-4 on Qwen3-1.7B, probability smoothing raised the mathematics mean from 42.35% to 43.52% (+1.17 pp), with branch centering off in both runs. With Phi-4 and DN 9B on Qwen3-8B, smoothing raised the cross-domain mean from 69.73% to 70.04% (+0.31 pp) (Table R38).
Table R38. Dual-teacher smoothing ablations in mathematics and cross-domain distillation.
| Dual-teacher setting | Original student (%) | Without smoothing (%) | With smoothing (%) | Difference (pp) |
|---|---|---|---|---|
| Qwen3-1.7B, JustRL + Phi-4, five math benchmarks | 36.45 | 42.35 | 43.52 | +1.17 |
| Qwen3-8B, Phi-4 + DN 9B, four math and code benchmarks | 69.15 | 69.73 | 70.04 | +0.31 |
References
[1] Feng et al. (2026). Weak-to-Strong Generalization via Direct On-Policy Distillation.
W4 — Several key comparisons are missing. The paper cites cross-tokenizer OPD methods such as SimCT and CompassOPD in its related work but does not compare against them; even a single-teacher comparison on Phi-4 would show whether the proposed top-k alignment transfers supervision better. Also, there is no same-family reference, such as OPD or Direct-OPD from a strong Qwen3 teacher at comparable compute.
We added a cross-tokenizer comparison with CompassOPD using Phi-4 and a same-family OPD baseline using a Qwen3-14B teacher. Our single-teacher method outscored CompassOPD at all three student sizes. Delta-MOPD also outscored the same-family OPD baseline on AIME24 and AIME25 at comparable GPU cost.
1. Cross-tokenizer OPD. With the same Phi-4 checkpoints, our single-teacher method scored 1.29, 0.55, and 1.64 pp higher than CompassOPD at 1.7B, 4B, and 8B (Table R39, Table 3 of the revised manuscript).
Table R39. Single-teacher comparison with CompassOPD using Phi-4.
| Method | Qwen3-1.7B | Qwen3-4B | Qwen3-8B |
|---|---|---|---|
| Original student | 36.45 ± 0.33 | 57.32 ± 0.22 | 59.61 ± 0.29 |
| CompassOPD (Phi-4) | 38.19 ± 0.27 | 57.32 ± 0.24 | 58.83 ± 0.41 |
| Our single-teacher method (Phi-4) | 39.48 ± 0.29 | 57.87 ± 0.28 | 60.47 ± 0.20 |
In the separate JustRL control on Qwen3-1.7B, strict alignment yielded an AIME24/25 mean of 50.05%, above SimpleOPD at 46.41% and close to native Direct-OPD at 51.30% (Table 1 of the revised manuscript).
2. Same-family OPD. Starting from Qwen3-4B's AIME24/25 mean of 68.75%, standard OPD with a Qwen3-14B teacher reached 69.64% (+0.89 pp). Delta-MOPD reached 72.66% (+3.91 pp), 3.02 pp above standard OPD (Table R40).
Table R40. Standard OPD with a Qwen3-14B teacher versus Delta-MOPD.
| Model / method | AIME24 | AIME25 | Avg. | Δ vs. original student |
|---|---|---|---|---|
| Qwen3-14B teacher | 80.31 | 72.19 | 76.25 | — |
| Original Qwen3-4B | 70.00 | 67.50 | 68.75 | — |
| Qwen3-4B + standard OPD | 71.56 | 67.71 | 69.64 | +0.89 |
| Qwen3-4B + Delta-MOPD | 75.94 | 69.38 | 72.66 | +3.91 |
Both methods ran for 300 steps on eight GPUs. Measured step times total 74.65 GPU-hours for OPD (111.97 s/step) and 86.47 GPU-hours for Delta-MOPD (Table 9 of the revised manuscript). The OPD run took 9.40 wall-clock hours, equivalent to 75.24 GPU-hours.
W5 — The training setup is limited. Training rollouts are capped at 2,048 tokens while evaluation allows 31,744, so supervision likely covers only the beginning of most reasoning traces, and the handling of truncated responses is not described. Also, since the sequential-MOPD and Delta-MOPD contain the same number of forward passes of teachers, the claim that Delta-MOPD beats sequential distillation “using 300 rather than 600 optimizer steps” should be accompanied by wall-clock or GPU-hour comparisons.
We clarified how truncated rollouts are handled and measured the training costs of joint and sequential distillation on the same hardware. When a training rollout reaches the token cap, we retain the generated prefix and compute the loss on its valid tokens. Joint distillation used less GPU time than either sequential order.
1. Truncation. At the 2,048-token cap, we keep the generated prefix and compute loss on valid tokens. We do not insert EOS at the cutoff and mask padding after a natural EOS. Evaluation allows 31,744 tokens for complete reasoning and answers (Section 4.1 of the revised manuscript).
2. Training cost. Delta-MOPD scores with four checkpoints per step. Sequential distillation scores with the active teacher's two checkpoints per step. On the same eight GPUs, the 300-step Delta-MOPD run used 86.47 GPU-hours, compared with 126.63 and 123.98 GPU-hours for the two 600-step sequential runs, reductions of 31.7% and 30.3%, respectively (Table R41 and Table 9 in Section 5.4 of the revised manuscript). The GRPO comparison in Table 10 of the revised manuscript uses a separate run.
Table R41. Cumulative step time and GPU-hours for joint and sequential distillation.
| Method | Optimizer steps | Cumulative step time (hours) | GPU-hours |
|---|---|---|---|
| Sequential distillation (JustRL → Phi-4) | 600 | 15.83 | 126.63 |
| Sequential distillation (Phi-4 → JustRL) | 600 | 15.50 | 123.98 |
| Delta-MOPD | 300 | 10.81 | 86.47 |
Thank you for your feedback. We added joint math and code distillation with three distinct tokenizers, expanded the coverage analysis, tested Phi-4 and smoothing, added OPD baselines, measured training costs, and evaluated capabilities outside the training domains.
Joint distillation across tokenizers. We originally paired JustRL, whose tokenizer is close to the student's, with Phi-4 to test shared supervision. We now also pair Phi-4-reasoning → Phi-4-reasoning-plus with Qwen3.5-9B → DN-MOPD-Qwen3.5-9B-teacher-code. The two teacher pairs and the Qwen3-8B student each use a distinct tokenizer, and both teachers score the same student rollouts.
We sampled 10,000 prompts each from Skywork-OR1-RL-Data (math) and Eurus-2-RL-Data (code) without replacement and shuffled them into a shared fixed 1:1 pool. Joint training raised the overall mean from 69.15% to 70.04% and outscored both single-teacher variants and domain routing in both domains (R30, revised Table 4).
Table R30. Joint math and code results on Qwen3-8B.
Phi-4 in mathematics-only distillation. All reported math-only runs used AIME23 for calibration. AMC23 was tested in preliminary experiments and mistakenly named in the submission (R31, revised Table 7).
Table R31. AIME23 calibration.
The reported 16% is Phi-4's pre-fusion shift magnitude share (revised Table 16). Phi-4 has a lower gain and greater mean student–reference distance than JustRL (0.6223 vs. 0.3475). R32 shows the weights using gains alone and using gains with the mean distances recorded during training (revised Table 8).
Table R32. Mathematics teacher weights.
Adding Phi-4 to smoothed JustRL raised the math mean by 0.55 pp at 4B and 0.91 pp at 8B (R33, revised Table 3).
Table R33. Five-benchmark math mean ± SD (%).
Phi-4 in joint math and code distillation. We calibrate on HMMT February for math and LiveCodeBench v5 for code, then select gains by input domain (R34, revised Table 7). The calculated weights favor Phi-4 on math and DN 9B on code. Their recorded weights averaged 39.84% and 58.20%, respectively. The sum is below 100% because some student tokens map to neither teacher (R35, revised Table 8 and Figure 5).
Table R34. Cross-domain calibration.
Table R35. Cross-domain teacher weights.
Adding Phi-4 to DN 9B raised the math mean from 70.49% to 71.86% (+1.37 pp), the code mean from 67.93% to 68.22% (+0.29 pp), and the overall mean by 0.83 pp (R30, revised Table 4).
In an AIME25 grid-coloring problem, domain routing missed cases. The Delta-MOPD student solved it correctly and more concisely with Boolean checks and one counting rule for all 16 configurations. Each binary variable represents a shared edge (revised Appendix D.1.1).