- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
They work on the on-policy self-distillation problem, where we have questions and a reference solution for each of them. The student generates an answer, and a teacher, which uses the same underlying model but is conditioned on the reference solution, provides distributions for computing the token-level distillation loss. They argue that it is better to have multiple teachers, conditioned on different versions of the reference solution, and select a teacher for each token. Their teacher pool includes both the full reference and truncated versions. Through intervention experiments, they show that the teacher producing the largest accuracy gain can differ across token positions. They then propose a rule for choosing which teacher to use for each token. For a given token generated by the student in a rollout, they check whether each teacher assigns it a higher or lower probability than the student does. This divides the teachers into two groups, and they select the group with more teachers. From that group, they select the teacher with the highest student-to-teacher KL divergence and use its distribution to guide the student.
Their results show improvements over OPSD, SDPO, and PAINT in average performance across three math datasets.
Critical strengths
- The model performance is very good.
- The pilot experiments that aim to justify the design choices are interesting.
Critical weaknesses
- Having multiple teachers increases training time significantly. Wall-clock time should be part of the comparisons of different methods in the tables.
- The experiments comparing the farthest teacher with the closest are interesting, but it would make more sense to evaluate the full range of teachers: the closest teacher, the second closest, etc. (table 4)
- They used a single seed for training. Their gains are large, but without additional runs with different seeds, it is hard to be certain about the robustness of the results across training runs.
Questions for authors
- Can you compare the best teacher identified in the pilot experiments with the teacher selected by your method? For example, report the accuracy of your teacher selection against that oracle, as well as the average gain from your teacher selection compared with the best and random teacher selections.
5 comments
Rebuttal Part2
W2: More Detailed Table 4
are interesting, but it would make more sense to evaluate the full range of teachers: the closest teacher, the second closest, etc. (table 4)
Thank you for the suggestions. We agree that reporting more details would be helpful for validating our pilot study.
Following your suggestion, we evaluated the full rank spectrum within the majority subset: the closest, the second closest, the farthest, and the second farthest teachers, given that there are 7 candidate teachers in total. As shown in Table 4, intervention performance exhibits a monotonic trend: as the forward-KL divergence increases from Nearest to Farthest, the intervention gain steadily rises (e.g., from +0.59% to +2.42% on 1.7B, and from +1.21% to +2.78% on 4B). This strongly confirms that within the majority-consensus subset, the teacher-student forward-KL distance serves as a reliable, continuous indicator of supervisory informativeness, directly validating our maximum-KL selection rule.
Table 4. Average gains from the farthest and nearest teachers.
| Teacher / Model | Qwen3-1.7B | Qwen3-4B |
|---|---|---|
| Farthest | +2.42% | +2.78% |
| Second Farthest | +1.48% | +2.39% |
| Second Nearest | +0.57% | +1.56% |
| Nearest | +0.59% | +1.21% |
We have added these results to Section 4.2 expanding Table 4.
W3: Multi-seed
They used a single seed additional runs with different seeds
Thank you for the suggestions. We agree that multi-seed experiments would be helpful to better evaluate the robutness of our method.
Our primary experiments in Table 5 were conducted with Seed 42. Following your suggestion, we repeated the full training of all methods on Qwen3-1.7B with an independent seed (Seed 100). As shown in Table 14, with Seed 100, MTA-OPSD achieves an average score of 56.6%, achieving a +5.5% gain over the base model and outperforming the strongest baseline (PAINT at 53.7%) by +2.9%. Moreover, across both seeds (57.0% on Seed 42 vs. 56.6% on Seed 100), MTA-OPSD demonstrates minimal variance (0.4%), confirming that our performance gains are robust and not artifacts of seed selection. These results shows a strong robustness of our method.
Table 14. Mathematical reasoning performance of seed 100.
| Method | AMC23 | AIME24 | AIME25 | Average |
|---|---|---|---|---|
| Qwen3-1.7B | 74.1 | 45.6 | 33.6 | 51.1 |
| OPSD | 75.0 | 48.1 | 35.3 | 52.8 |
| SDPO | 75.8 | 44.4 | 32.5 | 50.9 |
| Paint | 75.5 | 48.1 | 37.5 | 53.7 |
| MTA-OPSD | 78.2 | 51.9 | 39.7 | 56.6 |
We have added these results to Appendix D.4.
Q1: Pilot Experiment (Token-level vs. Block-level Selection)
Can you compare the best teacher identified in the pilot experiments with the teacher selected by your method?
Thank you for the suggestion. We agree that comparing against a pilot study variant which chooses teachers at every token by our method is critical.
Following your and Reviewer 2h2e's suggestions, we provide a comparison against the pilot study variant which chooses teachers at every token. In our original manuscript's pilot study, the teacher was chosen only at the first token of an intervention block and allowed to generate 64/256 tokens consecutively. Following your suggestion, we implemented a token-level variant under the identical experimental setup: for every token position within the intervention block, we dynamically identify the teacher using the same majority-voting + KL-based selection criterion. As shown in Table 5, in regular token positions, Token-level intervention consistently outperforms block-level intervention, even selecting the nearest teacher at each token matches or slightly exceeds the best block-level teacher. Moreover, totally following MTA-OPSD (selecting the farthest teacher after majority voting) provides further boosted performance, delivering nearly 2 times of the gains over the best block-level results. These results further validate the importance of choosing a teacher separately for every token.
Table 5. Comparison of pilot study.
| Teacher / Model | Qwen3-1.7B | Qwen3-4B |
|---|---|---|
| Block Level (Initial) | ||
| Best | +4.59% | +4.88% |
| Token Level | ||
| Farthest (MTA-OPSD) | +8.91% | +9.38% |
| Nearest | +4.43% | +4.98% |
| Random | +7.34% | +8.06% |
We have incorporated these discussion into Section 5.4 of the revised manuscript.
Gentle Reminder (24h Remaining)
Dear Reviewer Zomr,
We sincerely thank you for your careful evaluations and constructive suggestions. As the end of the discussion period is approaching (only 24 hours left), we are eagerly anticipating your feedback to ascertain if our responses have adequately addressed your concerns. We hope our additional experiments and clarifications address them. Your feedback will be instrumental in enhancing the quality of our manuscript.
Thank you once again for your time and consideration!
Best wishes,
Authors of Submission 49270
Rebuttal Part1
Dear Reviewer Zomr,
We sincerely appreciate your thoughtful feedback and constructive suggestions. Below, we address your concerns in detail and provide the requested multi-seed experiments.
W1: Training Cost
Thank you for the suggestions. We agree that reporting the training overhead would be helpful to better evaluate our method.
First, we are aware of the computation overhead. To mitigate it, following your and Reviewer 2h2e's suggestions, we provide an ablation variant of a small-teacher-pool. Specifically, we evaluated a smaller multi-teacher pool of 4 teacher configurations: {full reference, prefix-0.5, suffix-0.5, random-span-0.5}, against the default 7 configurations, using Qwen3-1.7B as the base model. Interestingly as shown in Table 12, a samller multi-teacher pool achieves an identical average score (57.0%), significantly outperforming OPSD (54.0%) and all single-teacher variants in Table 7 (Table 6 in the initial manuscript). This confirms that the benefits of MTA-OPSD stem from the adaptive selection mechanism over complementary context perspectives, rather than simply expanding the pool size. Moreover, it shows the potential of achieving better efficiency-performance trade-off for MTA-OPSD.
Table 12. Ablation of a smaller teacher pool.
Second, we report the training overhead of the main experiment also the small-teacher-pool ablation, on Qwen3-1.7B. It is shown in Table 13 that end-to-end step time for MTA-OPSD remains favorable. In particular, MTA-OPSD spends more time on the Policy Update stage, because it evaluates forward passes for candidate teachers. However, it remains controlable, especially considering the potential of switching to the small pool reduces the overhead (from 55s/step to 39s/step) while retaining the accuracy gain. Moreover, its lower trainer memory usage arises from dynamic batching in our implementation: the token budget for each micro-batch accounts for all candidate teacher contexts. Consequently, more teachers lead to fewer samples per micro-batch and smaller logits tensors in each forward pass, reducing peak memory usage at the cost of more micro-batches and slower updates. This explains why the default pool uses less memory but takes longer than the smaller pool.
Table 13. Computation Overhead on Qwen3-1.7B.
Note that our current implementation adapts the existing verl [4] batching infrastructure. Further optimization specifically for MTA-OPSD could increase parallelism and use more GPU memory to accelerate teacher evaluation. Thus, the observed memory–speed trade-off is controllable rather than an inherent limitation of the method.
Overall, it shows a favorable efficiency-performance trade-off.
We have added these results to Appendix D.3.