acceptodds
Under review as a conference paper at ICLR 2027

Every Token Needs Its Own Teacher: Multi-Teacher Adaptive On-Policy Self-Distillation

Abstract

On-policy Self-Distillation (OPSD) has proven effective for post-training large language models by constructing a teacher conditioned on privileged references. However, existing approaches keep the teacher fixed either throughout training by consistently exposing it to the full reference, or within each response by adaptively varying partial reference exposure only across responses. In this paper, we argue that neither of them is sufficient. Instead, every token needs its own teacher, as the teacher preference can vary token by token within the same response. Through a teacher-intervention pilot study, we validate that the teacher preference differs across tokens, and reveal that a preferred teacher can be identified for each token through majority voting and KL-based selection. Motivated by these findings, we propose Multi-teacher Adaptive On-policy Self-Distillation (MTA-OPSD), the first token-level adaptive OPSD method. Our method constructs a multi-teacher pool consisting of candidate teachers, and enables every token to adaptively select its preferred teacher during training. Experiments with Qwen3 models on mathematical reasoning tasks show that MTA-OPSD consistently outperforms existing OPSD approaches with an average gain of up to over the base model, highlighting the importance of adapting teacher supervision at the token level.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Self-distillation, Large Language Models, Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_ZomrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

They work on the on-policy self-distillation problem, where we have questions and a reference solution for each of them. The student generates an answer, and a teacher, which uses the same underlying model but is conditioned on the reference solution, provides distributions for computing the token-level distillation loss. They argue that it is better to have multiple teachers, conditioned on different versions of the reference solution, and select a teacher for each token. Their teacher pool includes both the full reference and truncated versions. Through intervention experiments, they show that the teacher producing the largest accuracy gain can differ across token positions. They then propose a rule for choosing which teacher to use for each token. For a given token generated by the student in a rollout, they check whether each teacher assigns it a higher or lower probability than the student does. This divides the teachers into two groups, and they select the group with more teachers. From that group, they select the teacher with the highest student-to-teacher KL divergence and use its distribution to guide the student.

Their results show improvements over OPSD, SDPO, and PAINT in average performance across three math datasets.

Critical strengths
  1. The model performance is very good.
  2. The pilot experiments that aim to justify the design choices are interesting.
Critical weaknesses
  1. Having multiple teachers increases training time significantly. Wall-clock time should be part of the comparisons of different methods in the tables.
  2. The experiments comparing the farthest teacher with the closest are interesting, but it would make more sense to evaluate the full range of teachers: the closest teacher, the second closest, etc. (table 4)
  3. They used a single seed for training. Their gains are large, but without additional runs with different seeds, it is hard to be certain about the robustness of the results across training runs.
Questions for authors
  1. Can you compare the best teacher identified in the pilot experiments with the teacher selected by your method? For example, report the accuracy of your teacher selection against that oracle, as well as the average gain from your teacher selection compared with the best and random teacher selections.
5 comments
AuthorsOpenReview ↗

Rebuttal Part1

Dear Reviewer Zomr,

We sincerely appreciate your thoughtful feedback and constructive suggestions. Below, we address your concerns in detail and provide the requested multi-seed experiments.

W1: Training Cost

Having multiple teachers increases training time significantly ⋯\cdots

Thank you for the suggestions. We agree that reporting the training overhead would be helpful to better evaluate our method.

First, we are aware of the computation overhead. To mitigate it, following your and Reviewer 2h2e's suggestions, we provide an ablation variant of a small-teacher-pool. Specifically, we evaluated a smaller multi-teacher pool of 4 teacher configurations: {full reference, prefix-0.5, suffix-0.5, random-span-0.5}, against the default 7 configurations, using Qwen3-1.7B as the base model. Interestingly as shown in Table 12, a samller multi-teacher pool achieves an identical average score (57.0%), significantly outperforming OPSD (54.0%) and all single-teacher variants in Table 7 (Table 6 in the initial manuscript). This confirms that the benefits of MTA-OPSD stem from the adaptive selection mechanism over complementary context perspectives, rather than simply expanding the pool size. Moreover, it shows the potential of achieving better efficiency-performance trade-off for MTA-OPSD.

Table 12. Ablation of a smaller teacher pool.

MethodAMC23AIME24AIME25Average
Qwen3-1.7B74.145.633.651.1
OPSD75.349.237.554.0
MTA-OPSD
 Default78.551.441.157.0
 Small Pool77.752.840.657.0

Second, we report the training overhead of the main experiment also the small-teacher-pool ablation, on Qwen3-1.7B. It is shown in Table 13 that end-to-end step time for MTA-OPSD remains favorable. In particular, MTA-OPSD spends more time on the Policy Update stage, because it evaluates forward passes for candidate teachers. However, it remains controlable, especially considering the potential of switching to the small pool reduces the overhead (from 55s/step to 39s/step) while retaining the accuracy gain. Moreover, its lower trainer memory usage arises from dynamic batching in our implementation: the token budget for each micro-batch accounts for all candidate teacher contexts. Consequently, more teachers lead to fewer samples per micro-batch and smaller logits tensors in each forward pass, reducing peak memory usage at the cost of more micro-batches and slower updates. This explains why the default pool uses less memory but takes longer than the smaller pool.

Table 13. Computation Overhead on Qwen3-1.7B.

MethodvLLM (GB)Trainer (GB)Total Memory (GB)Generation (s/step)Policy Update (s/step)Total Time (s/step)
OPSD5750107212455
SDPO5753110181947
PAINT5767124212354
MTA-OPSD
 Default572683215589
 Small Pool573693213971

Note that our current implementation adapts the existing verl [4] batching infrastructure. Further optimization specifically for MTA-OPSD could increase parallelism and use more GPU memory to accelerate teacher evaluation. Thus, the observed memory–speed trade-off is controllable rather than an inherent limitation of the method.

Overall, it shows a favorable efficiency-performance trade-off.

We have added these results to Appendix D.3.

AuthorsOpenReview ↗

Rebuttal Part2

W2: More Detailed Table 4

⋯\cdots are interesting, but it would make more sense to evaluate the full range of teachers: the closest teacher, the second closest, etc. (table 4)

Thank you for the suggestions. We agree that reporting more details would be helpful for validating our pilot study.

Following your suggestion, we evaluated the full rank spectrum within the majority subset: the closest, the second closest, the farthest, and the second farthest teachers, given that there are 7 candidate teachers in total. As shown in Table 4, intervention performance exhibits a monotonic trend: as the forward-KL divergence increases from Nearest to Farthest, the intervention gain steadily rises (e.g., from +0.59% to +2.42% on 1.7B, and from +1.21% to +2.78% on 4B). This strongly confirms that within the majority-consensus subset, the teacher-student forward-KL distance serves as a reliable, continuous indicator of supervisory informativeness, directly validating our maximum-KL selection rule.

Table 4. Average gains from the farthest and nearest teachers.

Teacher / ModelQwen3-1.7BQwen3-4B
Farthest+2.42%+2.78%
Second Farthest+1.48%+2.39%
Second Nearest+0.57%+1.56%
Nearest+0.59%+1.21%

We have added these results to Section 4.2 expanding Table 4.

W3: Multi-seed

They used a single seed ⋯\cdots additional runs with different seeds ⋯\cdots

Thank you for the suggestions. We agree that multi-seed experiments would be helpful to better evaluate the robutness of our method.

Our primary experiments in Table 5 were conducted with Seed 42. Following your suggestion, we repeated the full training of all methods on Qwen3-1.7B with an independent seed (Seed 100). As shown in Table 14, with Seed 100, MTA-OPSD achieves an average score of 56.6%, achieving a +5.5% gain over the base model and outperforming the strongest baseline (PAINT at 53.7%) by +2.9%. Moreover, across both seeds (57.0% on Seed 42 vs. 56.6% on Seed 100), MTA-OPSD demonstrates minimal variance (0.4%), confirming that our performance gains are robust and not artifacts of seed selection. These results shows a strong robustness of our method.

Table 14. Mathematical reasoning performance of seed 100.

MethodAMC23AIME24AIME25Average
Qwen3-1.7B74.145.633.651.1
OPSD75.048.135.352.8
SDPO75.844.432.550.9
Paint75.548.137.553.7
MTA-OPSD78.251.939.756.6

We have added these results to Appendix D.4.

Q1: Pilot Experiment (Token-level vs. Block-level Selection)

Can you compare the best teacher identified in the pilot experiments with the teacher selected by your method?

Thank you for the suggestion. We agree that comparing against a pilot study variant which chooses teachers at every token by our method is critical.

Following your and Reviewer 2h2e's suggestions, we provide a comparison against the pilot study variant which chooses teachers at every token. In our original manuscript's pilot study, the teacher was chosen only at the first token of an intervention block and allowed to generate 64/256 tokens consecutively. Following your suggestion, we implemented a token-level variant under the identical experimental setup: for every token position within the intervention block, we dynamically identify the teacher using the same majority-voting + KL-based selection criterion. As shown in Table 5, in regular token positions, Token-level intervention consistently outperforms block-level intervention, even selecting the nearest teacher at each token matches or slightly exceeds the best block-level teacher. Moreover, totally following MTA-OPSD (selecting the farthest teacher after majority voting) provides further boosted performance, delivering nearly 2 times of the gains over the best block-level results. These results further validate the importance of choosing a teacher separately for every token.

Table 5. Comparison of pilot study.

Teacher / ModelQwen3-1.7BQwen3-4B
Block Level (Initial)
 Best+4.59%+4.88%
Token Level
 Farthest (MTA-OPSD)+8.91%+9.38%
 Nearest+4.43%+4.98%
 Random+7.34%+8.06%

We have incorporated these discussion into Section 5.4 of the revised manuscript.

AuthorsOpenReview ↗

Gentle Reminder (24h Remaining)

Dear Reviewer Zomr,

We sincerely thank you for your careful evaluations and constructive suggestions. As the end of the discussion period is approaching (only 24 hours left), we are eagerly anticipating your feedback to ascertain if our responses have adequately addressed your concerns. We hope our additional experiments and clarifications address them. Your feedback will be instrumental in enhancing the quality of our manuscript.

Thank you once again for your time and consideration!

Best wishes,

Authors of Submission 49270

Reviewer_usxjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Multi-Teacher Adaptive On-Policy Self-Distillation (MTA-OPSD), which adaptively selects a teacher for each token from a pool of teachers conditioned on different portions of the reference solution. The selection is performed through majority voting followed by choosing the teacher with the largest student-teacher KL divergence. The authors motivate this design using a teacher-intervention study and demonstrate improvements over several OPSD baselines on mathematical reasoning benchmarks.

While the experimental results are promising, I have reservations about the central motivation and the practical significance of the proposed method. In particular, the current experiments do not convincingly establish that token-level teacher adaptation is necessary, and the benefits relative to the additional computational requirements remain unclear.

Critical strengths

The method achieves improvements over existing OPSD baselines across three model scales, with particularly noticeable gains on the 1.7B model. The authors also provide useful ablations comparing individual teachers (Table 6), alternative ways of combining multiple teachers (Table 7), and different components of the selection mechanism (Table 8).

These experiments strengthen the empirical contribution. In particular, the improvements over random teacher selection and mean-loss aggregation suggest that the proposed selection mechanism captures useful information beyond simply having access to multiple teachers. I consider this the strongest argument in favor of acceptance.

Critical weaknesses
  1. Insufficient evidence for the necessity of token-level teacher adaptation. The central claim that different tokens require different teachers is not convincingly established. The pilot study measures teacher preference through relatively long teacher-generated continuations (64–256 tokens), which may not reflect the suitability of teachers for individual token-level supervision. More importantly, the experiments lack a direct comparison with response-level adaptive teacher selection using the same teacher pool. Therefore, it remains unclear whether the gains genuinely come from token-level adaptation or from a more effective multi-teacher selection mechanism.

  2. Unclear cost-effectiveness of the proposed method. The method requires evaluating seven reference-conditioned teachers to select supervision for each token, potentially introducing substantial computational overhead. However, the paper does not report compute-normalized comparisons, and the improvements over strong baselines are sometimes modest (e.g., 57.0 vs. 56.2 for the best single teacher in Table 6, and 79.2 vs. 78.9 for Paint on Qwen3-8B). Without accounting for the additional training cost, it is difficult to assess whether such fine-grained teacher selection is practically worthwhile.

Questions for authors

How reliably does the proposed majority-voting and KL-based selection identify the teacher with the largest intervention gain at each token position? Are these preferred-teacher assignments stable across independent intervention samples?

5 comments
AuthorsOpenReview ↗

Rebuttal Part1

Dear Reviewer usxj,

We sincerely appreciate your thoughtful feedback and constructive suggestions. Below, we address your concerns in detail and provide the requested additional baseline.

W1&Q1: Necessity of Token-level Teacher Adaptation

The pilot study ⋯\cdots relatively long teacher-generated continuations (64–256 tokens), which may not reflect the suitability of teachers for individual token-level supervision. More importantly ⋯\cdots direct comparison with response-level adaptive teacher selection using the same teacher pool ⋯\cdots How reliably does the proposed majority-voting and KL-based selection identify the teacher with the largest intervention gain at each token position?

Thank you for the suggestions. We agree that addition experiments would provide more sufficient evidence for the motivation of adaptive token-level teacher.

First, following your suggestions, implemented a response-level adaptive baseline using the exact same multi-teacher pool and selection criterion as MTA-OPSD. The only difference is that the teacher selection criterion is calculated as the response average. As shwon in Table 15, the response-level adaptive baseline yields only modest improvements over the base model (52.2% vs. 51.1%), whereas MTA-OPSD reaches 57.0% (+4.8% over the response-level baseline). This confirms that the superior performance of MTA-OPSD does not arise from having access to a multi-teacher pool, but critically depends on the token-level teacher adaptation.

Table 15. Comparison with the response-level adaptive teacher selection baseline.

MethodAMC23AIME24AIME25Average
Qwen3-1.7B74.145.633.651.1
Response-level Baseline75.547.233.952.2
MTA-OPSD78.551.441.157.0

Second, we are aware of the need for a pilot study for individual token-level supervision. Following your and Reviewer 2h2e's suggestions, we provide a comparison against the pilot study variant which chooses teachers at every token.

In our original manuscript's pilot study, the teacher was chosen only at the first token of an intervention block and allowed to generate 64/256 tokens consecutively. Following your suggestion, we implemented a token-level variant under the identical experimental setup: for every token position within the intervention block, we dynamically identify the teacher using the same majority-voting + KL-based selection criterion. As shown in Table 5, in regular token positions, Token-level intervention consistently outperforms block-level intervention, even selecting the nearest teacher at each token matches or slightly exceeds the best block-level teacher. Moreover, totally following MTA-OPSD (selecting the farthest teacher after majority voting) provides further boosted performance, delivering nearly 2 times of the gains over the best block-level results. These results further validate the importance of choosing a teacher separately for every token.

Table 5. Comparison of pilot study.

Teacher / ModelQwen3-1.7BQwen3-4B
Block Level (Initial)
 Best+4.59%+4.88%
Token Level
 Farthest (MTA-OPSD)+8.91%+9.38%
 Nearest+4.43%+4.98%
 Random+7.34%+8.06%

We have incorporated these into Section 5.4, Section 6.3, and Section D.5 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal Part2

W2: Cost-effectiveness

⋯\cdots potentially introducing substantial computational overhead ⋯\cdots

Thank you for the suggestions. We agree that reporting the training overhead would be helpful to better evaluate our method.

First, we are aware of the computation overhead. To mitigate it, following your and Reviewer 2h2e's suggestions, we provide an ablation variant of a small-teacher-pool. Specifically, we evaluated a smaller multi-teacher pool of 4 teacher configurations: {full reference, prefix-0.5, suffix-0.5, random-span-0.5}, against the default 7 configurations, using Qwen3-1.7B as the base model. Interestingly as shown in Table 12, a samller multi-teacher pool achieves an identical average score (57.0%), significantly outperforming OPSD (54.0%) and all single-teacher variants in Table 7 (Table 6 in the initial manuscript). This confirms that the benefits of MTA-OPSD stem from the adaptive selection mechanism over complementary context perspectives, rather than simply expanding the pool size. Moreover, it shows the potential of achieving better efficiency-performance trade-off for MTA-OPSD.

Table 12. Ablation of a smaller teacher pool.

MethodAMC23AIME24AIME25Average
Qwen3-1.7B74.145.633.651.1
OPSD75.349.237.554.0
MTA-OPSD
 Default78.551.441.157.0
 Small Pool77.752.840.657.0

Second, we report the training overhead of the main experiment also the small-teacher-pool ablation, on Qwen3-1.7B. It is shown in Table 13 that end-to-end step time for MTA-OPSD remains favorable. In particular, MTA-OPSD spends more time on the Policy Update stage, because it evaluates forward passes for candidate teachers. However, it remains controlable, especially considering the potential of switching to the small pool reduces the overhead (from 55s/step to 39s/step) while retaining the accuracy gain. Moreover, its lower trainer memory usage arises from dynamic batching in our implementation: the token budget for each micro-batch accounts for all candidate teacher contexts. Consequently, more teachers lead to fewer samples per micro-batch and smaller logits tensors in each forward pass, reducing peak memory usage at the cost of more micro-batches and slower updates. This explains why the default pool uses less memory but takes longer than the smaller pool.

Table 13. Computation Overhead on Qwen3-1.7B.

MethodvLLM (GB)Trainer (GB)Total Memory (GB)Generation (s/step)Policy Update (s/step)Total Time (s/step)
OPSD5750107212455
SDPO5753110181947
PAINT5767124212354
MTA-OPSD
 Default572683215589
 Small Pool573693213971

Note that our current implementation adapts the existing verl [4] batching infrastructure. Further optimization specifically for MTA-OPSD could increase parallelism and use more GPU memory to accelerate teacher evaluation. Thus, the observed memory–speed trade-off is controllable rather than an inherent limitation of the method.

Overall, it shows a favorable efficiency-performance trade-off.

We have added these results to Appendix D.3.

AuthorsOpenReview ↗

Gentle Reminder (24h Remaining)

Dear Reviewer usxj,

We sincerely thank you for your careful evaluations and constructive suggestions. As the end of the discussion period is approaching (only 24 hours left), we are eagerly anticipating your feedback to ascertain if our responses have adequately addressed your concerns. We hope our additional experiments and clarifications address them. Your feedback will be instrumental in enhancing the quality of our manuscript.

Thank you once again for your time and consideration!

Best wishes,

Authors of Submission 49270

Reviewer_Yj6yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Multi-Teacher Adaptive On-Policy Self-Distillation (MTA-OPSD), addressing the limitation that existing adaptive self-distillation methods assign a single teacher to an entire generated response. A teacher-intervention pilot study motivates the design. Experiments on Qwen3 models of 1.7B, 4B, and 8B parameters demonstrate improvements over existing OPSD baselines on mathematical reasoning benchmarks.

Critical strengths
  1. Simple and practical method. The combination of majority voting and KL-based selection is straightforward and does not require training an additional teacher-selection network.

  2. Encouraging experimental results. MTA-OPSD achieves average scores of 57.0, 77.2, and 79.2 on Qwen3-1.7B, 4B, and 8B, respectively, outperforming the compared OPSD methods. The ablations also show that the selection mechanism performs better than random teacher selection and averaging teacher losses.

Critical weaknesses
  1. The selection criterion lacks a strong justification. The paper assumes that, after majority filtering, a larger teacher–student KL divergence indicates a better teacher. However, a larger KL can also reflect unreliable or overly confident supervision. The intervention results establish a correlation in the tested settings, but do not demonstrate that maximizing KL is generally the right selection rule.

  2. The contribution of KL-based selection is less convincing than claimed. In Table 8, majority voting followed by random selection achieves 56.6, compared with 57.0 for the proposed method. This relatively small difference suggests that majority voting may account for most of the improvement.

  3. Potentially substantial computational overhead. The method requires evaluating multiple teacher distributions for token-level selection. Although all teachers share model parameters, their conditioning contexts differ. The paper would benefit from reporting actual training time, memory consumption, and computational cost relative to the baselines.

  4. Limited evaluation scope. All main experiments use Qwen3 models and mathematical reasoning tasks. It remains unclear whether the approach generalizes to other model families, knowledge-intensive tasks, coding, or instruction following.

Questions for authors

please refer to the weakness part.

9 comments
AuthorsOpenReview ↗

Rebuttal Part1

Dear Reviewer Yj6y,

We sincerely appreciate your thoughtful feedback and constructive suggestions. Below, we address your concerns in detail and provide the requested experiments on additional models and benchmarks.

W1: Selection Criterion

The selection criterion lacks a strong justification ⋯\cdots after majority filtering, a larger teacher–student KL divergence indicates a better teacher. However, a larger KL can also reflect unreliable or overly confident supervision. The intervention results establish a correlation in the tested settings, but do not demonstrate that maximizing KL is generally the right selection rule.

We thank the reviewer for raising this critical point, and we welcome the opportunity to clarify the theoretical motivation behind our two-stage selection rule.

We would like to emphasize that, MTA-OPSD does not naively select the teacher with the maximum KL divergence in isolation. Instead, it is first to go through majority voting to obtain the reliable teacher subset, and then find the teacher with the largest KL within this subset. We would like to recall Table 9 (Table 8 in the initial manuscript), to show the difference between these 2 selection criterions. As shown below, maximizing KL without majority voting achieves only modest gains over the base model (52.9% vs. 51.1%), confirming your observation that a large KL alone can capture unreliability. However, pairing majority voting with KL selection boosts performance to 57.0% (+4.1% over unconstrained KL), empirically demonstrating that majority voting successfully eliminates misleading teachers.

Table 9. Ablation of majority voting and KL-based teacher selection.

Majority VotingKL SelectionAMC23AIME24AIME25Average
Base Modelbase Model74.145.633.651.1
NoFarthest75.848.933.952.9
YesFarthest78.551.441.157.0

We would also like to provide further discussion of the selection criterion. The mechanism of our method can be understood as decoupling the reliability of the supervision direction from the informativeness of the supervision signal. Majority voting first determines whether the teacher pool collectively supports or opposes the student-generated token yiy^i, KL-based selection then identifies the most informative teacher aligned with that direction. This ordering is crucial: KL distance is not used as a standalone measure of teacher quality, but only after directional consensus has filtered out potentially unreliable teachers.

Specifically, when VM(ci)=1V_M(c_i)=1, the majority teachers in Mmaj(ci)M_{\mathrm{maj}}(c_i) assign a higher probability to yiy^i than the student itself, and selecting the teacher with the largest KL therefore reinforces the student's current choice. Meanwhile, the selected teacher can also provide a substantially different overall probability distribution. Consequently, the supervision can strengthen the sampled token without being limited to simply imitating it, thereby retaining informative probabilities for other plausible predictions and potentially reducing overfitting to the student's current trajectory.

When VM(ci)=−1V_M(c_i)=-1, the majority teachers assign a lower probability to yiy^i, and the sampled token is collectively regarded as a less preferred choice. In this case, a teacher with a small KL distance may only weakly downweight this token or remain too close to the student distribution. In contrast, the farthest teacher is more likely to redistribute probability mass toward different candidate tokens, providing a stronger corrective signal together with more distinct alternative prediction directions. Such supervision may therefore help the student move away from a potentially erroneous local choice rather than merely making a marginal adjustment to it.

We have integrated these in Section 5.5, in our updated manuscript.

AuthorsOpenReview ↗

Rebuttal Part2

W2: The Contribution of KL-based Selection

The contribution of KL-based selection is less convincing than claimed. In Table 9 (Table 8 in the initial manuscript), majority voting followed by random selection achieves 56.6, compared with 57.0 for the proposed method ⋯\cdots

Thank you for this careful reading. We agree that, in Table 9 (Table 8 in the initial manuscript), majority voting accounts for the larger share of the improvement. However, we respectfully disagree with the conclusion that the choice of teacher within the majority subset is of minor importance. The two steps are designed to work together: majority voting determines a reliable supervision direction, and KL-based selection determines the most informative teacher along that direction. We clarify this below using results already in the paper.

1. A joint view of Tables 8 and 9

The Random-pick strategy in Table 8 (Table 7 in the initial manuscript) samples a teacher uniformly from the whole pool, which is exactly the "no majority voting + random selection" variant. Combining it with Table 9 (Table 8 in the initial manuscript) gives the following grid (Qwen3-1.7B, average over AMC23/AIME24/AIME25):

Selection within the candidate setw/o Majority Votingw/ Majority Voting
Random53.656.6
Nearest (smallest KL)–53.3
Farthest (largest KL, MTA-OPSD)52.957.0

This view leads to three observations.

  • The choice within the majority subset matters substantially. After majority voting, performance ranges from 53.3 (nearest) to 57.0 (farthest), a spread of 3.7 points. This is comparable to the gain from majority voting itself (+3.0, from 53.6 to 56.6, the best results in Table 8). If the within-subset choice were unimportant, the nearest teacher would perform close to the others. Instead, it loses almost all of the benefit of majority voting. The random variant comparing favorably with farthest indicates that a randomly drawn majority teacher often carries a sufficiently strong signal in our pool. KL-based selection makes this outcome deterministic: it always picks the most informative teacher and never falls into the low-signal regime represented by the nearest teacher.
  • The two steps interact rather than add. Without majority voting, choosing the farthest teacher is worse than random selection (52.9 vs. 53.6). Without a directional filter, the teacher with the largest divergence is often one whose supervision direction disagrees with the rest of the pool. With majority voting, the same criterion gives the best result (57.0). Majority voting is therefore what makes a large divergence a sign of informativeness rather than unreliability. The interaction term, (57.0−56.6)−(52.9−53.6)=+1.1(57.0-56.6)-(52.9-53.6)=+1.1, is larger than either marginal difference of KL selection alone.
  • The training results agree with the pilot study. In Table 4, within the majority subset, the farthest teacher yields a larger intervention gain than the nearest teacher (+1.83% for Qwen3-1.7B, +1.57% for Qwen3-4B). The gap between farthest and nearest in Table 9 (Table 8 in the initial manuscript) is the training-level counterpart of this token-level finding.

2. Why the farthest teacher is the principled choice As noted in our response to Reviewer 2h2e, Eqs. (3), (15), and (16) mistakenly write the reverse KL. In our implementation, both the teacher ranking and the distillation loss use the forward KL DKL(pT ∥ pθ)D_{\mathrm{KL}}(p_T\,\|\,p_\theta). The selection criterion is therefore the same quantity the student minimizes. Since the gradient of the forward KL with respect to the student logits is pθ−pTp_\theta - p_T, the farthest teacher in the majority subset provides the largest corrective signal in the agreed direction, and the nearest teacher provides the smallest. This explains the ordering nearest < random < farthest in Table 9 (Table 8 in the initial manuscript). KL-based selection is thus not an independent heuristic added on top of majority voting. It is the loss-consistent way to choose among teachers that majority voting has already deemed reliable, and it adds negligible cost because all teacher distributions are already computed for majority voting.

3. On the size of the gap We acknowledge that the difference between random and farthest selection after majority voting is small (+0.4), but we do not claim this difference alone as strong evidence. We believe the gap is small in this setting partly because all seven teachers are derived from the same model and differ only in how the reference is truncated. Teachers in the majority subset are therefore fairly similar, and a random choice is often close to the farthest one. The main evidence for KL-based selection is the full picture above: removing it in favor of the nearest teacher collapses performance, and using it without majority voting hurts. Only the combination of the two steps achieves the best result.

AuthorsOpenReview ↗

Rebuttal Part3

W3: Training Cost

Potentially substantial computational overhead ⋯\cdots The paper would benefit from reporting actual training time, memory consumption, and computational cost relative to the baselines.

Thank you for the suggestions. We agree that reporting the training overhead is helpful to better evaluate our method.

First, we are aware of the computation overhead. To mitigate it, following your and Reviewer 2h2e's suggestions, we provide an ablation variant of a small-teacher-pool. Specifically, we evaluated a smaller multi-teacher pool of 4 teacher configurations: {full reference, prefix-0.5, suffix-0.5, random-span-0.5}, against the default 7 configurations, using Qwen3-1.7B as the base model. Interestingly as shown in Table 12, a samller multi-teacher pool achieves an identical average score (57.0%), significantly outperforming OPSD (54.0%) and all single-teacher variants in Table 7 (Table 6 in the initial manuscript). This confirms that the benefits of MTA-OPSD stem from the adaptive selection mechanism over complementary context perspectives, rather than simply expanding the pool size. Moreover, it shows the potential of achieving better efficiency-performance trade-off for MTA-OPSD.

Table 12. Ablation of a smaller teacher pool.

MethodAMC23AIME24AIME25Average
Qwen3-1.7B74.145.633.651.1
OPSD75.349.237.554.0
MTA-OPSD
 Default78.551.441.157.0
 Small Pool77.752.840.657.0

Second, we report the training overhead of the main experiment also the small-teacher-pool ablation, on Qwen3-1.7B. It is shown in Table 13 that end-to-end step time for MTA-OPSD remains favorable. In particular, MTA-OPSD spends more time on the Policy Update stage, because it evaluates forward passes for candidate teachers. However, it remains controlable, especially considering the potential of switching to the small pool reduces the overhead (from 55s/step to 39s/step) while retaining the accuracy gain. Moreover, its lower trainer memory usage arises from dynamic batching in our implementation: the token budget for each micro-batch accounts for all candidate teacher contexts. Consequently, more teachers lead to fewer samples per micro-batch and smaller logits tensors in each forward pass, reducing peak memory usage at the cost of more micro-batches and slower updates. This explains why the default pool uses less memory but takes longer than the smaller pool.

Table 13. Computation Overhead on Qwen3-1.7B.

MethodvLLM (GB)Trainer (GB)Total Memory (GB)Generation (s/step)Policy Update (s/step)Total Time (s/step)
OPSD5750107212455
SDPO5753110181947
PAINT5767124212354
MTA-OPSD
 Default572683215589
 Small Pool573693213971

Note that our current implementation adapts the existing verl [4] batching infrastructure. Further optimization specifically for MTA-OPSD could increase parallelism and use more GPU memory to accelerate teacher evaluation. Thus, the observed memory–speed trade-off is controllable rather than an inherent limitation of the method.

Overall, it shows a favorable efficiency-performance trade-off.

We have added these results to Appendix D.3.

Reviewer_2h2eOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MTA-OPSD creates seven teacher distributions from the same model by varying the reference solution shown to it, then selects a teacher for each student token through majority voting followed by KL-based selection. A teacher-intervention study motivates this design, and experiments on Qwen3-1.7B, 4B, and 8B across AMC23, AIME24, and AIME25 report higher average accuracy than OPSD, SDPO, and Paint, with ablations testing individual teachers and different selection strategies.

Critical strengths

The pilot study keeps the student prefix fixed and changes which teacher briefly takes over generation. This provides concrete evidence that the most helpful teacher can depend on where the student is in its response.

The ablations include individual teachers, random teacher selection, and averaging teacher losses. The proposed selection rule reaches 57.0, compared with 53.6 for random selection and 53.7 for averaging, supporting the value of choosing teachers carefully.

The results cover three model sizes, with consistent improvements on AIME25. Gains over the strongest baseline on that benchmark are 2.5, 2.0, and 2.0 points for the 1.7B, 4B, and 8B models.

Critical weaknesses

The pilot lets a teacher generate 64–256 tokens at once, and the ablations do not compare against choosing one teacher per response or per block from the same pool. That comparison would show whether choosing a teacher separately for every token is necessary.

Equation (16) uses reverse KL, while the Full (OPSD) baseline in Table 6 is described in §6.1 as using forward KL with clipping. A single-teacher comparison using MTA-OPSD’s exact loss would help separate the benefit of teacher selection from the effect of changing the loss.

Table 6 shows 57.0 for MTA-OPSD versus 56.2 for the best single teacher, but MTA-OPSD evaluates seven reference variants. Training time and peak memory, including results with a smaller teacher pool, would help readers judge whether the extra 0.8 points justify the cost.

Questions for authors

With the same teacher pool and loss, how does choosing once per response or per 64-token block compare with choosing at every token?

Do the single-teacher runs in Table 6 use the same loss and clipping settings as MTA-OPSD?

Can a smaller teacher pool retain most of the improvement, and how does it affect runtime and memory?

The three Paint scores for 8B in Table 5 average approximately 78.1, but the reported average is 78.9; which entry needs correcting?

Since teachers are ranked by reverse KL, the choice of distillation loss could affect which teacher is preferred. Connecting this design to broader KD work on loss design, probability-mass allocation, and hard/soft supervision [1–2] would help explain when the selection rule should work; a conceptual discussion is enough.

[1] ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence. ICML, 2025.

[2] DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. ICML, 2025.

9 comments
AuthorsOpenReview ↗

Rebuttal Part1

Dear Reviewer 2h2e,

We sincerely appreciate your thoughtful feedback and constructive suggestions. Below, we address your concerns in detail and provide the requested pilot experiments.

W1&Q1: Pilot Experiment (Token-level vs. Block-level Selection)

The pilot lets a teacher generate 64–256 tokens at once ⋯\cdots With the same teacher pool and loss, how does choosing once per response or per 64-token block compare with choosing at every token? That comparison would show whether choosing a teacher separately for every token is necessary.

Thank you for the suggestion. We agree that comparing against a pilot study variant which chooses different teachers at every token is critical.

In our original manuscript's pilot study, the teacher was chosen only at the first token of an intervention block and allowed to generate 64/256 tokens consecutively. Following your suggestion, we implemented a token-level variant under the identical experimental setup: for every token position within the intervention block, we dynamically identify the teacher using the same majority-voting + KL-based selection criterion. As shown in Table 5, in regular token positions, Token-level intervention consistently outperforms block-level intervention, even selecting the nearest teacher at each token matches or slightly exceeds the best block-level teacher. Moreover, totally following MTA-OPSD (selecting the farthest teacher after majority voting) provides further boosted performance, delivering nearly 2 times of the gains over the best block-level results. These results further validate the importance of choosing a teacher separately for every token.

Table 5. Comparison of pilot study.

Teacher / ModelQwen3-1.7BQwen3-4B
Block Level (Initial)
 Best+4.59%+4.88%
Token Level
 Farthest (MTA-OPSD)+8.91%+9.38%
 Nearest+4.43%+4.98%
 Random+7.34%+8.06%

We have incorporated these into Section 5.4 of the revised manuscript.

W2&Q2: Experiment Setting

Equation (16) uses reverse KL, while the Full (OPSD) baseline in Table 6 is described in §6.1 as using forward KL with clipping ⋯\cdots Do the single-teacher runs in Table 6 use the same loss and clipping settings as MTA-OPSD?

We thank the reviewer for identifying this notation error. MTA-OPSD strictly uses forward KL with clipping, identical to the OPSD setup. This can be verified in our supplementary source code at Line 42, configs/math_deepmath/qwen3_1_7b_opsd2.yaml, and Line 2713-2722, third_party/verl/verl/trainer/ppo/core_algos.py. The use of reverse KL notation in Eqs. (3), (15), and (16) of the initial manuscript was an unfortunate typographical error. We have corrected these equations to forward KL in the revision.

All single-teacher ablations reported in Table use the exact same forward KL loss with clipping as MTA-OPSD and OPSD. The performance improvements of MTA-OPSD therefore stem purely from token-level adaptive teacher selection, rather than any loss discrepancy.

AuthorsOpenReview ↗

Rebuttal Part2

W3&Q3: Training Cost

⋯\cdots but MTA-OPSD evaluates seven reference variants. Training time and peak memory, including results with a smaller teacher pool ⋯\cdots Can a smaller teacher pool retain most of the improvement, and how does it affect runtime and memory?

Thank you for the suggestions. We agree that reporting the training overhead and exploring a smaller teacher pool are helpful to better evaluate our method.

To test whether MTA-OPSD can maintain its gains with fewer teachers, we evaluated a smaller multi-teacher pool of 4 teacher configurations: {full reference, prefix-0.5, suffix-0.5, random-span-0.5}, against the default 7 configurations, using Qwen3-1.7B as the base model. Interestingly as shown in Table 12, a samller multi-teacher pool achieves an identical average score (57.0%), significantly outperforming OPSD (54.0%) and all single-teacher variants in Table 7 (Table 6 in the initial manuscript). This confirms that the benefits of MTA-OPSD stem from the adaptive selection mechanism over complementary context perspectives, rather than simply expanding the pool size. Moreover, it shows the potential of achieving better efficiency-performance trade-off for MTA-OPSD.

Table 12. Ablation of a smaller teacher pool.

MethodAMC23AIME24AIME25Average
Qwen3-1.7B74.145.633.651.1
OPSD75.349.237.554.0
MTA-OPSD
 Default78.551.441.157.0
 Small Pool77.752.840.657.0

Second, we report the training overhead of the main experiment also the small-teacher-pool ablation, on Qwen3-1.7B. It is shown in Table 13 that end-to-end step time for MTA-OPSD remains favorable. In particular, MTA-OPSD spends more time on the Policy Update stage, because it evaluates forward passes for candidate teachers. However, it remains controlable, especially considering the potential of switching to the small pool reduces the overhead (from 55s/step to 39s/step) while retaining the accuracy gain. Moreover, its lower trainer memory usage arises from dynamic batching in our implementation: the token budget for each micro-batch accounts for all candidate teacher contexts. Consequently, more teachers lead to fewer samples per micro-batch and smaller logits tensors in each forward pass, reducing peak memory usage at the cost of more micro-batches and slower updates. This explains why the default pool uses less memory but takes longer than the smaller pool.

Table 13. Computation Overhead on Qwen3-1.7B.

MethodvLLM (GB)Trainer (GB)Total Memory (GB)Generation (s/step)Policy Update (s/step)Total Time (s/step)
OPSD5750107212455
SDPO5753110181947
PAINT5767124212354
MTA-OPSD
 Default572683215589
 Small Pool573693213971

Note that our current implementation adapts the existing verl [4] batching infrastructure. Further optimization specifically for MTA-OPSD could increase parallelism and use more GPU memory to accelerate teacher evaluation. Thus, the observed memory–speed trade-off is controllable rather than an inherent limitation of the method.

Overall, it shows a favorable efficiency-performance trade-off.

We have added these results to Appendix D.3.

Q4: Statistics

The three Paint scores for 8B in Table 6 (Table 5 in the initial manuscript) average approximately 78.1, but the reported average is 78.9; which entry needs correcting?

Thank you for catching this arithmetic oversight. The correct average score for PAINT on Qwen3-8B is indeed 78.1. We have corrected this entry in Table 6 (Table 5 in the initial manuscript) of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal Part3

Q5: Further Discusssions

Since teachers are ranked by reverse KL, the choice of distillation loss could affect which teacher is preferred. Connecting this design to broader KD work on loss design, probability-mass allocation, and hard/soft supervision [1–2] would help explain when the selection rule should work; a conceptual discussion is enough.

Thank you for this insightful question. First, a correction: Eqs. (3), (15), and (16) of the manuscript mistakenly write the reverse KL. In our implementation, both the teacher ranking and the distillation loss use the forward KL, consistent with OPSD. The correction makes the answer more direct: in MTA-OPSD, the divergence used to rank teachers is the same divergence the student optimizes. Below, we explain why this matters and connect it to ABKD [1] and DistiLLM-2 [2].

The selection rule is loss-consistent by design

With the correction, Eq. (15) is

D_{\mathrm{KL}}\left( p_{T_k}(\cdot \mid c_i) \,\|\, p_\theta(\cdot \mid c_i) \right).$$ which is exactly the per-token loss in Eq. (16). Among teachers that agree on the supervision direction, MTA-OPSD therefore picks the one that yields the largest learning signal under the loss actually being minimized. For the forward KL, the gradient with respect to the student logits $z$ is $\nabla_z D_{\mathrm{KL}}(p_T\|p_\theta)=p_\theta-p_T$, so a larger divergence corresponds to more probability mass to be moved. Seen this way, we agree with you that the preferred teacher depends on the loss, and we regard this as intended. The general form of our rule is $$p^*_T(c_i)=\arg\max_{p_{T_k}\in M_{\mathrm{maj}}(c_i)} \mathcal{D}_{\mathrm{loss}}\big(p_{T_k},p_\theta\big),$$ and forward KL is the instance we use. If the loss changes, the ranking criterion should change with it. **Connection to probability-mass allocation (ABKD)** ABKD [1] shows that, within the $\alpha$-$\beta$ divergence family, forward KL ($\alpha{=}1,\beta{=}0$) and reverse KL ($\alpha{=}0,\beta{=}1$) are two extremes of mass allocation. Forward KL has a weak hardness-concentration effect, because mismatches enter linearly, and no confidence-concentration effect, because all tokens are weighted equally. This makes it mass-covering and stable, but its supervision tends to be diffuse. Reverse KL has strong versions of both effects, which can over-concentrate the student on a single mode. ABKD further argues that for large vocabularies, a stronger hardness-concentration (smaller $\alpha$) helps. MTA-OPSD can be read through the same lens, with one difference. ABKD keeps the teacher fixed and reshapes the divergence; MTA-OPSD keeps the divergence fixed (forward KL) and reshapes the target, token by token. Selecting the forward-KL-farthest teacher in the direction-consistent subset concentrates supervision on targets with the largest teacher-student mismatch, which adds a hardness-concentration effect at the level of target selection. Because the loss itself remains forward KL, it introduces no confidence-concentration, so the student is not pushed toward the mode collapse associated with reverse KL. The two approaches are orthogonal, and an $\alpha$-$\beta$ loss could be combined with our selection by ranking teachers with the same $(\alpha,\beta)$-divergence. We present this as a conceptual analogy rather than a formal equivalence.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response & Gentle Reminder (36h Remaining)

We sincerely thank all reviewers for your careful evaluations and constructive suggestions. Based on these suggestions, we conducted extensive additional experiments and made revisions to our manuscript. Below, we summarize the experiments and revisions made in response to the feedback.

  1. Table 4 (Section 4.2, Page 6): we updated the pilot study results with a full range of teachers, in response to Reviewer Zomr's W2.

  2. Table 5 (Section 5.4, Page 8): we added a comparison between the token-level teacher intervention strictly following our method, and the initial block-level intervention, in response to Reviewer 2h2e's W1&Q1, Reviewer usxj's W1&Q1, and Reviewer Zomr's Q1.

  3. Table 15 (Section 6.3, Page 10, and Section D.5, Page 17): aside from Paint, we added a response-level adaptive teacher selection baseline, in response to Reviewer usxj's W1.

  4. Table 10 (Section D.1, Page 16): we extended our method to a new logical reasoning task, in response to Reviewer Yj6y's W4.

  5. Table 11 (Section D.2, Page 16): we extended our method to a new model family, Olmo3-7B, in response to Reviewer Yj6y's W4.

  6. Table 12 (Section D.3, Page 16 and 17): we attempted a smaller-teacher-pool variant of our method, to mitigate the computation overhead, in response to Reviewer 2h2e's W3&Q3, Reviewer Yj6y's W3, Reviewer usxj's W2, and Reviewer Zomr's W1.

  7. Table 13 (Section D.3, Page 17): we reported the training overhead of the main experiment also the small-teacher-pool ablation, in response to Reviewer 2h2e's W3&Q3, Reviewer Yj6y's W3, Reviewer usxj's W2, and Reviewer Zomr's W1.

  8. Table 14 (Section D.4, Page 17): we conducted multi-seed experiments, in response to Reviewer Zomr's W3.

  9. Section 5.5 (Page 8): we deeply analyzed the mechanism of our method, in response to Reviewer Yj6y's Q1.

Newly added text and tables are marked in yellow in the updated manuscript. Meanwhile, we also provided Table 16 in our rebuttal only, in response to Reviewer 2h2e's Q5.

As the end of the discussion period is approaching (only 36 hours left), we are eagerly anticipating your feedback to ascertain if our responses have adequately addressed your concerns. Your feedback will be instrumental in enhancing the quality of our manuscript.

We sincerely thank the reviewers and the Area Chair for their time and valuable feedback.

AuthorsOpenReview ↗

(Submission 49270] Polite request to remind reviewers (24h left in discussion)

Dear Area Chair,

Thank you very much for your time and efforts in handling our submission. We are writing to respectfully request your assistance in reminding all four Reviewers to review our rebuttal, as the end of the discussion period is approaching (only 24 hours left). We have not yet received any responses, and we are eager to confirm whether our responses have adequately addressed their concerns before the discussion phase ends.

Thank you once again for your support and dedicated efforts!

Best wishes,

Authors of Submission 49270

AuthorsOpenReview ↗

(Urgent, 2h Left) Positive Rebuttal Update & AC Assistance Request for Submission 49270

Dear Area Chair,

Thank you for your continued support. As only 2 hours remain in the discussion period, we would like to share an encouraging rebuttal update and respectfully seek your kind assistance.

We are pleased to report that Reviewer 2h2e was satisfied with our clarifications and has positively updated their rating to 3. Unfortunately, the other three reviewers have not yet responded to our rebuttal.

If feasible, we would respectfully request your assistance in giving the remaining reviewers (Yj6y, usxj, and Zomr) a final reminder. We are looking forward to their feedback. We also hope our thorough responses and the positive consensus reached with Reviewer 2h2e will be reflected in the final evaluation.

Thank you once again for your dedication and support!

Best wishes,

Authors of Submission 49270

AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

We would like to provide a concise summary. By the end of 18.11 AOE, our paper gets scores of 3222.

Core Strengths Recognized by Reviewers

The pilot study is interesting and provides concrete evidence for our method motivation. By Reviewers 2h2e and Zomr.

Our method is simple but practical. By Reviewer Yj6y.

The ablations are extensive, useful, and solid. By Reviewer 2h2e and usxj.

The main experiment results are encouraging, coving several model sizes and benchmarks, strengthening the empirical contribution. By All Reviewer.

Address all Concerns

Most of the reviewers' concerns are in common, and we address them all. We have provided a detailed summarization of the additional experiments and revisions in our comment "General Response & Gentle Reminder (36h Remaining)".

Reviewer 2h2e is satisfied with our rebuttal, "addressed my concerns to a considerable extent.", and he has raised his score from 2 to 3.

Reviewer Yj6y did not respond to our rebuttal until 20 mins before the deadline, which unfortunately prevents us from further discussion. Still, his concerns regarding "W1: Selection Criterion" and "W4: Additional Models and Benchmarks" are addressed. Regarding his remaining concerns of "limited benefit of KL-based selection" and efficiency, we have provided solid theoretical and empirical evidence in our rebuttal to him, in our response "Rebuttal Part2" and "Rebuttal Part3" (e.g., Table 8, 9, 12, 13)

Reviewer usxj and Zomr unfortunately have not responsed to our rebuttal. However, we believe we have addressed their concerns, with extensive additional experiments in our rebuttal to them (e.g., Table15, 5, 12, 13, 4, 14). Also, given that most of their concerns are in common with Reviewer 2h2e's and Reviewer 2h2e is satisfied with our rebuttal, we believe our rebuttal also provides sufficient response to them.

We kindly ask the AC to ask Reviewer usxj and Zomr to consider our rebuttal in the next phase, with new evidence from our additional experiments. Thank you once again for your support and dedicated efforts!

Thank you very much!

Best,

Authors of Submission 49270

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.