acceptodds
Under review as a conference paper at ICLR 2027

Collective Wisdom: On-Policy Self-Distillation with Synergistic Multi-Perspective Teacher Policy Fusion

Abstract

In On-Policy Self-Distillation (OPSD), the student receives dense token-level supervisory signals from a teacher branch conditioned on privileged information, thereby improving its reasoning ability without relying on a stronger external teacher. However, a single teacher perspective can introduce bias and incomplete coverage in the supervisory signals, causing student updates to rely excessively on local supervisory signals and leading to performance instability or even collapse during the distillation process. Although incorporating multi-perspective teachers can broaden the sources of supervisory signals, conflicts among the supervisory signals provided by different teachers can make inadequate policy fusion susceptible to performance fluctuations during the distillation process. To address these challenges, we propose Collective Wisdom (CW), a synergistic multi-perspective teacher policy fusion method for OPSD. Specifically, we first take the union of the top- vocabulary from each teacher to construct a joint policy space and normalize the individual teacher policies within this space to fuse them. We then realign the shared top- vocabulary based on the fused policy to distill a consensus policy, which is employed to optimize the student policy. This design enables the synergistic fusion and stable distillation of multi-perspective teacher policies. Furthermore, experiments on models of different sizes and architectures across four mathematical reasoning benchmarks show that our method outperforms representative baselines in stabilizing distillation and mitigating performance collapse.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_j6DZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets a practical pain point in on-policy self-distillation (OPSD). the training often jitters or even collapses, whether you use one privileged teacher or naively average several. The proposed fix, Collective Wisdom, builds three teacher views from the same backbone (answer-guided, contrast-with-reference, and outcome-feedback reflection), merges their top-k candidate tokens into a shared union space, renormalizes each teacher there, averages them into a consensus distribution, re-picks the top-k under that consensus, and finally distills the student with clipped forward KL on this aligned vocabulary.

Critical strengths

The collapse phenomenon is documented convincingly (Fig. 1) and the fix is simple, cheap to implement, and well-motivated — unifying the candidate vocabulary before averaging is a genuinely sensible way to keep teacher disagreement from injecting noise.

Critical weaknesses

technically the contribution is thin — it boils down to "union the top-k sets, renormalize, average, re-rank", i.e. uniform-weight probability averaging with a second top-k pass, and no theoretical justification is given for why this specific fusion should stabilize training rather than, say, learned or entropy-aware weighting.

The evaluation is confined to math reasoning with one 30k training set and very small test sets (AIME/HMMT have ~30 items each), so Mean@12 differences of ~1 point are well within noise, and no multi-seed runs or significance tests are reported.

Questions for authors

see the weakness above

3 comments

To Weakness-1, thank you for your valuable suggestions and careful reading of our paper. Each step of "union, normalization, averaging, and top-kk selection" addresses a specific problem in multi-teacher top-kk vocabulary distillation. The vocabulary union and normalization construct a shared policy space for all teachers. Without independent verification of teacher reliability, equal-weight averaging yields a consensus policy, which is the consensus distribution that minimizes the average forward KL over all teachers. Since policy fusion alters the probability ranking of candidate tokens, the consensus vocabulary must be realigned with the fused consensus policy. We thus select the top-kk highest-scoring tokens from the joint policy space, so that tokens jointly supported by multiple teachers, or assigned high probability by at least some teachers, are retained first. A single-teacher top-kk vocabulary, in contrast, is filtered only by that teacher's probability ranking. It tends to concentrate on the high-probability region of a single view and discard potentially valid candidates from other teachers, causing insufficient supervision coverage and unstable distillation.

In addition, under the main experimental settings of Qwen3-8B, Qwen3-4B, Qwen3-1.7B, and Olmo3-7B, we measured two metrics over the top-20 tokens at each position during CW training. The effective support ES(p~l)=1∑v∈Vl,Topkp~l,v2\mathrm{ES}(\tilde{p}_l)=\frac{1}{\sum_{v\in\mathcal{V}_{l,\mathrm{Topk}}}\tilde{p}_{l,v}^{2}} measures the number of tokens that provide effective supervision, with a larger value indicating more effective tokens. The coverage Cov(Vl,Topk)=1N∑j=1N∑v∈Vl,Topkpl,vTj\mathrm{Cov}(\mathcal{V}_{l,\mathrm{Topk}})=\frac{1}{N}\sum_{j=1}^{N}\sum_{v\in\mathcal{V}_{l,\mathrm{Topk}}}p^{T_j}_{l,v} measures the extent to which the distillation vocabulary retains each teacher's supervision; a larger value means broader coverage, while a smaller one means focus on a single teacher. Here, p~l\tilde{p}_l denotes the distillation target at position ll, and Vl,Topk\mathcal{V}_{l,\mathrm{Topk}} is the corresponding top-kk vocabulary. For a single teacher TjT_j, p~l=p~lTj\tilde{p}_l=\tilde{p}^{T_j}_l and Vl,Topk=Vl,TopkTj\mathcal{V}_{l,\mathrm{Topk}}=\mathcal{V}^{T_j}_{l,\mathrm{Topk}}; for CW, p~l=p~lT\tilde{p}_l=\tilde{p}^{T}_l and Vl,Topk=Vl,Topkshared\mathcal{V}_{l,\mathrm{Topk}}=\mathcal{V}^{\mathrm{shared}}_{l,\mathrm{Topk}}. pl,vTjp^{T_j}_{l,v} is the raw probability of token vv under teacher TjT_j over the full vocabulary.

As shown in the table below, CW achieves higher ES and Cov than every single teacher on all four models. This indicates that consensus policy fusion distributes supervision over more effective tokens and better retains each teacher's supervision. Meanwhile, the ES of CW remains below 2.5 out of k = 20, so the target is not diluted across the top-k vocabulary but stays focused on a few plausible candidates. We will add this discussion and the conclusions to the revision.

ModelTeacherESCov
Qwen3-1.7BTeacher-11.4370.972
Teacher-21.5780.961
Teacher-31.4540.973
CW1.6930.982
Qwen3-4BTeacher-12.1640.978
Teacher-22.2120.980
Teacher-32.0770.982
CW2.3160.986
Qwen3-8BTeacher-12.3720.978
Teacher-22.2580.976
Teacher-32.2290.979
CW2.4830.984
Olmo3-7BTeacher-12.1930.981
Teacher-22.1180.984
Teacher-32.0530.991
CW2.2470.992

To Weakness-1, thank you for your valuable suggestions and careful reading of our paper. Under the same settings as the main experiments, we implemented entropy-aware weighting following the idea of the EOPD paper: pˉl,vT=∑j=1NwlTjp^l,vTj,v∈Vl,UnionT\bar{p}^{T}_{l,v}=\sum_{j=1}^{N}w^{T_j}_{l}\hat{p}^{T_j}_{l,v},\quad v\in\mathcal{V}^{T}_{l,\mathrm{Union}}, where the entropy weight is wlTj=exp⁡(−HlTj/τ)∑i=1Nexp⁡(−HlTi/τ)w^{T_j}_{l}=\frac{\exp\left(-H^{T_j}_{l}/\tau\right)}{\sum_{i=1}^{N}\exp\left(-H^{T_i}_{l}/\tau\right)} and the entropy is HlTj=−∑v∈Vl,UnionTp^l,vTjlog⁡p^l,vTjH^{T_j}_{l}=-\sum_{v\in\mathcal{V}^{T}_{l,\mathrm{Union}}}\hat{p}^{T_j}_{l,v}\log\hat{p}^{T_j}_{l,v}. The table below shows that the Overall Var of CW on Qwen3-1.7B, Qwen3-4B, Qwen3-8B, and Olmo3-7B is 0.47, 0.26, 0.09, and 1.16, respectively, compared with 1.44, 2.12, 1.79, and 3.08 for the entropy-weighted variant. Overall, CW trains more stably than entropy-aware weighting. One possible reason is that entropy weighting tends to up-weight low-entropy teachers, which biases the fused result toward a few more confident teachers and weakens the complementary supervision from other views. Equal-weight fusion instead preserves supervision from different teachers and avoids over-reliance on a single low-entropy teacher, resulting in more stable training and competitive average performance. The revised version will include this comparison with entropy-aware weighting and the related analysis.

MethodModelAMC23 MeanAMC23 VarAIME24 MeanAIME24 VarAIME25 MeanAIME25 VarHMMT25 MeanHMMT25 VarALL MeanALL Var
EntropyQwen3-1.7B84.030.8747.351.0134.752.9621.732.8849.821.44
CWQwen3-1.7B84.750.7548.672.1934.322.4423.271.1850.600.47
EntropyQwen3-4B95.930.8973.801.3461.735.0242.653.9670.642.12
CWQwen3-4B96.570.4173.361.2963.460.9341.142.9970.780.26
EntropyQwen3-8B94.720.7573.982.1466.332.6743.123.3771.471.79
CWQwen3-8B95.950.0876.081.0567.530.8644.541.1372.940.09
EntropyOlmo3-7B95.091.9667.313.1162.283.3740.656.4868.543.08
CWOlmo3-7B96.920.3269.942.4865.651.9941.984.0970.801.16

To Weak-2, thank you for your valuable suggestions and for reading our paper in detail.

  • Evaluation granularity. For each method, we save 9 checkpoints at equal intervals during training. Each checkpoint is evaluated on 130 problems: AMC23 (40 problems) and AIME24, AIME25, and HMMT25 (30 problems each). We sample 12 responses per problem, compute Mean@12 accuracy, and weight the results by the number of problems. The Overall Avg in the table aggregates 9×130×12=14,0409 \times 130 \times 12 = 14{,}040 responses, giving 108 samples per problem in total. Thus, a 1-percentage-point shift in Overall Avg requires changes in about 140 responses, and fluctuations in a few random responses have a minimal effect on performance.
  • Training seeds. We again appreciate this suggestion, as it helps further examine the stability of our method and guards against bias from random seeds. We trained CW with three seeds (42/78/123), keeping all other settings identical to the main experiments. As shown in the table below, the standard deviation of CW's Overall Avg across the three seeds ranges from 0.04 to 0.11. This variation is smaller than the performance gaps underlying our conclusions, and the main conclusions hold under all three seeds.
  • Evaluation scope. We acknowledge that our current study focuses on mathematical reasoning and has not yet been extended to other domains, which we will explore in future work. The above discussion and conclusions will be added in the revision.
MethodModelSeed-42 Overall MeanSeed-42 Overall VarSeed-78 Overall MeanSeed-78 Overall VarSeed-123 Overall MeanSeed-123 Overall VarOverall Mean±SD
CWQwen3-1.7B50.600.4750.570.5450.420.3650.53±0.10
CWQwen3-4B70.780.2670.870.2170.660.2970.77±0.11
CWQwen3-8B72.940.0972.790.0773.010.0872.91±0.11
CWOlmo3-7B70.801.1670.721.2470.741.0670.75±0.04

Thank you for taking the time and effort to review our paper and for your valuable suggestions. We hope that our responses and the additional experimental results above have addressed your concerns. We would be grateful if you find our responses satisfactory and could consider updating your score. If you have any remaining concerns, please let us know, and we will do our best to address them during the discussion period.

Reviewer_goxCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses training instability in On-Policy Self-Distillation (OPSD) and proposes Collective Wisdom (CW). CW builds three teachers from the same model conditioned on different privileged information. It first merges the teachers' top-20 candidate tokens, averages the teacher distributions over this set, and then re-selects the top-20 tokens of the fused distribution for distillation. Across four backbone models and four mathematical reasoning benchmarks, CW shows the smallest fluctuation during training, with average accuracy comparable to the strongest baselines.

Critical strengths
  1. The paper describes the construction of the multi-perspective teachers, the fusion over the candidate vocabulary, and the distillation objective in sufficient detail, and the overall pipeline is intuitive.

  2. The experiments cover four models of different sizes and architectures and four mathematical reasoning benchmarks. CW achieves the lowest Overall Var on all four models while maintaining competitive average accuracy, which supports its practical value in improving training stability.

Critical weaknesses
  1. The method is mainly a combination of existing techniques: averaging multiple teacher distributions, top-k distillation, and KL clipping are already used in multi-teacher distillation, OPSD-Topk, and OPSD, respectively, and multiple privileged-information teachers have already been used in MT-OPSD and AVSD. Compared with the most closely related MT-OPSD, CW mainly replaces loss-level averaging with distribution-level fusion and re-selects the candidate vocabulary after fusion, which is a limited change. The contribution is therefore incremental.

  2. CW's advantage in average accuracy over the baselines is small. For example, compared with standard GRPO, its Overall Avg is only 0.1–0.8 percentage points higher.

  3. Thinking mode is disabled during training with a maximum generation length of only 1,024 tokens, but enabled during evaluation. This inconsistency makes it difficult to judge how the abilities learned during training transfer to the evaluation setting.

  4. The progressive comparisons in Section 5.3 do not fully isolate the effects of individual design choices. Moreover, Figure 3 only compares T1, T1+T2, and T1+T2+T3 on Qwen3-8B. These results show the gains from adding teachers in this particular order, but do not establish the standalone contribution of T2 or T3.

  5. The paper attributes training instability to conflicts among multiple teachers, but provides no direct evidence for this. Figure 1 only shows the performance degradation, and Figure 4 only reports the overlap rates of candidate tokens; neither shows a relationship between teacher conflict and performance degradation.

Questions for authors
  1. Are all results from a single training run? Could you report results over multiple random seeds to confirm that the differences of less than one percentage point in Table 1 are reliable?

  2. The prompts of T2 and T3 contain {student_response}. Is this the student’s full response, or the partial response up to the current position? If it is the full response, the teacher can already see the token the student wrote at each position when predicting it. Could this bias the supervision signal toward the student’s own output?

  3. The maximum generation length during training is only 1,024 tokens. Why is this truncation length used, and how many training responses are truncated? Truncated responses often have not yet reached a final answer; can they still provide a useful learning signal for the student?

4 comments

To Weakness-1, thank you for your careful reading of our paper. Each step of "union, normalization, averaging, and top-kk reselection" addresses a specific issue in multi-teacher top-kk vocabulary distillation. Taking the union of the teachers' vocabularies and normalizing over it builds a common policy space for all teachers. Without independent verification of teacher reliability, equal-weight averaging yields the consensus distribution that minimizes the average forward KL divergence from all teachers, i.e., pˉ=1N∑i=1Npi=arg⁡min⁡q1N∑i=1NKL(pi∥q)\bar{p}=\frac{1}{N}\sum_{i=1}^{N}p_i=\arg\min_{q}\frac{1}{N}\sum_{i=1}^{N}\mathrm{KL}(p_i\Vert q). Since policy fusion changes the probability ranking of candidate tokens, the consensus vocabulary must be realigned according to the fused consensus policy. We therefore select the top-kk highest-scoring tokens from the joint policy space. This preferentially retains tokens jointly supported by multiple teachers or assigned high probability by at least some of them. In contrast, a single-teacher top-kk vocabulary is selected solely by one teacher's probability ranking. Such a vocabulary tends to concentrate on the high-probability region of a single view and discards potentially valid candidates from other teachers, which leads to insufficient supervision coverage and unstable distillation.

To Weakness-2, thank you for taking the time to read our paper in detail. We consider GRPO a stronger baseline than OPSD in terms of stability. GRPO directly uses verifiable answer correctness as the reward, which matches the accuracy metric used in evaluation. OPSD-style methods instead give the student token-level supervision by matching the teacher distribution. Regardless of the privileged information the teacher is conditioned on, the student's training objective is to match the teacher distribution, and answer accuracy is not optimized directly. This may be one reason why OPSD-style methods struggle to achieve stable gains over GRPO. As shown in Table 1 of the paper, the seven OPSD-style baselines yield 28 comparisons with GRPO across four models, and only 7 of them show a higher Overall Avg than GRPO. No baseline surpasses GRPO on all four models. The closest, AVSD, ties GRPO on Qwen3-1.7B. In contrast, CW exceeds GRPO on every model with an average gain of 0.58 percentage points, larger than that of AVSD, the strongest of the other methods. Moreover, CW attains the lowest Overall Var of all methods on each of the four models. Compared with GRPO, it reduces Overall Var from 1.00 to 0.47 on Qwen3-1.7B, from 0.44 to 0.26 on Qwen3-4B, from 0.79 to 0.09 on Qwen3-8B, and from 1.77 to 1.16 on Olmo3-7B. These results further confirm that our method effectively improves distillation stability.

To Weakness-3, thank you for taking the time to read our paper in detail and for your valuable comments. On capability transfer, compared with the initial models with thinking mode enabled, CW improves the Overall Avg across 9 checkpoints from 70.08, 69.61, and 48.49 to 72.94, 70.78, and 50.60 on Qwen3-8B, Qwen3-4B, and Qwen3-1.7B, respectively. We also added experiments on CW with thinking mode disabled, where the Overall Avg across 9 checkpoints rises from 39.77, 33.87, and 26.51 to 41.92, 35.33, and 27.69 on the same three models. These results confirm that the capability gains hold in both modes.

To Weakness-4, thank you for this valuable suggestion. It led us to further analyze how the number and type of teachers affect performance. The number of teachers is a hyperparameter used mainly to assess the overall gain and training stability of multi-teacher supervision relative to single-teacher supervision. As more teachers are added, the consensus from multi-source supervision makes training more stable, while the performance gain gradually plateaus. To examine the effect of teacher type, we ran single-teacher distillation with each teacher (T1, T2, T3) under the main experimental configuration. As shown in the table below, the best-performing teacher changes from step to step with no consistent pattern. This indicates that distillation stability under a single teacher is insensitive to teacher type: instability occurs regardless of which teacher is used. We attribute this to single-teacher distillation fixing the student's reasoning path, which limits the coverage of supervision signals. Student updates then rely too heavily on local supervision, and this mechanism has little to do with teacher type.

MethodModelStep-0Step-1Step-2Step-3Step-4Step-5Step-6Step-7Step-8Step-9
Teacher-1Qwen3-8B70.0874.1072.4370.5169.2967.0567.2465.4566.1566.03
Teacher-2Qwen3-8B70.0874.4473.0170.3269.8768.3367.3165.7765.7164.94
Teacher-3Qwen3-8B70.0874.3772.5671.7968.8567.7667.6966.2265.3265.77

To Weakness-5, thank you for the valuable suggestion. We think that the differing overlap rates reflect inconsistent distillation vocabularies across teachers, as different teachers assign probability to different tokens. This inconsistency in turn misaligns the optimization objectives of multiple teachers. For example, directly fusing the distillation objectives of individual teachers, as in MT-OPSD, leads to unstable training. To further verify the policy differences among teachers, we computed the JS divergence during training in the main experiments to measure how teacher distributions differ on the same token: JSl(I)=H!(1∣I∣∑i∈Ipl(i))−1∣I∣∑i∈IH!(pl(i))\mathrm{JS}_l(\mathcal{I})=H\\!\left(\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}p^{(i)}_{l}\right)-\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}H\\!\left(p^{(i)}_{l}\right), where the entropy is H!(pl(i))=−∑v∈Vl,UnionTpl,v(i)log⁡2pl,v(i)H\\!\left(p^{(i)}_{l}\right)=-\sum_{v\in\mathcal{V}^{T}_{l,\mathrm{Union}}}p^{(i)}_{l,v}\log_2 p^{(i)}_{l,v} and I\mathcal{I} denotes the set of compared policies, with I=1,2,3\mathcal{I}=\\{1,2,3\\} or I=1,2,3,CW\mathcal{I}=\\{1,2,3,\mathrm{CW}\\}. The JS divergence among T1, T2, and T3 is 0.3047, indicating clear differences between teacher policies conditioned on different privileged information. In contrast, the JS divergence among T1, T2, T3, and CW is 0.2381, so the consensus policy of CW lies close to the center of the teacher policies. These results verify that our policy fusion method incorporates the consensus supervision signals from all teachers, thereby improving the stability of multi-teacher distillation.

To Question-1, thank you for the valuable suggestion. Training with multiple random seeds further tests our method's stability and guards against seed-induced bias. We ran experiments with three random seeds (42/78/123), keeping all other parameters identical to the main experimental configuration. As shown in the table below, the standard deviation of CW's Overall Avg across the three seeds is 0.04–0.11. This fluctuation is smaller than the performance gaps on which the paper's conclusions rely, and the main conclusions hold under all three seeds.

MethodModelSeed-42 Overall MeanSeed-42 Overall VarSeed-78 Overall MeanSeed-78 Overall VarSeed-123 Overall MeanSeed-123 Overall VarOverall Mean±SD
CWQwen3-1.7B50.600.4750.570.5450.420.3650.53±0.10
CWQwen3-4B70.780.2670.870.2170.660.2970.77±0.11
CWQwen3-8B72.940.0972.790.0773.010.0872.91±0.11
CWOlmo3-7B70.801.1670.721.2470.741.0670.75±0.04

To Question-2, thank you for taking the time to read our paper in detail and for your valuable comments. Regarding student responses and truncation, {student_response} refers to the complete student response. If a response is truncated without reaching a final answer, {eval_result} marks it as incorrect, since the model fails to produce the correct answer within the budget. We acknowledge that the complete response can bias the supervision signal toward the student's own outputs, and SDPO also reports that this lowers the entropy of the teacher model. However, the T2 and T3 teachers receive additional privileged information and are explicitly instructed to perform contrastive reflection rather than simply repeat the student's response. We measured the teacher entropy in two runs, both conditioned on the complete student response, one without and one with the additional privileged information, obtaining 0.2767 and 0.2882, respectively. This entropy increase suggests that our design partially mitigates the above issue.

To Question-3, thank you for taking the time to read our paper in detail and for your valuable comments. For the maximum generation length, we follow the OPSD configuration and keep all baselines consistent with it. OPSD compares student generation lengths of 1,024 and 4,096 on Qwen3-1.7B and finds that longer generation does not bring consistent improvements, which it attributes to early tokens being more critical for learning. Our hyperparameter experiments support the same conclusion. On Qwen3-8B, 2.78% of responses were truncated during training with a generation length of 1,024, compared with 1.36% at 4,096. A fourfold increase in generation length thus lowered the truncation rate by 1.42 percentage points. The experiments also showed a small performance gain between the two lengths, whereas training at 4,096 took 3 to 4 times as long as at 1,024. Considering both performance and efficiency, we set the maximum generation length to 1,024. Truncated responses still provide effective learning signals, since their generated prefixes contain the early tokens that matter most for learning. The {eval_result} marks these responses as incorrect, allowing the teacher to perform contrastive reflection on them as failed attempts.

Thank you for taking the time and effort to review our paper and for your valuable suggestions. We hope that our responses and the additional experimental results above have addressed your concerns. We would be grateful if you find our responses satisfactory and could consider updating your score. If you have any remaining concerns, please let us know, and we will do our best to address them during the discussion period.

Thank you for the detailed responses and additional experiments. I appreciate the multi-seed results, the evaluations under both thinking modes, and the clarification of truncation rates, which help address several of my original concerns. Regarding Weaknesses 1 and 4, the response clarifies the rationale for the fusion procedure, but the additional single-teacher experiments do not further isolate the contributions of distribution-level fusion and vocabulary reselection. My assessment of the methodological contribution as incremental remains largely unchanged. Regarding Weakness 5, the JS divergence analysis supports the existence of differences among teacher distributions, but the connection between these differences and training instability remains insufficiently established. In particular, the lower JS divergence after including CW in the compared policy set does not by itself demonstrate that harmful teacher conflicts have been mitigated. Having also considered the other reviewers’ comments, I believe my initial assessment remains appropriate and therefore maintain my original rating.

Reviewer_d5sZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Collective Wisdom (CW) to address training instability in OPSD caused by limited supervision coverage from a single teacher and conflicting supervision among multiple teachers. Specifically, CW constructs a joint policy space by taking the union of the top-k vocabularies from three teachers conditioned on different privileged information, and generates a consensus policy by normalizing and averaging their policies. It then reselects the top-k vocabulary based on the consensus policy and optimizes the student using forward KL divergence.

Critical strengths

S1. Performance degradation during prolonged OPSD training on mathematical reasoning tasks is a known issue (https://github.com/siyan-zhao/OPSD/issues/20). Addressing this problem is important for making OPSD a more reliable post-training method.

S2. The experiments cover models of different sizes and architectures, and evaluate training stability through performance variations across checkpoints.

Critical weaknesses

W1. The paper argues that averaging teacher-specific KL losses in existing MT-OPSD fails to effectively integrate conflicting supervision, and proposes averaging teacher policies before distillation to address this issue. However, excluding clipping and assuming the same vocabulary space, the two approaches yield mathematically identical gradients. Specifically,

LMT=1N∑j=1NDKL(pTj,∥,pS)=DKL(pˉT,∥,pS)+C,pˉT=1N∑j=1NpTj,\mathcal L_{\mathrm{MT}} = \frac{1}{N}\sum_{j=1}^{N} D_{\mathrm{KL}}(p_{T_j} \\,\Vert\\, p_S) = D_{\mathrm{KL}}(\bar p_T \\,\Vert\\, p_S) + C, \qquad \bar p_T = \frac{1}{N}\sum_{j=1}^{N} p_{T_j},

where CC is independent of the student parameters. Therefore,

∇θLMT=∇θDKL(pˉT,∥,pS).\nabla_\theta \mathcal L_{\mathrm{MT}} = \nabla_\theta D_{\mathrm{KL}}(\bar p_T \\,\Vert\\, p_S).

Thus, policy averaging itself does not provide a different optimization signal from conventional loss averaging. Consequently, without vocabulary selection and normalization, the proposed method is essentially equivalent to existing MT-OPSD, suggesting that the observed performance improvements may stem from these additional design choices rather than teacher policy fusion itself. This makes the independent contribution of policy fusion emphasized in the paper unclear.

W2. The paper's main claim is that single-teacher OPSD suffers from limited supervision coverage due to its bias toward a specific reasoning path, leading to training instability. To address this issue, the paper introduces multiple teachers. However, it is not demonstrated whether the resulting consensus policy pˉl,vT\bar p_{l,v}^{T} (Eq. 5) actually provides broader supervision coverage. Comparisons of entropy, probability concentration, or effective support size are missing, and the vocabulary overlap analysis in Figure 4 is insufficient to substantiate this claim.

Questions for authors

Please see the weaknesses section.

10 comments

To Weakness-1, thank you for the rigorous derivation, and sorry for the confusion.

We agree that averaging teacher losses and averaging teacher policies yield mathematically identical gradients under full-vocabulary distillation without truncation. The formulas in PRELIMINARIES are written over the full vocabulary. However, policy fusion in our method, as well as CW and MT-OPSD in our experiments, all operate on Top-kk vocabularies, as configured in Appendix A.2 EXPERIMENTAL SETUP. We will revise the formulas in PRELIMINARIES in the revised version.

Without the full vocabulary, each teacher selects its own Top-kk vocabulary for updates. MT-OPSD first distills over each teacher's Top-kk vocabulary and then weights the results, so the overall distillation vocabulary is heterogeneous. In contrast, CW first aligns the policy spaces of the Top-kk vocabularies selected by different teachers. It then fuses the policies, updates the policy space, and finally optimizes the model. As a result, the two methods produce different gradients owing to their different choices of distillation vocabulary.

We further provide the corresponding theoretical analysis and conclusion. Under full-vocabulary distillation, CW and MT-OPSD share the same gradient, consistent with the reviewer's conclusion. Under Top-kk distillation, their gradients coincide only when all teachers have exactly the same Top-kk vocabulary. All our CW and MT-OPSD experiments use Top-kk vocabularies, and we observed that this condition fails in the vast majority of cases.

Theoretical Analysis:

The Top-kk distillation loss of MT-OPSD at position ll is ℓlMT=1N∑j=1N∑v∈Vl,TopkTjp~l,vTj(log⁡p~l,vTj−log⁡p~l,vSj),p~l,vTj=pl,vTj∑u∈Vl,TopkTjpl,uTj,p~l,vSj=pl,vS∑u∈Vl,TopkTjpl,uS\ell^{\mathrm{MT}}_l=\frac{1}{N}\sum_{j=1}^{N}\sum_{v\in\mathcal{V}^{T_j}_{l,\mathrm{Topk}}}\tilde{p}^{T_j}_{l,v}\Big(\log\tilde{p}^{T_j}_{l,v}-\log\tilde{p}^{Sj}_{l,v}\Big),\quad \tilde{p}^{T_j}_{l,v}=\frac{p^{T_j}_{l,v}}{\sum_{u\in\mathcal{V}^{T_j}_{l,\mathrm{Topk}}}p^{T_j}_{l,u}},\quad \tilde{p}^{Sj}_{l,v}=\frac{p^{S}_{l,v}}{\sum_{u\in\mathcal{V}^{T_j}_{l,\mathrm{Topk}}}p^{S}_{l,u}}

The Top-kk distillation loss of CW at position ll is ℓlCW=∑v∈Vl,Topksharedp~l,vT(log⁡p~l,vT−log⁡p~l,vS),p~l,vT=pˉl,vT∑u∈Vl,Topksharedpˉl,uT,p~l,vS=pl,vS∑u∈Vl,Topksharedpl,uS\ell^{\mathrm{CW}}_l=\sum_{v\in\mathcal{V}^{\mathrm{shared}}_{l,\mathrm{Topk}}}\tilde{p}^{T}_{l,v}\Big(\log\tilde{p}^{T}_{l,v}-\log\tilde{p}^{S}_{l,v}\Big),\quad \tilde{p}^{T}_{l,v}=\frac{\bar{p}^{T}_{l,v}}{\sum_{u\in\mathcal{V}^{\mathrm{shared}}_{l,\mathrm{Topk}}}\bar{p}^{T}_{l,u}},\quad \tilde{p}^{S}_{l,v}=\frac{p^{S}_{l,v}}{\sum_{u\in\mathcal{V}^{\mathrm{shared}}_{l,\mathrm{Topk}}}p^{S}_{l,u}}

Lemma. For a target distribution qq over a vocabulary subset A\mathcal{A} and the student's normalized policy p~AS\tilde{p}^{S}_{\mathcal{A}} on A\mathcal{A}, we have ∇θ∑v∈Aqv(log⁡qv−log⁡p~A,vS)=−∑v∈A(qv−p~A,vS)∇θlog⁡pl,vS\nabla_\theta\sum_{v\in\mathcal{A}}q_v\big(\log q_v-\log\tilde{p}^{S}_{\mathcal{A},v}\big)=-\sum_{v\in\mathcal{A}}\big(q_v-\tilde{p}^{S}_{\mathcal{A},v}\big)\nabla_\theta\log p^{S}_{l,v}

Proposition 1 (Different vocabularies lead to different gradients). By the Lemma, the gradients of both methods take the form −∑vcl,v∇θlog⁡pl,vS-\sum_v c_{l,v}\nabla_\theta\log p^{S}_{l,v}, with coefficients cl,vMT=1N∑j=1N1[v∈Vl,TopkTj](p~l,vTj−p~l,vSj),cl,vCW=1[v∈Vl,Topkshared](p~l,vT−p~l,vS)c^{\mathrm{MT}}_{l,v}=\frac{1}{N}\sum_{j=1}^{N}\mathbb{1}\big[v\in\mathcal{V}^{T_j}_{l,\mathrm{Topk}}\big]\Big(\tilde{p}^{T_j}_{l,v}-\tilde{p}^{Sj}_{l,v}\Big),\quad c^{\mathrm{CW}}_{l,v}=\mathbb{1}\big[v\in\mathcal{V}^{\mathrm{shared}}_{l,\mathrm{Topk}}\big]\Big(\tilde{p}^{T}_{l,v}-\tilde{p}^{S}_{l,v}\Big)

  • Full vocabulary: Vl,TopkTj=Vl,Topkshared=V\mathcal{V}^{T_j}_{l,\mathrm{Topk}}=\mathcal{V}^{\mathrm{shared}}_{l,\mathrm{Topk}}=\mathcal{V}, so local normalization has no effect (p~l,vT=pˉl,vT=1N∑jpl,vTj\tilde{p}^{T}_{l,v}=\bar{p}^{T}_{l,v}=\frac{1}{N}\sum_{j}p^{T_j}_{l,v}). Therefore, cl,vMT=cl,vCW=1N∑jpl,vTj−pl,vSc^{\mathrm{MT}}_{l,v}=c^{\mathrm{CW}}_{l,v}=\frac{1}{N}\sum_{j}p^{T_j}_{l,v}-p^{S}_{l,v}, and the two gradients are identical.

  • Top-kk vocabulary: If the teachers' Top-kk vocabularies are not exactly identical, then ∣Vl,UnionT∣>k|\mathcal{V}^{T}_{l,\mathrm{Union}}|>k, and the set Vl,UnionT∖Vl,Topkshared\mathcal{V}^{T}_{l,\mathrm{Union}}\setminus\mathcal{V}^{\mathrm{shared}}_{l,\mathrm{Topk}} is non-empty. For any vv in this set, cl,vCW=0c^{\mathrm{CW}}_{l,v}=0. In contrast, cl,vMTc^{\mathrm{MT}}_{l,v} contains the student term p~l,vSj\tilde{p}^{Sj}_{l,v}, which varies with the student policy and is not identically zero. In other words, MT-OPSD still updates tokens supported by only some teachers, whereas CW does not. The two gradients coincide only when all teachers' Top-kk vocabularies are exactly the same.

To Weakness-2, thank you for the valuable suggestion and for reading our paper so carefully.

Analyzing these metrics helps support and validate our conclusions. Using the main experimental configurations of Qwen3-8B, Qwen3-4B, Qwen3-1.7B, and Olmo3-7B, we computed two metrics over the top-20 tokens at each position during CW training. The effective support (ES) measures the number of tokens that provide effective supervision signals: ES(p~l)=1∑v∈Vl,Topkp~l,v2\mathrm{ES}(\tilde{p}_l)=\frac{1}{\sum_{v\in\mathcal{V}_{l,\mathrm{Topk}}}\tilde{p}_{l,v}^{2}}, Larger values mean more effective tokens, and smaller values mean fewer. The coverage (Cov) quantifies how much of each teacher's supervision the distillation vocabulary retains: Cov(Vl,Topk)=1N∑j=1N∑v∈Vl,Topkpl,vTj\mathrm{Cov}(\mathcal{V}_{l,\mathrm{Topk}})=\frac{1}{N}\sum_{j=1}^{N}\sum_{v\in\mathcal{V}_{l,\mathrm{Topk}}}p^{T_j}_{l,v}, A larger value reflects broader coverage, whereas a smaller one suggests a focus on a single teacher. Here, p~l\tilde{p}_l denotes the distillation target at position ll, and Vl,Topk\mathcal{V}_{l,\mathrm{Topk}} is the corresponding top-kk vocabulary. For a single teacher TjT_j, p~l=p~lTj\tilde{p}_l=\tilde{p}^{T_j}_l and Vl,Topk=Vl,TopkTj\mathcal{V}_{l,\mathrm{Topk}}=\mathcal{V}^{T_j}_{l,\mathrm{Topk}}; for CW, p~l=p~lT\tilde{p}_l=\tilde{p}^{T}_l and Vl,Topk=Vl,Topkshared\mathcal{V}_{l,\mathrm{Topk}}=\mathcal{V}^{\mathrm{shared}}_{l,\mathrm{Topk}}. The term pl,vTjp^{T_j}_{l,v} denotes the original probability of teacher TjT_j over the full vocabulary.

As shown in the table below, CW achieves higher ES and Cov than every single teacher on all four models. This indicates that consensus policy fusion distributes supervision over more effective tokens and better retains each teacher's supervision. Meanwhile, the ES of CW remains below 2.5 out of k = 20, so the target is not diluted across the top-k vocabulary but stays focused on a few plausible candidates. We will include this discussion and conclusion in the revision.

ModelTeacherESCov
Qwen3-1.7BTeacher-11.4370.972
Teacher-21.5780.961
Teacher-31.4540.973
CW1.6930.982
Qwen3-4BTeacher-12.1640.978
Teacher-22.2120.980
Teacher-32.0770.982
CW2.3160.986
Qwen3-8BTeacher-12.3720.978
Teacher-22.2580.976
Teacher-32.2290.979
CW2.4830.984
Olmo3-7BTeacher-12.1930.981
Teacher-22.1180.984
Teacher-32.0530.991
CW2.2470.992

Thank you for the response.

W1. Based on the authors' explanation, the main contribution appears to lie in consensus-based top-k vocabulary selection rather than policy averaging itself. This distinction should be made clear in the paper.

However, the theoretical analysis only establishes that CW and MT-OPSD produce different gradients. It does not explain why the proposed vocabulary selection mitigates conflicting supervision or provides better supervision than MT-OPSD. Thus, my main concern regarding the justification of the proposed method remains unresolved. I would consider raising my score if the authors provide theoretical or empirical evidence addressing this concern.

W2. I suggest incorporating the additional experimental results into the paper to better substantiate the claim that CW improves supervision coverage.

Comments on the paper

1 comment
Area_Chair_poCwOpenReview ↗

Please start the discussion on the rebuttals

Dear Reviewers,

Thanks a lot for your help reviewing this submission! We have received the rebuttals of the authors, so I would like to hear everyone's opinion on the authors' rebuttal to help me form my final meta-review and recommendation for this paper.

Can I ask everyone to share a little bit about their post-rebuttal feelings about the paper by replying to this message thread, and update their original review/rating of the paper accordingly at your earliest convenience? Thanks a lot in advance!

Best, your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.