- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets a practical pain point in on-policy self-distillation (OPSD). the training often jitters or even collapses, whether you use one privileged teacher or naively average several. The proposed fix, Collective Wisdom, builds three teacher views from the same backbone (answer-guided, contrast-with-reference, and outcome-feedback reflection), merges their top-k candidate tokens into a shared union space, renormalizes each teacher there, averages them into a consensus distribution, re-picks the top-k under that consensus, and finally distills the student with clipped forward KL on this aligned vocabulary.
Critical strengths
The collapse phenomenon is documented convincingly (Fig. 1) and the fix is simple, cheap to implement, and well-motivated — unifying the candidate vocabulary before averaging is a genuinely sensible way to keep teacher disagreement from injecting noise.
Critical weaknesses
technically the contribution is thin — it boils down to "union the top-k sets, renormalize, average, re-rank", i.e. uniform-weight probability averaging with a second top-k pass, and no theoretical justification is given for why this specific fusion should stabilize training rather than, say, learned or entropy-aware weighting.
The evaluation is confined to math reasoning with one 30k training set and very small test sets (AIME/HMMT have ~30 items each), so Mean@12 differences of ~1 point are well within noise, and no multi-seed runs or significance tests are reported.
Questions for authors
see the weakness above
3 comments
To Weakness-1, thank you for your valuable suggestions and careful reading of our paper. Under the same settings as the main experiments, we implemented entropy-aware weighting following the idea of the EOPD paper: , where the entropy weight is and the entropy is . The table below shows that the Overall Var of CW on Qwen3-1.7B, Qwen3-4B, Qwen3-8B, and Olmo3-7B is 0.47, 0.26, 0.09, and 1.16, respectively, compared with 1.44, 2.12, 1.79, and 3.08 for the entropy-weighted variant. Overall, CW trains more stably than entropy-aware weighting. One possible reason is that entropy weighting tends to up-weight low-entropy teachers, which biases the fused result toward a few more confident teachers and weakens the complementary supervision from other views. Equal-weight fusion instead preserves supervision from different teachers and avoids over-reliance on a single low-entropy teacher, resulting in more stable training and competitive average performance. The revised version will include this comparison with entropy-aware weighting and the related analysis.
| Method | Model | AMC23 Mean | AMC23 Var | AIME24 Mean | AIME24 Var | AIME25 Mean | AIME25 Var | HMMT25 Mean | HMMT25 Var | ALL Mean | ALL Var |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Entropy | Qwen3-1.7B | 84.03 | 0.87 | 47.35 | 1.01 | 34.75 | 2.96 | 21.73 | 2.88 | 49.82 | 1.44 |
| CW | Qwen3-1.7B | 84.75 | 0.75 | 48.67 | 2.19 | 34.32 | 2.44 | 23.27 | 1.18 | 50.60 | 0.47 |
| Entropy | Qwen3-4B | 95.93 | 0.89 | 73.80 | 1.34 | 61.73 | 5.02 | 42.65 | 3.96 | 70.64 | 2.12 |
| CW | Qwen3-4B | 96.57 | 0.41 | 73.36 | 1.29 | 63.46 | 0.93 | 41.14 | 2.99 | 70.78 | 0.26 |
| Entropy | Qwen3-8B | 94.72 | 0.75 | 73.98 | 2.14 | 66.33 | 2.67 | 43.12 | 3.37 | 71.47 | 1.79 |
| CW | Qwen3-8B | 95.95 | 0.08 | 76.08 | 1.05 | 67.53 | 0.86 | 44.54 | 1.13 | 72.94 | 0.09 |
| Entropy | Olmo3-7B | 95.09 | 1.96 | 67.31 | 3.11 | 62.28 | 3.37 | 40.65 | 6.48 | 68.54 | 3.08 |
| CW | Olmo3-7B | 96.92 | 0.32 | 69.94 | 2.48 | 65.65 | 1.99 | 41.98 | 4.09 | 70.80 | 1.16 |
To Weak-2, thank you for your valuable suggestions and for reading our paper in detail.
- Evaluation granularity. For each method, we save 9 checkpoints at equal intervals during training. Each checkpoint is evaluated on 130 problems: AMC23 (40 problems) and AIME24, AIME25, and HMMT25 (30 problems each). We sample 12 responses per problem, compute Mean@12 accuracy, and weight the results by the number of problems. The Overall Avg in the table aggregates responses, giving 108 samples per problem in total. Thus, a 1-percentage-point shift in Overall Avg requires changes in about 140 responses, and fluctuations in a few random responses have a minimal effect on performance.
- Training seeds. We again appreciate this suggestion, as it helps further examine the stability of our method and guards against bias from random seeds. We trained CW with three seeds (42/78/123), keeping all other settings identical to the main experiments. As shown in the table below, the standard deviation of CW's Overall Avg across the three seeds ranges from 0.04 to 0.11. This variation is smaller than the performance gaps underlying our conclusions, and the main conclusions hold under all three seeds.
- Evaluation scope. We acknowledge that our current study focuses on mathematical reasoning and has not yet been extended to other domains, which we will explore in future work. The above discussion and conclusions will be added in the revision.
| Method | Model | Seed-42 Overall Mean | Seed-42 Overall Var | Seed-78 Overall Mean | Seed-78 Overall Var | Seed-123 Overall Mean | Seed-123 Overall Var | Overall Mean±SD |
|---|---|---|---|---|---|---|---|---|
| CW | Qwen3-1.7B | 50.60 | 0.47 | 50.57 | 0.54 | 50.42 | 0.36 | 50.53±0.10 |
| CW | Qwen3-4B | 70.78 | 0.26 | 70.87 | 0.21 | 70.66 | 0.29 | 70.77±0.11 |
| CW | Qwen3-8B | 72.94 | 0.09 | 72.79 | 0.07 | 73.01 | 0.08 | 72.91±0.11 |
| CW | Olmo3-7B | 70.80 | 1.16 | 70.72 | 1.24 | 70.74 | 1.06 | 70.75±0.04 |
Thank you for taking the time and effort to review our paper and for your valuable suggestions. We hope that our responses and the additional experimental results above have addressed your concerns. We would be grateful if you find our responses satisfactory and could consider updating your score. If you have any remaining concerns, please let us know, and we will do our best to address them during the discussion period.
To Weakness-1, thank you for your valuable suggestions and careful reading of our paper. Each step of "union, normalization, averaging, and top-k selection" addresses a specific problem in multi-teacher top-k vocabulary distillation. The vocabulary union and normalization construct a shared policy space for all teachers. Without independent verification of teacher reliability, equal-weight averaging yields a consensus policy, which is the consensus distribution that minimizes the average forward KL over all teachers. Since policy fusion alters the probability ranking of candidate tokens, the consensus vocabulary must be realigned with the fused consensus policy. We thus select the top-k highest-scoring tokens from the joint policy space, so that tokens jointly supported by multiple teachers, or assigned high probability by at least some teachers, are retained first. A single-teacher top-k vocabulary, in contrast, is filtered only by that teacher's probability ranking. It tends to concentrate on the high-probability region of a single view and discard potentially valid candidates from other teachers, causing insufficient supervision coverage and unstable distillation.
In addition, under the main experimental settings of Qwen3-8B, Qwen3-4B, Qwen3-1.7B, and Olmo3-7B, we measured two metrics over the top-20 tokens at each position during CW training. The effective support ES(p~l)=∑v∈Vl,Topkp~l,v21 measures the number of tokens that provide effective supervision, with a larger value indicating more effective tokens. The coverage Cov(Vl,Topk)=N1∑j=1N∑v∈Vl,Topkpl,vTj measures the extent to which the distillation vocabulary retains each teacher's supervision; a larger value means broader coverage, while a smaller one means focus on a single teacher. Here, p~l denotes the distillation target at position l, and Vl,Topk is the corresponding top-k vocabulary. For a single teacher Tj, p~l=p~lTj and Vl,Topk=Vl,TopkTj; for CW, p~l=p~lT and Vl,Topk=Vl,Topkshared. pl,vTj is the raw probability of token v under teacher Tj over the full vocabulary.
As shown in the table below, CW achieves higher ES and Cov than every single teacher on all four models. This indicates that consensus policy fusion distributes supervision over more effective tokens and better retains each teacher's supervision. Meanwhile, the ES of CW remains below 2.5 out of k = 20, so the target is not diluted across the top-k vocabulary but stays focused on a few plausible candidates. We will add this discussion and the conclusions to the revision.