- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Sequential Monte Carlo Knowledge Distillation (SMC-KD), a distillation method that leverages student-generated, teacher-guided reasoning trajectories for SFT. The teacher guidance, using the teacher–student likelihood ratio, calibrates the weights used to redistribute the partial-block responses among particles. Since the method only requires the likelihood ratio for each block, it naturally supports different teacher–student vocabularies. Experiments show that SMC performs between Best-of- and pass@, and SMC-KD outperforms conventional OPD on mathematical reasoning benchmarks, although most gains are modest.
Critical strengths
- Tokenizer-agnostic distillation. Unlike conventional OPD, SMC-KD does not require token-level vocabulary alignment between the teacher and student. This is a practical advantage that enables flexible distillation across different model families and tokenizers, although misalignment between the teacher and student might lead to undesirable distillation.
- Effective teacher-guided trajectory selection. SMC consistently outperforms BoN across different student–teacher pairs and , which is natural, as generation is steered toward directions preferred by the teacher with finer control.
Critical weaknesses
-
Modest gains. The main weakness of the paper is that the gains from SMC-KD are modest (an average improvement of 1.14 points over OPD), especially considering the mismatched training configurations (OPD: N=8 × 16 prompts × 750 updates with lr=2e-6; SMC-KD: N=8 × 32 prompts × 750 updates with lr=5e-6).
-
Not scalable to . This is one of the critical problems that might hinder scaling up the method. SMC-KD does not benefit from recursive improvement of the student model. OPD does not suffer from the diversity collapse issue mentioned in the paper, allowing it to learn from more training steps.
Questions for authors
-
Starting from the point raised in the weaknesses section, is there any reason why you used different training configurations for OPD and SMC-KD? It would be good to see results from a fair comparison.
-
Starting from the point raised in the weaknesses section, could you report the performance of OPD with larger K? I am curious how much knowledge OPD could extract when K>1.
-
Adding an oracle baseline could help reinforce the claim. I would recommend adding SFT on teacher-generated trajectories as an oracle baseline (although I expect this to yield better performance than SMC-KD) to check the performance ceiling.
-
How does the number of active particles after step h evolve during SMC sampling? I am concerned about the case where the student–teacher likelihood ratio falls below 1 for all active (unfinished) particles, causing the particle population to collapse into copies of the same short response.
4 comments
Q2 (OPD with larger ). is a parameter of SMC-KD, the number of times the student that generates the training data is refreshed, and it does not apply to OPD in the same way. OPD is fully on-policy, so it generates new rollouts from the current student before each of its 750 updates. SMC-KD$_{12}$ instead generates new training data only 12 times, and after each round it takes up to 64 updates on that data. Under this schedule, SMC-KD$_{12}$ outperforms OPD in 18 of 20 combinations. Running SMC-KD at this extreme, refreshing after every batch, was less stable, because data from a single small batch is noisy without correctness labels. If the question is whether OPD would gain from more training updates, our comparison keeps the number of updates matched, at 750 for both methods.
Q3 (Teacher SFT as an oracle baseline). Table 1 now includes Teacher SFT, which fine-tunes the student on 8 teacher-generated responses per prompt with the same SFT recipe as SMC-KD. Interestingly, it is not a ceiling, and SMC-KD$_{12}$ outperforms it in 17 of 20 combinations. Teacher SFT is the strongest baseline only for Llama-3.2-1B, where it exceeds SMC-KD$_{12}$ by 0.3 to 1.3 points on MATH-500, GSM8K, and IFEval. With the cross-family teacher it falls below the base student on four of the five benchmarks, and for Qwen2.5-1.5B it lowers HumanEval+ from 40.3% to 34.8%. This is consistent with exposure bias (Section 1). The student trains on teacher-written prefixes that it would not produce itself, whereas SMC-KD trains on responses the student generated.
Q4 (number of active particles). The population size stays at throughout the search. All particles start active, and at each block every active particle is extended by 64 tokens. A particle finishes when it emits the end-of-turn token or reaches the 2,048-token budget, and it then stays in the population with its weight. The number of active particles therefore falls as particles finish, and resampling can move it in either direction, since it replaces low-weight particles with copies of high-weight ones, whether active or finished. The search ends when no active particle remains.
The mechanism you describe can occur. Each block multiplies an active particle's weight by the teacher--student ratio of that block. This factor equals one on average, since the block is drawn from the student's full distribution, but for a typical block it is below one, because the teacher usually gives the student's own text somewhat lower probability than the student does. Most active particles therefore lose some weight at each block, while a finished particle keeps its weight, so a finished particle with a high weight can receive copies at the next resampling.
This does not make the population concentrate on short responses, for three reasons. First, both models score the end-of-turn token, so stopping early is penalized when the teacher finds it unlikely. Second, resampling is triggered only when the ESS falls below , so lineages are not replaced at every block. Third, we measured this directly. In the one-shot training populations, the response a population concentrates on is within 4% of the mean length of the distinct responses in its population. Across iterations we observe the opposite effect, a rising fraction of responses that reach the 2,048-token budget, which motivated the capped-response filter.
Thank you to the authors for the extensive revision.
However, after carefully reviewing the updated manuscript and I decided to remain my score. Due to the below reasons:
-
While both methods sample from the same 12,000 prompts and run 750 optimizer updates, OPD trains for 1 epoch with a batch size of 16 prompts, whereas SMC-KD trains for 2 epochs with a batch size of 32 prompts. The argument that duplicate prefix merging reduces an 8-particles to 2.8 unique responses does not justify doubling the prompt batch size and training epochs in an apples-to-apples comparison. Prefix collapse seems to be an inherent consequence of SMC resampling rather than an external constraint to be compensated for.
-
36% higher GPU-hour cost cannot be ignored, and the overall gains remain modest relative to the added complexity.
Thank you for the follow-up. We would like to address both points.
Batch size and epochs. To remove this difference, we reran SMC-KD$_{12}$ with exactly OPD's training configuration, namely 16-prompt batches, one epoch, and 750 updates, with everything else unchanged. The results are essentially unchanged from those in the paper, within 0.2 points on average, and SMC-KD$_{12}$ still outperforms OPD in 17 of the 20 pairing–benchmark combinations, by 1.8 points on average and by up to 7.6 points. The gains over OPD therefore do not come from the batch size or the number of epochs.
| Pairing | Method | Batch, epochs | MATH-500 | GSM8K | AMC23 | HumanEval+ | IFEval | Avg. |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-1.5B ← Qwen2.5-Math-1.5B | OPD | 16, 1 | 54.0 ± 0.1 | 72.9 ± 0.5 | 25.2 ± 1.3 | 40.9 ± 0.6 | 39.0 ± 0.8 | 46.4 |
| SMC-KD$_{12}$ (paper) | 32, 2 | 55.7 ± 0.1 | 73.8 ± 1.0 | 28.1 ± 3.0 | 43.7 ± 0.7 | 39.1 ± 0.9 | 48.1 | |
| SMC-KD$_{12}$ | 16, 1 | 55.5 ± 0.0 | 73.1 ± 1.2 | 27.0 ± 2.0 | 42.4 ± 2.2 | 39.1 ± 0.1 | 47.4 | |
| Llama-3.2-1B ← Llama-3.2-3B | OPD | 16, 1 | 24.2 ± 0.4 | 38.8 ± 1.1 | 11.4 ± 0.5 | 27.8 ± 0.7 | 44.9 ± 1.1 | 29.4 |
| SMC-KD$_{12}$ (paper) | 32, 2 | 26.4 ± 0.7 | 44.3 ± 1.2 | 13.9 ± 1.1 | 31.7 ± 0.6 | 45.9 ± 1.5 | 32.4 | |
| SMC-KD$_{12}$ | 16, 1 | 26.4 ± 0.2 | 42.2 ± 0.9 | 14.4 ± 0.4 | 32.0 ± 2.2 | 45.4 ± 1.2 | 32.1 | |
| Qwen2.5-0.5B ← Qwen2.5-Math-1.5B | OPD | 16, 1 | 32.0 ± 0.2 | 47.7 ± 0.5 | 13.9 ± 0.8 | 28.9 ± 0.9 | 26.6 ± 0.9 | 29.8 |
| SMC-KD$_{12}$ (paper) | 32, 2 | 33.0 ± 0.4 | 47.5 ± 0.2 | 14.5 ± 0.4 | 29.3 ± 1.1 | 28.9 ± 1.1 | 30.6 | |
| SMC-KD$_{12}$ | 16, 1 | 32.5 ± 0.4 | 47.9 ± 0.3 | 16.7 ± 0.7 | 28.0 ± 0.9 | 30.0 ± 0.7 | 31.0 | |
| Qwen2.5-0.5B ← Qwen3-1.7B | OPD | 16, 1 | 29.9 ± 0.3 | 43.1 ± 0.3 | 9.1 ± 1.2 | 23.2 ± 1.1 | 27.2 ± 0.7 | 26.5 |
| SMC-KD$_{12}$ (paper) | 32, 2 | 30.8 ± 0.7 | 47.9 ± 0.5 | 6.6 ± 1.4 | 30.9 ± 1.5 | 28.8 ± 0.7 | 29.0 | |
| SMC-KD$_{12}$ | 16, 1 | 31.6 ± 0.7 | 48.8 ± 0.4 | 5.9 ± 0.9 | 30.8 ± 0.4 | 27.0 ± 1.0 | 28.8 |
The Student SFT and Teacher SFT baselines in the paper lead to the same conclusion. They use exactly SMC-KD's training configuration, with the same prompts, batch size, epochs, learning rate, and number of updates. They differ from SMC-KD only in how the training responses are generated and weighted. Under this identical configuration, SMC-KD$_{12}$ outperforms Student SFT in 18 of the 20 combinations, by 2.0 points on average, and Teacher SFT in 17 of 20, by 3.4 points on average.
Cost and gains. The additional compute is spent in the search, which is the step that produces the training data, and the matched comparisons above show that this data is what drives the gains. The benefits of SMC-KD also go beyond accuracy. It does not require the teacher and student to share a tokenizer, which you noted as a strength, and relative to standard SFT it adds only resampling during generation and a weighted loss.
We hope these results address your concerns, and if so, we would appreciate it if you would consider updating your score.
Thank you for your constructive feedback. We are glad you found the tokenizer-agnostic design practical and teacher-guided SMC effective. We address your concerns below, and all changes in the revised paper are shown in blue.
W1 and Q1 (gains and training configurations).
Size of the gains. In the revision, we improved SMC-KD by removing responses that reach the 2,048-token budget from its training data, a filter we also apply to the Teacher SFT and Student SFT baselines (Appendix F.2, and W2 below). We also extended the evaluation with HumanEval+, IFEval, and three additional baselines. With these changes, SMC-KD$_{12}$ outperforms OPD in 18 of the 20 pairing--benchmark combinations, by 2.0 points on average and up to 7.7 points, with a 95% paired-bootstrap interval of [1.1,2.9] (Appendix F.5). Relative to the base student, SMC-KD$_{12}$ improves accuracy by 4.3 points on average, nearly twice the 2.3-point improvement of OPD.
Why the configurations differ. The two methods apply different kinds of update, and we used the standard configuration for each. Generation is matched, since both produce 8 responses for each of the 12,000 prompts and take 750 updates. The batch sizes differ because SMC-KD particles share prefixes. After merging duplicates, an 8-particle population contains 2.8 distinct responses on average, so a 32-prompt SMC-KD batch holds about 90 distinct responses, still fewer than the 128 rollouts of a 16-prompt OPD batch. The learning rates follow common practice, around 10−6 for OPD and 5×10−6 to 10−5 for SFT.
Comparison with identical configurations. The new Teacher SFT and Student SFT baselines use exactly SMC-KD's SFT configuration on the same prompts. Student SFT is SMC-KD$_1$ without block-by-block resampling of partial responses, and SMC-KD$_1$ outperforms it in 11 of 12 math combinations. SMC-KD$_{12}$ outperforms Teacher SFT in 17 of 20. OPD (PG) uses OPD's configuration with the same teacher information as SMC-KD, and SMC-KD$_{12}$ outperforms it in 18 of 20 with one tie. The gains therefore come from the data SMC produces, not from the optimizer settings.
W2 (scaling to K>1). The revised experiments show that SMC-KD does benefit from iteration. We found that a major cause of the earlier degradation was responses that reach the 2,048-token budget. Once they enter the training data, they accumulate across rounds. Removing them from the SFT data (Appendix F.2) improves SMC-KD$_{12}$ in 15 of the 20 combinations. With this filter, SMC-KD$_{12}$ improves on SMC-KD$_1$ in 17 of 20 combinations, including HumanEval+ in every pairing, and it is now our main variant. The main exception is the cross-family pairing, where long responses that do not reach an answer become more common because the teacher distinguishes promising particles less reliably (Section 4.3).
We agree that diversity matters when iterating. The iterative variant reported in the submission was trained on all responses, including capped ones, and Appendix F.3 shows that removing capped responses raises its pass@8 in every pairing. The one-shot variant exceeds OPD's pass@8 in three of the four pairings (Table 15). Reduced diversity therefore does not prevent SMC-KD from benefiting from more rounds. It is a known trade-off of SMC, whose resampling step copies high-weight prefixes, and this same concentration is what makes the selected responses more accurate (Section 4.2). Appendix F.4 shows one way to further improve diversity, using island particles (Vergé et al., 2015). Islands resample independently, so several lineages per prompt always survive, and they increase pass@N in all 12 settings. Applying islands and other diversity-improving methods to SMC-KD in settings where diversity matters more is a promising direction for future work.