acceptodds
Under review as a conference paper at ICLR 2027

SMC-KD: Knowledge distillation using Teacher-Guided Sequential Monte Carlo

Abstract

Small language models are becoming more capable, and in many domains they can generate a correct answer across multiple samples, even when they do not do so reliably in a single attempt. Distillation from a stronger teacher can turn these occasional successes into more reliable behavior, but its effectiveness depends on which responses the student is trained on. For example, fine-tuning on teacher-generated responses can lead to exposure bias: the student trains on prefixes it would not produce itself, which differ from those it sees at inference. Training on the student's own responses avoids this mismatch but needs feedback on those responses. Selecting good responses with Best-of- (BoN) or verifier-guided search typically relies on correctness labels or separately trained reward models. On-policy distillation (OPD) instead has the teacher score each student token, but a poorly matched teacher can hinder performance, and different teacher and student vocabularies complicate token-level supervision. In this paper, we introduce teacher-guided Sequential Monte Carlo Knowledge Distillation (SMC-KD), a method that uses teacher feedback to identify desirable trajectories among those generated by the student and distills them into the student's parameters. SMC-KD requires no correctness labels, process reward model, or semantic step boundaries, and naturally supports teacher and student models with different vocabularies. The student generates a population of partial trajectories, the teacher scores them by the teacher–student likelihood ratio, and SMC resampling allocates subsequent generation to higher-scoring prefixes. We then train the student on these trajectories through supervised fine-tuning (SFT). Before any training, we show that teacher-guided SMC selects more accurate responses than BoN under the same likelihood score, for every student–teacher pairing and population size we evaluate. After training, SMC-KD matches or improves on the base student in every setting and outperforms OPD in 10 of 12 comparisons across four student–teacher pairings and three math benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Distillation, Sequential Monte Carlo, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w8uBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Sequential Monte Carlo Knowledge Distillation (SMC-KD), a distillation method that leverages student-generated, teacher-guided reasoning trajectories for SFT. The teacher guidance, using the teacher–student likelihood ratio, calibrates the weights used to redistribute the partial-block responses shs_h among NN particles. Since the method only requires the likelihood ratio for each block, it naturally supports different teacher–student vocabularies. Experiments show that SMC performs between Best-of-NN and pass@NN, and SMC-KD outperforms conventional OPD on mathematical reasoning benchmarks, although most gains are modest.

Critical strengths
  • Tokenizer-agnostic distillation. Unlike conventional OPD, SMC-KD does not require token-level vocabulary alignment between the teacher and student. This is a practical advantage that enables flexible distillation across different model families and tokenizers, although misalignment between the teacher and student might lead to undesirable distillation.
  • Effective teacher-guided trajectory selection. SMC consistently outperforms BoN across different student–teacher pairs and NN, which is natural, as generation is steered toward directions preferred by the teacher with finer control.
Critical weaknesses
  • Modest gains. The main weakness of the paper is that the gains from SMC-KD are modest (an average improvement of 1.14 points over OPD), especially considering the mismatched training configurations (OPD: N=8 × 16 prompts × 750 updates with lr=2e-6; SMC-KD: N=8 × 32 prompts × 750 updates with lr=5e-6).

  • Not scalable to K>1K>1. This is one of the critical problems that might hinder scaling up the method. SMC-KD does not benefit from recursive improvement of the student model. OPD does not suffer from the diversity collapse issue mentioned in the paper, allowing it to learn from more training steps.

Questions for authors
  1. Starting from the point raised in the weaknesses section, is there any reason why you used different training configurations for OPD and SMC-KD? It would be good to see results from a fair comparison.

  2. Starting from the point raised in the weaknesses section, could you report the performance of OPD with larger K? I am curious how much knowledge OPD could extract when K>1.

  3. Adding an oracle baseline could help reinforce the claim. I would recommend adding SFT on teacher-generated trajectories as an oracle baseline (although I expect this to yield better performance than SMC-KD) to check the performance ceiling.

  4. How does the number of active particles after step h evolve during SMC sampling? I am concerned about the case where the student–teacher likelihood ratio falls below 1 for all active (unfinished) particles, causing the particle population to collapse into copies of the same short response.

4 comments

Thank you for your constructive feedback. We are glad you found the tokenizer-agnostic design practical and teacher-guided SMC effective. We address your concerns below, and all changes in the revised paper are shown in blue.

W1 and Q1 (gains and training configurations).

Size of the gains. In the revision, we improved SMC-KD by removing responses that reach the 2,048-token budget from its training data, a filter we also apply to the Teacher SFT and Student SFT baselines (Appendix F.2, and W2 below). We also extended the evaluation with HumanEval+, IFEval, and three additional baselines. With these changes, SMC-KD$_{12}$ outperforms OPD in 18 of the 20 pairing--benchmark combinations, by 2.0 points on average and up to 7.7 points, with a 95% paired-bootstrap interval of [1.1,2.9][1.1, 2.9] (Appendix F.5). Relative to the base student, SMC-KD$_{12}$ improves accuracy by 4.3 points on average, nearly twice the 2.3-point improvement of OPD.

Why the configurations differ. The two methods apply different kinds of update, and we used the standard configuration for each. Generation is matched, since both produce 8 responses for each of the 12,000 prompts and take 750 updates. The batch sizes differ because SMC-KD particles share prefixes. After merging duplicates, an 8-particle population contains 2.8 distinct responses on average, so a 32-prompt SMC-KD batch holds about 90 distinct responses, still fewer than the 128 rollouts of a 16-prompt OPD batch. The learning rates follow common practice, around 10−610^{-6} for OPD and 5×10−65\times10^{-6} to 10−510^{-5} for SFT.

Comparison with identical configurations. The new Teacher SFT and Student SFT baselines use exactly SMC-KD's SFT configuration on the same prompts. Student SFT is SMC-KD$_1$ without block-by-block resampling of partial responses, and SMC-KD$_1$ outperforms it in 11 of 12 math combinations. SMC-KD$_{12}$ outperforms Teacher SFT in 17 of 20. OPD (PG) uses OPD's configuration with the same teacher information as SMC-KD, and SMC-KD$_{12}$ outperforms it in 18 of 20 with one tie. The gains therefore come from the data SMC produces, not from the optimizer settings.

W2 (scaling to K>1K > 1). The revised experiments show that SMC-KD does benefit from iteration. We found that a major cause of the earlier degradation was responses that reach the 2,048-token budget. Once they enter the training data, they accumulate across rounds. Removing them from the SFT data (Appendix F.2) improves SMC-KD$_{12}$ in 15 of the 20 combinations. With this filter, SMC-KD$_{12}$ improves on SMC-KD$_1$ in 17 of 20 combinations, including HumanEval+ in every pairing, and it is now our main variant. The main exception is the cross-family pairing, where long responses that do not reach an answer become more common because the teacher distinguishes promising particles less reliably (Section 4.3).

We agree that diversity matters when iterating. The iterative variant reported in the submission was trained on all responses, including capped ones, and Appendix F.3 shows that removing capped responses raises its pass@8 in every pairing. The one-shot variant exceeds OPD's pass@8 in three of the four pairings (Table 15). Reduced diversity therefore does not prevent SMC-KD from benefiting from more rounds. It is a known trade-off of SMC, whose resampling step copies high-weight prefixes, and this same concentration is what makes the selected responses more accurate (Section 4.2). Appendix F.4 shows one way to further improve diversity, using island particles (Vergé et al., 2015). Islands resample independently, so several lineages per prompt always survive, and they increase pass@NN in all 12 settings. Applying islands and other diversity-improving methods to SMC-KD in settings where diversity matters more is a promising direction for future work.

Q2 (OPD with larger KK). KK is a parameter of SMC-KD, the number of times the student that generates the training data is refreshed, and it does not apply to OPD in the same way. OPD is fully on-policy, so it generates new rollouts from the current student before each of its 750 updates. SMC-KD$_{12}$ instead generates new training data only 12 times, and after each round it takes up to 64 updates on that data. Under this schedule, SMC-KD$_{12}$ outperforms OPD in 18 of 20 combinations. Running SMC-KD at this extreme, refreshing after every batch, was less stable, because data from a single small batch is noisy without correctness labels. If the question is whether OPD would gain from more training updates, our comparison keeps the number of updates matched, at 750 for both methods.

Q3 (Teacher SFT as an oracle baseline). Table 1 now includes Teacher SFT, which fine-tunes the student on 8 teacher-generated responses per prompt with the same SFT recipe as SMC-KD. Interestingly, it is not a ceiling, and SMC-KD$_{12}$ outperforms it in 17 of 20 combinations. Teacher SFT is the strongest baseline only for Llama-3.2-1B, where it exceeds SMC-KD$_{12}$ by 0.3 to 1.3 points on MATH-500, GSM8K, and IFEval. With the cross-family teacher it falls below the base student on four of the five benchmarks, and for Qwen2.5-1.5B it lowers HumanEval+ from 40.3% to 34.8%. This is consistent with exposure bias (Section 1). The student trains on teacher-written prefixes that it would not produce itself, whereas SMC-KD trains on responses the student generated.

Q4 (number of active particles). The population size stays at N=8N=8 throughout the search. All particles start active, and at each block every active particle is extended by 64 tokens. A particle finishes when it emits the end-of-turn token or reaches the 2,048-token budget, and it then stays in the population with its weight. The number of active particles therefore falls as particles finish, and resampling can move it in either direction, since it replaces low-weight particles with copies of high-weight ones, whether active or finished. The search ends when no active particle remains.

The mechanism you describe can occur. Each block multiplies an active particle's weight by the teacher--student ratio of that block. This factor equals one on average, since the block is drawn from the student's full distribution, but for a typical block it is below one, because the teacher usually gives the student's own text somewhat lower probability than the student does. Most active particles therefore lose some weight at each block, while a finished particle keeps its weight, so a finished particle with a high weight can receive copies at the next resampling.

This does not make the population concentrate on short responses, for three reasons. First, both models score the end-of-turn token, so stopping early is penalized when the teacher finds it unlikely. Second, resampling is triggered only when the ESS falls below N/2N/2, so lineages are not replaced at every block. Third, we measured this directly. In the one-shot training populations, the response a population concentrates on is within 4% of the mean length of the distinct responses in its population. Across iterations we observe the opposite effect, a rising fraction of responses that reach the 2,048-token budget, which motivated the capped-response filter.

Thank you to the authors for the extensive revision.

However, after carefully reviewing the updated manuscript and I decided to remain my score. Due to the below reasons:

  1. While both methods sample from the same 12,000 prompts and run 750 optimizer updates, OPD trains for 1 epoch with a batch size of 16 prompts, whereas SMC-KD trains for 2 epochs with a batch size of 32 prompts. The argument that duplicate prefix merging reduces an 8-particles to 2.8 unique responses does not justify doubling the prompt batch size and training epochs in an apples-to-apples comparison. Prefix collapse seems to be an inherent consequence of SMC resampling rather than an external constraint to be compensated for.

  2. 36% higher GPU-hour cost cannot be ignored, and the overall gains remain modest relative to the added complexity.

Thank you for the follow-up. We would like to address both points.

Batch size and epochs. To remove this difference, we reran SMC-KD$_{12}$ with exactly OPD's training configuration, namely 16-prompt batches, one epoch, and 750 updates, with everything else unchanged. The results are essentially unchanged from those in the paper, within 0.2 points on average, and SMC-KD$_{12}$ still outperforms OPD in 17 of the 20 pairing–benchmark combinations, by 1.8 points on average and by up to 7.6 points. The gains over OPD therefore do not come from the batch size or the number of epochs.

PairingMethodBatch, epochsMATH-500GSM8KAMC23HumanEval+IFEvalAvg.
Qwen2.5-1.5B ← Qwen2.5-Math-1.5BOPD16, 154.0 ± 0.172.9 ± 0.525.2 ± 1.340.9 ± 0.639.0 ± 0.846.4
SMC-KD$_{12}$ (paper)32, 255.7 ± 0.173.8 ± 1.028.1 ± 3.043.7 ± 0.739.1 ± 0.948.1
SMC-KD$_{12}$16, 155.5 ± 0.073.1 ± 1.227.0 ± 2.042.4 ± 2.239.1 ± 0.147.4
Llama-3.2-1B ← Llama-3.2-3BOPD16, 124.2 ± 0.438.8 ± 1.111.4 ± 0.527.8 ± 0.744.9 ± 1.129.4
SMC-KD$_{12}$ (paper)32, 226.4 ± 0.744.3 ± 1.213.9 ± 1.131.7 ± 0.645.9 ± 1.532.4
SMC-KD$_{12}$16, 126.4 ± 0.242.2 ± 0.914.4 ± 0.432.0 ± 2.245.4 ± 1.232.1
Qwen2.5-0.5B ← Qwen2.5-Math-1.5BOPD16, 132.0 ± 0.247.7 ± 0.513.9 ± 0.828.9 ± 0.926.6 ± 0.929.8
SMC-KD$_{12}$ (paper)32, 233.0 ± 0.447.5 ± 0.214.5 ± 0.429.3 ± 1.128.9 ± 1.130.6
SMC-KD$_{12}$16, 132.5 ± 0.447.9 ± 0.316.7 ± 0.728.0 ± 0.930.0 ± 0.731.0
Qwen2.5-0.5B ← Qwen3-1.7BOPD16, 129.9 ± 0.343.1 ± 0.39.1 ± 1.223.2 ± 1.127.2 ± 0.726.5
SMC-KD$_{12}$ (paper)32, 230.8 ± 0.747.9 ± 0.56.6 ± 1.430.9 ± 1.528.8 ± 0.729.0
SMC-KD$_{12}$16, 131.6 ± 0.748.8 ± 0.45.9 ± 0.930.8 ± 0.427.0 ± 1.028.8

The Student SFT and Teacher SFT baselines in the paper lead to the same conclusion. They use exactly SMC-KD's training configuration, with the same prompts, batch size, epochs, learning rate, and number of updates. They differ from SMC-KD only in how the training responses are generated and weighted. Under this identical configuration, SMC-KD$_{12}$ outperforms Student SFT in 18 of the 20 combinations, by 2.0 points on average, and Teacher SFT in 17 of 20, by 3.4 points on average.

Cost and gains. The additional compute is spent in the search, which is the step that produces the training data, and the matched comparisons above show that this data is what drives the gains. The benefits of SMC-KD also go beyond accuracy. It does not require the teacher and student to share a tokenizer, which you noted as a strength, and relative to standard SFT it adds only resampling during generation and a weighted loss.

We hope these results address your concerns, and if so, we would appreciate it if you would consider updating your score.

Reviewer_UUBeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SMC-KD uses a frozen student to generate a population of responses in 64-token blocks. Teacher/student likelihood ratios reweight prefixes, and ESS-triggered systematic resampling allocates further generation to promising prefixes. The student is then trained on all final particles using weighted SFT, either once or with refreshed student snapshots. No correctness labels, process reward model, or semantic step boundaries are used for training; ordinary student decoding is used at deployment. Under a common response space and the stated support assumptions, ratio guidance targets the teacher distribution, while the SFT projection minimizes forward KL. The paper distinguishes this projection from reverse-KL OPD. Direct-generation experiments compare SMC with identically scored BoN across four model pairings and particle counts 4-32. Training experiments use 12,000 MATH problem statements and evaluate MATH-500, GSM8K, and AMC23 over three training seeds. The selected one-shot/iterative configuration outperforms OPD in 10 of 12 comparisons; either fixed refresh setting wins 9 of 12. The appendices also document iteration-induced truncation and reduced multi-sample coverage.

Critical strengths
  1. The paper offers a coherent way to use teacher likelihood during data generation without requiring a learned verifier or aligned token-level targets. The sampling/projection decomposition and the distinction between forward-KL SFT and reverse-KL OPD are clearly explained (Sections 3.1-3.3, pp. 5-7). The strongest empirical result is the direct-generation comparison: ratio-guided SMC improves mean accuracy over BoN in all 16 pairing/population combinations of Table 1. For Qwen2.5-0.5B at N=32, the improvement is 45.1% to 53.4%. This provides useful evidence that prefix resampling can exploit a teacher score more effectively than terminal selection at the same particle count.

  2. The analysis of scorer compatibility and failure modes is informative. The 13-model scoring matrix shows why a stronger solver need not be a better scorer for another model's responses (Appendix D). Training results include three seeds, teacher-only versus ratio controls, both refresh schedules, capped-response rates, and pass@8. The Llama GSM8K gain over OPD (38.8% to 43.5%) is practically interesting. Reporting that iterative training can harm AMC23 performance and multi-sample coverage, rather than only presenting the favorable pass@1 results, makes the evidence more useful and delineates the method's intended benefit.

Critical weaknesses
  1. The training experiments do not isolate the value of SMC over simpler ways of constructing SFT data, and the resource comparison is incomplete. Table 2 compares only the base model, OPD, and SMC-KD. Missing controls include teacher-generated SFT and student-generated BoN-plus-SFT with the same teacher score and matched data/training budgets. The direct-generation advantage over BoN is valuable, but uses the highest-scoring final response, whereas training consumes the whole weighted particle population; it does not by itself establish the quality advantage of the training data. Table 3 uses 32 prompts with eight weighted particles for SFT versus 16 prompts with eight rollouts for OPD at the same 750 updates, implying twice as many trajectory presentations before accounting for length and duplication. No total generated/scored-token or GPU-hour comparison is reported. Since several Table 2 gains are only 0.4-1.9 points, these omissions materially limit attribution to the proposed method.

  2. Important details connecting the implemented sampler to the theoretical target remain unspecified. Table 3 gives only model-dependent Hugging Face sampling defaults. The likelihood-ratio denominator must correspond to the actual proposal, or an additional importance correction is needed; temperature or top-p/top-k sampling can otherwise change the claimed target and support. For different tokenizers, independently scoring the same text is a useful practical score, but the blockwise factorization and teacher-target interpretation require an explicit common outcome space and boundary convention. Retokenization can change the end of an earlier prefix. Section 2.1 specifies termination at EOS or the maximum number of blocks, and Section 2.3 explains that completed responses are carried forward with incremental weight one. What remains unclear is how EOS and length-capped responses enter the likelihood calculations and target-distribution interpretation, and how incremental teacher scores handle retokenization at student-block boundaries.It also gives no full-vocabulary OPD alignment details for the cross-family comparison. These are requests for implementation clarification, not a demonstrated error, but they matter to the theoretical interpretation and reproducibility, especially with no code available in the submission (p. 10).

Questions for authors
  1. Decisive additional evidence: Can you compare SMC-KD with teacher-generated SFT and teacher-scored BoN-plus-SFT while matching prompt exposure, student training tokens, and total generation/scoring cost? For the existing OPD comparison, please report data reuse, trajectory/token presentations, total teacher-scored tokens, and GPU-hours. Equal optimizer-step counts alone do not establish a matched budget.

  2. Clarification: What exact model checkpoints, chat templates, generation settings, and log-probabilities are used for each pairing? Does the denominator use the actual normalized decoding distribution after temperature and truncation? If it uses raw model probabilities instead, what target does the implemented sampler approximate? Please distinguish the ideal full-support derivation from the finite-length, possibly truncated implementation.

  3. Clarification: For different tokenizers, how are prefix potentials and incremental teacher scores computed when appending a student block changes the teacher's tokenization of the boundary? Please specify treatment of special tokens, EOS and capped responses, and how vocabulary correspondence is handled in the full-vocabulary OPD baseline. Also confirm that Eq. (6) uses the normalized residual SMC weights, carries weights forward when resampling is skipped, accounts for duplicate multiplicities, and uses the stated sequence-log-likelihood reduction rather than an unintended per-sequence token mean.

  4. Clarification and decisive evidence: How was K=1 versus K=12 chosen for each main-table pairing, and was that choice fixed independently of the three evaluation sets? The fixed-K results already win 9/12 comparisons, so this is not an allegation that the result depends entirely on selection. Please report paired uncertainty for the small gains, ideally resampling by problem as well as reporting training-seed variability. The tradeoff also deserves prominence: Llama pass@8 is 44.0% for iterative SMC-KD versus 53.5% for OPD (Table 12), and iterative cross-family Qwen2.5-0.5B falls to 6.1% on AMC23 (Table 11). Which deployment objective determines the recommended configuration?

4 comments

Thank you for the detailed and constructive review. We are glad you found the method coherent and the analyses informative. We added the requested controls, uncertainty estimates, and implementation details. All changes in the revised paper are shown in blue.

W1 and Q1 (SFT controls and matched budgets). We added both requested controls to Table 1, trained on the same 12,000 prompts.

  • Teacher SFT fine-tunes the student on 8 teacher-generated responses per prompt.
  • Student SFT is the teacher-scored BoN-plus-SFT control. It draws 8 independent student responses per prompt, weights them by the softmax of the same sequence-level score log⁡(πT/πS)\log(\pi_T/\pi_S), and applies the same weighted SFT.

Both use exactly SMC-KD's SFT recipe (batch size, learning rate, epochs, number of updates, and the capped-response filter). Student SFT also has the same generation and teacher-scoring budget as SMC-KD$_1$, so the two differ only in whether partial responses are resampled block by block.

These controls isolate the value of SMC. On the three math benchmarks, SMC-KD$_1$ outperforms Student SFT in 11 of 12 combinations, by 0.9 points on average (95% paired-bootstrap interval [0.1,1.8][0.1, 1.8]). Since both train on the whole weighted population, this supports a quality advantage of the training data, not only of the selected response. SMC-KD$_{12}$ outperforms Teacher SFT in 17 of 20 combinations.

For the OPD comparison, the generation and scoring budget is the same. Both methods generate 8 responses of at most 2,048 tokens for each of the 12,000 prompts, and the teacher scores all of them. The training configurations differ only to keep the number of updates equal. Because SMC-KD particles share prefixes, an 8-particle population contains 2.8 distinct responses on average after merging duplicates. We therefore use twice OPD's batch size (32 prompts versus 16), so that an SMC-KD batch holds about 90 distinct responses, still fewer than the 128 rollouts in an OPD batch. We then train for two epochs, so that both methods take 750 updates. SMC-KD also needs only the teacher's log-probability of each sampled token, while OPD needs its full distribution. On one A100 40GB GPU, an SMC-KD run takes 15 GPU-hours on average versus 11 for OPD (Appendix B).

W2 and Q2 (sampling settings, proposal, and target). Appendix B now has paragraphs on models and prompts, the proposal and potential, tokenization and termination, and OPD with a cross-family teacher, and Table 2 lists the exact sampling settings of every model. Every model uses its own chat template. Tokenization and the OPD alignment are covered under Q3 below.

Denominator. In every SMC run, log⁡πS\log\pi_S is the log-probability of each token under the normalized distribution it was actually sampled from, after temperature and any truncation. The weights are therefore proper importance weights with respect to the true proposal, and no extra correction is needed.

Target. For SMC-KD training data, the student samples from its full distribution at temperature TT, with no top-pp or top-kk, and the teacher is scored at the same TT. The proposal therefore has full support, and the target is the teacher tempered to TT. In the direct-generation experiments of Section 4.2, each student instead samples with its recommended settings (Table 2), and the denominator uses this same truncated distribution.

Finite length. Responses end at the end-of-turn token or at 2,048 tokens, and both models score every generated token, including end-of-turn. A capped response has no end-of-turn token, and its weight is the teacher--student ratio of the tokens it generated. Because the teacher's probability of a prefix equals its total probability over all continuations, the target is exact on a finite outcome space, namely complete responses of at most 2,048 tokens together with capped 2,048-token prefixes. Removing capped responses before SFT then corresponds to conditioning this target on finishing within the budget.

Q3 (tokenization, OPD alignment, and Equation 6).

Tokenization. In all of our SMC and SMC-KD pairings, the student and teacher assign the same token ids to ordinary text (Qwen2.5 and Qwen3 differ only in four special tokens). The teacher therefore scores the student's token ids directly, block boundaries coincide, and no retokenization occurs. Our claim that SMC-KD supports different vocabularies rests on the method needing only the teacher's likelihood of text and on the 13-model scoring analysis of Appendix D, and Appendix B now says this explicitly. It also gives the convention for different tokenizers. The teacher scores the decoded prefix text with its own tokenizer, and each block's weight update is the ratio of this score after the block to the score before it. When these updates are multiplied over all blocks, the intermediate scores cancel. The final weight therefore depends only on the teacher's likelihood of the complete response, and the tokenization at block boundaries affects only the intermediate weights.

OPD alignment. The Qwen2.5 student and Qwen3 teacher both have 151,936 logit columns, which pad their tokenizers to a common size, with the same ids for ordinary text, so the full-vocabulary reverse KL is computed over all columns. Qwen3's non-thinking chat template places an empty thinking block before the response, and we insert the same block in the teacher's input. Without it, the teacher puts almost all its probability on opening a thinking block, and OPD diverged.

Equation 6. We confirm all four points. The objective uses the normalized residual SMC weights after the last block. When resampling is skipped, each particle carries its weight forward, and finished particles keep their weight. Duplicate responses are merged by summing their weights. Each response contributes its summed token log-likelihood, not a per-token mean.

Q4 (choice of KK, uncertainty, and the multi-sample trade-off).

Choice of KK. Table 1 now reports both SMC-KD$_1$ and SMC-KD$_{12}$ for every pairing, so there is no per-pairing choice. All pairings use the same schedule, K=12K=12 rounds of B=1,024B=1{,}024 prompts, and we did not tune KK on the evaluation benchmarks. As Appendix B explains, we first ran SMC once with the base student (K=1K=1), and because this improved the student, a natural next step was to let the improved student generate the data for later prompts. Refreshing the data after every optimizer batch was less stable, since without correctness labels the data from a single small batch can be noisy. We therefore use 12 rounds of 1,024 prompts, a batch large enough for SMC to improve the data on average before the updated student generates the next round.

Uncertainty. Every trained entry is a mean over three training seeds, and the new Appendix F.5 adds a paired bootstrap that resamples problems. Averaged over the 20 combinations, SMC-KD$_{12}$ exceeds OPD by 2.0 points (95% interval [1.1,2.9][1.1, 2.9]). For individual combinations, the interval excludes zero in favor of SMC-KD$_{12}$ in 5 of the 20 and never in favor of OPD. As expected for differences of a few points, individual intervals are wider, and the evidence lies in the average and the consistent direction across combinations.

Trade-off and recommendation. Section 4.3 now discusses both issues you raise. On pass@8 (Table 15), the one-shot SMC-KD$_1$ exceeds OPD in three of the four pairings, is below it only for Llama-3.2-1B, and stays above the base student in every pairing. Based on these results, SMC-KD$_1$ can serve as a strong baseline. In general, we expect additional rounds of search and training to help further, since each round starts from an improved student, and SMC-KD$_{12}$ improves on SMC-KD$_1$ in 17 of the 20 combinations. The main exception is the cross-family pairing, which we trace to long responses that do not reach an answer, because the teacher distinguishes promising particles less reliably (Appendix F.2). When running several rounds, especially with a cross-family teacher, we recommend monitoring the search to check that the teacher still improves the selected responses and that overly long responses are not being generated.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer UUBe,

As the discussion period approaches its end, we would like to check whether our response has addressed your concerns. If any remain, we would be glad to clarify before the discussion closes. Otherwise, we would appreciate it if you would consider updating your assessment.

Thank you for the authors' response. After considering the rebuttal, I would prefer to maintain my original assessment.

Reviewer_APgPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses SFT data selection, exploring whether a teacher can guide the construction of student SFT data. The student model generates trajectories segment by segment, while the teacher model assigns scores based on likelihood ratios; Sequential Monte Carlo (SMC) resampling is then employed to retain high-scoring prefixes, ultimately resulting in weighted SFT on these trajectories. SMC consistently outperforms Best-of-N (BoN) sampling, and SMC-KD surpasses OPD in 10 out of 12 comparative evaluations.

Critical strengths

The idea is interesting, and the proposed method is a good way to improve exploration to collect more data for producing better students: by scoring student-generated trajectories from teacher logits and using these scores to resample and collect higher-quality trajectories, it is natural to put more training budget on superior data. This approach may help mitigate the "mismatch" issue that arises when teacher signals are used directly as rewards to update the student model.

Critical weaknesses
  • There is still a mismatch between the objective of promising prefix trajectories and the teacher scoring. The teachers may have different reasoning patterns, and the promising prefix for students may not align with the teacher's preference. Since the teacher still evaluates trajectories based on the student’s prefixes, has the impact of unreliable feedback merely shifted from the parameter-update phase to the data-selection phase? Also, how diverse are the resampled trajectories? Is there a risk of leading to mode collapse and limited reasoning paths because of repetitive prefixes?

  • Does SMC-KD genuinely enhance reasoning capabilities, or does it primarily compress the benefits of search and selection into the pass@1 metric? OPD has better pass@8.

  • Could we have a better way to use the resampling? It seems a bit wasteful: 1) to calculate the teacher's logits but only use them to sample data 2) it needs more sampling budget for collecting data, and a lot of bad trajectories could also help improve the student but be dropped. When accounting for the costs of generating teacher scores and SMC-based data, do the performance gains justify the overhead? Given that teacher logits are already computed, could the teacher’s distribution be reused to maximize the returns per unit of computational budget?

Questions for authors

Please the weakness

5 comments

Thank you for your thoughtful review. We are glad you found the idea interesting and a natural way to focus training on better data. We address your concerns below, and all changes in the revised paper are shown in blue.

W1 (teacher mismatch and diversity). We agree that SMC-KD still depends on the teacher's judgement of student-written prefixes. What changes is how much damage an unreliable judgement can do, for three reasons.

(i) The teacher can only reweight responses that the student wrote, so every training token comes from the student. Even if the teacher's weights carried no information, SMC-KD would reduce to SFT on the student's own samples. In OPD, by contrast, the teacher's next-token distribution is the training target at every position, so a mismatched teacher pulls every token toward its own style.

(ii) The weights are normalized within each prompt, so only the teacher's relative preference among the student's own candidates matters. If the teacher assigns low likelihood to the student's reasoning style in general, all candidates are lowered together and this cancels. In addition, because the potential is the ratio πT/πS\pi_T/\pi_S, a block is not favored merely because the student finds it likely. It gains weight only where the teacher assigns it more probability than the student does.

(iii) Resampling moves the population toward prefixes the teacher rates highly, which is where the teacher's later scores are most reliable. OPD instead queries the teacher on every student prefix, including ones the teacher would rarely produce.

You also point out that a prefix the teacher scores highly may not be the one most likely to lead the student to a correct answer. We tested this before any training. Resampling student prefixes by the teacher's ratio selects more accurate responses than BoN in every MATH-500 setting (Section 4.2). We also compare it with two reference points, search guided by the student's own likelihood and search guided by a PRM trained on correctness labels. Teacher-guided search closes 39% to 97% of the accuracy gap between them (Appendix E).

The cross-family experiment (Section 4.3) then tests a teacher that reasons differently. With the same Qwen2.5-0.5B student, we replace the Qwen2.5-Math-1.5B teacher with Qwen3-1.7B. OPD drops by up to 5.7 points and falls below the base student on AMC23 and HumanEval+, and Teacher SFT drops by up to 6.5 points. SMC-KD$_1$ loses at most 1.2 points and stays above OPD on all five benchmarks, and SMC-KD$_{12}$ stays above OPD on four of the five.

On diversity, we agree that reduced diversity is a known trade-off of resampling (Doucet & Johansen, 2009). Resampling copies high-weight prefixes, which is what lets SMC select more accurate responses than BoN. The copies of a prefix are then extended independently, so they can still lead to different responses. To answer your question on how diverse the trajectories are, each 8-particle population in our training data ends with 2.8 distinct responses on average. The new Appendix F.4 studies island particles (Vergé et al., 2015), a simple way to improve diversity. Islands split the NN particles into groups that resample only within their own group, so at least one independent lineage per island survives to the end. As the number of islands grows, pass@NN increases in all 12 settings. An intermediate split also selects better responses than BoN in all 12 settings and than plain SMC in 7. Applying islands and other diversity-improving methods to SMC-KD in settings where diversity matters more is a promising direction for future work. Even without islands, this concentration does not prevent the trained students from benefiting from repeated sampling, since the one-shot SMC-KD$_1$ improves pass@8 over the base student in every pairing and outperforms OPD in three of the four (Table 15).

W2 (reasoning or compression). Compressing the benefit of search into pass@1 is the intended goal of SMC-KD. The motivation in Figure 1 is that small students already reach correct answers across several samples but not reliably in one, and we agree that part of the gain is of this kind (Appendix F.3). Two results suggest that SMC-KD does more than that.

First, it transfers beyond the training domain. SMC-KD is trained only on math prompts, yet SMC-KD$_{12}$ has the highest HumanEval+ accuracy among all methods in all four pairings, 2.6 to 6.7 points above the base student. It also outperforms OPD on IFEval in all four pairings. Compressing math search into math pass@1 would not by itself explain gains on code and instruction following.

Second, the one-shot variant does not lose coverage. On pass@8 (Table 15), SMC-KD$_1$ matches or exceeds OPD in three of the four pairings and stays above the base student in every pairing.

The lower pass@8 reported in the submission came mainly from the iterative variant SMC-KD$_{12}$ trained on all responses, where each round of SFT on long, often repetitive capped responses makes them more common in the next round. Removing capped responses from its training data raises its pass@8 in every pairing.

W3 (cost and use of teacher scores). We would like to clarify that, in our experiments, SMC-KD and OPD use the same generation and teacher-scoring budget.

  • Same generation and teacher budget. Both generate 8 responses of at most 2,048 tokens for each of the 12,000 prompts and the teacher scores every generated token in both. Resampling does not create additional responses. It only moves the same 8-particle budget toward promising prefixes. Running the teacher is shared by all distillation methods, and SMC-KD needs less from it, only the log-probability of each sampled token rather than the full distribution.
  • The teacher's scores are the training signal. They decide which prefixes are extended and how much weight each final response receives in the weighted SFT objective (Equation 6), the same role the per-token loss plays in OPD, but at the level of whole responses. Completed responses with low weight also stay in the objective, with a smaller weight.
  • Gains at this budget. SMC-KD$_{12}$ outperforms OPD in 18 of 20 combinations, by 2.0 points on average (95% interval [1.1,2.9][1.1, 2.9], Appendix F.5).
  • Reusing the teacher's signal on pruned particles. We agree this is an interesting direction. The partial trajectories removed by resampling have already been scored by the teacher but are not used for training, and their token-level teacher signal could serve as an additional source of learning. The Conclusion now lists this as future work. Such a term would complement SMC-KD's guided generation. The OPD (PG) baseline, which uses the same per-token teacher information on its own as a reward, is outperformed by SMC-KD$_{12}$ in 18 of 20 combinations.

Thanks for the detailed feedback. I decided to keep my score and believe the work could benefit from better use of the teacher data in the optimization stage.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank all reviewers for their constructive feedback. We are glad that all reviewers found the paper clear and easy to follow. Reviewer APgP found the idea interesting and a good way to improve exploration when collecting data for better students. Reviewer UUBe found SMC-KD a coherent way to use teacher likelihood during data generation without requiring a learned verifier or aligned token-level targets, and found our analysis of scorer compatibility and failure modes informative. Reviewer w8uB highlighted tokenizer-agnostic distillation, since unlike conventional OPD, SMC-KD does not require token-level vocabulary alignment between the teacher and student, and effective teacher-guided trajectory selection.

We have carefully revised the manuscript to address every comment. Beyond the requested changes, we added further baselines, benchmarks, and analyses that strengthen the paper. All changes are shown in blue in the revised paper, and the main ones are listed below.

  1. Additional baselines. To test whether simpler ways of building the training data can match SMC-KD, Table 1 now compares SMC-KD with three additional baselines on the same prompts, in addition to the previously reported OPD. Student SFT (teacher-scored BoN-plus-SFT) is SMC-KD$_1$ without block-by-block resampling of partial responses, so it isolates the effect of SMC. Teacher SFT tests direct imitation of the teacher. OPD (PG) uses the same sampled-token teacher log-probabilities as SMC-KD as a token-level reward, so it tests whether this limited teacher information is better used to guide generation or as a training signal.
  2. Additional evaluations. To test whether SMC-KD improves the student beyond the training domain, rather than only compressing math search into math pass@1, Table 1 adds HumanEval+ and IFEval and reports both the one-shot (SMC-KD$_1$) and iterative (SMC-KD$_{12})variantsforeverypairing.SMC−KD) variants for every pairing. SMC-KD_{12}$ outperforms OPD and the base student in 18 of the 20 pairing--benchmark combinations each, and has the highest HumanEval+ accuracy in all four pairings. Because several individual gains are small, Appendix F.5 adds paired bootstrap intervals over problems. On average, SMC-KD$_{12}$ exceeds OPD by 2.0 points (95% interval [1.1,2.9][1.1, 2.9]).
  3. Teacher-guided SMC beyond math. SMC-KD needs only the teacher's log-probabilities, so we checked that this signal is useful outside math. We run SMC and BoN on HumanEval+ (Figure 2, Appendix E) and repeat the likelihood-scoring analysis on code (Appendix D.2). SMC matches or exceeds BoN in all 16 HumanEval+ settings.
  4. Capped responses. While analyzing the iterative results, we found that responses reaching the 2,048-token budget accumulate across rounds once they enter the training data. SMC-KD therefore no longer trains on them, and Appendix F.2 compares the variants trained with and without these responses. The filter improves SMC-KD$_{12}$ in 15 of the 20 combinations, and with it iteration improves on the one-shot variant in 17 of 20.
  5. Population diversity. The reviewers asked whether resampling limits the diversity of the training data. Resampling concentrates the population on high-scoring prefixes. This concentration is why SMC selects more accurate final responses than BoN, and it can also reduce the number of distinct responses. Appendix F.4 studies island particles, which resample within subgroups so that several independent lineages always survive. Islands increase pass@NN in all 12 settings while preserving the selection advantage over BoN. Applying islands and other diversity-improving methods to SMC-KD in settings where diversity matters more is a promising direction for future work.
  6. PRM reference. A process reward model trained with correctness labels is the strongest guidance available, so it shows how much of that benefit a label-free teacher signal can recover. On MATH-500, we add SMC and BoN guided by Qwen2.5-Math-PRM-7B (Appendix E). Without any labels or separately trained verifier, teacher-guided SMC closes 39% to 97% of the gap between self-guidance and the PRM.
  7. Implementation details. To connect the implementation to the theoretical target and support reproducibility, Appendix B now gives checkpoints, chat templates, exact sampling settings, the proposal and target under truncation and finite length, tokenization, the OPD alignment for the cross-family teacher, the choice of KK, the training budgets, and GPU-hours.

Table numbers refer to the revised paper, where the main results (Table 2 in the submission) are now Table 1 and the hyperparameters (Table 3) are now Table 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.