- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Few-Shot Self-Distillation (FSSD) for on-policy instruction tuning of pretrained language models. A frozen copy of the base model, conditioned on task-relevant demonstrations, provides token-level supervision along student-generated responses without accessing the current instruction’s reference answer. Experiments on Qwen3 models from 1.7B to 8B show improved assistant performance, fewer regressions on general-capability benchmarks, and smaller distributional shifts from the pretrained model compared with standard SFT.
Critical strengths
- FSSD uses the base model’s in-context learning capability to construct a teacher without requiring a separate, stronger model. Conditioning on related demonstrations provides a practical alternative to reference-answer conditioning.
- The paper investigates demonstration count, category aware selection, reference-answer conditioning, self-generated SFT data, and fixed versus current policy trajectories. These comparisons help clarify the contributions of the method’s components.
Critical weaknesses
- All experiments are restricted to Qwen3 models with at most 8B parameters and use LoRA. The constrained parameter updates under LoRA may themselves limit distributional changes, making it unclear whether the observed capability preservation would persist under full-parameter fine-tuning. The current results therefore provide insufficient evidence of generalization across model families or applicability to full-parameter fine-tuning of larger models.
- Replacing reference responses with few-shot demonstrations may introduce additional practical constraints. Obtaining multiple relevant, high-quality examples from the same task category can be more demanding than obtaining a reference answer for a given instruction, particularly for specialized or novel tasks. Furthermore, a frozen base model may lack the capabilities needed to solve the task even with these demonstrations, making its token-level supervision potentially unreliable. The paper does not investigate these limitations sufficiently to establish whether the approach can scale to a broader range of tasks.
Questions for authors
Have you assessed the reliability of the few-shot-conditioned teacher, particularly on tasks where the base model has low accuracy? Is there evidence that FSSD remains beneficial in these cases rather than reinforcing the teacher’s errors, and how does its performance compare with SFT using the available reference answers?
3 comments
Thank you for the clarifications and additional experiments. The full-parameter fine-tuning results partially address my concern that the benefits might depend on LoRA. However, in Table B, the clearest evidence for capability preservation is concentrated on Math-Hard. On the other benchmarks, the differences between FSSD and SFT are generally modest or mixed, with SFT performing comparably or better in several cases. These results therefore do not yet establish a broadly consistent advantage, and generalization beyond the evaluated Qwen3 family remains uncertain.
Nevertheless, I appreciate the authors’ efforts to address the concerns. The clarification that demonstrations reuse existing SFT data, together with the ablations on demonstration selection, helps alleviate the concern about additional data requirements. Considering these responses and the new full-parameter results, I am willing to modestly increase my score, while retaining the above reservations.
We thank the reviewer for their follow-up and for acknowledging the additional experimental evidence.
To directly address the reviewer's remaining concern regarding generalization beyond the Qwen3 model family, we conducted additional experiments using Ministral-3-3B. The results show that FSSD's improvements in open-ended assistant performance and reduced distributional shift extend to this model family as well.
| Model | Method | AlpacaEval 2.0 | Arena-Hard v2 | KL div |
|---|---|---|---|---|
| Ministral-3-3B | Base | 5.91±0.87 | 30.8±1.1 | 0.000±0.000 |
| SFT | 4.97±0.80 | 33.8±1.2 | 0.083±0.002 | |
| FSSD | 7.86±0.97 | 38.7±1.1 | 0.023±0.001 |
These results provide further evidence that FSSD's benefits extend beyond the Qwen3 model family.
We thank the reviewer for their constructive feedback and for the opportunity to address their concerns.
W1: LoRA and model generalization. Our choice of Qwen3-1.7B, 4B, and 8B follows the experimental setup of [1], which evaluates the same model family and scales using LoRA. This allows us to investigate FSSD in an established experimental setting, while extending the study from instruction-tuned models to pretrained base models. To address the reviewer's concern regarding full-parameter fine-tuning, we additionally conducted experiments on Qwen3-1.7B using full-parameter fine-tuning instead of LoRA. As expected, the results are consistent with our main findings, and the main conclusions remain unchanged. In fact, without LoRA, we observe even more pronounced capability degradation. Thus, these results further support our decision to employ high-rank LoRA as a computationally efficient alternative to full-parameter fine-tuning.
[1] Zhao, Siyan, et al. "Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models." arXiv preprint arXiv:2601.18734 (2026).
Table A. Open-ended assistant performance with Full-FT. Scores (%) are mean ± SEM.
Table B. General-capability performance under full-FT. Scores (%) are mean ± SEM; Avg is the mean across eight benchmarks. ARC-C = ARC Challenge, Math-H = Math-Hard, MMLU-P = MMLU-Pro, WG = WinoGrande, HE = HumanEval.
W2: Demonstration requirements. FSSD does not require additional annotated data beyond standard SFT. It reuses existing instruction-response pairs as demonstrations, sampled from the training dataset. Moreover, our ablations show that even a single demonstration yields substantial improvements, and that FSSD remains effective with randomly selected demonstrations (36.0 on Arena-Hard v2 versus 39.2 with category-aware selection; Table 4). This suggests that category-matched demonstrations are beneficial but not strictly necessary.
Q1: Teacher reliability and low-accuracy tasks. While we do not directly measure teacher accuracy conditional on base-model failures, our results provide evidence that low base-model accuracy does not preclude benefits from FSSD. For example, on Math-Hard, Qwen3-4B improves from 39.5 (base) to 45.6 with FSSD, whereas SFT decreases performance to 33.9. Similar improvements are observed at 1.7B and 8B.