acceptodds
Under review as a conference paper at ICLR 2027

Continually Post-training Small Models via Synthetic Self-distillation

Abstract

Continual post-training is notoriously challenging in both acquisition and retention: models must learn new capabilities without forgetting prior knowledge. While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-following abilities. Consequently, SDFT fails on smaller models critical for low-latency, on-device use. To enable continual learning at smaller scales, we first identify the root cause of failure. We show that poor acquisition and forgetting correlate linearly with the post-trained model's distribution shift, revealing the necessity of in-distribution learning. We thus introduce Synthetic Self-Distillation (SynthSelf), imitating SDFT's in-distribution rewrite by outsourcing the task to a minimally larger, same-family aligner. Coupled with off-policy supervision and rejection sampling, this provides the high-quality supervision small models require without breaking their original distribution. Evaluations spanning **4** acquisition datasets (including a mixture) and **14** retention benchmarks show SynthSelf consistently outperforms **6** methods (including off- and on-policy distillation) across **3** small model scales. Extensive further analyses confirm SynthSelf paves the way toward effective small-model continual learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Self-distillation, Fine-tuning, Small Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_96XgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a tension in the continual training of small models. Training on off-policy data can often induce catastrophic forgetting and poorer generalization. On the other hand, smaller models can be poorer in-context learner which renders self-distillation-style approaches weak. This paper aims to bridge the gap between the benefits of on-policy data and high quality supervision by retaining off-policy supervision while rewriting it via an aligner model and performing rejection sampling to preserve correctness. They identify desiderate for the aligner model: a model whose generations induce the lowest loss, while simultaneously achieving higher performance. Practically, they find that slightly larger variants of the same model family tend to achieve the best tradeoff and that the largest aligner is not necessarily the best. They demonstrate comparable retention to existing methods for leveraging on-policy data, while demonstrating greater acquisition across 3 continual learning settings. They demonstrate that these gains primarily arise from rewriting full demonstrations in a closer distribution to the student, rather than merely leveraging the stronger capabilities of the aligner via ablations.

Critical strengths
  • Problem Formulation: This work tackles a problem of practical and conceptual significance. Cases where a model’s context-learning ability are too weak to provide adequate supervision represent a key limitation for self-distillation-based approaches (which are an otherwise promising approach for continual learning) and this paper highlights and takes steps toward addressing this limitation.
  • Simple and Well-Motivated Approach: The approach presented by the paper is simple and presented in a relatively general way and appears to be broadly applicable. Moreover, the authors justify their choices via analysis of the weaknesses of existing approaches. Additionally, ablations generally provide justification for the design choices. Thus, I feel that the approach is likely to provide benefits across multiple settings and be generally applicable.
  • Well-Studied Ablations and Comparisons: The authors compare against a comprehensive range of benchmarks and provide analysis of their relative advantages compared to both fine-tuning based approaches, as well as existing distillation based approaches. The results they present support their conceptual claims and motivation for their results. They also demonstrate cases where their approach does not improve over the baselines (when the base model retains sufficient capability to produce supervision via context).
Critical weaknesses
  • Novelty: This paper may be less novel relative to existing work on generating synthetic data, including studies on understanding the impact of different synthetic data generators, the benefits of scaling generators, etc. While this may not necessarily be a reason for rejection, I feel that the article can be improved if the authors were to comment on this connection to prior work including commenting on (a) their novelty relative to this field (either in approach or problem formulation) and (b) the extent to which their results confirm or reject existing findings in this setting. I believe that without clarification of this point, the contribution of this paper is hard to evaluate that the authors can clarify further.

  • Limited Number of Settings and Examination of Task Difficulty: The authors use a set of three settings based off of the Self-Distillation Enables Continual Learning paper. While this is an appropriate choice to make comparisons between their approach and the baseline, I believe that examining a larger gradient of task types/difficulties could make their claims stronger. They present a claim on requiring just a slightly larger model of the same family for performing alignment. However, I feel that making this claim thoroughly would require examining potentially more challenging tasks or maybe understanding how the minimal size of the aligner model varies across tasks. Intuitively, there must be some scaling of the task difficulty/novelty/etc with the model size necessary to achieve a task right? If my assertion here is incorrect, would be helpful for the authors to clarify.

  • SDFT+Rejection Sampling: Apologies if I have missed this in the paper, but is/could SDFT also be performed with rejection sampling in order to partially mitigate the impact of noisy supervision when using the same student/teacher? I understand that this is not the exact setting examined by the original SDFT paper and also it is not technically a baseline. However, I think that it would be of conceptual interest

  • Sequence of Continual Learning: As far as I can understand, the experiments in this paper use a single continual learning stage. However, one other setting that SDFT examines is a sequence of continued training stages. I wonder whether SelfSynth would encounter difficulty by drifting away from the aligner in this setting? I wonder if the authors could comment on whether this is a concern/potentially run some experiment establishing that this is not an issue.

Questions for authors

Please respond to the weaknesses I raised

Reviewer_U9oYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper proposes Synthetic Self-Distillation, a continual post-training method for small language models that aims to improve the acquisition of new capabilities while mitigating catastrophic forgetting. The authors first investigate the relationship between distribution shift and continual learning performance, and identify the limitations of existing self-distillation methods when applied to smaller models. Building on these observations, SynthSelf uses a larger model from the same family to rewrite training demonstrations, then applies rejection sampling and off-policy supervised fine-tuning. The method is evaluated on Qwen2.5 models at three scales, covering science, tool-use, and medical domains, together with 14 retention benchmarks.

Critical strengths

The paper addresses the acquisition–retention trade-off in continual post-training for small language models. The authors examine the relationship between distribution shift and catastrophic forgetting, providing empirical observations that motivate their approach. The analysis of why existing self-distillation methods become less effective at smaller model scales also helps establish the motivation for the proposed method.

Critical weaknesses
  1. The proposed method primarily combines existing techniques, including teacher-generated data rewriting, knowledge distillation, rejection sampling, and supervised fine-tuning. Although these components are motivated by the limitations of self-distillation on small models, the resulting training pipeline appears to be a relatively straightforward combination of established approaches. The main distinction lies in selecting a slightly larger model from the same family to generate distribution-aligned demonstrations. It remains unclear whether this constitutes a sufficiently substantial methodological contribution compared with existing teacher–student distillation and synthetic data generation methods. The paper would benefit from a clearer explanation of its technical novelty beyond the combination and empirical validation of these components.
  2. All student models in the main experiments belong to the Qwen2.5 family, and the proposed approach relies heavily on the assumption that larger models from the same family generate more distribution-aligned supervision. However, it is unclear whether this observation generalizes to other model families with different architectures, pre-training procedures, or distillation strategies.
  3. Despite emphasizing continual post-training, the main experiments primarily evaluate model performance after adapting to a single target domain or a fixed mixture of domains. This setting does not fully capture the challenges of continual learning, where models sequentially acquire multiple tasks or capabilities and must preserve previously learned knowledge over successive updates. In particular, the paper does not investigate how performance evolves across multiple training stages or whether forgetting accumulates as additional tasks are introduced. A sequential learning evaluation involving multiple domains and training stages would be necessary to demonstrate the effectiveness of SynthSelf in more realistic continual learning scenarios.
  4. Although the paper includes several fine-tuning and distillation baselines, it does not sufficiently isolate the benefits of same-family alignment from those of teacher-generated synthetic data and data filtering. In particular, a stronger comparison would involve conventional teacher-based SFT with matched training data sizes, filtering criteria, and generation budgets. Additionally, while the rejection sampling ablation demonstrates its effectiveness, the paper does not fully examine how different filtering thresholds, rejection rates, or the number of generated candidates affect the acquisition–retention trade-off. These experiments would help determine whether the improvements arise from the proposed distribution-alignment mechanism or from more general improvements in supervision quality.
  5. A central claim of the paper is that reducing distribution shift is critical for improving acquisition and retention. However, the analysis relies primarily on empirical correlations between KL divergence and downstream performance, which do not establish a causal relationship. Furthermore, the proposed aligner selection criterion uses the negative log-likelihood of generated demonstrations under the base model as a proxy for distribution alignment, but the relationship between this criterion and subsequent forgetting is not formally established.
Questions for authors

Please see the above comments.

Reviewer_87xxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the acquisition-retention trade-off in continual post-training of small language models. In this paper, the authors observe that Self-Distillation Fine-Tuning (SDFT) becomes less effective at smaller model scales, attributing this degradation to weaker in-context learning and self-supervision capabilities. They introduce SYNTHSELF, a framework that uses a larger, same-family aligner to rewrite training demonstrations into outputs that are more compatible with the student model's distribution. The rewritten demonstrations are filtered using answer-based rejection sampling, followed by standard supervised fine-tuning. Experiments on Qwen2.5 models at 0.5B, 1.5B, and 3B scales evaluate acquisition across science, tool-use, medicine, and a science mixture, alongside retention on 14 benchmarks. Results demonstrate improvements over SFT, SDFT, DFT, ASFT, and off-/on-policy distillation.

Critical strengths
  1. There is a strong practical motivation and promising empirical results. The proposed solution is straightforward and practical, requiring only offline rewriting, filtering, and SFT rather than a more complex on-policy training procedure. The experiments also demonstrate meaningful acquisition improvements across several domains.
  2. The analyses are quite useful as well, with good baselines. Beyond standard baseline comparisons, the paper studies different aligner models, compares off-policy and on-policy distillation, evaluates prompt-matched SDFT variants, and ablates rejection sampling. These experiments provide useful evidence that the method is not solely benefiting from a larger teacher model or a particular prompting strategy. The observation that smaller, same-family aligners can outperform substantially larger models while reducing generation costs is particularly interesting from an efficiency perspective.
Critical weaknesses
  1. The experiments do not sufficiently isolate the proposed distribution-alignment mechanism. A central claim is that SYNTHSELF succeeds primarily because same-family rewriting produces supervision closer to the student's distribution. However, several aspects of the experimental design make this conclusion difficult to establish: (1) The primary comparisons against off-policy and on-policy distillation in Table 1 are not information-matched. (2) The distributional alignment measure in Eq. 4 is the negative log-likelihood of a single rewritten sequence under the base model, which, from my understanding, should be a divergence between the full teacher and student output distributions.
  2. Limited validation of continual learning and generalization beyond the evaluated setting. Although the paper is motivated by continual post-training, the main experiments evaluate isolated single-stage fine-tuning rather than sequential acquisition of multiple capabilities. Consequently, it remains unclear whether SYNTHSELF continues to preserve earlier acquired skills when a model is repeatedly updated on new domains. Furthermore, all student experiments use one model family, Qwen2.5, with three relatively small model sizes. The reported experiments also use a single random seed, without variance estimates or statistical significance analysis. The gains are also not uniform across settings. For example, the improvements in medicine are relatively modest, and some individual configurations provide limited gains over the original model or stronger baselines.
Questions for authors
  1. Could the authors include information-matched baselines in which the off-policy distillation teacher has access to the same gold demonstration as SYNTHSELF?
  2. Additionally, how does SYNTHSELF compare against standard SFT performed on the same subset of examples retained by rejection sampling?
  3. How does SYNTHSELF perform when the same model is trained sequentially on multiple domains? Maybe something like science -> tool-use -> medicine?
  4. Could the authors clarify whether the NLL reported in Fig. 4 and Table 2 is normalized by generated sequence length?
  5. Could the authors report results across multiple training seeds and, ideally, at least one additional model family?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.