acceptodds
Under review as a conference paper at ICLR 2027

MixSD: Mixed Contextual Self-Distillation for Knowledge Injection

Abstract

Supervised fine-tuning (SFT) is widely used to inject new knowledge into language models, but it often degrades pretrained capabilities such as reasoning, instruction following, and general-domain performance. We argue that this forgetting arises because standard fine-tuning targets are written by humans or external systems whose outputs diverge from the model’s own autoregressive distribution, forcing the optimizer to imitate low-probability token sequences. To address this problem, we propose MixSD, a simple teacher-free method for distribution-aligned knowledge injection. Instead of training on fixed targets, MixSD constructs supervision dynamically by mixing tokens from two conditionals of the base model itself: an expert conditional that observes the injected fact in context, and a naive conditional that reflects the model’s original prior. The resulting supervision sequences preserve the factual learning signal while remaining substantially closer to the base model's distribution. We evaluate MixSD on two synthetic corpora that we construct to study factual recall and arithmetic function acquisition in a controlled setting, together with established benchmarks for open-domain factual question answering and knowledge editing. Across multiple model scales and settings, MixSD consistently achieves a substantially improved memorization-retention trade-off compared to SFT and on-policy self distillation baselines, retaining up to 100% of the base model’s held-out capability while maintaining near-perfect training accuracy, whereas standard SFT retains as little as 21%. We further show that MixSD produces substantially lower-NLL supervision targets under the base model and reduces harmful movement along Fisher-sensitive parameter directions. These results suggest that aligning supervision with the model’s native generation distribution is a simple and effective principle for knowledge injection that mitigates catastrophic forgetting.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Knowledge Injection, Catastrophic Forgetting, Supervised Fine Tuning, Self Distillation, Parametric Knowledge, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_PykUOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MIXSD, a self-distillation method for injecting new knowledge into language models while reducing catastrophic forgetting. MIXSD constructs training sequences by mixing tokens from an expert conditional that observes the target knowledge and a naive conditional that reflects the base model’s original behavior, followed by standard NLL training on the verified mixed outputs. Experiments on synthetic factual and arithmetic tasks, SimpleQA, and MQuAKE show that MIXSD generally achieves a better knowledge-acquisition and capability-retention trade-off than SFT, OPSD, LoRA, and several knowledge-editing baselines across multiple model scales.

Critical strengths
  • MIXSD provides a simple and intuitive way to inject new knowledge while keeping the supervision closer to the base model’s original generation distribution.

  • The experiments are comprehensive, covering factual recall, arithmetic function learning, open-domain knowledge, knowledge editing, multiple model scales, and several relevant baselines and ablations.

  • The paper provides useful analyses beyond aggregate accuracy, including target likelihood, parameter-update directions, error patterns, training-seed variation, and the acquisition-retention trade-off.

Critical weaknesses

i) The experiments do not clearly isolate the effect of mixed token-level sampling from the other changes introduced during data construction.

MIXSD replaces the short answers used by standard fine-tuning with substantially longer reasoning sequences generated by the base model. The resulting training data therefore differ not only in how tokens are sampled, but also in sequence length, style, total training tokens, and optimization steps. More importantly, the =0 variant already provides much of the reported retention improvement in several settings, even though it uses only the expert-conditioned distribution and performs no mixing with the naive distribution. This makes it difficult to determine how much of the gain actually comes from the proposed mixed contextual distillation.

Prior work has shown that training on self-generated data can itself preserve a language model’s existing capabilities [a]. The paper should therefore include controlled experiments that match sequence length, training-token count, and optimization steps across variants. Comparisons between mixed sequences and equally long sequences generated entirely from the expert or base-model distribution would help establish whether token-level mixing provides an additional benefit.

ii) The method relies heavily on verification and repeated generation, but the contribution of this selection process is not separated from the proposed distillation mechanism.

MIXSD retains a generated sequence only when it passes a task-specific correctness check, with up to 100 generation attempts for each example. Table 6 reports high retry rates in several settings, while the paper does not report the average number of attempts or generated tokens required for each accepted sequence. This filtering process changes the training distribution by keeping responses that the base model can already express correctly, and it may account for part of the improvement attributed to MIXSD. Related work has shown that selecting or regenerating self-produced rationales according to answer correctness can substantially improve downstream training [b].

The dependence on automatic verification also limits the current evidence for broader knowledge injection. Exact checks are relatively straightforward for the factual and functional tasks considered here, but many realistic updates involve long-form, ambiguous, or open-ended knowledge for which correctness cannot be determined automatically. The paper should separate the effects of mixing and verification through an ablation without rejection filtering, report the full generation cost, and explain how the method would operate when a reliable verifier is unavailable.

[a] Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal. ACL 2024.

[b] STaR: Bootstrapping Reasoning With Reasoning. NeurIPS 2022.

Questions for authors
  1. When sequence length, total training tokens, and optimization steps are matched, does mixed token-level sampling still outperform sequences generated entirely from the expert-conditioned or base-model distribution?

  2. How much of the improvement comes from verification and repeated sampling rather than contextual mixing, and how would MIXSD be applied when no reliable automatic verifier is available?

Reviewer_6Q7MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MixSD, a method for teaching language models new knowledge while reducing catastrophic forgetting by changing the answers used for supervised fine-tuning (SFT). For each training question, a fixed copy of the original model generates a new answer one token at a time: at each step, it proposes a token with the correct answer provided in context and another token without that information, both conditioned on the same generated prefix, and randomly chooses between them. After verifying the completed answer’s correctness, MixSD uses the original question as the SFT input and the generated answer as the target, so the student learns to answer without receiving the ground truth in its context. Experiments primarily on Qwen3 models across synthetic factual and arithmetic datasets, SimpleQA, and MQuAKE show that this approach often preserves more general math, coding, and knowledge capability than standard SFT while learning new information, although stronger mixing can reduce knowledge acquisition.

Critical strengths
  1. The method shows substantial improvements in the acquisition–retention trade-off in compared to standard SFT and this seems to benefit better on larger model - among all the tested model the Qwen3 8B model has smallest loss (even some shows small gain) on OOD domains, and the tradeoff cost little on knowledge injection performance.
  2. The paper uses controlled datasets and experiment to help distinguish learning from forgetting. Synthetic dataset and fictional entities reduce reliance on pretrained factual knowledge. Retrieval-based evaluation tests whether the model can still use supplied facts. And the OOD datasets include various math datasets of different difficulty and coding datasets. I appreciate the clear and transparency on testing these important domains rather than only the MMLU and iFEval which can hide a lot of capability loss.
  3. The error and parameters analyses provide interesting insights on how models learn and forget. I find the observation interesting that capability forgetting correlate less with overall magnitude changes in parameters rather the movement in a small group of sensitive parameters.
Critical weaknesses
  1. In table 1 and 2 author claims presentation of "Bold and underline indicate the best and second-best methods in each column, respectively" excludes Base. However, MixSD remains below Base on average held-out capability for the 1.7B and 4B models in both main tables; in Table 2, this also holds for 8B. That means substantial forgetting remains even with the proposed method, and this presentation emphasizes relative improvements too much. Reporting overall capability loss relative to Base alongside acquisition would make the practical trade-off clearer.
  2. The main experiments use synthetic datasets like KGFact and KGFunc which are good to provide this controlled experiment setting but it would be better to demonstrate the method in more realistic knowledge acquisition with maybe real world knowledge and documents with certain date cutoff to prevent contamination. The simpleQA experiment is moving towards more realistic knowledge learning but the mixsd method seems to hurt more on factual accuracy compared to normal SFT compared to the synthetic setting (like the drop from mixedsd is more severe than in synthetic case).
Questions for authors
  1. Question 1: see weakness 1. Even if the proposed method is better than others, it still sacrifices a lot on math capabilities and that leads to overall performance lower than Base on most of setups. My conclusion from this might be finetuning based knowledge injection then is just not worth pursuing as retrieval-based method doesn't have any such problem? What's author's opinion or justification for fine-tuning based knowledge injection work? Alternatively, in strength I have pointed out the observation that this seems to get better on larger models. If the authors can add even larger size models like 14B 27B and if in that regime we see no general capability loss that would also be encouraging.
  2. See weakness 2: it would be nice to provide more supporting evidence that this method works as good on realistic knowledge injection.
Reviewer_M3KgOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MixSD, a framework that injects new knowledge into LLMs while mitigating catastrophic forgetting. It argues that standard SFT fails not simply because parameters change, but because human- or externally authored targets often lie off the base model’s autoregressive distribution, forcing learning from low-probability tokens and damaging prior capabilities. MixSD replaces fixed targets with self-generated mixed supervision: at each step, it samples from an expert conditional that sees the injected fact and a naive conditional that reflects the model’s prior. The resulting Bernoulli-mixed sequence is not a strict teacher joint distribution. It provides token-level, low-NLL alignment that preserves factual signal while reducing harmful update directions. Experiments show improved memorization-retention trade-offs over SFT and OPSD.

Critical strengths
  • The paper is clearly presented, with a well-motivated problem formulation linking catastrophic forgetting to target-distribution mismatch and an intuitive method description that makes MixSD easy to understand and position.

  • MixSD shows consistent improvement over SFT and OPSD in the memorization-retention trade-off, maintaining strong knowledge acquisition while preserving more held-out general capability across most reported settings.

Critical weaknesses
  • My major concern lies in the fact that MixSD’s core alignment operates at the token level: it lowers base-model NLL by constructing Bernoulli-mixed trajectories from expert and naive conditionals. However, this artificial mixture is not a natural teacher joint distribution, and token-level plausibility may not fully guarantee high-level semantic stability of knowledge injection or forgetting mitigation. In particular, it remains unclear whether the same mixed supervision can consistently preserve teacher-like behavior while robustly injecting new knowledge across more complex, conflicting, or long-horizon settings.

  • I notice that MixSD relies on verification, retry, and discard steps to ensure correct mixed targets. This extra filtering may change the training distribution and complicate direct fairness comparisons with SFT baselines trained on unfiltered canonical targets.

Questions for authors
  • Would semantic- or phrase-level mixing, rather than token-level Bernoulli mixing, better preserve coherent teacher-conditioned trajectories while maintaining MixSD’s memorization-retention trade-off and improving the stability of high-level knowledge injection?

  • How much of MixSD’s improvement comes from the mixing strategy itself, versus the additional verification, retry, and discard pipeline used to filter generated supervision? The authors could add controls such as SFT trained on the same verified subset, MixSD without retry, MixSD without discard, and compute-matched verification budgets.

  • How sensitive is MixSD to the choice of λ\lambda across datasets, model scales, and knowledge types, especially when the injected knowledge is more complex or conflicting?

Reviewer_ZqvqOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MIXSD, a teacher-free knowledge-injection method that constructs supervision by token-level Bernoulli mixing between an expert conditional, which sees the gold fact in context, and a naive conditional, which does not. The mixed targets are trained with standard NLL. The authors build KGFACT and KGFUNC for controlled factual and arithmetic injection, and also evaluate SimpleQA and MQuAKE. Experiments use Qwen3-1.7B/4B/8B and Llama-3.2-1B against SFT and OPSD, with additional analyses of target NLL, Fisher alignment, error modes, seeds, LoRA, and compute.

Critical strengths
  1. The controlled KGFACT/KGFUNC setup, multiple scales, and seed checks are a genuine strength; they separate acquisition from retention and make the forgetting claim testable.

  2. The NLL, Fisher-alignment, and error-mode analyses give a plausible mechanism and are more informative than aggregate accuracy alone.

Critical weaknesses
  1. Limited novelty. The method is essentially context distillation plus verified self-training: an expert conditional with the gold fact, a naive conditional, and Bernoulli token mixing. The two closest uncited works: Bai et al. (2021), A General Language Assistant as a Laboratory for Alignment (context distillation), and Shenfeld et al. (2026), Self-Distillation Enables Continual Learning, are not discussed or compared, so the claimed new principle is not established.

Refs: [1] Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., ... & Kaplan, J. (2021). A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. [2] Shenfeld, I., Damani, M., Hübotter, J., & Agrawal, P. (2026). Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897.

  1. Narrow and confounded evaluation. Experiments cover only Qwen3-1.7/4/8B and Llama-3.2-1B, with no Qwen3.5/3.8-class backbone; a compute-matched context-distillation/rejection-sampling baseline is also missing. Because MIXSD uses up to 100 retries and verification, its gains over SFT/OPSD are confounded with extra generation and filtering.

  2. How is λ selected for each row in Tables 1–2 without using held-out capability, and do the gains survive a single fixed λ across settings?

  3. How much of the gain comes from retry/verification? Retry rates reach 91%; please report SFT/context distillation on the same verified subset and with matched generation compute.

  4. The experiments are text-only and cover only Qwen3-1.7/4/8B and Llama-3.2-1B; there is no multimodal LLM. Since the paper frames MIXSD as a general knowledge-injection principle, this limits the claim; at minimum one MLLM probe (e.g., LLaVA-1.5-7B or Qwen3-VL or Qwen3.5) should be added.

  5. The paper should discuss [a,b,c], which cover layer-wise injection, encoder-decoder knowledge injection, and knowledge editing, respectively.

Refs: [a] Jiang, K., Ding, Y., Ren, Y., Jiang, N., Gao, Z., Zheng, Z., ... & Li, Q. (2026, April). When large multimodal models confront evolving knowledge: Challenges and explorations. In International Conference on Learning Representations (Vol. 2026, pp. 72306-72351). [b] Chen, T., Tan, Z., Gong, T., Wu, Y., Chu, Q., Liu, B., ... & Yu, N. (2024, November). Llama slayer 8b: Shallow layers hold the key to knowledge injection. In Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 5991-6002). [c] Jiang, K., Jiang, H., Jiang, N., Gao, Z., Bi, J., Ren, Y., ... & Li, Q. (2025). KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Augmentations and Constraints.

Questions for authors

See weaknesses.

Comments on the paper

4 comments

Thank you for the detailed responses and additional experiments. The new context-distillation controls provide stronger evidence for the benefit of mixing, and the reported retry counts and generation costs are helpful. These additions clarify several of my concerns and strengthen my confidence in the empirical results. Exact training-token matching and reliance on verification remain limitations. I maintain my original rating of 3 (weak acceptance).

Thank you for the additional experiments and clarifications. The expanded comparisons across backbones and datasets address most of my concerns. I will keep my overall score unchanged and increase my confidence.

Thank you! We are pleased that the additional experiments have addressed your concerns. Your suggestions really helped us strengthen the experimental evaluation and improve the paper!

We thank all reviewers for their constructive feedback! Reviewers found MixSD simple and intuitive (PykU, M3Kg), the controlled KGFact/KGFunc setup a genuine strength (Zqvq, 6Q7M), the gains over SFT and OPSD consistent, especially on larger models (M3Kg, 6Q7M), and the NLL, parameter-update and error analyses informative (Zqvq, 6Q7M, PykU). We summarize how we addressed the main concerns:

  1. Whether mixing helps beyond context distillation and SDFT (Zqvq, PykU): Against context distillation controls matched in length or chosen as the lowest-NLL of 16 samples, MixSD retains more at every matched step on Qwen3-1.7B and has the highest held-out average in 6 of 9 dataset–backbone settings. SDFT keeps much less held-out ability than MixSD at every scale. (Revised: Section 5 and 6; Appendix E.3–E.5.)
  2. Verification, retries and compute cost (Zqvq, M3Kg, PykU, 6Q7M): Verification and retries are part of MixSD and filter out no question. MixSD needs 1.0–3.8 attempts per question, generation costs less than 1/10 of one training run on almost every dataset and backbone. (Revised: Appendix F.1 and H.3.)
  3. Realistic knowledge with a date cutoff, and acquisition on SimpleQA (6Q7M, M3Kg): On Wiki2026 (1,200 questions from 390 Wikipedia articles about 2026 events, uploaded in the supplementary material), MixSD learns nearly as many facts as SFT while keeping 1.7–4.7× its held-out ability. On SimpleQA trained for three epochs, MixSD keeps far more held-out ability than SFT at matched acquisition on all three models. (Revised: Abstract; Section 1, 5 and 6; Appendix J.2 and J.3.)
  4. Remaining gap to Base, and larger, newer and multimodal backbones (6Q7M, Zqvq): We added a Δ vs. Base column to Tables 1–2 and six new backbones. On Qwen3-14B, Qwen3-32B and Qwen3-VL-8B, MixSD stays within 0.3, 3.8 and 1.5 points of Base, whereas SFT loses 29.1, 26.0 and 37.1 points. On the multimodal benchmark MMEVOKE, MixSD learns more than SFT and forgets less. (Revised: Tables 1–2; Section 5 and 6; Appendix J.1 and J.4.)
  5. Mixing granularity and sensitivity to λ (M3Kg, Zqvq): Phrase-level and clause-level mixing stay within 2 points of token-level mixing on every model. With one fixed λ=0.3, MixSD keeps more held-out ability than SFT in all 15 dataset–model settings, by +9.8 to +62.6 points. (Revised: Appendix G.3 and H.1.)
  6. Relation to prior work (Zqvq, PykU): We now discuss context distillation, SDFT, SSR, STaR, MMEVOKE, KORE and SLayer. These works change how the model is updated or train on targets fully conditioned on the hint, while MixSD changes what the model is trained on. (Revised: Section 2.)

We have uploaded a revised PDF in which all changes are highlighted in blue. We believe MixSD brings a simple and effective principle to knowledge injection, and we thank the reviewers and the Area Chair again for their time and valuable suggestions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.