acceptodds
Under review as a conference paper at ICLR 2027

LMATE: Learning New Tasks without Forgetting through Teacher-Guided Revision

Abstract

Continual instruction tuning seeks to fine-tune an instruction tuned model to a new target task while avoiding to lose its general capabilities. Supervised Fine-Tuning (SFT) trains the model to mimic ground-truth demonstrations from the dataset that may differ substantially from their existing response distribution, causing unnecessary changes and leading to catastrophic forgetting. We present LLM as a Teacher (LMATE), which adopts a teacher/student approach that constructs task-informed training targets from the student’s own answers. For each prompt, the student generates an initial response, and a stronger teacher with access to the ground-truth demonstration provides targeted feedback. Conditioned on its initial response and this feedback, the student produces a revised response through in-context learning. We then fine-tune the student on these revised answers. To limit learning from suffixes that may drift after an early correction, we apply the training loss only to the revised prefix ending at the -th token-level difference. This concentrates the update on the earliest teacher-guided corrections while excluding more extensively revised suffixes. We evaluate LMATE on scientific question answering, tool use, and factual knowledge acquisition, measuring retention with IFEval and MMLU-Pro. LMATE achieves a competitive adaptation–retention trade-off across these tasks. On Tool-Use it reaches 66.19% accuracy compared with 61.03% for SFT, and for Timely Events it reaches 22.44 F1-score compared with 17.19 for SFT, while retaining higher IFEval and MMLU-Pro accuracy on both datasets.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Large Language Model, post-training, continual instruction tuning, teacher-student methods

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_VFkjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies instruction tuning for new-task adaptation while retaining general capabilities. It proposes LMATE, which constructs training targets through three rounds of teacher-guided revision: a stronger teacher accesses the reference demonstration and critiques a frozen student's response, and the student revises its answer using this feedback. The student is then fine-tuned on the revised responses, with the loss restricted to a prefix ending at the k_diff-th token-level edit relative to the initial response. The paper also introduces SELFTEACHSFT, a mixture of reference-model completions and ground-truth demonstrations. Experiments use Qwen2.5-3B-Instruct on chemistry question answering, tool use, and recent-event knowledge acquisition, measuring retention with IFEval and MMLU-Pro. LMATE improves over SFT on tool use and timely events, although its advantage over SELFTEACHSFT is task-dependent and substantial forgetting remains in the skill-learning settings.

Critical strengths
  1. A well-motivated approach to adaptation with retention. Teacher-guided revision and prefix supervision offer a practical way to incorporate task information while limiting changes to student-generated responses.

  2. Evaluation of both adaptation and retention. The paper reports results across three tasks with multiple random seeds and includes a useful simple baseline. LMATE improves both target performance and retention over SFT on Tool-Use and Timely Events.

Critical weaknesses
  1. Limited evidence for the incremental contribution. The main ideas build on existing critique-guided training and self-distillation approaches. The ablations do not sufficiently establish the added value of student-side revision and the edit-bounded prefix objective.

  2. Insufficient explanation of task-dependent performance. On Science Q&A, LMATE has mixed results relative to SFT and lower mean scores than SELFTEACHSFT across all three metrics. The paper does not adequately explain when or why its additional mechanisms are beneficial.

  3. Unclear computational cost-benefit trade-off. LMATE requires a larger teacher and multiple feedback–revision rounds, but the additional data-construction cost is not quantified. Its practical advantage over the simpler SELFTEACHSFT baseline therefore remains unclear.

  4. Insufficient evidence for distribution preservation. Limiting token edits does not necessarily keep the training targets close to the student's distribution. The paper does not directly validate that this mechanism limits distribution shift and thereby reduces forgetting.

Questions for authors
  1. Can you clarify whether the existing experiments include training on the full revised responses, and provide a comparison against a prefix control matched for supervised-token count? The retained-prefix length distribution and loss normalization would help determine whether edit locations, rather than reduced supervision alone, account for the gains.

  2. Can you provide a matched-retention comparison with SELFTEACHSFT and report the end-to-end cost of constructing the LMATE targets? This would clarify whether LMATE improves the practical trade-off over the simple baseline, particularly given the Science Q&A and Timely Events results.

Reviewer_69RmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LMATE adapts a language model by training it on its own answers after revision using feedback from a larger teacher. The teacher sees the ground-truth answer, while the student uses the feedback to revise its response. Training loss is restricted to the revised prefix ending at the k_diff-th changed token. The paper also introduces SelfTeachSFT, which mixes ground-truth answers with frozen-reference outputs. Experiments cover science QA, tool use, and recent factual knowledge, with IFEval and MMLU-Pro measuring retention.

Critical strengths

1.LMATE improves on standard SFT in two settings: Tool-Use accuracy rises from 61.03 to 66.19, and Timely Events F1 from 17.19 to 22.44, with better retention in both cases (Tables 1 and 3). Reporting five-seed means and standard deviations makes these results more informative.

2.I found SelfTeachSFT a useful contribution. It is simple, competitive, and provides a clear way to study the adaptation–retention trade-off. The prompts and pseudocode also make the methods easy to follow.

3.The analysis of the revision process is helpful (Appendix G, Figure 4). Feedback improves answer quality much more on Science Q&A than Tool-Use, and extra rounds do not always help Timely Events. This gives a useful picture of where teacher feedback works and where it falls short.

Critical weaknesses
  1. The case for the full LMATE pipeline is weak. SelfTeachSFT with γ = 0.1 has higher means on all three Science Q&A metrics (Table 2). On Timely Events, it reaches 32.09 F1 versus LMATE's 22.44, although LMATE retains more general ability (Table 3). LMATE's Tool-Use accuracy gain over this baseline is only 1.04 points. Given the 32B teacher and three revision rounds, I would expect a clearer benefit at comparable adaptation or retention, supported by a cost comparison. The paper does not establish when the extra complexity is worthwhile.

  2. The central prefix-masking idea lacks a direct test. Appendix H varies k_diff, but does not report comparisons with full-response training on the same revised answers or an ordinary prefix with a matched supervision length. I therefore cannot tell whether identifying early edits helps, or whether the result comes from supervising fewer tokens. This is a missing test of the main contribution, and the current ablation does not resolve it.

  3. The knowledge-acquisition results are hard to interpret. Section 6.2 notes that formatting failures affect Timely Events scores, while Appendix G shows that longer answers can lose F1 despite containing correct information. Without format-validity rates and a clear answer-extraction procedure, it is hard to tell how much of the reported gain reflects learning new facts rather than changes in output format or length.

Questions for authors

1.Can you compare the proposed mask with full-response SFT and a length-matched prefix baseline, using the same revised answers?

2.At similar target-task performance or retention, where does LMATE improve over SelfTeachSFT, and how much additional generation and training time does it require?

3.Section 6.2 and Appendix G note that formatting and answer length affect Timely Events F1. How are answers extracted, and does LMATE's advantage over SFT remain after accounting for these effects?

1 comment

As the author has not yet responded to my questions, the original score remains unchanged.

Reviewer_zecuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors are focused on the problem of continually training an LLM to improve on new tasks while mitigating catastrophic forgetting on old ones. They specifically focus their attention on prior work that indicates that SFT can cause forgetting on old tasks and aim to devise methods that can achieve a better trade-off between adaptation and forgetting. To this end, they propose two methods: SelfTeachSFT, which performs SFT on a mixture of ground-truth solutions and initial model generated rollouts, and LMATE, which involves performing SFT on trajectories generated by conditioning on feedback given by a stronger teacher on initial model solutions. The authors test their method on three adaptation tasks.

Critical strengths
  • The initially proposed method of SelfTeachSFT is lightweight and appears competitive with (if not better than when the hyperparameter is set to 0.1) the actually proposed LMATE method and, by extension, the other baselines that the authors considered.
  • The problem of mitigating catastrophic forgetting is important.
Critical weaknesses

While the paper reads as a good-faith effort to provide a contribution in the space of mitigating catastrophic forgetting during finetuning, the paper has several critical flaws.

The most important issue concerns baselines and novelty.

  • Firstly, with respect to baselines, I would have expected there to be a proper RL baseline with GRPO. One of the key results within this space from the past year is that RL provides a strong alternative over SFT in terms of improving model performance while retaining task capabilities. The authors should be aware of this, as they do cite Shenfeld et al's "RL’s Razor: Why Online Reinforcement Learning Forgets Less" paper.
  • Secondly, however, if we are to restrict our attention to pure SFT finetuning, then the "Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting" paper of Chen et al (which the authors also cite) both provides relevant baselines and harms this paper's novelty. In this paper, the authors investigate explore alternative finetuning formulations to SFT, such as Self-SFT (where the model is finetuned on outputs generated by the initial policy, similar to the proposed SelfTeachSFT) and Iterative-SFT, where new SFT trajectories are sampled from the current policy each epoch. Given that both of the proposed methods involve some form of SFT on generations by the initial policy, this is an important omission.

Even when we ignore these issues, however, there are other problems:

  • The paper's framing of its contribution is rather odd. Is SelfTeachSFT or LMATE the contribution? If it is the latter, then the writing introducing SelfTeachSFT as a contribution only to negate that two paragraphs later in the introduction is confusing. Crucially, SelfTeachSFT is competitive with LMATE despite require far fewer resources.
  • On the matter of resources, LMATE is expensive. It requires not only access to ground-truth solutions but also a stronger LLM to provide feedback for the student. This is exacerbated by the fact that there already exists cheaper alternatives (like GRPO) or whether, for more difficult tasks such as math or agentic settings, the student can correctly respond to the teacher's feedback.
  • The authors additionally make various design decisions without empirically justifying them in the main paper. Section 4.1 focuses on the importance of the teacher focusing on the first error the student makes. Are there any experiments investigating whether having the teacher focus on all errors is better or worse? Similarly, the authors propose an "edit-bound prefix-tuning" element to training. This not only complicates regular SFT but is not ablated in the main paper.
  • Performing experiments exclusively with Qwen2.5-3B is an issue. How do results generalize to different scales or model families?

Given these, it is sadly hard for me to suggest acceptance to the current state of the paper.

Questions for authors

How does GRPO or the various SFT alternatives considered in "Retaining by Doing" perform against the proposed method(s)?

1 comment

I am maintaining my scores due to the lack of response from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.