acceptodds
Under review as a conference paper at ICLR 2027

Online Representation Preservation for Continual Instruction Tuning

Abstract

Continual instruction tuning requires language models to acquire new capabilities while maintaining performance on previously learned tasks. Replay trains the model on stored or selected responses, whose prefixes may differ from those generated by the current student. We propose ReTAP (Representation-based Task Anchoring and Preservation), which matches the student’s hidden states to those of frozen, task-specific reference models on freshly sampled student responses. Before each training stage, each reference is selected from completed student checkpoints based on validation performance on the corresponding task. This representation loss is combined with supervised learning on the current task, without requiring independently trained expert models. On the eight-task TRACE benchmark, ReTAP achieves the highest final average performance and backward transfer among the evaluated continual-learning methods across all three model backbones. Its average scores fall only 0.91, 1.47, and 0.32 points below independently trained single-task benchmarks on Qwen2.5-7B-Instruct, Qwen3-1.7B, and Qwen3-4B, respectively, compared with deficits of 2.85, 4.23, and 2.71 points for the best competing method on each backbone. These results demonstrate effective historical capability preservation within a single continually updated model.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
fine-tuning, large language model, in-context learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_98eTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReTAP replaces fixed-answer replay with aligning the student's hidden states to those of a frozen, task-specific reference model on the student's own online-generated responses, thereby sidestepping the mismatch between the prefixes of stored answers and those the student actually generates.

Critical strengths

All reference models are drawn from the existing checkpoints of the same training run (no separately trained experts, and no added inference-time cost), and it consistently achieves the highest Avg and BWT across three backbones, four capacities, and three replay ratios, while being nearly insensitive to K and yielding robust conclusions.

Critical weaknesses

I consider this a competent and rigorous paper, but not one deserving of the highest rating. The method is not particularly novel, essentially combining hidden-state alignment with the characteristics of SDFT, and it offers no distinctive highlight. The experiments show good results mainly on LoRA, while the gains under FFT remain limited, which constrains the paper's potential contribution to the community.

Questions for authors

Core unresolved issue: The authors do not adequately justify the connection between prefix mismatch and hidden-state alignment. Existing work attributes hidden-state alignment mainly to alignment at the knowledge and reasoning levels (DPRD), which appears to contradict the prefix-mismatch claim. I went through the entire paper and found Section 3.4, yet it does not resolve my confusion; the theoretical grounding is very thin.

The experimental setup is largely reasonable, and I have no further requests there. I would be glad to raise my score if the authors can resolve my core concern. I would also suggest the authors report the method's additional overhead, particularly runtime and GPU cost, since this matters for others trying to gauge the time complexity of adopting it.

4 comments

Thank you for recognizing the rigor of our experiments, the practicality of using historical references, and the continual-learning results. We particularly appreciate your question about the relationship between prefix mismatch and representation alignment. We address this central concern first, followed by the relationship to SDFT, applicability to full fine-tuning, and computational overhead.

Q1: How is prefix mismatch related to hidden-state alignment? Does this conflict with knowledge- or reasoning-level preservation?

The two interpretations are compatible: the prefix distribution determines where preservation constraints are applied, while hidden-state alignment determines what is preserved at those states.

Concretely, the current student generates a response to a historical-task prompt. The student and the frozen historical reference then process exactly the same response prefixes, and their hidden states are aligned at corresponding positions. The reference does not need to independently generate the same response; it provides supervision on prefixes already visited by the student.

For example, when the student answers an old-task question using wording different from the stored answer, its subsequent generation follows different prefixes. Preservation constraints on the stored answer do not directly cover these prefixes. ReTAP instead evaluates the historical reference on the student-generated prefixes and constrains the student’s representations there.

This explains how the knowledge/reasoning interpretation relates to our motivation: knowledge- or reasoning-level preservation concerns the content that representation constraints may preserve, whereas prefix coverage concerns the input states on which those constraints operate. The same hidden-state loss can be applied to either fixed answers or student-generated responses; their distinction lies in the distribution of supervised states.

To further examine the empirical value of representation preservation, we supplemented the existing reverse-KL comparison with forward-KL output distillation:

Experiment groupHistorical preservation objectiveλAvg ↑BWT ↑
Original paperBest tested reverse KL1064.0145−2.6064
Original paperRepresentation MSE1066.6580−0.2132
Additional experimentsBest tested forward KL5063.6158−2.7955
Additional experimentsRepresentation MSE1066.1161−0.8285

The additional forward-KL experiments tested λ ∈ {1, 10, 50}; the table reports the best tested configuration. Within the respective comparison groups, MSE improves over reverse KL by 2.64 Avg / 2.39 BWT points and over forward KL by 2.50 Avg / 1.97 BWT points.

These results support the practical value of representation preservation in our setting.

Q2: Is ReTAP simply a combination of hidden-state alignment and SDFT?

Our contribution concerns how to use capabilities retained in the history of a single continual-training run to construct preservation targets for multiple previous tasks while learning the current task. The distinctions from SDFT are concrete:

  • Learning objective: SDFT uses additional information from demonstrations to support task learning. ReTAP introduces explicit historical-task preservation constraints alongside current-task cross-entropy training.
  • Reference source: SDFT provides its teacher with demonstration-conditioned information. ReTAP uses historical checkpoints from the same training run as references for previous tasks, without separately training task experts.
  • Supervision and role: SDFT transfers supervision through output-distribution distillation. ReTAP evaluates historical references on the current student’s old-task trajectories and uses their representations as preservation targets.

These choices have complementary roles: historical checkpoints specify the reference capabilities, student trajectories specify the states receiving preservation constraints, and the representation loss implements those constraints. Together, these components form a continual-preservation procedure that uses historical capabilities to support learning across tasks.

Q3: Are the benefits largely restricted to LoRA, with limited gains under FFT?

We added complete eight-task FFT experiments on Qwen3-1.7B and Qwen3-4B, including OPR-RU as a comparator. Each cell below reports Avg / BWT, using seed 1234:

MethodQwen3-1.7BQwen3-4BQwen2.5-7B
Replay53.7150 / −4.806462.6204 / −1.031565.7609 / −2.4659
OPR-RU55.7308 / −2.385863.7859 / −0.761365.5404 / −2.3761
ReTAP:56.7478 / +0.413564.0296 / −0.560866.5442 / −1.2158

ReTAP outperforms both Replay and OPR-RU in Avg and BWT on all three evaluated models under FFT. Relative to OPR-RU, ReTAP improves by 1.02 Avg / 2.80 BWT points on Qwen3-1.7B, 0.24 Avg / 0.20 BWT points on Qwen3-4B, and 1.00 Avg / 1.16 BWT points on Qwen2.5-7B.

Compared with Replay, the corresponding improvements are 3.03 Avg / 5.22 BWT, 1.41 Avg / 0.47 BWT, and 0.78 Avg / 1.25 BWT points, respectively. These results support the applicability of ReTAP to FFT across the evaluated model sizes, with improvements in both final performance and historical retention.

Q4: What is the additional computational overhead, and how can it be reduced?

The additional computation mainly comes from response generation on historical prompts, reference-model forward passes, and reference selection in the default configuration. These operations occur during training. Deployment uses only the final student model and adds no teacher inference path.

We evaluated a simplified K1 / last1 / Task-end configuration: one sampled response per historical prompt, final-layer representation supervision, and a fixed reference saved immediately after learning each historical task. This removes validation-based reference selection.

On two A800-80GB GPUs, three paired seed/task-order settings give the following results; ± denotes sample standard deviation:

MethodAvg ↑BWT ↑Mean algorithm time ↓
OPR-RU64.5780 ± 0.9213−2.6324 ± 0.98167.5726 h
OPR-SC64.0813 ± 1.0441−3.5795 ± 0.74677.5417 h
ReTAP: K1 / last1 / Task-end66.5728 ± 0.4764−0.4912 ± 1.03086.8685 h

The simplified ReTAP configuration improves Avg by 1.99 / 2.49 points over OPR-RU/SC while reducing mean algorithm time by 9.30% / 8.93%, respectively. All three paired settings show improvements in Avg, BWT, and algorithm time. Compared with the default K8 / all-layer / Selected configuration’s mean of 14.8809 hours in these experiments, the simplified configuration reduces algorithm time by approximately 53.84%.

The reported algorithm time covers training-stage computation, including model training, response generation, reference-model forward passes, and reference selection.

These explanations clarify the complementary roles of prefix coverage and representation preservation. The output-distillation comparisons, FFT extensions, and three paired runtime measurements provide concrete evidence addressing your concerns about the mechanism, contribution, and practical cost.

Thank you again for your thoughtful feedback. We hope that these clarifications and additional results address your concerns, and we would greatly appreciate your reconsideration of the score in light of this response.

Dear Reviewer,

Thank you for taking the time to read our paper and share your thoughtful comments. We considered each of your points carefully and responded to them individually.

We hope our responses address your concerns. If there is anything we have not fully addressed or explained clearly, we would be happy to discuss it further.

Thank you again for your time and helpful feedback.

Best regards,
The Authors

Dear Reviewer 98eT, thank you for pressing us on the connection between prefix mismatch and hidden-state alignment. If you have a chance before the discussion closes, we’d especially welcome your thoughts on Q1 in our response above. The example there illustrates how student-generated prefixes determine where supervision is applied and how hidden-state alignment defines the training signal at those prefixes. We’d be happy to clarify anything that remains unclear.

Reviewer_1iKPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ReTAP studies the problem of catastrophic forgetting during continual instruction tuning. For the current task, the model uses the standard supervised cross-entropy loss. For previous tasks, it samples new answers from historical prompts and matches the hidden states of the current student model with the representations from frozen historical checkpoints selected for each task. ReTAP achieves the best final average performance and backward transfer on eight tasks from TRACE and three Qwen backbones. Its performance is also close to that of separately trained single-task models.

Critical strengths
  1. The motivation is clear and the method design makes sense. ReTAP preserves representations based on the answer prefixes actually generated by the current student, which avoids the problem in traditional replay methods where fixed historical answers may not match the current student's generation trajectory.
  2. The experiments cover a relatively wide range of settings. The paper evaluates the method on different types of tasks, three backbones, and several design choices. It also provides ablation studies on the number of rollouts, preservation strength, supervision layers, replay ratio, LoRA capacity, and output distillation.
Critical weaknesses
  1. There is no comparison under a matched compute budget. By default, ReTAP samples (K=8) answers for each historical prompt and also requires extra forward passes for the student and reference models. However, the paper mainly matches the number of historical samples across methods. It is hard to tell whether the performance gains are enough to justify the extra computation.
  2. Checkpoint selection based on the validation set may bring additional gains. ReTAP selects a reference checkpoint for each historical task based on its validation score, and this choice directly affects the training target. The paper does not provide enough ablation studies comparing validation-selected references with the latest, fixed, random, or merged references.
  3. The novelty compared with existing checkpoint-teacher methods is not clear enough. GTR-Turbo[1] has already proposed using historical checkpoints as free teachers to guide later online training. Although ReTAP uses task-specific checkpoints and hidden-state matching, it does not directly compare with this type of method. The paper also does not sufficiently show why these differences should be considered a separate core contribution.

[1] Wei T, Yang Y, Zhang C, et al. Gtr-turbo: Merged checkpoint is secretly a free teacher for agentic VLM training[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Questions for authors

See Weakness

2 comments

Dear Reviewer,

Thank you for taking the time to read our paper and share your thoughtful comments. We considered each of your points carefully and responded to them individually.

We hope our responses address your concerns. If there is anything we have not fully addressed or explained clearly, we would be happy to discuss it further.

Thank you again for your time and helpful feedback.

Best regards,
The Authors

Dear Reviewer 1iKP, thank you for raising the questions about compute fairness and checkpoint selection. If time permits before the discussion closes, we’d be grateful if you could take a look at the parts of our response on training cost and the reference-selection ablations. Please feel free to let us know if either point would benefit from further clarification.

Reviewer_6yPuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual instruction tuning of LLMs on a sequence of tasks, where a single model must learn each new task while retaining earlier ones. The authors argue that replay-based methods supervise fixed answer prefixes that can drift away from the prefixes the current student actually generates, and that the latest checkpoint may already have forgotten earlier tasks. They propose ReTAP. At each stage, a small set of historical prompts is drawn. For each previous task, a frozen reference is chosen from the completed student checkpoints (LoRA adapters) according to validation score on that task. The current student samples K fresh responses to each historical prompt, and the loss matches the student's hidden states to those of the reference on the same sampled sequences. Experiments use the 8-task TRACE benchmark with Qwen2.5-7B-Instruct, Qwen3-1.7B and Qwen3-4B, three seed/task-order pairs, and baselines Sequential SFT, Replay, OPR-RU, OPR-SC and an adapted LwF. ReTAP reaches the best average score and backward transfer (BWT), within 0.3-1.5 points of single-task models.

Critical strengths

Consistent and sizeable gains under a carefully controlled protocol. On all three backbones ReTAP has the best Avg and BWT in Table 1 with BWT near zero against -2.5 to -5 for the replay variants. The runs use three paired seed/order settings, reported SDs and a common evaluator, and the authors avoid mixing in numbers from other evaluation protocols. The advantage also holds across replay ratios (Figure 4) and LoRA ranks (Figure 5), and the stage-wise tables in Appendix E show less forgetting on harder tasks such as MeetingBank. This is the main reason the paper deserves attention.

A simple, practical recipe with clear implementation details. ReTAP needs no external expert, only LoRA snapshots over a shared base, and inference uses a single adapter. The optimizer, decoding limits, sampling procedure and baseline adaptations are documented in Appendix A, and the λ and K sweeps show the method is fairly insensitive once λ is large enough (Figure 3). Selecting a reference per task from the model's own checkpoints is a sensible and inexpensive way to avoid training experts. A practitioner could adopt such a recipe easily.

Critical weaknesses

The contribution relative to the closest prior work is thin and is not separated empirically. The paper's own Related Work describes OPRD as "closely related": it already aligns teacher and student hidden states on student-generated rollouts. What remains specific to ReTAP is (i) applying on-policy hidden-state matching to continual instruction tuning and (ii) choosing each task's reference among the model's own earlier checkpoints by validation score. This is a reasonable engineering combination, but it is closer to a recombination of known components than to a new principle, and Proposition 1 adds little explanatory power.

The evidence base is narrow, and tuning, cost and fairness are under-documented. All results are on one benchmark (TRACE, eight tasks), with backbones up to 7B and LoRA as the main setting. The one full fine-tuning run (Figure 5, single seed) shows a margin over Replay of only 0.78 Avg, much smaller than under LoRA, so generality beyond the main setting is not established.

Questions for authors

Could you compare against OPRD, GKD-style distillation on historical prompts, and SDFT under the same TRACE protocol and budget? This is the most direct way to show what ReTAP adds over existing on-policy distillation methods.

How were λ, K and the layer set chosen, and on which split? Since Table 4 is single-seed and λ=5 outperforms the default λ=10 in Avg, could you report multi-seed results for the λ and layer sweeps, and describe the tuning done for the baselines (LwF coefficient, OPR settings, learning rate)?

6 comments

Thank you for recognizing the consistent improvements and practical appeal of using the model’s own historical checkpoints. We address your questions about the contribution beyond existing distillation methods, the scope of the empirical results, and the choice and cost of the training configuration through the additional experiments below.

1. Contribution beyond OPRD: task-specific preservation throughout continual learning

OPRD establishes hidden-state alignment on student-generated trajectories. ReTAP builds on this operation to address a specific continual-learning problem: which historical behavior should be preserved as a shared model learns successive tasks?

At stage t, ReTAP combines current-task instruction learning with representation preservation on historical-task prompts. For a historical task j, the current student generates a response, and both the student and the historical reference process that same response. Their representations are aligned at corresponding token positions. The reference comes from the learner’s own checkpoint history, allowing different historical tasks to retain different reference behaviors without training separate task experts.

This distinction matters because a single recent checkpoint may already have forgotten an earlier task. A task-specific reference preserves an explicit target for that task while the student continues learning. Accordingly, the contribution is the continual preservation formulation and its historical-reference mechanism, with student-rollout representation alignment serving as the underlying operation.

We examined the reference mechanism directly. Keeping K=8 and all-layer supervision fixed, we compared task-specific Selected references against Latest, which uses the immediately preceding stage’s checkpoint for all historical tasks:

Reference policyAvg ↑BWT ↑
Latest66.27 ± 0.36−0.79 ± 0.49
Selected66.80 ± 0.59−0.31 ± 0.58

Results are mean ± sample standard deviation over three paired seed/task-order configurations.

Task-specific selection improves Avg by 0.52 points and BWT by 0.47 points on average, supporting the value of task-specific historical references.

2. Comparison with output-level distillation: both forward and reverse KL

To examine the preservation objective, we evaluated output-level distillation on student-generated historical responses using both KL directions. The original reverse-KL experiments and the additional forward-KL experiments are reported separately, each with its corresponding representation-MSE result:

Experiment groupHistorical preservation objectiveAvg ↑BWT ↑
Original experimentsReverse KL, best tested coefficient64.01−2.61
Original experimentsRepresentation MSE66.66−0.21
Additional experimentsForward KL, best tested coefficient63.62−2.80
Additional experimentsRepresentation MSE66.12−0.83

Representation MSE improves Avg/BWT by 2.64/2.39 points over reverse KL and 2.50/1.97 points over forward KL in the respective experiment groups. These results support representation preservation as an effective objective within the historical-reference training pipeline.

This comparison also clarifies the relationship with GKD-style training: both can use student-generated trajectories, while the preservation target differs—output distributions versus hidden representations. SDFT uses demonstration-conditioned self-distillation to support learning from demonstrations; ReTAP supplies task-specific historical representation targets during a continual task sequence.

3. Computational cost: strong retention with one rollout, one supervised layer, and no teacher search

We evaluated a streamlined configuration with:

  • K=1: one sampled response per historical prompt;
  • last-layer supervision;
  • Task-end references: task j always uses the checkpoint saved immediately after learning task j, eliminating checkpoint-selection evaluation.

Across three paired seed/task-order configurations on Qwen2.5-7B with LoRA, this configuration obtains:

MethodAvg ↑BWT ↑Training-pipeline time ↓
OPR-RU64.58 ± 0.92−2.63 ± 0.987.57 ± 0.12 h
OPR-SC64.08 ± 1.04−3.58 ± 0.757.54 ± 0.14 h
ReTAP: K1 / last1 / Task-end66.57 ± 0.48−0.49 ± 1.036.87 ± 0.35 h

The streamlined configuration improves Avg by 1.99 points over OPR-RU and 2.49 points over OPR-SC, while reducing measured runtime by 9.3% and 8.9%, respectively. The improvements in Avg, BWT, and runtime hold in all three paired configurations.

Runtime was measured on two A800 GPUs and covers training-stage computation, including model training, response generation, reference-model computation, and checkpoint selection where applicable.

Thus, the observed performance advantage remains when using one rollout, supervising only the final layer, and removing teacher search. ReTAP also retains a single student model for inference.

4. Full fine-tuning: additional evidence across three model sizes

We added complete eight-task FFT experiments on Qwen3-1.7B and Qwen3-4B, including OPR-RU as a comparator. Each cell below reports Avg / BWT, using seed 1234:

MethodQwen3-1.7BQwen3-4BQwen2.5-7B
Replay53.7150 / −4.806462.6204 / −1.031565.7609 / −2.4659
OPR-RU55.7308 / −2.385863.7859 / −0.761365.5404 / −2.3761
ReTAP:56.7478 / +0.413564.0296 / −0.560866.5442 / −1.2158

ReTAP outperforms both Replay and OPR-RU in Avg and BWT on all three evaluated models under FFT. Relative to OPR-RU, ReTAP improves by 1.02 Avg / 2.80 BWT points on Qwen3-1.7B, 0.24 Avg / 0.20 BWT points on Qwen3-4B, and 1.00 Avg / 1.16 BWT points on Qwen2.5-7B.

Compared with Replay, the corresponding improvements are 3.03 Avg / 5.22 BWT, 1.41 Avg / 0.47 BWT, and 0.78 Avg / 1.25 BWT points, respectively. These results support the applicability of ReTAP to FFT across the evaluated model sizes, with improvements in both final performance and historical retention.

5. Hyperparameter selection and sensitivity

Thank you for asking for a fuller account of hyperparameter selection and sensitivity. We examine the preservation-loss weight, rollout count, and supervised layers separately, and use repeated experiments to assess whether a simpler configuration retains the benefits.

(a) Preservation-loss weight λ: multiple values achieve strong accuracy and retention.

The coefficient λ controls the weight of historical representation preservation relative to current-task learning. We tested {0.01, 0.1, 1, 5, 10, 20, 50}. The following results use the same seed/task-order configuration:

ConfigurationAvg ↑BWT ↑
Replay62.90−3.70
λ=0.0159.69−7.61
λ=0.161.49−6.08
λ=164.81−1.75
λ=566.80−0.41
λ=10 (default)66.66−0.21
λ=2065.96+0.31
λ=5067.07+0.54

Two observations follow. First, very small preservation weights correspond to poor retention, consistent with insufficient historical supervision. Second, at the four tested values 5, 10, 20, and 50, Avg ranges from 65.96 to 67.07 and BWT from −0.41 to +0.54; every configuration improves Avg over Replay by at least 3.06 points and BWT by at least 3.29 points. The gains extend across multiple preservation weights spanning an order of magnitude.

Compared with λ=10, λ=5 achieves 0.14 points higher Avg, while λ=10 achieves 0.20 points higher BWT. Both settings deliver strong final accuracy and historical retention.

We used λ=10 as a common default specified before these experiments. The sensitivity analysis shows strong performance across λ ∈ {5, 10, 20, 50}.

(b) Rollout count K: repeated experiments show strong performance with a single rollout.

Keeping the other settings fixed, we evaluated K ∈ {1, 2, 4, 8, 16}, with three paired seed/task-order configurations for each value:

KAvg ↑BWT ↑
166.83 ± 0.24−0.28 ± 0.63
266.90 ± 0.42−0.15 ± 0.80
466.48 ± 0.21−0.61 ± 0.36
866.70 ± 0.32−0.40 ± 0.34
1666.86 ± 0.37−0.29 ± 0.92

Results are mean ± sample standard deviation.

Across the tested rollout counts, mean Avg varies by 0.42 points, with no monotonic improvement as the number of samples increases. In particular, K=1 achieves mean Avg and BWT comparable to the original K=8 default.

We therefore use K=1 in the additional low-cost experiments, maintaining strong performance while reducing the number of responses generated and processed for each historical prompt.

(c) Supervised layers: the final layer already provides an effective preservation signal.

We added a layer comparison with a consistent historical-reference policy, fixing K=1 and λ=10 and changing only the supervised layers:

Supervised layersAvg ↑BWT ↑
All layers66.11−0.73
Final layer66.27−0.62
Final four layers66.09−0.20

Avg differs by at most 0.18 points across these configurations. Final-layer supervision achieves the highest Avg, while supervising the final four layers achieves the best BWT. These results support final-layer supervision as an effective and efficient preservation strategy.

The final hidden state directly feeds into the output prediction, giving final-layer preservation a concrete behavioral interpretation; the experiment provides empirical evidence of its effectiveness.

We further combined K=1, final-layer supervision, and λ=10 with the Task-end reference policy, which requires no teacher search. Across three paired seed/task-order configurations, this setup achieves 66.57 ± 0.48 Avg and −0.49 ± 1.03 BWT, compared with 64.58 ± 0.92 and −2.63 ± 0.98 for OPR-RU. This establishes a practical result: reducing rollout count and supervised layers while removing teacher search still yields strong overall performance across repeated runs.

The K sweep and the combined streamlined configuration each use three repeated runs; each individual λ and layer configuration is evaluated in one complete task-sequence run.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.