Online Representation Preservation for Continual Instruction Tuning

Continual instruction tuning requires language models to acquire new capabilities while maintaining performance on previously learned tasks. Replay trains the model on stored or selected responses, whose prefixes may differ from those generated by the current student. We propose ReTAP (Representation-based Task Anchoring and Preservation), which matches the student’s hidden states to those of frozen, task-specific reference models on freshly sampled student responses. Before each training stage, each reference is selected from completed student checkpoints based on validation performance on the corresponding task. This representation loss is combined with supervised learning on the current task, without requiring independently trained expert models. On the eight-task TRACE benchmark, ReTAP achieves the highest final average performance and backward transfer among the evaluated continual-learning methods across all three model backbones. Its average scores fall only 0.91, 1.47, and 0.32 points below independently trained single-task benchmarks on Qwen2.5-7B-Instruct, Qwen3-1.7B, and Qwen3-4B, respectively, compared with deficits of 2.85, 4.23, and 2.71 points for the best competing method on each backbone. These results demonstrate effective historical capability preservation within a single continually updated model.