TRIBOS: Training Before Supervision for Continual Learning

Applying supervised fine-tuning to sequential tasks is a standard practice in continual learning. However, direct SFT relies heavily on the independent and identically distributed assumption. Consequently, adaptation to newly encountered input distributions becomes entangled with the core objective of learning task semantics. As a result, the model may expend substantial parameter capacity on fitting superficial distributional variations, such as writing style and structural patterns, when processing continual text streams. This substantially dilutes the supervisory signal for core task semantics and leads to fragmented representations, thereby degrading continual-learning performance and exacerbating catastrophic forgetting. To address this problem, we introduce $\text{TR}\text{a}\text{I}\text{ning}$ $\text{B}\text{ef}\text{O}\text{re}$ $\text{S}\text{upervision}$ (TRIBOS), a two-stage optimization framework that explicitly decouples distribution adaptation from semantic alignment. For each incoming task, TRIBOS first performs prompt-only self-supervised adaptation using a causal language-modeling objective to absorb distributional shifts without label leakage. Subsequently, it applies answer-only SFT to learn the target mapping. Extensive experiments on the TRACE benchmark demonstrate that TRIBOS consistently outperforms existing parameter-efficient continual learning methods. Across three backbone models, TRIBOS significantly improves final average performance by 11.06 to 17.22 percentage points and reduces forgetting by 12.90 to 17.61 percentage points compared with Sequential LoRA. Furthermore, TRIBOS effectively preserves general knowledge and reasoning capabilities on multiple mainstream datasets after long task sequences.