acceptodds
Under review as a conference paper at ICLR 2027

TRIBOS: Training Before Supervision for Continual Learning

Abstract

Applying supervised fine-tuning to sequential tasks is a standard practice in continual learning. However, direct SFT relies heavily on the independent and identically distributed assumption. Consequently, adaptation to newly encountered input distributions becomes entangled with the core objective of learning task semantics. As a result, the model may expend substantial parameter capacity on fitting superficial distributional variations, such as writing style and structural patterns, when processing continual text streams. This substantially dilutes the supervisory signal for core task semantics and leads to fragmented representations, thereby degrading continual-learning performance and exacerbating catastrophic forgetting. To address this problem, we introduce (TRIBOS), a two-stage optimization framework that explicitly decouples distribution adaptation from semantic alignment. For each incoming task, TRIBOS first performs prompt-only self-supervised adaptation using a causal language-modeling objective to absorb distributional shifts without label leakage. Subsequently, it applies answer-only SFT to learn the target mapping. Extensive experiments on the TRACE benchmark demonstrate that TRIBOS consistently outperforms existing parameter-efficient continual learning methods. Across three backbone models, TRIBOS significantly improves final average performance by 11.06 to 17.22 percentage points and reduces forgetting by 12.90 to 17.61 percentage points compared with Sequential LoRA. Furthermore, TRIBOS effectively preserves general knowledge and reasoning capabilities on multiple mainstream datasets after long task sequences.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, large language models, catastrophic forgetting, parameter-efficient fine-tuning, self-supervised adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K2FGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TRIBOS, a two-stage continual learning method that first adapts to new task inputs through self-supervised learning, then performs supervised fine-tuning. It uses Routed LoRA to separate the two stages. Results on TRACE show promising improvements in performance and forgetting across three backbone models.

Critical strengths
  1. The two-stage design is simple and intuitive, and the method is easy to follow.
  2. The results on TRACE are promising, and the ablation studies help explain the design choices.
Critical weaknesses
  1. Only one task order is evaluated. Since the method aims to reduce interference across task transitions, I think testing multiple task orders is important. The current results do not show how sensitive TRIBOS is to task ordering.
  2. The benefit of the two-stage design is relatively small. In Table 25, TTT→SFT improves Final Avg. by only about 1 point over SFT-only under the same architecture and nominal training budget. With vanilla LoRA, forgetting actually increases. This makes it difficult to tell how much of the main improvement comes from the two-stage design itself.
  3. The choice of inference scaling needs clarification. The authors mention that γ = 0.25 was selected after inspecting benchmark results. I would prefer to see γ selected on a separate validation set, or results consistently reported for γ = 0.25, 0.5, and 1.0.
  4. Most baselines and ablations are single runs. Given the relatively small gain in Table 25, results across multiple random seeds would help establish whether the improvement is consistent. This is separate from the task-order issue.
  5. Computational cost is not sufficiently reported. The paper focuses on parameter efficiency, but I would also like to see training time, GPU-hours, and memory usage compared with the baselines. Matching training passes does not necessarily mean matching compute.
  6. The evaluation is limited to TRACE. MMLU, GSM8K, and BBH measure general capability retention, but they are not additional continual-learning benchmarks. Testing another continual-learning benchmark would make the conclusions more convincing.
  7. Scalability remains unclear. The main experiments use Qwen3-8B, and the largest model tested is Qwen3-14B. A 32B or larger model would provide stronger evidence that the method scales.
Questions for authors
  1. Have the authors tested TRIBOS with different task orders? How consistent are the improvements in forgetting and final performance across different sequences?

  2. In Table 25, TTT→SFT improves Final Avg. by only about 1 point over SFT-only. Is this improvement consistent across random seeds? How much of the overall gain comes from the two-stage training rather than RoutedLoRA itself?

  3. Since γ = 0.25 was chosen after inspecting benchmark results, how would the main conclusions change if γ were selected on a separate validation set or fixed to 1?

  4. Could the authors report training time, GPU-hours, and peak memory usage compared with the main baselines? This would help assess the actual computational cost of the method.

Reviewer_Ej4VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that direct supervised fine-tuning (SFT) in continual learning couples prompt-distribution adaptation and answer-supervised learning, potentially interfering with the learning of task-specific semantics and exacerbating catastrophic forgetting. Inspired by Test-Time Training(TTT), this paper proposes TRIBOS, a two-stage framework that first performs prompt-only self-supervised adaptation to the input distribution, followed by answer-only supervised fine-tuning for task learning. The authors claim that this is the first integration of TTT into continual learning to explicitly decouple distribution adaptation from semantic alignment. Experiments on the TRACE benchmark demonstrate improved continual learning performance and reduced catastrophic forgetting across multiple backbone models.

Critical strengths
  1. Traditional SFT couples input distribution adaptation with task semantic learning within a single optimization process, requiring the model to simultaneously learn distributional features and task objectives. The authors argue that this may cause interference in the supervisory signal and thereby exacerbate catastrophic forgetting in continual learning.
  2. The authors evaluate TRIBOS on eight heterogeneous tasks from the TRACE benchmark using three backbone models. The experiments include comparisons with various continual learning methods, controlled ablation studies on training order and parameter allocation, and evaluations of general capability retention on MMLU, GSM8K, and BBH.
Critical weaknesses
  1. The novelty of the proposed two-stage training strategy seems to to incremental. As acknowledged in Section 2, task-adaptive pretraining already performs self-supervised adaptation on unlabeled task data before supervised fine-tuning. TRIBOS follows a similar paradigm by combining prompt-only TTT with answer-only SFT, without clearly demonstrating its methodological novelty over existing approaches in the context of continual learning.

  2. The independent contribution of the proposed two-stage training strategy is insufficiently demonstrated. As shown in Table 25, TTT→SFT improves Final Avg. by only 1.04 percentage points over SFT-only training under the same RoutedLoRA architecture, substantially less than the gains reported over Sequential LoRA. Moreover, applying TTT→SFT to vanilla LoRA increases forgetting from 23.91 to 32.26, while final average performance remains nearly unchanged (34.37 vs. 34.44). These results suggest that the reported improvements may depend heavily on the RoutedLoRA architecture, raising concerns about the independent effectiveness and generalizability of the proposed training strategy.

Questions for authors

1.The proposed two-stage training strategy closely resembles existing task-adaptive pretraining, which also performs self-supervised adaptation before supervised fine-tuning. Could the authors clarify the methodological differences between TRIBOS and task-adaptive pretraining, particularly what new contributions the proposed strategy introduces to continual learning?

2.The contribution of the two-stage training strategy remains unclear. As shown in Table 25, TTT→SFT improves Final Avg. only from 46.87 to 47.92 over SFT-only training under the same RoutedLoRA architecture, while its application to vanilla LoRA increases forgetting. Could the authors provide further evidence to distinguish the contributions of the training strategy and RoutedLoRA architecture to knowledge retention?

3.The independent contribution of the two-stage training strategy is insufficiently demonstrated. As shown in Table 25, TTT→SFT improves Final Avg. by only 1.04 percentage points over RoutedLoRA SFT-only training, substantially less than the gains reported over Sequential LoRA. Could the authors provide further evidence to clarify how much of the improvement comes from the two-stage training strategy rather than the RoutedLoRA architecture?

4.The hyperparameter selection procedure may introduce evaluation bias and undermine the fairness of experimental comparisons. As stated in Appendix A.5, the inference scaling factor γ = 0.25 was selected through sensitivity sweeps on the reported benchmarks rather than an independent validation set. This raises concerns about whether the reported performance gains are influenced by evaluation-based hyperparameter tuning. The authors should provide results using γ values selected on an independent validation set.

5.Figure 2 contains relatively small text, and the overall training workflow could be presented more clearly. Could the authors improve the figure's readability and provide a clearer explanation of the parameter updates and model state transitions between the two training stages and across tasks?

Reviewer_Y4isOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRIBOS, a two-stage training framework for parameter-efficient continual learning of large language models, aiming to mitigate catastrophic forgetting when learning new tasks.The authors argue that conventional supervised fine-tuning must simultaneously adapt to changes in the input distribution and learn task-specific semantic mappings. The coupling of these two objectives may introduce representational interference and exacerbate forgetting during continual learning.To address this issue, the paper introduces two sequential training stages, implemented using a fixed-capacity Routed LoRA architecture that assigns parameter updates to different components across the two stages. The authors evaluate the proposed method on the eight-task TRACE sequence using three backbone models, reporting final task performance, forgetting, and general capability retention.

Critical strengths
  1. The paper attempts to separate input-distribution adaptation from supervised task learning, investigating representational interference in continual learning from an optimization perspective. Compared with approaches that primarily rely on parameter regularization or task isolation to mitigate catastrophic forgetting, this work provides a different perspective on the problem.

  2. The fixed-capacity design, which requires neither experience replay nor task identity at inference time, offers practical value.

  3. The paper further investigates the effectiveness of the proposed method through ablation studies on training order, computational budget, and parameter update strategies, providing useful empirical evidence for understanding the role of the two-stage training approach.

Critical weaknesses
  1. In the architecture-matched comparison with a total budget of 48 training epochs and an inference scaling factor of 1, TTT→SFT achieves a Final Avg. of 47.92, compared with 46.87 for SFT-only, representing an improvement of only approximately 1.04 percentage points (Table 25). In contrast, the larger improvements reported in the main results use an inference scaling factor of 0.25, which was selected post hoc after inspecting performance on the reported benchmarks (Appendix E.4). These observations suggest that the main performance advantages of TRIBOS may depend substantially on the specific Routed LoRA architecture rather than solely on the proposed training order.

  2. The baselines in the main results are evaluated using single runs, whereas TRIBOS is evaluated over three runs. In addition, some comparisons involve differences in model capacity and inference settings, which may affect the fairness and reliability of the reported results.

Questions for authors
  1. Could the authors use a consistent, independent validation-based hyperparameter selection protocol for TRIBOS and the most relevant baselines, and report paired results under the same random seeds?
  2. Under matched architectures and computational budgets, is the approximately one-percentage-point improvement consistent across different random seeds and task orders?
  3. In the first task, the zero-initialized expert matrices prevent the shared projection and router from being updated during the strict TTT stage (Appendix C.5). How does this limitation affect the interpretation of the proposed two-stage optimization mechanism?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.