- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TRIBOS, a two-stage continual learning method that first adapts to new task inputs through self-supervised learning, then performs supervised fine-tuning. It uses Routed LoRA to separate the two stages. Results on TRACE show promising improvements in performance and forgetting across three backbone models.
Critical strengths
- The two-stage design is simple and intuitive, and the method is easy to follow.
- The results on TRACE are promising, and the ablation studies help explain the design choices.
Critical weaknesses
- Only one task order is evaluated. Since the method aims to reduce interference across task transitions, I think testing multiple task orders is important. The current results do not show how sensitive TRIBOS is to task ordering.
- The benefit of the two-stage design is relatively small. In Table 25, TTT→SFT improves Final Avg. by only about 1 point over SFT-only under the same architecture and nominal training budget. With vanilla LoRA, forgetting actually increases. This makes it difficult to tell how much of the main improvement comes from the two-stage design itself.
- The choice of inference scaling needs clarification. The authors mention that γ = 0.25 was selected after inspecting benchmark results. I would prefer to see γ selected on a separate validation set, or results consistently reported for γ = 0.25, 0.5, and 1.0.
- Most baselines and ablations are single runs. Given the relatively small gain in Table 25, results across multiple random seeds would help establish whether the improvement is consistent. This is separate from the task-order issue.
- Computational cost is not sufficiently reported. The paper focuses on parameter efficiency, but I would also like to see training time, GPU-hours, and memory usage compared with the baselines. Matching training passes does not necessarily mean matching compute.
- The evaluation is limited to TRACE. MMLU, GSM8K, and BBH measure general capability retention, but they are not additional continual-learning benchmarks. Testing another continual-learning benchmark would make the conclusions more convincing.
- Scalability remains unclear. The main experiments use Qwen3-8B, and the largest model tested is Qwen3-14B. A 32B or larger model would provide stronger evidence that the method scales.
Questions for authors
-
Have the authors tested TRIBOS with different task orders? How consistent are the improvements in forgetting and final performance across different sequences?
-
In Table 25, TTT→SFT improves Final Avg. by only about 1 point over SFT-only. Is this improvement consistent across random seeds? How much of the overall gain comes from the two-stage training rather than RoutedLoRA itself?
-
Since γ = 0.25 was chosen after inspecting benchmark results, how would the main conclusions change if γ were selected on a separate validation set or fixed to 1?
-
Could the authors report training time, GPU-hours, and peak memory usage compared with the main baselines? This would help assess the actual computational cost of the method.