- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Although the proposed framework is technically sophisticated, several important concerns remain. First, the fundamental premise of SKH, which assumes that task-specific structural importance can indicate regions of pretrained knowledge requiring protection, lacks sufficient theoretical motivation and empirical evidence. Second, the approach does not fundamentally address the scalability challenge of continual learning, as maintaining independent LoRA experts for each task may lead to increasing memory and inference costs. Finally, the experimental validation is conducted on relatively limited task sequences and does not adequately investigate long-term scalability, computational efficiency, memory consumption, or robustness under more diverse and challenging continual learning scenarios.
Critical strengths
The writting is generally good.
Critical weaknesses
-
SKH increases training complexity but incurs no analysis cost.
-
The issue of the unlimited growth in the number of LoRA experts has not been resolved.
-
RDSA's prototype routing may rely excessively on the visual domain.
-
There is a lack of theoretical explanation for relation distillation. Why does maintaining the relationship preserve task identity?
-
RDSA maintains a task prototype for each task and employs cosine similarity for routing. This approach relies on the implicit assumption that a task can be effectively represented by a single direction within the projected image feature space. For real-world, complex tasks, this assumption is highly questionable. A single task may encompass multiple visual-semantic clusters. For instance, medical datasets often contain a mix of organs, imaging modalities, and disease types. Relying on a single prototype risks "prototype averaging" or even causing the centers of distinct tasks to converge too closely.
-
The authors employ the projector from the previous stage as a "teacher," guiding the current projector to align its relation distribution regarding old prototypes with that of the previous stage. This effectively constitutes a "teacher chain," where the teacher from stage guides the projector at stage , while the teacher at is itself the product of prior distillation. Consequently, over long task sequences, there is a risk of cumulative error. However, as the paper’s benchmarks involve relatively short task sequences, it remains unproven whether stability can be maintained across sequences comprising dozens or even hundreds of tasks.
Questions for authors
Please see the weakness section.