- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoLoRA-Med, a continual learning framework for medical vision-language models. It decomposes LoRA adapters into shared and task-specific branches to balance knowledge transfer and catastrophic forgetting. The method preserves previously learned task-specific LoRA components, conditionally aligns related task updates or separates conflicting ones through orthogonality constraints, and protects important shared parameters through regularization. At inference, a question-based router selects the appropriate task-specific LoRA without requiring ground-truth task identity. The method is validated on 11 medical datasets across seven imaging modalities and five tasks.
Critical strengths
The paper is well motivated by the need for medical vision-language models to learn heterogeneous tasks over time, rather than repeatedly adapting to one task type. Learning classification, multi-label classification, detection, cell counting, and report generation in one continual-learning stream makes the problem practically relevant. The evaluation is also reasonably broad, covering 11 datasets, seven imaging modalities, two backbones, and several continual LoRA baselines.
Critical weaknesses
- The methodological novelty is not sufficiently clear relative to existing continual LoRA methods. The overall design of shared adaptation with task-specific components has clear precedent in methods such as CL-LoRA, HiDe-LLaVA, and LiLoRA. In addition, the proposed orthogonality constraint is closely related to O-LoRA, while the protection of important shared parameters is similar in spirit to RegLoRA-style regularization. The main novelty therefore seems to lie in combining these mechanisms and conditionally applying them based on update similarity, rather than introducing a fundamentally new continual-learning structure. The paper should better distinguish this contribution from these closely related approaches.
- The query-only routing mechanism is not fully convincing. The router selects the task-specific LoRA using only textual features from the question, without considering the image. Since the five tasks have very different question formats, the reported 99.96% routing accuracy may largely reflect easily identifiable task-specific wording. It is unclear whether this routing strategy would remain reliable when different tasks use similar prompts or when task identity depends on image content. A harder routing evaluation with overlapping prompts or a comparison with multimodal routing would strengthen the paper.
- The experimental evidence does not fully support a consistent advantage of the proposed method. CoLoRA-Med is not consistently the best-performing method in Table 1, particularly on InternVL, while the abstract highlights improvements over the average of five low-rank continual learning baselines rather than over the strongest baseline. Moreover, the ablation study mainly reports normalized retention metrics rather than actual task-level performance. Since strong normalized retention does not necessarily imply strong task learning, it would be important to report the task-level metrics for each ablation under the same setting as Table 1. This would better show whether the proposed components improve retention without sacrificing plasticity.
Questions for authors
- Could the authors report the ablation results using the same training setting as Table 1 and include the actual task-level metrics, rather than only normalized retention metrics?
- Could the authors clarify how the shared and task-specific LoRA components are combined at inference, merged or computed separately?
- The experiments use a single task order. Could the authors evaluate multiple task orders to assess whether the reported gains are robust to task order?
Response to Reviewer v8Tj
We thank the reviewer for their time and constructive feedback, which has helped us strengthen the manuscript, and we have revised the paper.
Response to CL-LoRA, HiDe-LLaVA, LiLoRA, O-LoRA, RegLoRA): O-LoRA applies orthogonality to all past tasks unconditionally, whereas CoLoRA-Med applies it only to private input factors of conflicting task pairs. RegLoRA applies a binary mask that drives the new update toward zero at the 2% largest past entries, whereas CoLoRA-Med weights the drift of shared factors continuously, entry by entry, by gradient importance. Unlike HiDe-LLaVA, which fixes an offline layer split and protects nothing during training, CoLoRA-Med uses similarity during training to choose between alignment, orthogonality and no constraint, which none of these methods does. We added Table 4 to the appendix. It compares CoLoRA-Med with APD, ACL, O-LoRA, RegLoRA, CL-LoRA, HiDe-LLaVA and LiLoRA.
Response to question-based routing:
We agree that the reported 99.96% reflects in-distribution wording, since validation questions share templates with training. In a new held-out-wording experiment, when the router is fit on half of the templates and tested on unseen ones, accuracy is 93.2 ± 3.7%: at least 99% for four tasks, but 81.6% for report generation, whose short instructions lack distinctive format cues. Ambiguous questions receive near-zero routing margins, so the margin can flag them. We revised the limitations paragraph in the Discussion to report this. However, the alternatives face similar or stronger limitations: prompting the VLM to infer the task is itself sensitive to wording, while requiring a task ID shifts the burden to clinicians and fails whenever the task is unspecified. Question-based routing keeps task selection automatic and adds an explicit confidence signal, the routing margin, that prompting alone does not provide. We also used the same question-based router for LiLoRA for a fair comparison.
Additionally, we route on the question rather than the image because, in our setting, task identity is defined by the requested output, not by image content: the same ultrasound images appear in both classification (BUSI, BUS-UCLM) and detection (BUSI-det, BUS-UCLM-det), and chest X-rays appear in both multi-label classification and report generation, so an image-based or multimodal router cannot separate these tasks, whereas the question specifies the expected output. Question-based routing also avoids an additional image encoder at inference.
Response to advantage over baselines:
We have added results on the medical backbone MedGemma-1.5-4B-IT (Table 1), where the advantage is clearest. Compared with the strongest baseline, O-LoRA, CoLoRA-Med reduces forgetting from 27.34% to 18.73% (a 31.5% relative reduction), raises worst-task retention from 48.57% to 84.04%, and improves ANP from 84.16% to 91.47%; these gaps are well beyond the seed variation. We agree that on Qwen the forgetting gap to O-LoRA (6.94 vs. 7.05) is within one standard deviation. Because normalized metrics measure retention relative to each method's own initial performance, final task-level scores give a complementary view: across all three backbones, CoLoRA-Med achieves the best or second-best multi-label macro-F1 and cell-counting MAE (e.g., 43.38 vs. 24.25 macro-F1 for O-LoRA on MedGemma). Finally, we note that the ICLR reviewer guidelines explicitly recognize that scientific significance does not necessarily require state-of-the-art results, but can arise from new, relevant theoretical or empirical knowledge. Consistent with this criterion, our objective was to establish (i) a replay-free shared/private LoRA framework; (ii) similarity-conditioned interaction combined with importance-weighted protection of shared parameters; and (iii) an evaluation across 11 medical datasets showing that forgetting is most severe on a medical backbone, where our gains are largest, and that no single method dominates.
We now report final task-level scores for every ablation (Appendix Table 9).