acceptodds
Withdrawn from ICLR 2027

CoLoRA-Med: Continual Medical Learning with Low-Rank Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models must continually adapt to new tasks without catastrophic forgetting of earlier ones, a requirement that is especially critical in medical applications, where reliability depends on retaining previously acquired capabilities. Existing low-rank adaptation methods for continual learning either constrain parameter updates or isolate them per task, and thus struggle to simultaneously share knowledge across related tasks and prevent interference between conflicting ones. We introduce CoLoRA-Med, a continual learning method that decomposes a low-rank adapter into shared and task-specific components. CoLoRA-Med freezes the components learned for previous tasks and guides new updates according to their relationship with prior ones: updates aligned with related tasks are encouraged to reuse shared directions, conflicting updates are steered into separate subspaces, and changes to parameters important for earlier tasks are regularized. We evaluate CoLoRA-Med on 11 medical visual question answering datasets across seven imaging modalities and five tasks: classification, multi-label classification, detection, cell counting, and report generation. Compared with the average of five low-rank continual learning baselines, CoLoRA-Med improves final average normalized performance by 3% and reduces mean normalized forgetting from 10.2% to 6.9%, a 32% relative reduction. These results show that combining shared adaptation with selective task specialization effectively balances plasticity and stability. Hence, CoLoRA-Med moves medical AI toward systems that keep learning new clinical skills without losing the ones clinicians already rely on. Code and checkpoints will be released.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Medical vision-language models, continual learning, low-rank adaptation, catastrophic forgetting, knowledge transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v8TjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoLoRA-Med, a continual learning framework for medical vision-language models. It decomposes LoRA adapters into shared and task-specific branches to balance knowledge transfer and catastrophic forgetting. The method preserves previously learned task-specific LoRA components, conditionally aligns related task updates or separates conflicting ones through orthogonality constraints, and protects important shared parameters through regularization. At inference, a question-based router selects the appropriate task-specific LoRA without requiring ground-truth task identity. The method is validated on 11 medical datasets across seven imaging modalities and five tasks.

Critical strengths

The paper is well motivated by the need for medical vision-language models to learn heterogeneous tasks over time, rather than repeatedly adapting to one task type. Learning classification, multi-label classification, detection, cell counting, and report generation in one continual-learning stream makes the problem practically relevant. The evaluation is also reasonably broad, covering 11 datasets, seven imaging modalities, two backbones, and several continual LoRA baselines.

Critical weaknesses
  1. The methodological novelty is not sufficiently clear relative to existing continual LoRA methods. The overall design of shared adaptation with task-specific components has clear precedent in methods such as CL-LoRA, HiDe-LLaVA, and LiLoRA. In addition, the proposed orthogonality constraint is closely related to O-LoRA, while the protection of important shared parameters is similar in spirit to RegLoRA-style regularization. The main novelty therefore seems to lie in combining these mechanisms and conditionally applying them based on update similarity, rather than introducing a fundamentally new continual-learning structure. The paper should better distinguish this contribution from these closely related approaches.
  2. The query-only routing mechanism is not fully convincing. The router selects the task-specific LoRA using only textual features from the question, without considering the image. Since the five tasks have very different question formats, the reported 99.96% routing accuracy may largely reflect easily identifiable task-specific wording. It is unclear whether this routing strategy would remain reliable when different tasks use similar prompts or when task identity depends on image content. A harder routing evaluation with overlapping prompts or a comparison with multimodal routing would strengthen the paper.
  3. The experimental evidence does not fully support a consistent advantage of the proposed method. CoLoRA-Med is not consistently the best-performing method in Table 1, particularly on InternVL, while the abstract highlights improvements over the average of five low-rank continual learning baselines rather than over the strongest baseline. Moreover, the ablation study mainly reports normalized retention metrics rather than actual task-level performance. Since strong normalized retention does not necessarily imply strong task learning, it would be important to report the task-level metrics for each ablation under the same setting as Table 1. This would better show whether the proposed components improve retention without sacrificing plasticity.
Questions for authors
  1. Could the authors report the ablation results using the same training setting as Table 1 and include the actual task-level metrics, rather than only normalized retention metrics?
  2. Could the authors clarify how the shared and task-specific LoRA components are combined at inference, merged or computed separately?
  3. The experiments use a single task order. Could the authors evaluate multiple task orders to assess whether the reported gains are robust to task order?
1 comment
AuthorsOpenReview ↗

Response to Reviewer v8Tj

We thank the reviewer for their time and constructive feedback, which has helped us strengthen the manuscript, and we have revised the paper.

Response to CL-LoRA, HiDe-LLaVA, LiLoRA, O-LoRA, RegLoRA): O-LoRA applies orthogonality to all past tasks unconditionally, whereas CoLoRA-Med applies it only to private input factors of conflicting task pairs. RegLoRA applies a binary mask that drives the new update toward zero at the 2% largest past entries, whereas CoLoRA-Med weights the drift of shared factors continuously, entry by entry, by gradient importance. Unlike HiDe-LLaVA, which fixes an offline layer split and protects nothing during training, CoLoRA-Med uses similarity during training to choose between alignment, orthogonality and no constraint, which none of these methods does. We added Table 4 to the appendix. It compares CoLoRA-Med with APD, ACL, O-LoRA, RegLoRA, CL-LoRA, HiDe-LLaVA and LiLoRA.

Response to question-based routing:

We agree that the reported 99.96% reflects in-distribution wording, since validation questions share templates with training. In a new held-out-wording experiment, when the router is fit on half of the templates and tested on unseen ones, accuracy is 93.2 ± 3.7%: at least 99% for four tasks, but 81.6% for report generation, whose short instructions lack distinctive format cues. Ambiguous questions receive near-zero routing margins, so the margin can flag them. We revised the limitations paragraph in the Discussion to report this. However, the alternatives face similar or stronger limitations: prompting the VLM to infer the task is itself sensitive to wording, while requiring a task ID shifts the burden to clinicians and fails whenever the task is unspecified. Question-based routing keeps task selection automatic and adds an explicit confidence signal, the routing margin, that prompting alone does not provide. We also used the same question-based router for LiLoRA for a fair comparison.

Additionally, we route on the question rather than the image because, in our setting, task identity is defined by the requested output, not by image content: the same ultrasound images appear in both classification (BUSI, BUS-UCLM) and detection (BUSI-det, BUS-UCLM-det), and chest X-rays appear in both multi-label classification and report generation, so an image-based or multimodal router cannot separate these tasks, whereas the question specifies the expected output. Question-based routing also avoids an additional image encoder at inference.

Response to advantage over baselines:

We have added results on the medical backbone MedGemma-1.5-4B-IT (Table 1), where the advantage is clearest. Compared with the strongest baseline, O-LoRA, CoLoRA-Med reduces forgetting from 27.34% to 18.73% (a 31.5% relative reduction), raises worst-task retention from 48.57% to 84.04%, and improves ANP from 84.16% to 91.47%; these gaps are well beyond the seed variation. We agree that on Qwen the forgetting gap to O-LoRA (6.94 vs. 7.05) is within one standard deviation. Because normalized metrics measure retention relative to each method's own initial performance, final task-level scores give a complementary view: across all three backbones, CoLoRA-Med achieves the best or second-best multi-label macro-F1 and cell-counting MAE (e.g., 43.38 vs. 24.25 macro-F1 for O-LoRA on MedGemma). Finally, we note that the ICLR reviewer guidelines explicitly recognize that scientific significance does not necessarily require state-of-the-art results, but can arise from new, relevant theoretical or empirical knowledge. Consistent with this criterion, our objective was to establish (i) a replay-free shared/private LoRA framework; (ii) similarity-conditioned interaction combined with importance-weighted protection of shared parameters; and (iii) an evaluation across 11 medical datasets showing that forgetting is most severe on a medical backbone, where our gains are largest, and that no single method dominates.

We now report final task-level scores for every ablation (Appendix Table 9).

Reviewer_99ThOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoLoRA-Med combines shared and task-specific LoRA branches for continual medical vision-language learning. It freezes previous private branches, applies similarity-conditioned alignment or orthogonality to new private updates, protects important shared parameters, and routes questions to private branches. Evaluation covers five task types, seven imaging modalities, and 11 datasets using two 4B backbones.

Critical strengths

The heterogeneous medical evaluation shows useful task-specific gains. In Table 1, Qwen multi-label F1 reaches 35.47 versus 30.56 for the next-best continual baseline, and cell-counting MAE is lowest among the compared continual methods on both backbones. These results give the work practical value.

Critical weaknesses
  • Limited methodological novelty. Shared/private adapters, similarity-based stabilization, and orthogonality constraints have clear precedents, as Section 2 acknowledges. The main increment is their conditional combination with shared-parameter protection. The methodological advance over closely related designs such as CL-LoRA and LiLoRA is modest.
  • A robust advantage over strong baselines is not established. In Table 2, forgetting is 6.94±1.69 versus 7.05±0.66 for O-LoRA, a difference of only 0.11 percentage points; ANP is 98.54±1.94 versus 97.48±0.69. The reported 32% forgetting reduction is relative to the average of five baselines. With only two seeds and one task order, and rankings varying across tasks and backbones, the evidence for a reliable advantage remains limited. ———————————————— While the authors add some externel experiments which partially addressed my concerns, the novelty distinction from CL-LoRA and LiLoRA remains not conviencable. Therefore, I decide to rasie my score from 2 to 3 instead of 4.
Questions for authors
  • Could you clarify the main technical distinction from CL-LoRA and LiLoRA, and the specific limitation that conditional interaction addresses?
  • Given the small forgetting gap to O-LoRA in Table 2, which existing task-level results or analyses best support the method's advantage, and where is that advantage most pronounced?
  • Could you explain the trade-off in Table 3, where the full configuration has lower forgetting but removing interaction losses gives higher ANP, and the rationale for choosing the full configuration?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 99Th

We thank the reviewer for their time and constructive feedback, which has helped us strengthen the manuscript, and we have revised the paper.

Response to (CL-LoRA, LiLoRA):

CL-LoRA splits capacity by layer (early layers shared, late layers task-specific) and always applies orthogonality to block weights, and LiLoRA nests each task's private residual inside a shared input factor A, using cosine similarity with the previous task only to scale a uniform penalty on B₀. In CoLoRA-Med, each private slice has its own A_t and B_t in every layer, and similarity with every earlier task selects the type of constraint. This addresses a limitation of fixed constraints on heterogeneous medical tasks: always-on orthogonality blocks transfer between related tasks, while always-on sharing lets conflicting tasks, such as detection and report generation, interfere. We added Table 4 to the appendix. It compares CoLoRA-Med with APD, ACL, O-LoRA, RegLoRA, CL-LoRA, HiDe-LLaVA and LiLoRA.

Response to advantage over baselines:

We have added results on the medical backbone MedGemma-1.5-4B-IT (Table 1), where the advantage is clearest. Compared with the strongest baseline, O-LoRA, CoLoRA-Med reduces forgetting from 27.34% to 18.73% (a 31.5% relative reduction), raises worst-task retention from 48.57% to 84.04%, and improves ANP from 84.16% to 91.47%; these gaps are well beyond the seed variation. We agree that on Qwen the forgetting gap to O-LoRA (6.94 vs. 7.05) is within one standard deviation. Because normalized metrics measure retention relative to each method's own initial performance, final task-level scores give a complementary view: across all three backbones, CoLoRA-Med achieves the best or second-best multi-label macro-F1 and cell-counting MAE (e.g., 43.38 vs. 24.25 macro-F1 for O-LoRA on MedGemma). Finally, we note that the ICLR reviewer guidelines explicitly recognize that scientific significance does not necessarily require state-of-the-art results, but can arise from new, relevant theoretical or empirical knowledge. Consistent with this criterion, our objective was to establish (i) a replay-free shared/private LoRA framework; (ii) similarity-conditioned interaction combined with importance-weighted protection of shared parameters; and (iii) an evaluation across 11 medical datasets showing that forgetting is most severe on a medical backbone, where our gains are largest, and that no single method dominates.

In the ablations, we choose the full configuration because it achieves the lowest forgetting (6.14%) among all variants while keeping ANP within 0.5 pp of the best variant, within one standard deviation; removing any component increases forgetting (6.74–9.10%).

The two seeds and one task order are already acknowledged in the limitations paragraph. Prior work shows that retention in heterogeneous medical VQA depends on both the CL strategy and the task order (Shaaban et al., 2026); a systematic study of task-order sensitivity is beyond the scope of this work and remains an important direction for future research.

Reviewer_o8aGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoLoRA-Med, a LoRA-based continual learning method that decomposes model adaptations into task-shared and task-specific components. Task boundaries and identities are known during training, but task identities are unavailable at inference. The method combines conditional interaction between task-specific LoRA updates, importance-based regularization of shared parameters, and question-based routing to select task-specific adapters at inference. The authors evaluate the method on heterogeneous medical vision-language tasks and report improvements in task performance and retention over existing LoRA-based continual learning baselines, although the gains vary across tasks and backbones.

Critical strengths

The method combines shared and task-specific LoRA components with relatively straightforward regularization mechanisms. The overall framework is easy to understand and appears compatible with existing LoRA-based adaptation pipelines.

Critical weaknesses

Limited methodological novelty.

Decomposing a model into task-shared and task-specific components is well established in continual learning. For example, Additive Parameter Decomposition (APD, ICLR 2020) decomposes parameters into shared and task-adaptive components, while Adversarial Continual Learning (ACL, ECCV 2020) separates shared and task-specific representations. Although their implementations differ, both approaches use this separation to support knowledge sharing and preserve previously learned capabilities. CoLoRA-Med should more clearly explain its conceptual advance beyond this established framework. The conditional interaction mechanism and shared-parameter protection introduce additional design choices, but it remains unclear whether their combination constitutes a sufficiently substantial methodological contribution.

  • APD: Scalable and Order-robust Continual Learning with Additive Parameter Decomposition, ICLR'20.
  • ACL: Adversarial Continual Learning, ECCV'20.

Unclear connection between the medical motivation and the proposed method.

Medical continual learning is prominently emphasized in the title and introduction, but the proposed algorithm does not incorporate medical-specific knowledge or constraints. The shared/private decomposition, update-similarity regularization, importance-based parameter protection, and question-based routing are all domain-agnostic mechanisms. While evaluation on heterogeneous medical datasets is valuable, the paper does not sufficiently explain which characteristics of medical tasks motivate these design choices or what medical-specific insights emerge from the approach. The contribution associated with the medical setting therefore needs to be articulated more clearly.

Limited evaluation of question-based routing robustness.

The method relies on question-based routing to select a task-specific adapter when task identity is unavailable at inference. Appendix C reports routing accuracy of at least 99.96%. However, this near-perfect result may reflect how easily the benchmark’s tasks can be distinguished by their question wording. When different tasks use similar or ambiguous question formulations, identifying the appropriate adapter may be more challenging. Since routing errors could directly affect downstream performance, the reported accuracy alone does not establish robustness under such conditions.

Questions for authors

Q1. What is the key conceptual advance of CoLoRA-Med over prior shared/private decomposition approaches such as APD and ACL? How does the conditional interaction mechanism extend these ideas beyond their implementation with LoRA and additional regularization?

Q2. Which specific characteristics of medical continual learning motivate the proposed design choices? Would the same reasoning apply to general-domain heterogeneous VQA tasks? If so, what does the medical framing contribute beyond the choice of evaluation datasets?

Q3. How robust is question-based routing when different tasks use similar or ambiguous question formulations? How would routing errors affect the downstream performance of CoLoRA-Med?

1 comment
AuthorsOpenReview ↗

Response to Reviewer o8aG

We thank the reviewer for their time and constructive feedback, which has helped us strengthen the manuscript, and we have revised the paper.

Response to (APD, ACL):

APD splits full network weights additively, and ACL uses separate shared and private encoders; in both, the interaction between tasks is fixed: APD applies a uniform ℓ2 drift penalty, and ACL applies an adversarial loss and shared–private orthogonality to every task. CoLoRA-Med's conceptual advance is that the relation between each pair of tasks, measured by the cosine similarity of their private updates, decides how they interact: aligned updates are pulled together to share directions, conflicting updates are pushed into orthogonal input subspaces, and unrelated ones are left unconstrained, while shared factors are protected entry by entry using accumulated gradient importance. We added Table 4 to the appendix. It compares CoLoRA-Med with APD, ACL, O-LoRA, RegLoRA, CL-LoRA, HiDe-LLaVA and LiLoRA.

Response to the connection between the medical motivation and the proposed method:

The mechanisms are domain-agnostic and could apply to general heterogeneous VQA, but the medical setting is where their motivating conditions hold together and are binding:

  1. Shared and task-specific knowledge: In medicine, unlike general benchmarks, the same modality is annotated for different clinical purposes, from labels to boxes, counts and reports. Thus, our method shares visual knowledge in shared factors and keeps output-specific behavior in private factors.

  2. Conditional interaction: In medicine, tasks can share visual evidence yet require incompatible outputs, so related and conflicting task pairs occur in the same sequence. Thus, our method uses update similarity to align, separate or leave each pair unconstrained.

  3. Task selection at inference: In medicine, the same image can appear under several tasks, so image-based routing cannot identify the task, and clinicians should not have to specify it. Thus, our method routes based on the question wording.

  4. Protecting what matters most: In medicine, earlier capabilities may already be clinically validated, so losing them is costly. Thus, our method weights protection of each shared parameter by its gradient importance for earlier tasks, rather than restricting all parameters equally.

  5. Rehearsal-free retention: In general-domain CL, being rehearsal-free is allowed, whereas in medicine, privacy regulations and data-use agreements often make it a requirement. Thus, our method retains earlier tasks without replay, and its router stores only hashed question prototypes. We now state which medical properties motivate each component (Introduction, paragraph 2).

Response to question-based routing:

We agree that the reported 99.96% reflects in-distribution wording, since validation questions share templates with training. In a new held-out-wording experiment, when the router is fit on half of the templates and tested on unseen ones, accuracy is 93.2 ± 3.7%: at least 99% for four tasks, but 81.6% for report generation, whose short instructions lack distinctive format cues. Ambiguous questions receive near-zero routing margins, so the margin can flag them. We revised the limitations paragraph in the Discussion to report this. However, the alternatives face similar or stronger limitations: prompting the VLM to infer the task is itself sensitive to wording, while requiring a task ID shifts the burden to clinicians and fails whenever the task is unspecified. Question-based routing keeps task selection automatic and adds an explicit confidence signal, the routing margin, that prompting alone does not provide. We also used the same question-based router for LiLoRA for a fair comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.