acceptodds
Withdrawn from ICLR 2027

Med-PRISM: Spectrally Anisotropic Memory for Continual Medical Multimodal Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical multimodal models are expected to continually learn skills with different prediction requirements, from diagnostic labels to spatial localization and image-based reasoning. Retaining earlier skills is difficult when past data cannot be revisited. Freezing past adapters is not enough because new updates can still change earlier predictions. A past parameter update can be strong along some directions and weak along others. We propose Med-PRISM, a replay-free continual learning framework that allows shared adjustment while giving stronger protection to high-energy historical directions. The shared adapter learns under a drift constraint. Private adapters are frozen after each skill and reused together. Rather than treating all historical directions equally, we use the spectrum of each past private update to record its directions and their relative strengths. It discourages new private updates from using high-energy historical directions while leaving lower-energy directions more plasticity. No past samples or activations are stored, and inference requires no task-specific routing. We derive a bound on changes in earlier-task losses and identify the interference term. Experiments on MedicalSkill-CL, a benchmark we construct for continual learning across five medical skills, show that Med-PRISM surpasses the strongest evaluated baseline by 1.94 and 2.71 points in terms of Avg and Last, respectively.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, MLLM, Medical Image

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_qSsLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Med-PRISM, a replay-free continual learning framework for medical multimodal models. The method combines a shared LoRA adapter trained under a drift constraint with task-specific private LoRA adapters that are frozen after each stage and cumulatively activated at inference. The central contribution is a regularization mechanism called SAMC, which performs SVD on each frozen private update and uses the resulting spectral energy distribution to penalize new private updates more strongly along high-energy historical directions while leaving lower-energy directions more flexible. The authors also construct a five-task medical benchmark called MedicalSkill-CL and provide a theoretical bound on old-task loss drift.

Critical strengths

S1. The paper constructs MedicalSkill-CL, a five-task benchmark covering heterogeneous medical skills with different prediction formats, from classification to grounding to multi-image reasoning. The construction pipeline includes format harmonization, overlap checks, and stratified sampling, addressing a genuine gap in evaluating continual learning across diverse medical tasks.

S2. The theoretical analysis cleanly decomposes old-task loss drift into four interpretable terms and identifies the first-order interference component controlled by SAMC. The spectral diagnostics in Section 4.4 provide meaningful empirical evidence that the learned weighting behaves as the theory predicts.

Critical weaknesses

W1. The architecture lacks novelty. Equation 1 is simply the base weight plus a shared LoRA plus the sum of all private LoRA updates, essentially the same superposition as O-LoRA. SAMC merely replaces uniform orthogonality penalties with spectral-energy-weighted penalties, a straightforward extension of GPM and SGP that the paper itself acknowledges in Section 2.3. The two-point gain over the strongest baseline does not justify acceptance at a top venue.

W2. Scalability is unaddressed. Inference activates all historical private adapters without routing, so memory and computation grow linearly with tasks. Appendix F reports 53.7M retained parameters at stage 5, yet the paper evaluates only five tasks and discusses no merging, pruning, or bounded-memory mechanism.

Questions for authors

See the Critical Weaknesses.

Reviewer_FLsAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Med-PRISM, a replay-free continual learning framework for medical multimodal models. The method combines a shared LoRA pathway with task-specific private adapters and introduces Spectrally Anisotropic Memory Consolidation (SAMC) to reduce interference from later tasks. SAMC uses the spectral structure of previous private updates to give stronger protection to high-energy historical directions while leaving other directions more flexible. The paper also provides a theoretical analysis linking this spectral constraint to old-task loss drift. Experiments are conducted on the proposed MedicalSkill-CL benchmark, which contains five sequential medical skills covering VQA, diagnosis, concept recognition, visual grounding, and reasoning VQA. The results show improved overall continual learning performance over the evaluated replay-free baselines, with additional ablations on task order, SAMC weighting, shared adaptation, and spectral behavior.

Critical strengths
  1. The loss-drift analysis and spectral diagnostics directly examine the interference targeted by SAMC.
  2. Med-PRISM gives stronger protection to high-energy directions from previous private updates while leaving lower-energy directions more flexible. The spectral weighting gives the method a clear difference from uniform protection.
Critical weaknesses
  1. The baseline budgets are not matched. Med-PRISM trains 23.0M parameters per task and retains 53.7M after task 5, while several baselines use smaller ranks, fewer trainable parameters, or fewer retained parameters. The reported gains are hard to separate from differences in capacity and training budget.
  2. The loss-drift bound covers the first-order term inside the historical spectral support, while the outside-support gradient, shared drift, and smoothness terms remain in the bound. It does not directly account for the full forgetting behavior measured in the experiments.
Questions for authors
  1. The overall Avg and Last combine tasks with different metrics and output types. What do these aggregate scores represent when comparing methods across the five skills?
  2. The theory controls only part of the old-task loss change. How closely does this result relate to the forgetting measured by Avg, Last, and BWT?
Reviewer_XoQ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores how a medical multimodal model can learn new skills without forgetting earlier ones, without replaying old training data. Med-PRISM uses a shared LoRA adapter and adds a private adapter for each new task. The main idea is to reduce forgetting by limiting interference with directions learned in earlier tasks. SAMC applies stronger penalties when new updates overlap with historical directions that have higher energy. The authors give a local bound on changes in old-task loss and test the method on five medical skills. Using Qwen3-VL-8B, Med-PRISM improves Avg and Last by 1.94 and 2.71 points over the strongest tested baseline. Ablation results suggest that SAMC helps reduce forgetting.

Critical strengths
  1. The method addresses a clear source of forgetting. Freezing old adapters does not stop new updates from changing earlier predictions. Med-PRISM limits changes to the shared adapter and uses SAMC to reduce interference from the new private adapter. It does not need to store old training samples or activations, and all learned adapters are used together without choosing one for each task.

  2. The ablations show that SAMC helps reduce forgetting. Removing it lowers performance on earlier tasks, while the comparison with uniform weighting suggests that spectral weighting provides an additional benefit.

Critical weaknesses
  1. Historical update energy may not fully reflect what needs protection. SAMC gives more protection to directions with higher historical energy, but Figure 4 mainly checks their relationship with first-order gradient sensitivity. A small gradient does not rule out a large loss increase in a high-curvature direction. Although the theorem includes a nonlinear remainder and a term for sensitivity outside the stored subspace, their sizes are not reported. This makes it unclear how much of actual forgetting the analysis explains.

  2. The extra benefit of spectral weighting needs stronger evidence. Its gains over uniform weighting are modest, and the experiments use only one random seed and one backbone (Table 2; Section 4.1). Results from multiple seeds would help show whether this benefit is consistent.

  3. The role of the shared adapter needs more evidence. Updating it across tasks does not by itself show that it learns useful knowledge across tasks, or that the private adapters mainly learn task-specific knowledge. The training objective does not explicitly enforce this separation. The No-shared ablation also removes parameters, so it cannot tell whether the benefit comes from knowledge sharing or extra capacity.

Questions for authors
  1. Historical energy and actual forgetting: Can you show whether historical energy predicts old-task loss increases under equal-sized weight perturbations along different directions, especially when gradients are small? It would also help to compare the protected first-order term with the other terms in the loss-change bound and relate them to the observed loss changes.

  2. Statistical robustness: Does the advantage over Uniform persist across multiple random seeds? Means, standard deviations, and paired differences for Avg, Last, and BWT would help assess whether the approximately 0.75-point Last gain reliably supports spectrum-aware weighting.

  3. Role of the shared adapter: What evidence shows that the shared adapter learns knowledge useful across tasks? The training objective does not explicitly separate shared and task-specific knowledge. Also, removing the shared adapter reduces the number of trainable parameters. Could the improvement come from the extra parameters rather than knowledge sharing?

Reviewer_Z4ZXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Med-PRISM, a replay-free continual learning framework for medical multimodal models, aiming to mitigate catastrophic forgetting when learning heterogeneous medical skills sequentially. The framework combines shared and private LoRA adapters. Historical private adapters are frozen and cumulatively reused during inference, while the shared adapter is regularized to limit parameter drift.

The main contribution is Spectrally Anisotropic Memory Consolidation (SAMC), which applies singular value decomposition to historical LoRA updates to extract their spectral directions and relative energies. These spectral energies are then used to impose non-uniform constraints on new private adapters, providing stronger protection for high-energy historical directions while preserving flexibility in lower-energy directions.

Critical strengths
  1. SAMC extends conventional subspace regularization by assigning different protection strengths to historical directions according to their spectral energies, providing a reasonable approach to balancing knowledge retention and plasticity. The additional analyses of old-task gradient sensitivity, subspace overlap, and theoretical interference bounds improve the interpretability of the proposed mechanism. 
  2. The proposed MedicalSkill-CL benchmark covers medical VQA, diagnosis, concept recognition, visual grounding, and reasoning VQA, enabling evaluation across diverse prediction requirements. The experiments include several continual learning baselines, task-order variations, and component ablations, demonstrating improvements in overall performance and historical knowledge retention.
Critical weaknesses
  1. The main novelty of this work lies in assigning non-uniform protection weights to historical directions based on their spectral energies. However, Table 2 shows that replacing spectral weighting with uniform weighting only reduces Avg from 52.37 to 51.88 and Last from 50.78 to 50.02. These differences are relatively small compared with the performance degradation observed when SAMC is entirely removed. This suggests that the primary improvement may come from subspace regularization itself, while the additional benefit of spectral weighting is less substantial. Furthermore, although Figure 4 demonstrates a correlation between spectral energy and old-task sensitivity, the analysis is restricted to the stored historical subspace and does not fully establish whether spectral energy reliably captures the overall importance of historical knowledge. Stronger evidence is therefore needed to justify the necessity of non-uniform spectral weighting over existing uniform or soft orthogonality-based constraints. 
  2. The main experiments are conducted using only Qwen3-VL-8B and a single random seed. Considering that Med-PRISM improves Avg and Last over the strongest baseline, Octopus, by only 1.94 and 2.71 points, respectively, the absence of multiple-seed evaluations makes it difficult to assess the statistical reliability of these improvements. Moreover, although different task orders are considered, the evaluation remains limited to sequences of five medical skills. Since the framework progressively accumulates historical private adapters and their spectral information, its effectiveness under longer continual learning sequences or different multimodal backbones remains unclear.
Questions for authors

Q1. Necessity of spectral weighting: Table 2 shows relatively modest improvements of SAMC over uniform weighting. Could the authors further investigate this difference and provide additional evidence supporting the independent benefit of non-uniform spectral weighting? In addition, comparisons with more recent soft orthogonality or importance-aware subspace regularization methods, such as ASO-LoRA, would help clarify the necessity of the proposed mechanism.

[1] ASO-LoRA: Attribution Scores-based Soft Orthogonality Low-Rank Adaptation for Large Language Model Continual Learning, ACL 2026.

Q2. Experimental robustness and generalizability: Could the authors provide results across multiple random seeds to verify the stability of the reported improvements? Additionally, does Med-PRISM maintain similar advantages under longer task sequences or different multimodal backbones?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.