Med-PRISM: Spectrally Anisotropic Memory for Continual Medical Multimodal Learning

Medical multimodal models are expected to continually learn skills with different prediction requirements, from diagnostic labels to spatial localization and image-based reasoning. Retaining earlier skills is difficult when past data cannot be revisited. Freezing past adapters is not enough because new updates can still change earlier predictions. A past parameter update can be strong along some directions and weak along others. We propose Med-PRISM, a replay-free continual learning framework that allows shared adjustment while giving stronger protection to high-energy historical directions. The shared adapter learns under a drift constraint. Private adapters are frozen after each skill and reused together. Rather than treating all historical directions equally, we use the spectrum of each past private update to record its directions and their relative strengths. It discourages new private updates from using high-energy historical directions while leaving lower-energy directions more plasticity. No past samples or activations are stored, and inference requires no task-specific routing. We derive a bound on changes in earlier-task losses and identify the interference term. Experiments on MedicalSkill-CL, a benchmark we construct for continual learning across five medical skills, show that Med-PRISM surpasses the strongest evaluated baseline by 1.94 and 2.71 points in terms of Avg and Last, respectively.