Recoverable but Not Preserved: Hidden Forgetting in Continual Self-Supervised Medical Pretraining

Continual self-supervised pretraining is a natural strategy for keeping medical foundation encoders current, yet its standard evaluation protocol, fine-tuning the updated encoder on a source task, can itself recover degraded representations and thereby obscure underlying forgetting. Continuing CheXWorld, a JEPA-style chest-radiograph ViT, on RSNA Pneumonia leaves NIH ChestX-ray14 fine-tuning performance nearly unchanged (79.6–80.1 mAUROC), while a frozen linear probe on the same checkpoints loses 1.6 points. These trends diverge consistently across three training seeds and individual pathologies: Nodule, Pneumothorax, and Emphysema lose 3.5–4.2 probe points that fine-tuning largely recovers. This degradation occurs even though RSNA images are drawn from NIH and grows to 12.5 points after a single PadChest hop. We then ask what mechanisms preserve frozen representational accessibility. Prediction surprise, used to gate a fast/slow EMA dual memory, is a valid plasticity signal, tracking which samples induce larger representation changes ($\rho =0.51$), but per-sample target arbitration retains no more than a constant gate matched to its mean. More broadly, methods whose targets track the learner single, dual, gated, or rate-modulated converge to similar retention (77.8–78.2 mAUROC), whereas methods that explicitly constrain the learner toward the pre-update model retain up to 80.0 mAUROC in single-seed comparisons; EWC exhibits the same trend as regularization strength increases. Finally, changing only the probe early-stopping rule shifts the estimated retention by 2.4 points. Together, these results distinguish recoverability from preservation and plasticity from vulnerability, and motivate a reporting protocol for evaluating continually updated encoders.