acceptodds
Withdrawn from ICLR 2027

Recoverable but Not Preserved: Hidden Forgetting in Continual Self-Supervised Medical Pretraining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual self-supervised pretraining is a natural strategy for keeping medical foundation encoders current, yet its standard evaluation protocol, fine-tuning the updated encoder on a source task, can itself recover degraded representations and thereby obscure underlying forgetting. Continuing CheXWorld, a JEPA-style chest-radiograph ViT, on RSNA Pneumonia leaves NIH ChestX-ray14 fine-tuning performance nearly unchanged (79.6–80.1 mAUROC), while a frozen linear probe on the same checkpoints loses 1.6 points. These trends diverge consistently across three training seeds and individual pathologies: Nodule, Pneumothorax, and Emphysema lose 3.5–4.2 probe points that fine-tuning largely recovers. This degradation occurs even though RSNA images are drawn from NIH and grows to 12.5 points after a single PadChest hop. We then ask what mechanisms preserve frozen representational accessibility. Prediction surprise, used to gate a fast/slow EMA dual memory, is a valid plasticity signal, tracking which samples induce larger representation changes (), but per-sample target arbitration retains no more than a constant gate matched to its mean. More broadly, methods whose targets track the learner single, dual, gated, or rate-modulated converge to similar retention (77.8–78.2 mAUROC), whereas methods that explicitly constrain the learner toward the pre-update model retain up to 80.0 mAUROC in single-seed comparisons; EWC exhibits the same trend as regularization strength increases. Finally, changing only the probe early-stopping rule shifts the estimated retention by 2.4 points. Together, these results distinguish recoverability from preservation and plasticity from vulnerability, and motivate a reporting protocol for evaluating continually updated encoders.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, self-supervised learning, representation learning, catastrophic forgetting, foundation models, stability-plasticity, domain shift, medical imaging

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TeDQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates hidden forgetting in continual self-supervised medical pretraining. The authors show that frozen-probe performance declines while fine-tuning performance remains stable. They also examine dual-memory mechanisms and evaluation sensitivity.

Critical strengths

Addresses a meaningful evaluation problem in continual learning.

The main finding is supported by three seeds and pathology-level analysis.

Critical weaknesses
  1. Limited generalizability. The experiments are limited to one medical image encoder. It would be interesting to see whether similar findings hold for other models or self-supervised learning approaches.

  2. The anchored methods retain more information from the original task, but their performance on the new task is less clear. Without this comparison, it is difficult to assess the stability–plasticity trade-off.

  3. Limited experimental evidence. Some important results, particularly those involving CaSSLe, PFR, and PadChest, are based on a single seed. The PadChest experiments also involve different numbers of training updates, which makes the comparison harder to interpret.

  4. The paper mainly relies on linear probes to measure forgetting. I wonder whether the same conclusions would hold with more expressive probes, since lower linear-probe performance does not necessarily mean the information has been lost.

Questions for authors
  1. Do the main findings hold for another medical foundation encoder or self-supervised objective?

  2. Could the authors provide target-task performance for CaSSLe, PFR, and EWC to better assess the stability–plasticity trade-off?

  3. Does the severe PadChest forgetting persist under matched optimization steps?

  4. Would a nonlinear frozen probe lead to similar conclusions about representation preservation?

Reviewer_ejH8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors explore representations and performance in encoders using continual pretraining in medical images, using chest x-rays as their task of choice. They use CheXWorld to demonstrate that continual SSL pretraining reduces downstream linear-probe performance. However, this is contrasted by full fine-tuning which largely recovers the degradation leading to conflations with retention of performance instead of relearning. They further investigate sequential chaining of datasets and compare representational similarity with performance. They also discuss adaptive EMA-based teacher models and their ability to prevent forgetting and propose evaluation recommendations to focus on issues uncovered in their experimentation.

Critical strengths
  1. The authors clearly follow a distinct chain of ideas with extensive experimentation at both representation and performance level. Their central idea of the conflation of retention vs relearning of representations is introduced by the probing vs full finetuning results which is important.
  2. They not only investigate the idea but also propose and test a possible solution termed SGDM to mitigate their uncovered issues.
Critical weaknesses
  1. One of the strongest negatives about the paper is that it is jargon heavy to the point that it distracts from extracting useful information from it. It is obviously not short of scientific content, but it is unfortunately extremely hard to follow the writing of the authors without probably having experienced the same body of literature the authors themselves have. This results in a paper that is unnecessarily jargon laden without any attempt to soften it for readers. For example,

A single PadChest hop costs 12.52 points without a matching gain on its own task.

Maybe a reader understands with access to full context, but it still sounds unnecessary. And the text is similar in many places. Another few examples,

Prediction surprise, used to gate a fast/slow EMA dual memory, is a valid plasticity signal, tracking which samples induce larger representation changes (ρ = 0.51), but per-sample target arbitration retains no more than a constant gate matched to its mean.

RSNA consists of reannotated NIH images and is nearly indistinguishable from NIH in representation space, yet it costs more frozen accessibility than the more distant COVIDx.

The first part of the second quote is perfectly understandable, while I am certain it would help if they explained frozen accessibility to a reader unfamiliar with their terminology, before using it. It would be a good idea for the authors to simplify and self-contain their text and this issue is a detriment towards acceptability from my end.

  1. The authors are sometimes aggressive in their conclusions based on a fractional change in their performance metrics. For example, they state

Yet at matched epoch 19, RSNA costs more frozen accessibility (−1.46) than COVIDx (−0.92).

While the CKA difference to the NIH without losing performance on COVIDx is interesting in their sequential chaining, RSNA and COVIDx are being compared to each other with a ~0.5 difference in AUROC. This is even ignoring the idea of seed-variance in sequential chaining (though they do so in their initial NIH probe). Overall, this weakens some of their claims.

  1. Sometimes the authors seem to conflate some effects in their experimental workflow. For example, even in the discrepancy between probing and finetuning, it is possible that the continual SSL pretraining on their specific data and specific optimization is the cause of the degradation on their downstream probing and finetuning and not specifically a factor of continual SSL pretraining.
Questions for authors
  1. Can the authors clarify what should be the takeaways in terms of representation learning by readers of the paper?

Measuring the frozen encoder directly, documenting the readout, separating source retention from target utility, and matching adaptive rules to their averages make these distinctions visible.

This seems yet again jargon heavy instead of a clear recommendation to the readers of the paper. Also, if Table 3 is supposedly the takeaway from the issues observed here, is there a gap in scientific literature motivating this list of recommendations or are they formalizing their own experimentational outcomes?

  1. Can the authors expand on their understanding of what happened with the PadChest? As they state it is their most destructive dataset, but it is not clear what the reasons could be if it is not explained by distribution shift alone.
Reviewer_6YnoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies forgetting during continual self-supervised medical pretraining, showing that CheXWorld’s frozen-probe performance can decline while full fine-tuning performance remains stable. It compares surprise-gated EMA memories with fixed-reference regularization and distillation, and proposes separate reporting of preservation, recoverability, and target-domain utility.

Critical strengths

Across three RSNA training seeds, frozen-probe mAUROC falls from 79.62 to 77.90 while full fine-tuning shows no corresponding decline. The pathology-level analysis strengthens the evidence that the choice of evaluation protocol matters for frozen-encoder deployment.

The mean-matched constant-gate baseline provides a targeted control for the proposed adaptive routing mechanism. The resulting negative finding is informative about this intervention and is appropriately reported.

Critical weaknesses

The conceptual contribution is incremental relative to existing work on representation forgetting and prior-model distillation (Davari et al., 2022; Fini et al., 2022). The complementary medical-imaging result is useful, but the study offers limited explanation of when or why the evaluations diverge beyond its single encoder and source benchmark.

Despite the probe checks in Section 4.2, the main discrepancy has not been tested across probe optimization settings; the anchoring comparisons also use one seed and omit corresponding target-task gains. These limitations weaken the evidence that the measured degradation is robust and that fixed references preserve useful adaptation rather than mainly restricting updates.

Questions for authors

Beyond the complementary medical-imaging example, what new understanding of forgetting does the paper establish relative to prior representation-forgetting and distillation work, and which experiment supports it?

Does the probe–fine-tuning discrepancy persist when the original and updated encoders are evaluated with converged probes under a small, validation-selected learning-rate and regularization grid?

For CaSSLe and PFR, what target-task gains accompany the reported source retention, and does the retention advantage persist across additional seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.