acceptodds
Under review as a conference paper at ICLR 2027

Examining What Self-Supervised Models Learn from Longitudinal 3D Medical Images

Abstract

In longitudinal medical studies, representations of three-dimensional (3D) images are expected to capture meaningful changes within a subject over time. However, these representations may encode noise factors or omit information about meaningful temporal changes. We introduce the Longitudinal Representation Audit (LRA) framework for assessing what information is gained through the training process. LRA compares each trained encoder with its matched initialization and, where possible, controls for measured noise factors to isolate the information of interest. We apply LRA across several self-supervised learning (SSL) models on longitudinal knee magnetic resonance imaging (MRI) data using two training budgets. The experiments reveal that features from a randomly initialized 3D backbone in one configuration already achieve 0.630 temporal-order prediction accuracy before SSL training. Across all five models, training produces no statistically supported improvement in this accuracy over matched initialization. Training nevertheless benefits phenotype prediction: Static SSL improves mean AUROC by 0.118 over initialization, despite showing no detectable improvement in temporal-order accuracy. Favourable representation geometry also does not guarantee improved temporal-order performance: after 100 epochs, IET achieves the highest participation ratio (33.76), yet its temporal-order accuracy is 0.109 below initialization. Together, these findings show that decodability, non-collapse, and favourable representation geometry alone cannot establish that longitudinal information was acquired through training. LRA therefore shifts evaluation from what can be decoded? to what did training actually add?

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Longitudinal Representation Learning, Self-Supervised Learning, Representation Auditing, Medical Image Analysis, 3D MRI, Knee MRI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5MKmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tries to address whether the information we can decode from self-supervised representations of longitudinal 3D medical images was actually learned during training, or was already there before training. To answer this, the authors propose the Longitudinal Representation Audit (LRA), which compares each trained encoder with its matched untrained initialization on four axes (temporal order, identity, phenotype, and controlled change), using patient-level paired bootstrap, nuisance residualization, and 32 low-level image statistics as a learning-free reference. On OAI knee MRI, with a ViT-S trained under five SSL objectives for 10 and 100 epochs, they find that temporal order is already decodable at initialization (0.630, vs. 0.636 for the image statistics), that no objective gives a significant temporal gain, that Static SSL improves phenotype AUROC by 0.118, and that geometry measures such as the participation ratio do not track these gains.

Critical strengths

I think the execution is careful and honest. If it were submitted to a related workshop or a more medical conference / journal, I'd consider it to be a better fit. The comparisons are paired at the patient level with bootstrap intervals and Holm correction, part of the analysis was pre-registered (Table 8, p. 28), and the authors openly discuss their limitations and even the inconsistencies they found in their own records (pp. 44-45). I appreciate this, and it makes the numbers easy to interpret within the tested setting. It does not change my overall assessment, though.

Critical weaknesses
  1. I feel the novelty is limited and the main finding is not strong enough for ICLR. The core of LRA is to subtract the probe score of a matched random initialization from that of the trained encoder. Random-weight baselines are a standard control in representation learning, and the paper itself cites this line of work (Jarrett et al., 2009; Saxe et al., 2011; p. 3). Yet Sections 3 and 4 spend about four pages formalizing this comparison, while the results take about two. The main finding, that temporal order is decodable before training, is also what I would expect, given that 32 low-level image statistics alone already reach 0.636 (p. 7). I also noticed that Appendix F.1 (p. 35) says "IET was built before LRA, and LRA grew out of the need to check whether this design actually acquired the longitudinal information it was meant to encode." In the main text, IET is called "a longitudinal stress test rather than a proposed method" (p. 7), but it still has its own design rationale and ablation study (Appendix F). So to me the paper reads more like a careful negative result on the authors' own objective, reframed as an evaluation framework. Without a new method or a strong conclusion, I do not think it meets the ICLR bar, although I think it would be a good fit for a workshop.

  2. I think the conclusions go well beyond what a single, small-scale setting can support. All results come from one cohort (OAI SAG-3D-DESS), one backbone (ViT-S), and short training on 1,268 scans for 10 or 100 epochs, with one shared configuration across objectives. The 100-epoch results, including the +0.118 and -0.109 reported in the abstract, come from a single seed (pp. 14, 20, 44). With this budget, I feel the missing temporal gain may simply reflect undertraining rather than a property of the objectives. Claims such as decodability and geometry "cannot establish" acquired longitudinal information, or that the framework "is not specific to knee MRI" (p. 44), would need evidence from other datasets or modalities, or from stronger pretraining.

Other weaknesses:

  • The caption letters (b)-(d) of Fig. 10 do not match the panel titles (p. 30).
  • Fig. 13d is titled "Supervised bound", while Appendices D.9 and E.5 say the supervised model is not an upper bound.
  • The statement that "44 of 45 encoder pairs are incomparable" (p. 8) has no supporting table.
  • The ethics statement says the work does not involve human participants (p. 10). The data are human knee MRI from OAI, so I think a sentence on the data use agreement would be appropriate.
Questions for authors
  1. I wonder whether the temporal gain stays at zero with a larger training budget, or when starting from publicly available pretrained 3D medical encoders. Even one such setting would help tell undertraining apart from a limitation of the objectives.
  2. (Clarification) The holdout half is described as "sealed ... remains unused" (pp. 7, 14), but the pre-registered O21 analysis uses the full 1,380-patient cohort (pp. 28, 36). Was the holdout used, and which results does this affect?
  3. (Clarification) Following the exclusion rule in Appendix H.1 (p. 42), Table 10 suggests that Patient-CL 100 and JEPA 10 should also have been excluded from the clinical stage, but both appear in Table 22. Could the authors explain this?
Reviewer_eB9bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an audit of longitudinal medical-image representations through comparisons with matched initialization and low-level image statistics. On knee MRI, it finds that temporal-order information is already decodable before training, while improved phenotype prediction does not necessarily accompany improved temporal-order prediction.

Critical strengths
  • The central question what training actually adds beyond initialization? is worthwhile. The results provide a useful caution against interpreting post self-supervised training performance as evidence of training-acquired longitudinal information.

  • Matched-initialization comparisons, low-level references, and patient-level uncertainty estimates are sensible evaluation choices. The disagreement between representation geometry and task performance is also interesting.

Critical weaknesses
  • The presentation obscures a valuable central idea. Sections 3–4 introduce numerous terms and quantities before providing an accessible account of the actual procedure. The algebraic decomposition in Eq. (5) adds notation without substantially clarifying the experimental reasoning. A concise workflow and one worked example would communicate the contribution much more effectively.

  • The evaluation protocol is unnecessarily difficult to reconstruct. Encoder training, probe fitting, nuisance control, and data partitioning must be pieced together across the main text and appendices. Algorithm 1 is deferred to Appendix D.2, although this procedure should anchor the main presentation. Readers should immediately understand what is trained, what is frozen, and what each comparison establishes.

  • The results need a clearer connection to the claims. Table 1 places raw phenotype gains, acquisition-controlled reference margins, and absolute synthetic-change selectivity together, although these quantities answer different questions. Meanwhile, identity retrieval and controlled temporal gains appear only in Table 9. The main text should clearly identify each metric’s comparator, control, and evidentiary status, including which measurements support a training-gain claim and which remain descriptive.

Questions for authors
  • Which components are essential to LRA, and which conclusions require more than the matched-initialization comparison?

  • Could the authors consolidate the protocol and principal results into a self-contained presentation that explicitly connects each claim to its supporting comparison?

Reviewer_C4oROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper introduces the Longitudinal Representation Audit, which compares trained encoders with their matched random initializations across temporal ordering, knee identity, phenotype prediction, and controlled synthetic changes.

  • it additionally evaluates low-level image statistics and measured nuisance controls.

  • Experiments examine five self-supervised objectives using a common 3D backbone on longitudinal OAI knee MRI

Critical strengths
  • The untrained backbone achieves 0.630 temporal-order accuracy, while a low-level image-statistics reference achieves 0.636. So the matched-initialization comparison changes the interpretation of the results.

  • Patient-disjoint evaluation, paired patient-clustered bootstrap intervals, multiple information axes, and three-seed experiments at 10 epochs strengthen the study and the analysis includes substantial controls and reports their limitations transparently.

Critical weaknesses
  • Appendix I.2 states that the provenance of the MOAKS export and its binarization could not be established. The validity of the phenotype targets remains insufficiently established. If label availability varies systematically by visit, phenotype prediction could partly reflect those differences.

  • At 10 epochs, seed 0 was trained with a different code revision from seeds 1–2, and the Static SSL decline does not reproduce.

  • The replication evidence does not fully support conclusions about training duration and objective-level behavior. The authors appropriately acknowledge these limitations, but the results support a narrower conclusion: no positive gain was established under the tested linear-probe protocol.

Questions for authors
  • What changed between the training-code revisions used for seed 0 and seeds 1–2? Can the key comparisons be verified under one fixed revision, particularly the IET 100-epoch result and the Static SSL phenotype gain?

  • Can you identify the MOAKS reading project, label-construction rules, and treatment of missing or unscored items?

  • Appendix I.2 describes a sealed 690-patient holdout, while Table 8 and Appendix F.2 report an earlier analysis using the full 1,380-patient diagnostic cohort. Which patients had been examined before the holdout was designated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.