- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tries to address whether the information we can decode from self-supervised representations of longitudinal 3D medical images was actually learned during training, or was already there before training. To answer this, the authors propose the Longitudinal Representation Audit (LRA), which compares each trained encoder with its matched untrained initialization on four axes (temporal order, identity, phenotype, and controlled change), using patient-level paired bootstrap, nuisance residualization, and 32 low-level image statistics as a learning-free reference. On OAI knee MRI, with a ViT-S trained under five SSL objectives for 10 and 100 epochs, they find that temporal order is already decodable at initialization (0.630, vs. 0.636 for the image statistics), that no objective gives a significant temporal gain, that Static SSL improves phenotype AUROC by 0.118, and that geometry measures such as the participation ratio do not track these gains.
Critical strengths
I think the execution is careful and honest. If it were submitted to a related workshop or a more medical conference / journal, I'd consider it to be a better fit. The comparisons are paired at the patient level with bootstrap intervals and Holm correction, part of the analysis was pre-registered (Table 8, p. 28), and the authors openly discuss their limitations and even the inconsistencies they found in their own records (pp. 44-45). I appreciate this, and it makes the numbers easy to interpret within the tested setting. It does not change my overall assessment, though.
Critical weaknesses
-
I feel the novelty is limited and the main finding is not strong enough for ICLR. The core of LRA is to subtract the probe score of a matched random initialization from that of the trained encoder. Random-weight baselines are a standard control in representation learning, and the paper itself cites this line of work (Jarrett et al., 2009; Saxe et al., 2011; p. 3). Yet Sections 3 and 4 spend about four pages formalizing this comparison, while the results take about two. The main finding, that temporal order is decodable before training, is also what I would expect, given that 32 low-level image statistics alone already reach 0.636 (p. 7). I also noticed that Appendix F.1 (p. 35) says "IET was built before LRA, and LRA grew out of the need to check whether this design actually acquired the longitudinal information it was meant to encode." In the main text, IET is called "a longitudinal stress test rather than a proposed method" (p. 7), but it still has its own design rationale and ablation study (Appendix F). So to me the paper reads more like a careful negative result on the authors' own objective, reframed as an evaluation framework. Without a new method or a strong conclusion, I do not think it meets the ICLR bar, although I think it would be a good fit for a workshop.
-
I think the conclusions go well beyond what a single, small-scale setting can support. All results come from one cohort (OAI SAG-3D-DESS), one backbone (ViT-S), and short training on 1,268 scans for 10 or 100 epochs, with one shared configuration across objectives. The 100-epoch results, including the +0.118 and -0.109 reported in the abstract, come from a single seed (pp. 14, 20, 44). With this budget, I feel the missing temporal gain may simply reflect undertraining rather than a property of the objectives. Claims such as decodability and geometry "cannot establish" acquired longitudinal information, or that the framework "is not specific to knee MRI" (p. 44), would need evidence from other datasets or modalities, or from stronger pretraining.
Other weaknesses:
- The caption letters (b)-(d) of Fig. 10 do not match the panel titles (p. 30).
- Fig. 13d is titled "Supervised bound", while Appendices D.9 and E.5 say the supervised model is not an upper bound.
- The statement that "44 of 45 encoder pairs are incomparable" (p. 8) has no supporting table.
- The ethics statement says the work does not involve human participants (p. 10). The data are human knee MRI from OAI, so I think a sentence on the data use agreement would be appropriate.
Questions for authors
- I wonder whether the temporal gain stays at zero with a larger training budget, or when starting from publicly available pretrained 3D medical encoders. Even one such setting would help tell undertraining apart from a limitation of the objectives.
- (Clarification) The holdout half is described as "sealed ... remains unused" (pp. 7, 14), but the pre-registered O21 analysis uses the full 1,380-patient cohort (pp. 28, 36). Was the holdout used, and which results does this affect?
- (Clarification) Following the exclusion rule in Appendix H.1 (p. 42), Table 10 suggests that Patient-CL 100 and JEPA 10 should also have been excluded from the clinical stage, but both appear in Table 22. Could the authors explain this?