Examining What Self-Supervised Models Learn from Longitudinal 3D Medical Images

In longitudinal medical studies, representations of three-dimensional (3D) images are expected to capture meaningful changes within a subject over time. However, these representations may encode noise factors or omit information about meaningful temporal changes. We introduce the Longitudinal Representation Audit (LRA) framework for assessing what information is gained through the training process. LRA compares each trained encoder with its matched initialization and, where possible, controls for measured noise factors to isolate the information of interest. We apply LRA across several self-supervised learning (SSL) models on longitudinal knee magnetic resonance imaging (MRI) data using two training budgets. The experiments reveal that features from a randomly initialized 3D backbone in one configuration already achieve 0.630 temporal-order prediction accuracy before SSL training. Across all five models, training produces no statistically supported improvement in this accuracy over matched initialization. Training nevertheless benefits phenotype prediction: Static SSL improves mean AUROC by 0.118 over initialization, despite showing no detectable improvement in temporal-order accuracy. Favourable representation geometry also does not guarantee improved temporal-order performance: after 100 epochs, IET achieves the highest participation ratio (33.76), yet its temporal-order accuracy is 0.109 below initialization. Together, these findings show that decodability, non-collapse, and favourable representation geometry alone cannot establish that longitudinal information was acquired through training. LRA therefore shifts evaluation from what can be decoded? to what did training actually add?