Structure Before Time: Anatomy-Preserving Representation Learning for Longitudinal 3D Brain MRIs

Longitudinal modeling of high-resolution 3D brain MRI requires representations that preserve localized disease information while remaining tractable as patient histories grow. This is particularly challenging in Alzheimer's Disease (AD), where structural changes are spatially heterogeneous, subtle at early disease stages such as Mild Cognitive Impairment (MCI), and observed over irregular follow-up. We introduce **Structure Before Time (SBT)**, an anatomy-preserving representation learning principle for longitudinal brain MRI. SBT uses neuroanatomical structure to constrain representation formation before global and temporal aggregation, yielding compact, anatomically informed representations for longitudinal modeling. Controlled experiments show that the resulting gains arise from meaningful anatomical organization and its early introduction into the framework. Across AD diagnosis, MCI progression, and survival prediction, SBT consistently improves over single-scan and longitudinal baselines, while benefiting increasingly from accumulated history. The learned representation further transfers across cohorts, remains effective under alternative anatomical parcellation, extends to Frontotemporal dementia (FTD), and localizes controlled abnormalities with known regional ground truth in a synthetic dataset. These results support anatomy-preserving representation formation as an effective inductive bias for high-dimensional longitudinal neuroimaging.