acceptodds
Under review as a conference paper at ICLR 2027

Structure Before Time: Anatomy-Preserving Representation Learning for Longitudinal 3D Brain MRIs

Abstract

Longitudinal modeling of high-resolution 3D brain MRI requires representations that preserve localized disease information while remaining tractable as patient histories grow. This is particularly challenging in Alzheimer's Disease (AD), where structural changes are spatially heterogeneous, subtle at early disease stages such as Mild Cognitive Impairment (MCI), and observed over irregular follow-up. We introduce **Structure Before Time (SBT)**, an anatomy-preserving representation learning principle for longitudinal brain MRI. SBT uses neuroanatomical structure to constrain representation formation before global and temporal aggregation, yielding compact, anatomically informed representations for longitudinal modeling. Controlled experiments show that the resulting gains arise from meaningful anatomical organization and its early introduction into the framework. Across AD diagnosis, MCI progression, and survival prediction, SBT consistently improves over single-scan and longitudinal baselines, while benefiting increasingly from accumulated history. The learned representation further transfers across cohorts, remains effective under alternative anatomical parcellation, extends to Frontotemporal dementia (FTD), and localizes controlled abnormalities with known regional ground truth in a synthetic dataset. These results support anatomy-preserving representation formation as an effective inductive bias for high-dimensional longitudinal neuroimaging.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Longitudinal Modelling, T1-weighted sMRI scans, Alzheimer's Disease

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LKZ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a framework for predicting dementia outcome from longitudinal T1 imaging data. The idea is to first get parcel-specific embeddings before longitudinal modeling. The experiments are extensive to show the effectiveness of the proposed method.

Critical strengths

The breadth of validation analyses is amazing. Even a survival analysis, robustness to preprocessing, and external-data generalizability analysis are included. Overall I don't find any major flaws in the presentation.

Critical weaknesses
  1. My main concern is simply the limited methodological novelty. Learning parcel embeddings followed by longitudinal modeling seems incremental. There is a large body of literature on longitudinal MRI modeling, particularly on AD and related dementia. I don't think benchmarking classification accuracy on these public datasets represents a breakthrough.

  2. The clinical interpretation remains a weakness as in most MRI-based learning frameworks.

  3. A moderate concern is that the paper implies that the framework does not rely on a specific atlas, but comparing table 1 and table 2 suggests that there is a roughly 5% accuracy difference.

Questions for authors

The attention rollout visualization is not very informative. How does the saliency tell us about AD? I don't think showing regions being "influential" really helps us understand AD any better.

5 comments
AuthorsOpenReview ↗

Response to Reviewer LKZ4

We sincerely thank the reviewer for their time and for their highly encouraging assessment of our experimental rigor. We are thrilled that the reviewer found the "breadth of validation analyses is amazing". We address your concerns below and welcome all suggestions in refining the manuscript.

W1: Regarding the novelty

  • We would invite the reviewer to view SBT as a representation-learning principle inspired from the biological lens of neurodegeneration in Alzheimer's disease (Sec 1). Brain regions have meaningful anatomical correspondence across visits. In AD, progression appears as subtle, region-specific trajectories that unfold unevenly over time. SBT draws on this observation to provide an anatomical inductive bias for longitudinal learning. We acknowledge the extensive literature on longitudinal MRI (Sec 2, Appendix C.3). As summarized in Fig. 1, a central challenge remains: preserving fine-grained regional evidence from high-resolution scans while keeping computation tractable as patient histories grow. Existing approaches address this challenge through spatial downsampling, restrictions to paired or fixed-length histories, or compression into global scan embeddings. These strategies make different trade-offs among spatial detail, available temporal context, and explicit regional correspondence which may discard some useful information about the disease. Bringing these requirements together is therefore a substantive representation-learning problem and is non-trivial.

  • SBT addresses this problem by introducing anatomical structure during representation formation, before global spatial and temporal aggregation. The aim is to retain localized evidence while supporting high-resolution inputs and irregular, variable-length histories at a manageable computational cost. Our experiments demonstrate a favorable predictive performance and inference-cost trade-off (Fig. 5B), while being interpretable at parcel-level across visits. This combination of regional specificity, longitudinal flexibility, and computational scalability is the contribution we seek to establish. To the best of our knowledge, no prior longitudinal framework offers these properties together.

  • We believe that characterisation of SBT as “Parcel embeddings followed by longitudinal modeling” is a fair but understated one-line summary. We would invite the reviewer to consider the many possible design choices that can fit the same description, and how they behave. Each of the following is a natural first choice:

    • Early versus late anatomical atlas introduction: The late-grouping alternative uses atlas after global contextualization and then does parcel-pooling. It’s substantially lower performance than SBT shows that benefits only arise when the atlas is introduced before contextualization of parcel representations (Fig. 5A).
    • Probabilistic versus hard parcel membership: Assigning each patch exclusively to its highest-membership parcel reduces BAcc by ~11% and AUC by ~6%, supporting the value of retaining uncertain anatomical correspondence at regional boundaries (Table 4).
    • Learned versus uniform regional aggregation: In the matched GeM pooling comparison without parcel dropout, replacing learned parcel scoring with mean parcel aggregation reduces BAcc by ~7% and AUC by ~8% (Table 3).
    • Anatomical versus non-anatomical context: With the same encoder, anatomical grouping outperforms both random partitions and spatial blocks, supporting the value of meaningful anatomical context (Fig. 5A).
    • Scan-level versus parcel-level longitudinal aggregation: SBT-SL aggregates regional representations before temporal modeling, whereas SBT-PL retains parcel correspondence through temporal modeling. Their comparison examines the implications of these two levels of longitudinal integration (Table 1 and Appendix Sec. E.4).
    • Dependence on the temporal module: Replacing the Transformer with a GRU produces comparatively smaller performance differences, while preserving SBT’s advantage over the evaluated global-representation baselines (Table 3).
  • These comparisons support a specific finding: the anatomical context, the stage at which it shapes representation formation and the aggregation method matter. Sections 4.5-4.8 test this representation principle. Furthermore, SBT demonstrates generalizability to other tasks like time-to-event analysis and other diseases like frontotemporal dementia. Together, these analyses investigate the broader utility of the SBT’s representation principle.

AuthorsOpenReview ↗

Response (continued)

W2: Clinical interpretation

We thank the reviewer for raising this point. As the reviewer notes, clinical interpretation is a challenge shared by most MRI-based learning frameworks, not just SBT. It stems from the absence of regional ground truth in the publicly available dataset: MRI scans lack regional annotations of the anomalies that drive a diagnosis, so the attributions of any framework are difficult to verify directly. Likewise, it is also important to test the model's interpretations for reliability. The interpretability analyses of SBT are designed with this in mind (Sec 4.8 and Appendix Sec E) and serve two purposes:

  • The first is to verify the framework’s inherent interpretability and localization capabilities of SBT. The parcel weights, produced by the model for the anatomical representations, indicate which regions the model weighs on to classify a scan. To test whether these weights can be relied on, we constructed a synthetic dataset in which the abnormal regions and their severities are known by construction. Trained only on CN/ATR labels, the model's parcel weights recover the abnormal regions and track their severity. This experiment demonstrates that the model’s parcel-level explanations are spatially aligned with the underlying abnormalities of the synthetic dataset, thereby testing SBT’s reliability.
  • Second, a complementary plausible assessment in this realm is to assess anatomical concordance with the extant clinical literature and to check if the model follows the behavior. To assess the same, two tests were conducted viz cohort-level analysis and scan-level analysis (Sec 4.8 and Appendix Sec E). Cohort-level analysis, possible with our setup, showed similar expected patterns of affected regions in the dataset, consistent with the clinical literature. At the individual scan level, the parcel saliencies identify the most influential regions, and, in most cases, these fall within clinically established regions affected in AD pathology.

W3: Regarding alternative atlas

We appreciate your concern. We clarify that we don’t claim atlas independence or atlas invariance with respect to performance. The intended claim is that the benefit of anatomical contextualization persists under a different parcellation. As stated in Section 4.4, “the longitudinal advantage of SBT is preserved”. This is in concordance with the observation that using the Neuromorphometrics atlas, SBT continues to outperform the longitudinal baselines. Although the framework can adapt to different atlases, as discussed in Appendix Sec B2, “Atlas sensitivity must therefore be interpreted with respect to both anatomical coverage and volumetric support”, given HO has 69 parcels and Neuromorphometrics has 136 parcels. These properties jointly change the anatomical contexts available to the encoder and may contribute to the performance difference. The comparison therefore supports the persistence of SBT’s benefit across the two evaluated parcellations. We have clarified this in Sec 4.4 in the revised manuscript.

Q: Interpretations:

Parcel saliency expresses the explanation in anatomical units familiar to clinicians. Clinical assessment of AD starts from the patient’s functional deficits, such as impaired memory, disorientation or changes in executive function. Clinicians link these deficits to anatomically defined regions, such as the hippocampus, etc. The progression of AD is also described along such regions, starting in medial temporal structures and spreading to broader cortical areas (Braak & Braak, 1991). A high parcel saliency therefore points to a named structure with a known functional role, and a clinician can check it directly against the patient’s symptoms. For example, high saliency in the hippocampus and parahippocampal gyrus fits a patient whose main complaint is memory loss. Explanations organized by anatomy, and checked for reliability against known ground truth, remain an active area of exploration in longitudinal neuroimaging. SBT attempts to offer one. However, the current scope of SBT was to address the representation and computational limitations of longitudinal MRI modeling while preserving access to regional evidence across visits. Establishing their correspondence with individual patients’ symptoms requires further multimodal validation using cognitive assessments, clinical histories, and complementary biomarkers. We see this as a natural direction for future work, with SBT providing the regional and longitudinal outputs needed to investigate that correspondence.

We thank the reviewer again for the constructive assessment. We hope that our response will address your concerns. If you have any other questions, we will be happy to discuss them with you.

Reviewer_LKZ4OpenReview ↗

Contribution of the work

I appreciate your responses. I still feel you misspecified your core contribution and novelty. The longitudinal analysis of AD, or practically any brain disorder, is historically based on the "structure-before-time" principle. From extracting regional measures by Freesurfer and using linear mixed-effects models to model trajectories, to using CNN to extract local or global image features and using LSTM/RNN to model trajectories (also recent attention-based variants), SBT has always been a first basic design choice. On a side note, MRIs are often affinely registered beforehand, so all previous deep learning works have the capability to extract localized corresponding features.

So having "structure should shape representation formation before time is modeled" as your core hypothesis and "Structure-before-time(SBT) learning principle" as your core contribution actually trivializes your work.

I'm not denying that you might have found a specific design choice / combination of modules that might work better, but the current writing over-states, and more importantly, mis-represents the real contribution.

Reviewer_ZLujOpenReview ↗
Rating
2(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents the longitudinal learning of Brain MRI features with respect to Alzheimer’s disease and progression. The representation learning is done by learning the correspondence of features among and within the parcellations and anatomy of the brain over time. These features are considered deliberately before the temporal nature of the data is modelled with their Structure-before-time (SBT). Experiments show that their method is accurate across a number of well known datasets such as ADNI and AIBL for Alzheimer’s disease (AD).

Critical strengths
  • The approach has merit since AD is a global atrophy of brain tissue and parcellations maintain a sense of shape and size by definition.
  • Learning structure before longitudinal changes are modelled also makes sense and their analysis confirms their assertions well.
  • The authors do a thorough analysis of the model, its evidence and its outcomes with a strong discussion of each in the paper.
  • The use of ADNI and AIBL etc. are good as these are expected datasets for AD imaging analyses.
  • The results seems promising at first glance with clear trends in representation learning as claimed.
Critical weaknesses
  • The paper breaches the formatting requirements of an ICLR submission with a number of tables that use very small fonts (<< 10 size) and figures that are barely legible at 400% Zoom. This may seem like a small issue at first glance, but they are in-place for equity for readability that applies to all submission and unfairly advantages the present submission by fitting in as many results as possible.
  • The paper also presents experiments on the classification of AD, which is has a large literature that is completely been ignored in the review of related works. This deserves at least a paragraph of several references that have done similar work in this area. AIBL is mentioned as one of the datasets but “All scans are processed” in the text seems to imply that models are trained on datasets combined. This is not how it is usually done in the literature and for good reason, AIBL is an independent test set not used for training and all reported scores are therefore much lower. The combined dataset with a 10% val/test split will also inflate the true performance of any model despite the 5-fold split, since medical imaging datasets are usually small.
  • The single scan accuracies presented in table 1 appear to mis-represent the true accuracies of the reported method in the literature. For example, M3T method achieves AUC:0.9258 Acc: 0.9327 reported in their paper but is surprising low AUC: 83.06% Acc; 66.50% as reported in the paper.
  • The overall accuracies of the method are low when compared to the true accuracies of the classification of AD in the literature. ADNI classification accuracy is actually > 0.9 for AD/CN and > 0.8 for MCI tasks. Thus, the method presented is not state-of-the-art in performance in single image.
  • The main result is the Any length history in Table 1, yet it is only compared to the variants of the ViT with no other state of the art method in the literature. This is a severe limitation that points to a low accuracy of the method in general, see previous comment.
  • The method itself has no technical novelties to speak of, models involved are MLPs and ViTs, although not a strict requirement for acceptance and the novelty is in the combination of methods and training which can still be OK.
  • The synthetic results (especially in Figure 7) do not really mean much because of its simplicity and lack of similarity to human brains. In terms of Figure 7, it would be better to spend the real-estate on larger images of the brain results shown It is a valid pre-training strategy but this is nothing new.
Questions for authors

Could you explain the poor performance of the methods when compared to those reported in the literature (WRT to M3T et al.), even despite the combined dataset that seems to be used?

Why was a combined dataset approach used rather the train on ADNI test on AIBL approach in the literature?

The work makes mention of a “Longitudinal Transformer” but is never full defined other than the reference of the original ViT paper. There is substantial work in time series and longitudinal analysis of data that has not been used or mentioned, which seems strange. What was the reason for this design choice?

Reviewer_71vrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Structure Before Time (SBT), which learns anatomically localized representations before spatial and temporal aggregation for longitudinal 3D brain MRI. Its framework supports irregular, variable-length scan histories through scan-level or parcel-level temporal modeling. Validation across four cohorts reports improvements in diagnosis, MCI progression, and survival prediction, while controlled ablations, cross-cohort evaluations, and synthetic experiments support the benefits of early anatomical structuring and preservation of regional disease information.

Critical strengths
  1. Controlled ablations isolate the benefit of early anatomical structuring, strengthening the central methodological claim.

  2. Cross-cohort and synthetic evaluations provide complementary evidence for generalization and preservation of localized disease signals.

Critical weaknesses
  1. The deconfounding results are misinterpreted. A sex-prediction AUC of 0.14 becomes 0.86 when the scores are reversed, and an age correlation of −0.52 still indicates substantial dependence. These results do not establish effective confound removal.

  2. Table A1 lists 305 sMCI and 282 pMCI subjects, which sum to 587, not 687. Please clarify the discrepancy.

  3. The C-index increase across landmarks cannot be attributed solely to additional MRI history. The risk set shrinks from 640 to 419 subjects, with remaining events falling from 235 to 77. History lengths should be compared within the same landmark cohort.

  4. The claim of atlas independence seems overstated. Changing atlases reduces SBT-SL’s AUC from 94.84% to 89.35% for AD/CN and from 85.96% to 81.04% for MCI, showing appreciable sensitivity to atlas choice.

  5. The synthetic experiments mainly compare SBT variants using fixed regions and severity-derived labels. Spatial-grouping and global-model baselines are needed to establish whether SBT offers a localization advantage.

Questions for authors
  1. Were the confound probes retrained after residualization, and are the sex AUC of 0.14 and age correlation of −0.52 reported correctly?

  2. What are the correct sMCI/pMCI cohort counts, and what explains the discrepancy between the listed 305 + 282 subjects and the reported total of 687?

  3. Was the benefit of additional MRI history assessed within the same landmark risk set, or inferred from comparisons across landmarks with different patient populations?

  4. Were spatial-grouping or global-model baselines evaluated on the same synthetic localization task? If so, how did their localization results compare with SBT?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion

We sincerely thank all reviewers for their time, careful assessment, and constructive feedback. Their engagement has helped us strengthen the experimental evidence, clarify the scope and contribution of SBT, and improve the manuscript’s presentation.


We appreciate the positive assessments across the reviews:

  • Approach: “The approach has merit since AD is a global atrophy of brain tissue and parcellations maintain a sense of shape and size by definition.”
  • Methodological evidence: “Controlled ablations isolate the benefit of early anatomical structuring, strengthening the central methodological claim.”
  • Generalization and localization: “Cross-cohort and synthetic evaluations provide complementary evidence for generalization and preservation of localized disease signals.”
  • Experimental breadth: “The breadth of validation analyses is amazing,” with explicit recognition of survival analysis, robustness to preprocessing, and external-data generalizability.
  • Thoroughness: “The authors do a thorough analysis of the model, its evidence and its outcomes with a strong discussion of each in the paper.”
  • Relevance: In the follow-up discussion, Reviewer ZLuj described the novelty as “meaningful for longitudinal analysis” and stated that it “fits representation learning of ICLR.”


We believe our responses and manuscript revisions address all major concerns raised. Specifically, we have:

  • Clarified the subject-wise, within-dataset evaluation protocol, distinguished cross-dataset experiments, and explained the metrics and baseline comparisons, including an additional M3T comparison using matched high-resolution inputs.
  • Retrained confound probes after residualization, revised the interpretation of these results, clarified the MCI cohort counts, and made the within-landmark survival comparisons explicit.
  • Clarified the findings under alternative parcellations and added synthetic localization comparisons with spatial-grouping and global-aggregation baselines.
  • Explained the scope and validation of parcel-level interpretations, including the role of known synthetic abnormalities and the further validation needed for patient-specific clinical correspondence.
  • Refined the hypothesis and contribution statements, enlarged tables and figures, and moved the synthetic-sample illustration to the supplementary material.

SBT’s central contribution concerns how anatomical structure shapes representation formation in deep learning-based longitudinal models. SBT constrains feature contextualization within anatomical regions before broader contextual and temporal aggregation, preserving regional evidence that can be related across visits. Controlled comparisons examine both the anatomical context used and the stage at which it is introduced. Together with evaluations across cohorts, temporal architectures, and prediction tasks, these results support the value of this structural inductive bias for learning from high-resolution, irregular, variable-length MRI histories. The resulting framework offers several advantages: support for high-resolution 3D inputs, scalability to irregular, variable-length longitudinal histories, a favorable predictive performance–inference cost trade-off, and parcel-level interpretability across visits.


We are grateful to all reviewers for helping us make the contribution, supporting evidence, and scope more precise. We sincerely hope that the revisions and clarifications address the reviewers’ concerns, and we would be grateful for their favorable consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.