- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes MICViT, a 3D Vision Transformer for multimodal brain MRI that learns intra-modal and cross-modal interactions at local and global scales through four attention mechanisms. The method is evaluated on brain-age prediction across four datasets (UK Biobank, SOOP, ADNI, and Cam-CAN) and four MRI modalities. The authors report improved performance over established CNN and Transformer baselines, with larger gains when incorporating additional modalities. They also evaluate the proposed method on disease-onset prediction in UK Biobank and Alzheimer's disease classification in ADNI.
Critical strengths
-
Relevant research problem: Multimodal representation learning for neuroimaging is an important and challenging research direction. Combining complementary MRI modalities has the potential to capture different aspects of brain anatomy and pathology.
-
Motivated architectural design: Explicitly modeling modality-specific and cross-modal representations at both local and global scales is a reasonable approach to integrating multimodal information in 3D MRI.
Critical weaknesses
-
Limited methodological novelty: The proposed architecture combines modality-specific and cross-modal attention mechanisms at local and global scales. While this design is intuitive, the contribution appears incremental relative to existing Transformer architectures and multimodal fusion approaches. The paper does not sufficiently establish what fundamentally new methodological insight or generalizable principle MICViT introduces beyond combining these components.
-
Unclear significance of the reported improvements: Although MICViT reportedly outperforms the considered baselines, it is difficult to assess whether the gains are statistically reliable and practically meaningful.
-
Limited evidence of clinical relevance: The additional evaluations on disease-onset prediction and Alzheimer's disease classification are valuable. However, the extent to which MICViT improves these tasks over appropriate task-specific baselines, and whether these improvements are meaningful in practice, remains unclear. Better chronological brain-age prediction alone does not establish that the learned representations capture clinically relevant variation or improve clinical assessment.
-
Limited positioning relative to large-scale self-supervised foundation models: The evaluation focuses on supervised CNN and Transformer baselines. Recent work such as Neuro-JEPA explores self-supervised multimodal brain MRI representation learning through latent prediction, using a substantially larger pretraining corpus and evaluating representations across diverse downstream tasks. Although direct comparisons may be complicated by differences in pretraining data and protocols, the paper would benefit from discussing this line of work and clarifying what the proposed supervised architecture contributes beyond existing representation-learning approaches. This is particularly relevant given the limited sample sizes typical of neuroimaging studies.
-
Insufficient methodological detail for reproducibility: The loss function used to train MICViT is not explicitly specified in the description available for this review. The authors should clearly state the optimization objectives and relevant training details to enable independent reproduction and interpretation of the results.
-
Limited evidence of cross-dataset generalization: Evaluating on multiple datasets is a strength, but does not necessarily demonstrate transfer to unseen cohorts or acquisition settings. Cross-dataset evaluation would provide stronger evidence that the method learns transferable representations despite the heterogeneity of neuroimaging data.
Questions for authors
-
Methodological novelty: What are the fundamental differences between MICViT and existing local/global Transformer architectures and multimodal fusion methods? Beyond combining modality-specific and cross-modal attention at two spatial scales, what new methodological insight does the proposed architecture provide?
-
Contribution to clinical tasks: How does MICViT compare with appropriate task-specific baselines on disease-onset prediction and Alzheimer's disease classification? How substantial and consistent are the improvements, and what evidence supports their clinical relevance?
-
Relation to self-supervised foundation models: How does the proposed approach relate to recent self-supervised brain MRI foundation models, such as Neuro-JEPA? Although a direct comparison may not be straightforward given differences in training data and protocols, can the authors clarify the expected advantages of MICViT relative to large-scale pretrained representations?
-
Reproducibility: What loss function is used to train MICViT for each prediction task? Please specify the objective functions and relevant implementation details.
-
Generalization across datasets: Have the authors evaluated transfer to unseen cohorts or acquisition settings, for example, by training on one dataset and testing on another? Such experiments would help determine whether MICViT learns transferable representations across heterogeneous neuroimaging datasets.
5 comments
to Q4:
Loss functions: Brain Age Prediction
MICViT was trained as a univariate regression model using a linear regression head outputting a scalar prediction. We used mean squared error (MSE) loss between the predicted and the chronological age and mean absolute error (MAE) as evaluation metrics.
Disease-Onset Prediction
Both MICViT and the other baselines were trained as multi-task models to predict survival (risk) for 38 different diseases (phecodes) by optimizing a (mini-batch) approximation to the Cox partial log-likelihood with risk sets computed within mini-batches, separately for each of the 38 endpoints and adding the resulting losses (scaled by the number of events in the mini-batch for the considered endpoint, with a minimum scaling of one; endpoints for which no event is present in a mini-batch contribute a null loss). Each model outputs one log-risk per endpoint and we use (Harrell's) C-index (both per-endpoint and averaged over the 38 endpoints) as evaluation metric.
Alzheimer’s Disease Classification (ADNI)
Both MICViT and the other baselines were trained to discriminate Alzheimer’s Disease (AD) patients from cognitively normal (CN) subjects optimizing a class-balanced focal loss (combining the effective-number-of-samples based class weighting proposed in Cui et al. (CVPR 2019) with focal loss (Lin et al., 2017)) with beta = 0.999 and gamma = 2 (class weights normalized to add up to two and separately computed for each split to avoid information leakage). Each model outputs two logits (for CN and AD) and focal loss is implemented on top of (sigmoid-based) binary cross-entropy separately applied to the two logits.
Optimization and model selection
For all experiments we use Schedule-free AdamW with decoupled weight decay of 0.05 (acting as our main regularization tool together with stochastic depth with drop-path rate of 0.2; no further explicit regularization is used in the losses) and bfloat16 mixed precision training. For brain age prediction we use a learning rate of 1e-3 and a batch size of 2; for disease-onset prediction a learning rate of 1e-3 and a batch size of 4 (the largest we could use without running out of memory given the (multimodal) 3D nature and high resolution of the input MRIs); for AD classification a learning rate of 1e-4, a batch size of 2.. Models are trained for at most 200 epochs using early stopping (patience of 10 for brain age prediction and 20 for the other tasks) and the checkpoint attaining the best validation score (MSE for brain age prediction, Cox loss for disease-onset prediction and unweighted softmax cross-entropy, instead of focal loss, for AD classification) is used for testing. Our main goal is not to develop clinically meaningful risk prediction models (clinical validation on independent (hospital) data will be the focus of a future study) but to use disease-onset prediction as an additional task to comparatively assess the predictive performance of different neural network architectures (MICViT vs. baselines trained using the same training protocol).
to Q5:
MICViT was trained and evaluated on four different datasets (i.e., UKBB, Cam-CAN, ADNI, and SOOP). For each dataset, we trained MICViT from scratch with random weight initializations. We kept the network architecture and training strategies the same for all datasets except for the learning rate, which was selected based on performance on the validation set. All hyperparameter selections were done using only the training and validation sets. Our results show that MICViT can be effectively trained on different datasets collected from different populations with different MRI modalities without changing the network architecture. We did not investigate transferring the learned representations of MICViT pretrained on one dataset to another unseen dataset. This could be an interesting future work to further investigate the generalizability of MICViT. Note that transferring the pretrained weights of MICViT from one dataset to another could be challenging since the MRI modalities used in each dataset are different (only T1 MRI is common across all datasets; please see Table 1 for MRI modalities used in each dataset:
| Characteristic | Cam-CAN | ADNI | SOOP | UKBB |
|---|---|---|---|---|
| n | 613 | 2,384 | 1,062 | 41,404 |
| Women (%) | 49.4 | 51.3 | 47.5 | 52.9 |
| Age (yr) | 54.7 ± 18.6 | 74.5 ± 7.6 | 63.8 ± 13.7 | 63.9 ± 7.7 |
| Modalities | T1, T2, DWI, MTI | T1, FLAIR, STAR, DTI | T1, FLAIR, TRACE, ADC | T1, FLAIR, SWI, DWI |
Therefore, to transfer the weights of MICViT trained on one dataset to another, one might need to only transfer the weights learned using T1 MRI since other MRI modalities are not necessarily acquired using similar MRI protocols and might not be directly transferable. In addition, one should also be careful about whether or not the prediction targets are similarly defined in both datasets. We agree with the reviewer that it could be interesting to investigate if the representations learned using one dataset can generalize to other datasets (even using only T1 MRI for transferability study). We will mention this as a future work. We will also clarify that our current study shows that our proposed network architecture can be trained effectively on different datasets rather than the pretrained weights of our model.
External validation on ADNI (UKBB → ADNI)
Table: MAE (years) for brain age prediction on ADNI when models are trained on UKBB (mean ± SD over 5 splits)
To assess robustness against distributional shift, we further evaluated brain age models trained on UKBB on an external dataset, namely ADNI (n ≈ 471–484 subjects per split, age range: 55–93 years), on which only T1 and FLAIR sequences were available. Predicted brain ages were rescaled to years using mean and standard deviation computed on UKBB. We find that MiCViT outperforms mmFormer and Flex-MoE in terms of MAE at all dropout rates (Bonferroni corrected p < 0.001, Mann–Whitney U test on MAEs across runs). In particular, whereas the performance of Flex-MoE strongly fluctuates across data splits and dropout rates (MAE ranging from 6.81 to 9.86 years), MiCViT MAE remains relatively constant (ranging from 5.26 to 6.08 years across dropout rates). Averaged over all dropout rates (0.05–0.50) and data splits, MiCViT achieves a MAE of 5.68 ± 0.59 years, outperforming mmFormer (7.50 ± 0.89 years) and Flex-MoE (8.69 ± 1.73 years). Most of the error of the two baselines can be explained by a systematic underestimation of age on ADNI (brain-age gap: −2.51, −6.05 and −7.90 years for MiCViT, mmFormer and Flex-MoE respectively), suggesting that the two baselines systematically underestimate the participants’ age on this dataset. Overall, these results show that MiCViT generalises better than baselines to unseen datasets with a reduced number of imaging modalities, without any form of fine-tuning on the target domain.
| Dropout | MICViT MAE [years] | mmFormer MAE [years] | Flex-MoE MAE [years] |
|---|---|---|---|
| 0.05 | 6.07 ± 0.83 | 7.14 ± 0.41 | 7.57 ± 1.07 |
| 0.10 | 5.70 ± 0.48 | 7.96 ± 0.66 | 7.82 ± 0.76 |
| 0.15 | 5.75 ± 0.61 | 7.61 ± 1.36 | 9.45 ± 2.32 |
| 0.20 | 5.42 ± 0.49 | 7.70 ± 0.53 | 9.11 ± 1.32 |
| 0.25 | 6.08 ± 0.34 | 7.75 ± 0.88 | 9.86 ± 1.74 |
| 0.30 | 5.26 ± 0.13 | 7.03 ± 1.11 | 9.24 ± 1.84 |
| 0.35 | 5.47 ± 0.33 | 7.58 ± 0.86 | 9.61 ± 1.87 |
| 0.40 | 5.53 ± 1.06 | 7.39 ± 1.22 | 9.14 ± 0.98 |
| 0.45 | 6.04 ± 0.58 | 7.30 ± 0.67 | 8.25 ± 2.27 |
| 0.50 | 5.44 ± 0.28 | 7.53 ± 1.13 | 6.81 ± 0.51 |
| All | 5.68 ± 0.59 | 7.50 ± 0.89 | 8.69 ± 1.73 |
to Q3:
MICViT factorizes the attention in two aspects: (i) spatial context (local or global) and (ii) modality interaction (within-modality or across-modality). This leads to the design of four attention modules, i.e., Separated Local (SL), Separated Global (SG), Cross Local (CL), and Cross Global (CG). Such a factorization scheme systematically explores both intra-modal and inter-modal interactions at different spatial scales and thus achieves effective multimodal modeling. In addition, to verify the effectiveness of the proposed MICViT, we also compare our method with other Transformer-based methods, including ViT, Swin Transformer, GCViT with different fusion strategies, FlexMoE (MoE-based), and mmFormer (multimodal Transformer). Our MICViT outperforms these methods under the same experimental setting, which further verifies the effectiveness of our method. Although Neuro-JEPA aims to learn transferable representations for brain MRI via self-supervised learning, our MICViT mainly focuses on designing an effective Transformer for modeling intra-modal and inter-modal interactions. Thus, these two methods tackle different problems and are complementary to each other. Particularly, Neuro-JEPA develops a ViT-based model with MoE and trains it with a JEPA-based self-supervised method. Different from Neuro-JEPA, our MICViT explicitly models modality-specific features and feature interactions between different MRI contrasts in both local and global manners. Based on the above analysis, we argue that our MICViT is not directly comparable with Neuro-JEPA because (i) these two methods tackle different problems and (ii) they adopt different training strategies. Specifically, Neuro-JEPA learns general representations for brain MRI via self-supervised learning, while our MICViT is trained in a supervised manner for different brain MRI analysis tasks. Therefore, we argue that it is meaningful to further explore the effectiveness of our MICViT as the basic architecture for self-supervised representation learning. Based on our experimental results that MICViT outperforms other ViT-based and MoE-based methods, we conjecture that our MICViT with local-global and intra-modal-inter-modal modeling may also be effective for self-supervised learning. We will explore such an interesting problem in our future work. Note that it is meaningful but non-trivial to incorporate our MICViT into the self-supervised learning framework developed in Neuro-JEPA because our MICViT has a hierarchical architecture to model multimodal data.