acceptodds
Under review as a conference paper at ICLR 2027

Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI

Abstract

Brain MRIs pose a fundamental challenge for machine learning: models must learn from high-dimensional 3D data spanning multiple co-registered modalities, despite the limited sample sizes typical of neuroimaging studies relative to the diversity in anatomy, pathology, and acquisition conditions. While multimodal imaging provides complementary information critical for clinical interpretation, effectively integrating these signals remains difficult. We propose Multimodal Intra- and Cross-Context Vision Transformer (MICViT), a 3D Vision Transformer that explicitly models both modality-specific representations and cross-modal interactions across local and global contexts. Concretely, MICViT combines four attention mechanisms: modality-specific local and global attention for intra-modal feature learning, and cross-modal local and global attention to capture interactions between modalities. We evaluate MICViT on brain age prediction across four heterogeneous datasets (UK Biobank, SOOP, ADNI, Cam-CAN) using multiple MRI modalities (e.g., T1, FLAIR, DWI, SWI). MICViT consistently achieves better results than established CNN and Transformer baselines in 3D settings. Notably, it benefits more strongly from multimodal inputs, yielding larger performance gains as additional modalities are incorporated. Beyond age prediction, we further evaluate the learned representations on disease-onset prediction in UK Biobank and Alzheimer’s disease classification in ADNI, where MICViT shows competitive to strong performance across clinically relevant endpoints. These results suggest that explicitly modeling intra- and cross-modal interactions is a promising direction for representation learning in multimodal neuroimaging. Code is available at https://anonymous.4open.science/r/MICViT-C057

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal MRI Representation Learning, 3D Medical MRI Analysis, Vision Transformers, Cross-Modal Attention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eEg3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes MICViT, a 3D Vision Transformer for multimodal brain MRI that learns intra-modal and cross-modal interactions at local and global scales through four attention mechanisms. The method is evaluated on brain-age prediction across four datasets (UK Biobank, SOOP, ADNI, and Cam-CAN) and four MRI modalities. The authors report improved performance over established CNN and Transformer baselines, with larger gains when incorporating additional modalities. They also evaluate the proposed method on disease-onset prediction in UK Biobank and Alzheimer's disease classification in ADNI.

Critical strengths
  • Relevant research problem: Multimodal representation learning for neuroimaging is an important and challenging research direction. Combining complementary MRI modalities has the potential to capture different aspects of brain anatomy and pathology.

  • Motivated architectural design: Explicitly modeling modality-specific and cross-modal representations at both local and global scales is a reasonable approach to integrating multimodal information in 3D MRI.

Critical weaknesses
  • Limited methodological novelty: The proposed architecture combines modality-specific and cross-modal attention mechanisms at local and global scales. While this design is intuitive, the contribution appears incremental relative to existing Transformer architectures and multimodal fusion approaches. The paper does not sufficiently establish what fundamentally new methodological insight or generalizable principle MICViT introduces beyond combining these components.

  • Unclear significance of the reported improvements: Although MICViT reportedly outperforms the considered baselines, it is difficult to assess whether the gains are statistically reliable and practically meaningful.

  • Limited evidence of clinical relevance: The additional evaluations on disease-onset prediction and Alzheimer's disease classification are valuable. However, the extent to which MICViT improves these tasks over appropriate task-specific baselines, and whether these improvements are meaningful in practice, remains unclear. Better chronological brain-age prediction alone does not establish that the learned representations capture clinically relevant variation or improve clinical assessment.

  • Limited positioning relative to large-scale self-supervised foundation models: The evaluation focuses on supervised CNN and Transformer baselines. Recent work such as Neuro-JEPA explores self-supervised multimodal brain MRI representation learning through latent prediction, using a substantially larger pretraining corpus and evaluating representations across diverse downstream tasks. Although direct comparisons may be complicated by differences in pretraining data and protocols, the paper would benefit from discussing this line of work and clarifying what the proposed supervised architecture contributes beyond existing representation-learning approaches. This is particularly relevant given the limited sample sizes typical of neuroimaging studies.

  • Insufficient methodological detail for reproducibility: The loss function used to train MICViT is not explicitly specified in the description available for this review. The authors should clearly state the optimization objectives and relevant training details to enable independent reproduction and interpretation of the results.

  • Limited evidence of cross-dataset generalization: Evaluating on multiple datasets is a strength, but does not necessarily demonstrate transfer to unseen cohorts or acquisition settings. Cross-dataset evaluation would provide stronger evidence that the method learns transferable representations despite the heterogeneity of neuroimaging data.

Questions for authors
  1. Methodological novelty: What are the fundamental differences between MICViT and existing local/global Transformer architectures and multimodal fusion methods? Beyond combining modality-specific and cross-modal attention at two spatial scales, what new methodological insight does the proposed architecture provide?

  2. Contribution to clinical tasks: How does MICViT compare with appropriate task-specific baselines on disease-onset prediction and Alzheimer's disease classification? How substantial and consistent are the improvements, and what evidence supports their clinical relevance?

  3. Relation to self-supervised foundation models: How does the proposed approach relate to recent self-supervised brain MRI foundation models, such as Neuro-JEPA? Although a direct comparison may not be straightforward given differences in training data and protocols, can the authors clarify the expected advantages of MICViT relative to large-scale pretrained representations?

  4. Reproducibility: What loss function is used to train MICViT for each prediction task? Please specify the objective functions and relevant implementation details.

  5. Generalization across datasets: Have the authors evaluated transfer to unseen cohorts or acquisition settings, for example, by training on one dataset and testing on another? Such experiments would help determine whether MICViT learns transferable representations across heterogeneous neuroimaging datasets.

5 comments

MICViT factorizes the attention in two aspects: (i) spatial context (local or global) and (ii) modality interaction (within-modality or across-modality). This leads to the design of four attention modules, i.e., Separated Local (SL), Separated Global (SG), Cross Local (CL), and Cross Global (CG). Such a factorization scheme systematically explores both intra-modal and inter-modal interactions at different spatial scales and thus achieves effective multimodal modeling. In addition, to verify the effectiveness of the proposed MICViT, we also compare our method with other Transformer-based methods, including ViT, Swin Transformer, GCViT with different fusion strategies, FlexMoE (MoE-based), and mmFormer (multimodal Transformer). Our MICViT outperforms these methods under the same experimental setting, which further verifies the effectiveness of our method. Although Neuro-JEPA aims to learn transferable representations for brain MRI via self-supervised learning, our MICViT mainly focuses on designing an effective Transformer for modeling intra-modal and inter-modal interactions. Thus, these two methods tackle different problems and are complementary to each other. Particularly, Neuro-JEPA develops a ViT-based model with MoE and trains it with a JEPA-based self-supervised method. Different from Neuro-JEPA, our MICViT explicitly models modality-specific features and feature interactions between different MRI contrasts in both local and global manners. Based on the above analysis, we argue that our MICViT is not directly comparable with Neuro-JEPA because (i) these two methods tackle different problems and (ii) they adopt different training strategies. Specifically, Neuro-JEPA learns general representations for brain MRI via self-supervised learning, while our MICViT is trained in a supervised manner for different brain MRI analysis tasks. Therefore, we argue that it is meaningful to further explore the effectiveness of our MICViT as the basic architecture for self-supervised representation learning. Based on our experimental results that MICViT outperforms other ViT-based and MoE-based methods, we conjecture that our MICViT with local-global and intra-modal-inter-modal modeling may also be effective for self-supervised learning. We will explore such an interesting problem in our future work. Note that it is meaningful but non-trivial to incorporate our MICViT into the self-supervised learning framework developed in Neuro-JEPA because our MICViT has a hierarchical architecture to model multimodal data.

Loss functions: Brain Age Prediction

MICViT was trained as a univariate regression model using a linear regression head outputting a scalar prediction. We used mean squared error (MSE) loss between the predicted and the chronological age and mean absolute error (MAE) as evaluation metrics.

Disease-Onset Prediction

Both MICViT and the other baselines were trained as multi-task models to predict survival (risk) for 38 different diseases (phecodes) by optimizing a (mini-batch) approximation to the Cox partial log-likelihood with risk sets computed within mini-batches, separately for each of the 38 endpoints and adding the resulting losses (scaled by the number of events in the mini-batch for the considered endpoint, with a minimum scaling of one; endpoints for which no event is present in a mini-batch contribute a null loss). Each model outputs one log-risk per endpoint and we use (Harrell's) C-index (both per-endpoint and averaged over the 38 endpoints) as evaluation metric.

Alzheimer’s Disease Classification (ADNI)

Both MICViT and the other baselines were trained to discriminate Alzheimer’s Disease (AD) patients from cognitively normal (CN) subjects optimizing a class-balanced focal loss (combining the effective-number-of-samples based class weighting proposed in Cui et al. (CVPR 2019) with focal loss (Lin et al., 2017)) with beta = 0.999 and gamma = 2 (class weights normalized to add up to two and separately computed for each split to avoid information leakage). Each model outputs two logits (for CN and AD) and focal loss is implemented on top of (sigmoid-based) binary cross-entropy separately applied to the two logits.

Optimization and model selection

For all experiments we use Schedule-free AdamW with decoupled weight decay of 0.05 (acting as our main regularization tool together with stochastic depth with drop-path rate of 0.2; no further explicit regularization is used in the losses) and bfloat16 mixed precision training. For brain age prediction we use a learning rate of 1e-3 and a batch size of 2; for disease-onset prediction a learning rate of 1e-3 and a batch size of 4 (the largest we could use without running out of memory given the (multimodal) 3D nature and high resolution of the input MRIs); for AD classification a learning rate of 1e-4, a batch size of 2.. Models are trained for at most 200 epochs using early stopping (patience of 10 for brain age prediction and 20 for the other tasks) and the checkpoint attaining the best validation score (MSE for brain age prediction, Cox loss for disease-onset prediction and unweighted softmax cross-entropy, instead of focal loss, for AD classification) is used for testing. Our main goal is not to develop clinically meaningful risk prediction models (clinical validation on independent (hospital) data will be the focus of a future study) but to use disease-onset prediction as an additional task to comparatively assess the predictive performance of different neural network architectures (MICViT vs. baselines trained using the same training protocol).

MICViT was trained and evaluated on four different datasets (i.e., UKBB, Cam-CAN, ADNI, and SOOP). For each dataset, we trained MICViT from scratch with random weight initializations. We kept the network architecture and training strategies the same for all datasets except for the learning rate, which was selected based on performance on the validation set. All hyperparameter selections were done using only the training and validation sets. Our results show that MICViT can be effectively trained on different datasets collected from different populations with different MRI modalities without changing the network architecture. We did not investigate transferring the learned representations of MICViT pretrained on one dataset to another unseen dataset. This could be an interesting future work to further investigate the generalizability of MICViT. Note that transferring the pretrained weights of MICViT from one dataset to another could be challenging since the MRI modalities used in each dataset are different (only T1 MRI is common across all datasets; please see Table 1 for MRI modalities used in each dataset:

CharacteristicCam-CANADNISOOPUKBB
n6132,3841,06241,404
Women (%)49.451.347.552.9
Age (yr)54.7 ± 18.674.5 ± 7.663.8 ± 13.763.9 ± 7.7
ModalitiesT1, T2, DWI, MTIT1, FLAIR, STAR, DTIT1, FLAIR, TRACE, ADCT1, FLAIR, SWI, DWI

Therefore, to transfer the weights of MICViT trained on one dataset to another, one might need to only transfer the weights learned using T1 MRI since other MRI modalities are not necessarily acquired using similar MRI protocols and might not be directly transferable. In addition, one should also be careful about whether or not the prediction targets are similarly defined in both datasets. We agree with the reviewer that it could be interesting to investigate if the representations learned using one dataset can generalize to other datasets (even using only T1 MRI for transferability study). We will mention this as a future work. We will also clarify that our current study shows that our proposed network architecture can be trained effectively on different datasets rather than the pretrained weights of our model.

External validation on ADNI (UKBB → ADNI)

Table: MAE (years) for brain age prediction on ADNI when models are trained on UKBB (mean ± SD over 5 splits)

To assess robustness against distributional shift, we further evaluated brain age models trained on UKBB on an external dataset, namely ADNI (n ≈ 471–484 subjects per split, age range: 55–93 years), on which only T1 and FLAIR sequences were available. Predicted brain ages were rescaled to years using mean and standard deviation computed on UKBB. We find that MiCViT outperforms mmFormer and Flex-MoE in terms of MAE at all dropout rates (Bonferroni corrected p < 0.001, Mann–Whitney U test on MAEs across runs). In particular, whereas the performance of Flex-MoE strongly fluctuates across data splits and dropout rates (MAE ranging from 6.81 to 9.86 years), MiCViT MAE remains relatively constant (ranging from 5.26 to 6.08 years across dropout rates). Averaged over all dropout rates (0.05–0.50) and data splits, MiCViT achieves a MAE of 5.68 ± 0.59 years, outperforming mmFormer (7.50 ± 0.89 years) and Flex-MoE (8.69 ± 1.73 years). Most of the error of the two baselines can be explained by a systematic underestimation of age on ADNI (brain-age gap: −2.51, −6.05 and −7.90 years for MiCViT, mmFormer and Flex-MoE respectively), suggesting that the two baselines systematically underestimate the participants’ age on this dataset. Overall, these results show that MiCViT generalises better than baselines to unseen datasets with a reduced number of imaging modalities, without any form of fine-tuning on the target domain.

DropoutMICViT MAE [years]mmFormer MAE [years]Flex-MoE MAE [years]
0.056.07 ± 0.837.14 ± 0.417.57 ± 1.07
0.105.70 ± 0.487.96 ± 0.667.82 ± 0.76
0.155.75 ± 0.617.61 ± 1.369.45 ± 2.32
0.205.42 ± 0.497.70 ± 0.539.11 ± 1.32
0.256.08 ± 0.347.75 ± 0.889.86 ± 1.74
0.305.26 ± 0.137.03 ± 1.119.24 ± 1.84
0.355.47 ± 0.337.58 ± 0.869.61 ± 1.87
0.405.53 ± 1.067.39 ± 1.229.14 ± 0.98
0.456.04 ± 0.587.30 ± 0.678.25 ± 2.27
0.505.44 ± 0.287.53 ± 1.136.81 ± 0.51
All5.68 ± 0.597.50 ± 0.898.69 ± 1.73
Reviewer_op3yOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper primarily investigates feature representation learning and information fusion in multimodal 3D brain magnetic resonance imaging (MRI). Inspired by the reading and reasoning processes of clinical radiologists, it proposes a novel 3D vision Transformer architecture named MICViT (Multimodal Intra- and Cross-Context Vision Transformer). In clinical radiological reading, radiologists typically evaluate focal lesions in specific brain regions within the macroscopic whole-brain structure of that modality, and integrate complementary histological contrast information from other MRI sequences at the same anatomical locations. Inspired by this, MICViT extends the 2D Global Context Vision Transformer (GC ViT) to the 3D multimodal domain, explicitly decoupling intra-modal feature extraction from cross-modal feature interaction within a unified framework and integrating them across both local and global scales.

Critical strengths
  1. The experimental scale is substantial and the empirical evidence is extremely solid: the study spans 4 heterogeneous datasets with a total of over 45,000 samples, and rigorous 5-fold cross-validation and multi-scale parameter scaling analyses were completed on large-scale real-world data (41,000 cases from UKBB). In-depth training and evaluation were conducted on full-resolution (2253225^3) inputs, and rigorous comparisons with statistical significance testing (pp-values) were provided.

  2. The baselines are strong and highly convincing: the comparison baselines are broad, covering general 3D CNNs (ResNet, ConvNeXt), 3D ViTs (Vanilla ViT, Swin-T, GC ViT), and SOTA models specialized for multimodal medical data (mmFormer, Flex-MoE), with a complete comparison against Late-fusion variants. To ensure the fairness and competitiveness of the baselines, the authors performed 100 Bayesian hyperparameter optimization trials for ConvNeXt, constructing an extremely strong comparison target.

  3. The ablation studies are extremely detailed, with a closed causal chain among components: in the ablation experiments, not only was the independent removal of each module tested (Table 4 and Table 13), but Appendix Table 5 even exhaustively enumerated all permutations of the 4 attention modules within a single-layer network, empirically and rigorously demonstrating the synergistic optimality of the "SL →\to SG →\to CL →\to CG" pipeline order. Combined with Grad-CAM attribution heatmap visualizations, it clearly demonstrates the interpretability of the 4 modules in responding respectively to fine gray-white matter structures, ventricular enlargement, cross-modal lesion alignment, and whole-brain aging.

  4. The architecture has excellent theoretical complexity (linear scaling): owing to 3D local window partitioning and a shared global query mechanism, the computational complexity of self-attention scales linearly with the number of spatial tokens, O(N)\mathcal{O}(N), avoiding the catastrophic quadratic complexity O(N2)\mathcal{O}(N^2) of full 3D voxel self-attention, thereby providing practical computational feasibility for processing large-sized volumetric MRI.

Critical weaknesses
  1. Downstream clinical tasks lack dense local lesion prediction (it is recommended to add a brain tumor segmentation task). Shortcoming: all current tasks (brain age prediction, disease survival analysis, AD classification) are essentially global regression and classification tasks at the whole-brain image level (image-level / subject-level). Recommendation: since one of the authors' core innovations lies in "cross-modal local interaction (CL) extracting microscopic complementary signals from different sequences at the same anatomical location," the task that can best demonstrate this capability should be a highly challenging local dense prediction task—such as multimodal brain tumor segmentation (BraTS dataset) or multiple sclerosis white matter lesion segmentation. The absence of validation on segmentation tasks means that the model's power at the level of fine-grained local modeling has not been most intuitively demonstrated.

  2. Modality-missing handling remains rather passive, and comparison with cutting-edge pretraining paradigms (e.g., BrainMVP/M3AE) is lacking. Shortcoming: although the paper designs an evaluation of modality missing on ADNI and outperforms mmFormer and Flex-MoE under this setting, its network design still presumes complete modality alignment in its original intent. Recommendation: the current frontier of multimodal medical imaging places strong emphasis on large-scale self-supervised pretraining via cross-modal masked autoencoders (e.g., BrainMVP, M3AE), which naturally overcome modality missing and learn highly robust cross-modal reconstruction representations. If the paper could experimentally compare MICViT with such representative masked pretraining frameworks or discuss them in depth, it would take a further step toward methodological completeness.

Questions for authors

Overall, this is a good paper. I would consider raising my score if all the weaknesses above were addressed.

3 comments

Q1:

We thank the reviewer for this valuable suggestion. MICViT is also versatile to other dense prediction tasks. Following the reviewer’s valuable suggestion, we further incorporate a U-Net decoder to MICViT and conduct additional experiments on the multimodal brain tumor segmentation task using BraTS 2024. All the models are trained from scratch without pretraining in the same experiment protocol. They are evaluated on 5 folds using the same hold-out test set. The mean Dice ± std is reported.

ModelWT Dice ↑TC Dice ↑ET Dice ↑
MICViT (ours)0.8624 ± 0.00200.7796 ± 0.00370.5760 ± 0.0042
MICViT (without CL/CG)0.8314 ± 0.02280.7168 ± 0.04080.5258 ± 0.0273
mmFormer0.8457 ± 0.00160.7503 ± 0.00630.5629 ± 0.0089

Comparison with mmFormer. MICViT achieved higher mean Dice scores than mmFormer across all three tumor regions, with absolute differences of +0.0167 (WT), +0.0293 (TC), and +0.0131 (ET) in the evaluated setting. Cross-modal attention comparison. We additionally compared the full SL–SG–CL–CG architecture with a parameter-comparable SL–SG–SL–SG variant, replacing cross-modal attention with additional intra-modal attention while maintaining the same number of attention modules. The full architecture achieved higher mean Dice scores by +0.0310 (WT), +0.0628 (TC), and +0.0502 (ET). These additional experiments provide preliminary evidence that MICViT can be adapted to voxel-level segmentation and that incorporating explicit cross-modal attention may be beneficial for this task. Further targeted ablations and evaluation across additional datasets would be necessary to establish the generality of these observations.

to Q2.1:

Table: Brain-age prediction under controlled training-modality missingness on UK Biobank. Values represent MAE (years, mean ± SD across five matched splits). All models are evaluated on held-out test subjects with all four MRI modalities available. Lower is better.

Model0%15%30%35%50%
MICViT2.816 ± 0.0622.893 ± 0.0182.982 ± 0.1423.209 ± 0.0393.187 ± 0.112
mmFormer3.179 ± 0.0943.199 ± 0.0363.311 ± 0.0443.290 ± 0.0583.335 ± 0.089
FlexMoE3.086 ± 0.1453.153 ± 0.0513.204 ± 0.0613.277 ± 0.1683.453 ± 0.120
GCViT3.342 ± 0.0653.458 ± 0.0523.661 ± 0.1383.761 ± 0.3073.633 ± 0.116
ConvNeXt3.447 ± 0.0393.618 ± 0.0373.793 ± 0.0423.848 ± 0.0453.909 ± 0.101

Missingness rates denote the proportion of training subjects missing exactly one, two, or three modalities, respectively. At 30%, only 10% of training subjects retain all four modalities. At ≥35%, no complete-modality training subjects remain. At these higher rates, subjects are distributed approximately equally across the three missingness levels. We appreciate the reviewer’s suggestion to compare MICViT with other methods for handling missing modalities, especially self-supervised pretraining methods such as BrainMVP and M3AE. We would like to point out that although MICViT can be trained on data with missing modalities, it explicitly handles missing modalities by modality-aware attention masking. More specifically, after patch embedding, the tokens from missing modalities are masked out, i.e., not included as image queries and keys in both intra-modal (SL/SG) and cross-modal (CL/CG) attention; they are also masked out before feature aggregation (tokens for register are still available in attention so that the attention operation is well-defined). Therefore, MICViT can accept various (non-empty) modality combinations without changing the network architecture and parameters.

To further validate MICViT’s effectiveness when trained on incomplete multimodal data, we conducted more experiments on brain-age prediction with UK Biobank using four modalities (T1, FLAIR, DWI, SWI). For each subject, we followed dataset-split-level missingness to simulate heterogeneous acquisition protocols, i.e., binary masks indicating the availability of modalities were generated in advance and remained the same in each training batch, so that subjects with missing modalities do not have access to complete modalities during different training iterations. In total, we considered 10 missing modalities settings with different missing rates (5–50%) in addition to the baseline setting without missing modalities (0%) and repeated the experiments using 5-fold cross-validation. Missing rate here indicates the percentage of subjects with 1/2/3 modalities missing. For example, when missing rate is set to 15%, 15% of the subjects will randomly have one of the four modalities missing, 15% will randomly have two modalities missing, 15% will randomly have three modalities missing, and the remaining 55% will have all modalities available.

When tested on the subjects with complete modalities (four modalities available), as shown in Table, MICViT consistently achieved the best (lowest) brain-age prediction performance (MAE) averaged over 5-fold cross-validation among the five compared methods (MICViT, GCViT, ConvNeXt, mmFormer, FlexMoE) for all the 11 training settings. Particularly, although trained on data with missing modalities, when missing rate was set to 50%, MICViT still achieved a competitive brain-age prediction performance (3.187 ± 0.112 years in MAE), slightly higher than that trained on complete multimodal data (2.816 ± 0.062 years in MAE, +13.2% MAE), and outperformed other compared methods (mmFormer: 3.335 ± 0.089, FlexMoE: 3.453 ± 0.120, GCViT: 3.633 ± 0.116, ConvNeXt: 3.909 ± 0.101). These results further demonstrated MICViT’s effectiveness in learning informative multimodal representations even when trained on substantially incomplete multimodal data.

to Q2.2:

Masked (multimodal) pretraining as in BrainMVP and M3AE is a promising direction to improve multimodal medical image analysis. While MICViT explicitly models local and global information within and across imaging modalities in a structured way, BrainMVP and M3AE mainly focus on designing effective pretraining tasks to encourage the network to learn informative representations by reconstructing (masked) data across modalities. We believe these two types of methods are complementary to each other, e.g., MICViT can be used as the encoder for multimodal pretraining by additionally introducing masked multimodal reconstruction tasks. We will discuss this in the manuscript. It would be interesting to further validate the effectiveness of MICViT as the encoder for masked multimodal pretraining and compare with other pretraining methods such as BrainMVP and M3AE on various downstream tasks, which we believe can be better addressed in a separate work.

To investigate the potential advantages of self-supervised learning we also perform pretraining with M3AE [Liu et al., AAAI 2023]. The encoder and decoder are the same as for MICViT-UNet (attn_full, dim 96). Following the original paper we use 16³ patches, a masking ratio of 87.5 %, multimodal dropout (MDP, randomly dropping up to three modalities), modality-completion learning (adding an extra learnable image to complete missing modalities) and train for preliminary 75 epochs (due time and server constraints) using MSE loss between the four input modalities and their reconstructions. We pretrain on the training and validation data from the cross validation (CV) setup (to avoid information leakage we do not use the test data for pretraining since this is the same for all CV folds) and afterwards transfer all weights except the last layer (strict shape check). We use the same training setup as for training from scratch (GSAM and schedule-free AdamW optimiser, full resolution of 224³) except for initialising from the pretrained weights. Results:

ModelWT Dice ↑TC Dice ↑ET Dice ↑
M3AE-pretrained0.8653 ± 0.00410.7851 ± 0.00720.5839 ± 0.0081

Given the availability of clusters and limited time, we have not exhaustively explored the potential of self-supervised pretraining. We would like to look into this more systematically, given that our current results show a positive trend.

Reviewer_HMtCOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents MICViT, 3D vision transformer for analysing multimodal brain MRI. MICViT architecture uses separate input stems for each MRI modality and combines four attention mechanisms—separated local, separated global, cross local, and cross global attention—to learn features within and across modalities. Evaluate is conducted on brain-age prediction using UK Biobank, SOOP, Cam-CAN, and ADNI, and further assess its performance on predicting 38 disease-onset endpoints in UK Biobank and distinguishing cognitively normal individuals from those with Alzheimer’s disease in ADNI, including settings with missing modalities. The reported mean absolute error for brain-age prediction on UK Biobank is 2.816 years and 2.404 years.

MICViT is evaluated on four cohorts consist of several modality combinations, two resolutions, a scaling study, two additional downstream tasks, and performed 38-endpoint survival analysis. In additon, MICViT is tested on explicit intra- and cross-modal design at both local and global scales.

Critical strengths

MICViT 3D vision transformer for analysing multimodal brain MRI data, uses separate input stems for each MRI modality and combines four attention mechanisms—separated local, separated global, cross local, and cross global attention to learn features within and across modalities. MICViT is evalauted on brain-age prediction using UK Biobank, SOOP, Cam-CAN, and ADNI, and further assess its performance on predicting 38 disease-onset endpoints in UK Biobank and distinguishing cognitively normal individuals from those with Alzheimer’s disease in ADNI, including settings with missing modalities. Evaluated on four cohorts, several modality combinations, two resolutions, a scaling study, two additional downstream tasks, and performed 38-endpoint survival analysis. In addition, MICViT is tested on explicit intra- and cross-modal design at both local and global scales.

Critical weaknesses

Several methodological issues which made some of its claims difficult to analyze and assess. Ablation studies appear to use much smaller MICViT configuration than MICViT behind the headline results, which raise concerns about whether conclusions regarding individual attention components generalize to the full model.

Similarly, evaluations described as external are conducted independently within each dataset rather than across cohorts, and several performance advantages over strong baselines are not statistically significant.

Importance of all attention components and the evaluation of all attention permutations also appear stronger than the reported evidence supports. In addition, the contribution of the proposed convolutional stem is not isolated, the cross-global attention mechanism requires further clarification, and the interpretability analysis lacks quantitative validation.

Noticed several inconsistencies in sample sizes, reported performance values, and input resolution across sections and tables. Finally, the efficiency analysis is incomplete, downstream prediction results lack uncertainty estimates, and comparisons with recent 3D multimodal transformer and foundation-model approaches are limited. Addressing these issues would improve the reproducibility of the experiments and provide a clearer picture of MICViT’s actual contributions and limitations.

Questions for authors

Why do the all-available and complete-subset ADNI confusion matrices have identical totals? How many subjects are in each setting? What does the CG block compute when its queries are identical across modalities? How much of the gain comes from MICViT’s modality-specific stem? Why are Tables 10 to 12 labeled “external” when training and testing occu Also clarfiy: ablation studies appear to use a much smaller MICViT configuration than the model behind the headline results, raising concerns about whether the conclusions regarding individual attention components generalize to the full model.

3 comments
AuthorsOpenReview ↗

ADNI confusion matrices

To Q1:

The same total number of subjects in the confusion matrices is due to the fact that in both scenarios we use the same test set consisting of subjects with all 4 modalities available. On the other hand, in the "Complete-subset" scenario we only use for training subjects with all 4 modalities available while in the "All-available" scenario we additionally use subjects with less than 4 modalities available for training (so the set of subjects used for training in the "Complete-subset" scenario is a subset of the subjects used for training in the "All-available" scenario). We think that this way of presenting results allows a fair comparison of MICViT against mmFormer and FlexMoE and allows us to evaluate if using data with incomplete set of modalities during training is helpful. We will make that clearer in the manuscript.

To Q2:

In the ADNI dataset, there are 10,639 MRI sessions with seven different modality-availability combinations. For the complete-subset setting, we consider 2,384 sessions, for which all four types of MRI are available. For the all-available setting, we use all 10,639 sessions.

T1DTIFLAIRSTARMRI sessions
✓✗✗✗4,011
✓✗✓✓3,230
✓✓✓✓2,384
✓✗✓✗875
✓✗✗✓62
✓✓✓✗39
✓✓✗✓38
Total10,639

To Q3:

Thanks for your valuable questions. The difference between CG and SG lies in different ways of modeling global dependency. SG adopts global query for each modality to model global dependency within each modality. In contrast, CG employs a shared global query for all modalities to model global dependency across different modalities. In CG, although the queries are shared across different modalities, the attention scores are calculated based on not only query-key similarity but also learnable spatial bias and modality relative positional bias. The modality relative bias is determined by the query and key modalities. As a result, even with the same query, different attention scores can be obtained according to different target modalities to aggregate different global information from keys and values. In other words, CG decouples the shared query generation from target modality related global information aggregation. The shared queries are generated in a data dependent manner, i.e., calculated based on the concatenation of feature representations from all modalities, which encode global information from all modalities. Target modality related global information aggregation is realized through attention scores calculation with modality relative biases. We will add related formulations and explanations in the revised manuscript.

To Q4:

We thank the reviewer for this important question. To further evaluate the effectiveness of modality-specific feature extraction in MICViT, we performed an additional ablation study by replacing the modality-specific stems (Separated Stem) in MICViT with a single stem (Shared Stem) shared by all MRI modalities, while keeping other parts of the network unchanged. Both configurations were evaluated using five matched data splits. The results are summarized below:

ModelMean MAE (years)Std MAE (years)Paired t-test (two-sided p)trainable_param
Shared stem3.0120.1280.0001596,702,597
Separated stem2.2720.046Reference7,597,029

(*During the rebuttal period, we performed more experiments on full-resolution MRIs with different patch sizes. We found that a larger patch size (i.e., 9) leads to better performance for MICViT, with an average MAE of 2.272 ± 0.046 (years) over 5 data splits, compared to 2.404 ± 0.016 (years)). Using modality-specific stems significantly (two-sided paired t-test, t(4) = 13.825, p = 0.000159) decreases the mean MAE by 0.740 years (24.6%) as compared to using a shared stem. Although the model using separated stem has around 13.3% more trainable parameters than that using shared stem and thus the improvement brought by separated stem may result from not only modality-specific feature extraction but also increased model capacity, the ablation results still highlight the importance of modality-specific stems in improving the prediction performance of MICViT and justify our design of performing feature extraction for different MRI contrasts separately before applying intra-modal and cross-modal attentions.

It is worth noting that mmFormer and FlexMoE also utilize modality-specific stems. Moreover, ViT-LF, ResNet-LF, and ConvNeXt-LF are late-fusion methods that adopt different feature extraction methods for each modality. Hence, the performance advantage of MICViT over these methods cannot be solely attributed to the usage of modality-specific feature extraction, despite its effectiveness revealed in our ablation study. It may also be linked to our proposed intra-modal and cross-modal attention modules, although their individual contributions cannot be fully isolated from these comparisons.

To Q5:

The word external may be misleading and give the impression that the models were trained on the UKBB dataset and tested on other datasets (i.e. an external validation). However, we meant by the word external that these were evaluations on datasets other than UKBB, i.e. that models were trained and tested separately on these datasets. To avoid confusion we will use the expression evaluation on additional cohorts instead of external evaluation and adapt the captions of Tables 10 to 12 and the related text accordingly. UKBB was mainly used to develop and investigate MICViT systematically (e.g. ablation, incrementally adding modalities, scaling) and then tested on other datasets and scanning protocols to evaluate its performance on other datasets.

To Q6:

Thanks for your careful check. We have highlighted our two-stage evaluation strategy. Specifically, we first conducted an ablation study to validate the effectiveness of our proposed SL, SG attention as well as CL, CG attention and their different combinations. For this part, we use MICViT-Small to conduct ablation study under different settings since it is computation-consuming to conduct ablation study under different settings for larger MICViT models. To further validate the effectiveness of different attention blocks in the full resolution MRI data, we conduct ablation study on MICViT-Large by removing one type of attention block (Separated Local (SL), Separated Global (SG), Cross Local (CL), Cross Global (CG) attention) while keeping other settings the same as MICViT-Large. All MICViT-Large models are trained and evaluated using five-fold cross-validation with the same data splits as MICViT-Small for paired statistical analysis. The new results are summarized below:

ArchitectureMAE ± SD (years)ΔMAE (years)p-value
MICViT-Large (Full)2.272 ± 0.046——
Without SL2.463 ± 0.118+0.1900.03329
Without SG2.659 ± 0.131+0.3870.00093
Without CL2.736 ± 0.152+0.4640.00314
Without CG2.556 ± 0.133+0.2840.00825

Results in Table show that removing each attention block leads to significantly (unadjusted two-sided paired t-test, p < 0.05) increased prediction error, among which removing CL attention block results in the most increased prediction error (+0.464 years; 20.4%), followed by removing SG (+0.387 years; 17.0%), CG (+0.284 years; 12.5%), and SL (+0.190 years; 8.4%) attention blocks. These results provide evidence of the effectiveness of different attention blocks (capturing local and global spatial attention as well as intra-modal and cross-modal attention) in our proposed model architecture in predicting brain age from full resolution MRI data using MICViT-Large. We will add these results in the revised manuscript. Performance of MICViT-Large ablated by removing different attention blocks in predicting brain age from full resolution MRIs using five-fold cross-validation. Mean ± std of prediction error (in years) are reported. Removing each attention block significantly (unadjusted two-sided paired t-test, p < 0.05) increases prediction error compared to MICViT-Large.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.