- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates neurodegenerative disease diagnosis with incomplete multimodal data. Instead of directly deploying multiple large foundation models or reconstructing missing modalities, the paper proposes to distill modality-specific foundation models into lightweight encoders while aligning their representations in a shared semantic space. It then constructs a patient-modality hypergraph to integrate information from available modalities and similar patients. To this end, it introduces FMD-IMHF, a framework that combines foundation model distillation, cross-modal alignment, and hypergraph fusion. The method is evaluated on ADNI, AIBL, SCAN, and PPMI for AD diagnosis and disease progression prediction. The results demonstrate improvements in diagnostic performance under missing-modality settings while reducing computational costs.
Critical strengths
-
This paper addresses an important problem in multimodal medical diagnosis, where missing modalities are common in real clinical data. I find the combination of foundation model distillation and hypergraph fusion reasonable, as it reduces the computational overhead of multiple foundation models while leveraging information from similar patients to compensate for missing modalities.
-
The evaluation covers multiple datasets and different missing-modality settings. The authors evaluate the method on AD diagnosis and disease progression prediction, and conduct ablation studies to investigate the contributions of different components. The results provide empirical support for the effectiveness of the proposed framework.
Critical weaknesses
-
I have some concerns about the comparison with existing multimodal methods. Several multimodal baselines perform substantially worse than PET-only, even when all modalities are available. For MCI progression prediction, PET-only achieves a bACC of 0.761, while the proposed method achieves only 0.750. Moreover, the unimodal baselines use distilled student encoders, but it is unclear whether the other multimodal baselines benefit from the same pretrained representations. This raises concerns about the fairness of the comparison, making it difficult to determine whether the reported improvements come from the proposed fusion method or differences in feature extraction.
-
The evaluation mainly focuses on binary classification tasks, such as AD vs. NC and sMCI vs. pMCI, without considering more challenging multiclass diagnosis, such as NC vs. MCI vs. AD. Furthermore, the external validation cohorts are relatively small, with only 23 subjects in AIBL and 90 subjects in SCAN. These limitations make it difficult to assess whether the proposed method can generalize to more complex clinical settings and larger independent cohorts.
Questions for authors
-
Why do several multimodal baselines perform worse than PET-only? Are all compared methods evaluated using the same pretrained modality encoders?
-
Why does the evaluation focus only on binary classification tasks? Can the proposed method achieve similar improvements on more challenging three-class diagnosis, such as NC vs. MCI vs. AD?
-
What are the exact clinical features used as model inputs? Although CDR-SB is explicitly excluded, is global CDR also excluded? Are all clinical features collected at baseline rather than during follow-up?
11 comments
The Response to Weakness 1 (Part II)
(2) Regarding the Performance of Multimodal Baselines and the Fairness of Comparisons
Among the compared methods, only Q-Former (ICLR 2026) and our method are designed to incorporate pretrained foundation models as modality-specific feature encoders. Therefore, pretrained foundation models are used only in these two methods, while the other methods follow their respective original implementations.
This comparison setting was adopted because cross-modal knowledge distillation from foundation models is one of the core contributions of this work. Since the other compared methods, except Q-Former, do not incorporate pretrained foundation models, their original implementations were retained. However, this setup did not adequately control for the potential performance gains from pretrained representations. Consequently, their contributions cannot be fully disentangled from those of the proposed fusion method, potentially affecting the fairness of the comparison.
To further disentangle the effects of foundation-model representations from those of fusion methods, an additional set of controlled experiments was conducted. All compared methods used the same distilled student models as modality-specific feature encoders while retaining their respective downstream fusion and prediction modules. Specifically, it employs BrainMVP (UniFormer-S) for MRI, SegAnyPET (ViT-B) for PET, and Qwen2.5-7B for clinical feature extraction, followed by knowledge distillation to obtain lightweight student encoders identical to those used in our method. The Flex-MoE method retained its original feature extraction and fusion pipeline because its fusion mechanism requires spatially structured 3D patch features, whereas the pretrained representations are 512-dimensional global vectors.
The specific experimental results are presented in our comment entitled “The Response to Weakness 1 (Part III)”.
Under the unified feature encoder setting, our method performs well on AD versus NC classification. It outperforms all competing methods across metrics with complete modalities and maintains strong performance with naturally missing modalities, indicating good robustness to modality incompleteness.
For sMCI versus pMCI classification, the advantage is more modest. With complete modalities, our method achieves the highest AUC and sensitivity, while bACC, ACC, and specificity remain below the best competing results. With naturally missing modalities, it achieves the highest AUC, bACC, ACC, and sensitivity; its specificity is slightly lower than that of Q-Former. Overall, our method performs consistently on AD versus NC classification and shows strong discriminative ability on sMCI versus pMCI, particularly under naturally missing modalities. The significance test results further support its overall performance advantage.
In addition, we have added a hypergraph-based comparison method, which also employs a unified feature encoder; the related results and analysis are provided in our comment entitled “The Response to Weakness 1 (Part IV)”.
These experimental results have been added to Sections C.2 (Complete Comparison Results on ADNI) and C.3 (Complete Comparison Results on ADNI Without Unified Feature Encoders) of the Appendix. In addition, the comparison experiments in Table 1 of the manuscript have also been revised accordingly.
The Response to Weakness 1 (Part III)
Below are the comparison experimental results obtained using the unified distilled student encoder, excluding the newly added hypergraph-based comparison method.
Table: Comparison experiment of AD vs. NC for unified feature encoder
(1) Complete multimodal setting
| Method | AUC | bACC | ACC | SEN | SPE |
|---|---|---|---|---|---|
| TMC | 0.893±0.057* | 0.819±0.054** | 0.819±0.059** | 0.827±0.087** | 0.812±0.094* |
| MMIN | 0.950±0.015** | 0.874±0.026** | 0.870±0.028** | 0.907±0.041* | 0.840±0.031** |
| ShaSpec | 0.943±0.021** | 0.856±0.046** | 0.852±0.047** | 0.896±0.054* | 0.817±0.081* |
| DFTD | 0.940±0.029* | 0.866±0.037** | 0.860±0.037** | 0.897±0.055* | 0.836±0.054** |
| 3MT | 0.925±0.039* | 0.841±0.056** | 0.841±0.050** | 0.854±0.095* | 0.829±0.045** |
| DrFuse | 0.922±0.044* | 0.848±0.045** | 0.844±0.047** | 0.875±0.042** | 0.822±0.052** |
| HEALNet | 0.948±0.020* | 0.881±0.021** | 0.875±0.020** | 0.914±0.047* | 0.847±0.036** |
| Flex-MoE | 0.775±0.062** | 0.702±0.073** | 0.695±0.086** | 0.755±0.134* | 0.649±0.200* |
| IMDL | 0.921±0.042* | 0.858±0.037** | 0.854±0.042** | 0.883±0.040** | 0.833±0.072** |
| Q-Former | 0.969±0.015* | 0.886±0.041* | 0.888±0.039* | 0.868±0.089* | 0.904±0.062 |
| Ours | 0.996±0.007 | 0.976±0.011 | 0.974±0.012 | 0.991±0.012 | 0.961±0.018 |
(2) Naturally incomplete multimodal setting
| Method | AUC | bACC | ACC | SEN | SPE |
|---|---|---|---|---|---|
| TMC | 0.889±0.025** | 0.802±0.023** | 0.818±0.027** | 0.763±0.069** | 0.841±0.055* |
| MMIN | 0.940±0.012** | 0.870±0.027** | 0.875±0.024** | 0.857±0.046** | 0.883±0.028** |
| ShaSpec | 0.939±0.010** | 0.865±0.020** | 0.873±0.020** | 0.846±0.045** | 0.885±0.033** |
| DFTD | 0.939±0.012** | 0.868±0.021** | 0.879±0.024** | 0.840±0.040** | 0.896±0.037* |
| 3MT | 0.965±0.009** | 0.907±0.022* | 0.913±0.022* | 0.894±0.039** | 0.921±0.029 |
| DrFuse | 0.929±0.018** | 0.859±0.019** | 0.873±0.013** | 0.824±0.055** | 0.894±0.029* |
| HEALNet | 0.935±0.014** | 0.872±0.017** | 0.876±0.011** | 0.863±0.043** | 0.881±0.019** |
| Flex-MoE | 0.806±0.026** | 0.717±0.032** | 0.748±0.026** | 0.640±0.091** | 0.794±0.053** |
| IMDL | 0.917±0.026** | 0.861±0.025** | 0.870±0.022** | 0.840±0.038** | 0.883±0.022** |
| Q-Former | 0.956±0.011** | 0.885±0.031* | 0.883±0.039* | 0.889±0.030** | 0.881±0.053* |
| Ours | 0.990±0.006 | 0.966±0.013 | 0.966±0.016 | 0.965±0.012 | 0.966±0.022 |
Table: Comparison experiment of sMCIvs. pMCI for unified feature encoder
(1) Complete multimodal setting
| Method | AUC | bACC | ACC | SEN | SPE |
|---|---|---|---|---|---|
| TMC | 0.740±0.061* | 0.649±0.056* | 0.675±0.064 | 0.599±0.042* | 0.699±0.071 |
| MMIN | 0.750±0.039* | 0.650±0.064* | 0.676±0.058 | 0.599±0.077* | 0.701±0.052 |
| ShaSpec | 0.726±0.064* | 0.632±0.072 | 0.653±0.150 | 0.580±0.174 | 0.684±0.226 |
| DFTD | 0.726±0.056* | 0.640±0.050* | 0.676±0.155 | 0.555±0.178 | 0.725±0.242 |
| 3MT | 0.833±0.045 | 0.755±0.029 | 0.772±0.073 | 0.729±0.094 | 0.782±0.116 |
| DrFuse | 0.691±0.063** | 0.618±0.068* | 0.680±0.043 | 0.495±0.210 | 0.741±0.100 |
| HEALNet | 0.658±0.049** | 0.586±0.047** | 0.611±0.102 | 0.531±0.269 | 0.641±0.213 |
| Flex-MoE | 0.659±0.037** | 0.596±0.047** | 0.633±0.059* | 0.524±0.195* | 0.667±0.130 |
| IMDL | 0.762±0.035** | 0.691±0.035** | 0.701±0.039 | 0.674±0.028 | 0.709±0.042 |
| Q-Former | 0.839±0.040 | 0.752±0.039 | 0.779±0.057 | 0.700±0.089 | 0.803±0.090 |
| Ours | 0.850±0.031 | 0.750±0.044 | 0.751±0.089 | 0.748±0.082 | 0.752±0.137 |
(2) Naturally incomplete multimodal setting
| Method | AUC | bACC | ACC | SEN | SPE |
|---|---|---|---|---|---|
| TMC | 0.684±0.066** | 0.633±0.033** | 0.643±0.070* | 0.609±0.205 | 0.658±0.190 |
| MMIN | 0.731±0.048** | 0.668±0.045** | 0.665±0.051* | 0.676±0.132 | 0.660±0.108 |
| ShaSpec | 0.762±0.046* | 0.666±0.062* | 0.649±0.109 | 0.715±0.126 | 0.617±0.207 |
| DFTD | 0.758±0.042** | 0.674±0.058* | 0.673±0.117 | 0.679±0.135 | 0.670±0.230 |
| 3MT | 0.843±0.045 | 0.752±0.041* | 0.750±0.047 | 0.759±0.104 | 0.746±0.091 |
| DrFuse | 0.745±0.070** | 0.671±0.054** | 0.665±0.042* | 0.687±0.144 | 0.655±0.081 |
| HEALNet | 0.715±0.039** | 0.630±0.026** | 0.606±0.068** | 0.700±0.164 | 0.561±0.171 |
| Flex-MoE | 0.718±0.065** | 0.651±0.051** | 0.635±0.031** | 0.696±0.131 | 0.606±0.052* |
| IMDL | 0.740±0.050* | 0.675±0.034** | 0.683±0.039 | 0.660±0.025 | 0.690±0.043 |
| Q-Former | 0.854±0.032 | 0.761±0.048 | 0.768±0.030 | 0.743±0.171 | 0.780±0.095 |
| Ours | 0.863±0.033 | 0.793±0.026 | 0.778±0.041 | 0.834±0.032 | 0.752±0.070 |
The Response to Weakness 1(Part I)
We thank the reviewer for raising this concern.
(1) Regarding the Lower Performance of Multimodal Methods Compared with PET-only
The superior performance of some unimodal baselines over multimodal methods was mainly attributable to two issues in our original experimental design. First, unimodal baselines used all available subjects for each modality, whereas complete multimodal methods were evaluated only on subjects with all three modalities (545 for AD vs. NC and 615 for sMCI vs. pMCI), resulting in larger cohorts for the unimodal baselines. Second, the unimodal baselines employed modality-specific student encoders distilled from foundation models, providing stronger feature representations, whereas the other multimodal baselines did not consistently use the same distilled encoders. To address these issues, we revised the experimental design and conducted additional controlled experiments to ensure fair comparisons.
We repeated the experiments for all three unimodal baselines using the same complete-case subjects as the corresponding multimodal methods, with consistent data splits and evaluation protocols. The results before and after revision are presented below:
Table: Performance of unimodal methods on AD vs. NC.
(1) Before Revision
(2) After Revision
Note: Asterisks indicate statistically significant differences from Ours based on two-sided paired t-tests (*p < 0.05; **p < 0.01).
Table: Performance of unimodal methods on sMCI vs. pMCI.
(1) Before Revision
(2) After Revision
Note: Asterisks indicate statistically significant differences from Ours based on two-sided paired t-tests (*p < 0.05; **p < 0.01).
The revised results reveal substantial changes in several performance metrics. Accordingly, we have updated Table 1(a) in the main manuscript and Table A4 in the Appendix to reflect these changes. We thank the reviewer again for pointing out this issue, which has helped us improve the fairness and reliability of our experimental evaluation.