- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SCMT (Semantic-Guided Cross-Modal Transfer) to address the scarcity of paired functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI) multimodal data in neuroimaging. To tackle the challenge that source-domain unimodal data and target-domain multimodal data cannot be directly transferred due to discrepancies in their structures and representation spaces, SCMT adopts brain regions of interest (ROIs) defined by a brain atlas as a shared bridge for atlas-level knowledge transfer.
Critical strengths
- The migration interface is designed to be lightweight, circumventing the difficulty of cross-architecture parameter transfer: Instead of forcibly transferring the weights of deep encoders or subject-specific embeddings, it extracts ROI-level topological prior matrices (S and R) that are independent of specific model architectures, enabling decoupled cross-modal prior transfer.
- The dual-branch design and annealing mechanism are complementary in logic: It separates prior-free basic cross-modal learning and population semantic prior guidance into two independent branches. Combined with the annealing mechanism, the model regularizes the representation space with priors at the early training stage and gradually fits the real subject data in the later stage. This helps alleviate the suppression of individual heterogeneity imposed by strong priors.
Critical weaknesses
-
The compared methods are outdated and lack representative state-of-the-art baselines. Among the baseline models compared in the paper, the unimodal baselines include many earlier classic models (e.g., BrainNetCNN 2017, GAT 2018, BrainGNN 2021), while the multimodal comparisons are mainly limited to MME-GCN (2022) and a small number of graph models. In the past two years, numerous advanced methods have emerged in self-supervised graph representation learning and multimodal fusion for brain connectomes, especially pretraining for missing medical modalities. The failure to comprehensively include recent strong baselines in the horizontal comparison makes it difficult to objectively reflect SCMT’s true relative level in the field of multimodal brain representation.
-
Performance fluctuates sharply across datasets, and cross-task validation is insufficient (it is recommended to extend to tasks such as brain tumor classification). The cross-domain transfer effect is extremely weak: on same-source transfer ABIDE, the model achieves a significant gain (AUC improvement of about 0.045), but on cross-disease transfer ADNI, AUC only increases marginally from 0.6544 to 0.6556 (an improvement of only 0.0012), accuracy rises slightly by 0.6%, and the standard deviation is large. This large performance gap across datasets directly weakens the persuasiveness of the model’s claim to have “general cross-modal brain atlas transfer capability.” The downstream tasks are single: current validation is limited to binary classification tasks for neuropsychiatric/cognitive degenerative diseases (autism ASD and mild cognitive impairment MCI). It is recommended to extend validation to tasks with clearer clinical phenotypes and more pronounced histopathological differences (e.g., brain tumor subtype classification/grading) to further test the robustness of the semantic transfer mechanism on generalized neuroimaging tasks.
-
The correctness and medical rigor of LLM-generated and LLM-encoded semantics cannot be guaranteed. The paper uses offline ChatGPT to generate functional and anatomical descriptions of each brain region in the AAL atlas, and uses frozen language model (Llama-Encoder) encodings as the source of priors. Risk of hallucination and medical accuracy: general large models are highly prone to hallucination or overly generalized (vague and imprecise) descriptions when generating specialized brain science and anatomical functional descriptions, and the paper has not undergone cross-audit by neuro-medical experts or clinical validation. Lack of causal evidence for the contribution of linguistic semantics: the ablation experiments do not prove the effectiveness of “textual semantics” itself. If the text is replaced with random Gaussian noise, one-hot brain-region IDs, or a statistical correlation matrix computed solely from source-domain FC, would the performance remain the same? In Fig. 5(b), the affinity matrix is significantly correlated with the source-domain mean FC (ρ = 0.396), raising the suspicion that the prior matrix essentially takes effect through source-domain connectome topological statistics rather than genuine “language understanding.”
Questions for authors
see weaknesses above
3 comments
2/3 Response to Reviewer jVaT: Recent Baselines, Cross-Disease Transfer, and Semantic Prior Reliability
Comment 2 — Cross-Disease Transferability and Task Scope
Reviewer Concern: The reviewer highlights the limited ADNI improvement and asks whether the transfer mechanism generalizes beyond ASD/MCI binary classification, including brain-tumor tasks.
Response: We appreciate this concern. We distinguish reuse of an atlas-level transfer mechanism across cohorts from obtaining equally large gains across different diseases. The two target evaluations represent different transfer settings.
Different source–target relationships. Source pretraining uses 655 ABIDE subjects, excluding sites represented in the paired ABIDE target cohort. The ABIDE evaluation therefore examines cross-site transfer within the same disease and classification task. ADNI uses the same ABIDE-derived prior for MCI classification, without ADNI-specific source pretraining, and additionally requires AAL116-to-AAL90 ROI correspondence (Sec. 4.1). Since the transferred relations are calibrated using source FC and diagnosis supervision, equal improvements across these settings are not assumed.
Table R3. Full SCMT versus SCMT-Lite across target cohorts (AUC, mean ± SD).
| Target | SCMT-Lite | Full SCMT | ΔAUC |
|---|---|---|---|
| ABIDE | 0.6496 ± 0.0639 | 0.6943 ± 0.0647 | +0.0447 |
| ADNI | 0.6544 ± 0.0324 | 0.6556 ± 0.0336 | +0.0012 |
Cross-disease performance. On ABIDE, ACC and F1 improve by 4.25 and 5.30 percentage points. On ADNI, SCMT also achieves positive mean gains across all three metrics: +0.60 percentage points in ACC, +0.0012 in AUC, and +0.35 percentage points in F1 (Sec. 4.4). ADNI is a more challenging transfer setting: ABIDE-derived priors are applied to MCI classification without ADNI-specific source pretraining, across different clinical populations and connectivity patterns, with AAL116-to-AAL90 correspondence. Despite these challenges, SCMT reaches 0.6556 AUC on ADNI, exceeding BrainGFM fine-tuned from official pretrained weights (0.6015) and AFR-Net (0.5205). These comparisons demonstrate competitive mean performance in the cross-disease setting, while the smaller gain over SCMT-Lite suggests that source–target differences may affect the benefit of transferred priors.
Additional evidence from source-data scaling. Figure 3 and Appendix B.2 already examine the effect of source-pretraining scale with the ADNI target cohort and downstream settings fixed. Increasing the source pool from 25% to 100% raises ADNI AUC by 0.019, ACC by 2.1 percentage points, and F1 by 2.9 percentage points. This is a distinct source-scale comparison, not the Full-versus-Lite gain. It indicates that source-learning conditions remain relevant to ADNI performance even under cross-disease transfer.
Task scope. ASD and MCI provide two different clinical settings, but both are binary connectivity-based classification tasks. Brain-tumor subtyping and grading are valuable future evaluations; rigorous testing of this same mechanism would require paired fMRI/DTI connectomes with suitable pathological labels and validated ROI correspondence. We will present SCMT as a source-guided cross-modal framework evaluated across two distinct clinical cohorts, highlighting its positive mean improvements and competitive ADNI results while identifying brain-tumor tasks as a valuable direction for further validation.
3/3 Response to Reviewer jVaT: Recent Baselines, Cross-Disease Transfer, and Semantic Prior Reliability
Comment 3 — Semantic Reliability and Prior Specificity
Reviewer Concern: The reviewer questions the medical reliability of LLM-derived ROI descriptions and whether the prior's benefit reflects anatomical alignment, source-FC statistics, or language-derived information.
Response: We thank the reviewer for this important question. We clarify corpus provenance, the role of source-stage calibration, and additional controlled prior replacements.
(1) Corpus provenance and quality. SCMT reuses the fixed AAL ROI descriptions employed by BrainPrompt (MICCAI 2025) and SABER (ICME 2026) (App. E). Section 3.1 of the published BrainPrompt paper reports that domain experts reviewed the original ROI prompts for clinical relevance and correctness. This was an expert review in the source study, not a new independent clinical validation of SCMT. The descriptions specify ROI identity, anatomy, and general function, and are encoded offline using a frozen Llama-Encoder-1.0B. No patient-specific text is generated at training or inference time. We will provide the complete descriptions, resource provenance, and ROI correspondence in the revision.
(2) Interpretation of the transferred prior. The affinity prior is not simply raw text-embedding similarity: source FC reconstruction and diagnostic supervision calibrate the ROI representations before transfer. Therefore, its correlation with source connectivity is consistent with how it is constructed.
Table R4. Anatomical and connectivity properties of the transferred prior (Figure 5 audit).
| Analysis | Result |
|---|---|
| Left–right homologous ROI pairs | 54 |
| Mean homologous-pair weight increase over other pairs | 3.10% |
| Mean homologous-pair weight percentile | 92.6 |
| Spearman ρ: visualized prior vs. source healthy-reference FC | 0.396 |
The relatively high homologous-pair weights support anatomical organization. Figure 5 visualizes a transformed affinity matrix, whereas target learning uses a standardized mapping of the cosine-affinity matrix S. A code audit gives ρ = 0.4227 between S and the same healthy-reference FC. We will distinguish the displayed transformation from the matrix actually deployed.
(3) Controlled prior replacement: anatomical alignment and prior provenance. We designed two complementary controls to distinguish whether the gains depend on correctly aligned ROI relations or can be reproduced by source-connectivity statistics.
Jointly permuted prior. We apply a common ROI permutation to both rows and columns: S_perm = P S Pᵀ. This preserves symmetry, all matrix entries, and the eigenvalue spectrum, but reassigns relational weights to different anatomical ROIs. Thus, it changes anatomical correspondence without changing the prior's global value distribution. We tested three independent permutations (instances 17, 29, and 43); their mean AUCs were 0.6549, 0.6557, and 0.6502, respectively. Their average (0.6536) is 0.0407 below full SCMT (0.6943), supporting the importance of matching transferred relations to the correct anatomical regions.
Source-FC statistical prior. To test whether source-connectivity statistics can reproduce the result, we replaced the text-derived affinity prior with the mean FC of source-training subjects (Table 2(b)). We kept the healthy reference R, target-side consistency objective, fusion mechanism, and training configuration unchanged. This control achieves 0.6786 ± 0.0604 AUC, compared with 0.6943 ± 0.0647 for full SCMT (a mean difference of 0.0157). Source FC statistics provide useful guidance, but this replacement does not match the full configuration's mean performance.
Table R5. Prior replacement and anatomical alignment on ABIDE (AUC, mean ± SD).
| Configuration | ABIDE AUC |
|---|---|
| Full SCMT | 0.6943 ± 0.0647 |
| Joint permutation (instance 17) | 0.6549 ± 0.0787 |
| Joint permutation (instance 29) | 0.6557 ± 0.0572 |
| Joint permutation (instance 43) | 0.6502 ± 0.0788 |
| Source-FC statistical prior | 0.6786 ± 0.0604 |
Evidence for anatomically correct ROI alignment. The joint permutations retain the original relational weights, symmetry, element distribution, and eigenvalue spectrum; the key intervention is their reassignment to different anatomical ROIs. Across instances 17, 29, and 43, mean AUC decreases by 0.0394, 0.0386, and 0.0441, respectively, relative to full SCMT. This consistent reduction despite unchanged global matrix statistics provides direct controlled evidence that correct ROI–anatomical correspondence matters for SCMT's performance. Moreover, replacing the text-derived affinity prior with source-FC statistics yields a lower mean AUC (0.6786 versus 0.6943). Together, these results strengthen the case for transferring source-calibrated regional relations with their correct anatomical alignment. We will include the corpus provenance, matrix-processing definitions, and controlled results in the revision.
Response to Reviewer jVaT: Recent Baselines, Cross-Disease Transfer, and Semantic Prior Reliability
Comment 1 — Recent Baselines and Comparative Evaluation
Reviewer Concern: The reviewer requests stronger recent baselines, particularly brain-graph pretraining and multimodal FC–SC fusion methods.
Response: We thank the reviewer for this suggestion. We have extended our comparisons to BrainGFM (ICLR 2026) and AFR-Net (ICML 2026), representing recent brain-graph pretraining and multimodal FC–SC fusion (Sec. 4.2 and Table 1). BrainGFM is evaluated both from scratch and by fine-tuning its officially released pretrained weights. The two settings use the same architecture and target-side tuning budget. AFR-Net is adapted from its official implementation, retaining adaptive flow routing and weighted SC inputs. All supplementary comparisons use the same target subjects, ROI correspondence, and five outer splits.
Table R1. Recent baseline comparisons on ABIDE (mean ± SD).
Table R2. Recent baseline comparisons on ADNI (mean ± SD).
Compared with pretrained BrainGFM, SCMT achieves higher mean AUC by 0.0206 on ABIDE and 0.0541 on ADNI; compared with AFR-Net, the differences are 0.1446 and 0.1351. SCMT retains the highest mean AUC among the listed configurations on both cohorts. The pretraining resources and input modalities differ across methods and will be explicitly documented.
The original manuscript also includes more recent methods—ALTER (2024), BrainNPT (2024), BrainMass (2024), Cross-GNN (2024), and MMBNF (2025)—in Section 4.2 and Table 1. We will make their coverage more visible and discuss missing-modality pretraining work such as BrainMVP (CVPR 2025) and BrainAnytime (MICCAI 2026), distinguishing volumetric-image settings from our FC–SC connectivity-graph setting. These additions broaden the context for evaluating SCMT against recent approaches.