acceptodds
Under review as a conference paper at ICLR 2027

SCMT: Semantic-guided Cross-modal Transfer for Multimodal Brain Network Learning

Abstract

Learning from functional and structural brain networks is constrained by the scarcity of paired neuroimaging data. Larger functional-only cohorts offer additional information, but differences in modalities and representations make direct transfer challenging. We propose SCMT, a semantic-guided cross-modal transfer framework that uses atlas-defined brain regions as a shared interface. SCMT calibrates language-derived regional descriptions with source functional connectivity and diagnosis supervision, then transfers regional affinities and a healthy-connectivity reference to multimodal learning. A dual-branch reconstruction model combines subject-specific structure–function relationships with an auxiliary functional view guided by the transferred priors. An annealed consistency objective progressively shifts emphasis from population guidance toward the observed paired data. Experiments on ABIDE I and ADNI show improved classification over the base reconstruction model under label-blind transductive adaptation. Fusion and source-scale analyses further characterize the role of transferred information. These findings support atlas-level transfer as an approach to learning multimodal brain representations from limited paired data. Code will be made publicly available upon acceptance.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Multimodal

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jVaTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCMT (Semantic-Guided Cross-Modal Transfer) to address the scarcity of paired functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI) multimodal data in neuroimaging. To tackle the challenge that source-domain unimodal data and target-domain multimodal data cannot be directly transferred due to discrepancies in their structures and representation spaces, SCMT adopts brain regions of interest (ROIs) defined by a brain atlas as a shared bridge for atlas-level knowledge transfer.

Critical strengths
  1. The migration interface is designed to be lightweight, circumventing the difficulty of cross-architecture parameter transfer: Instead of forcibly transferring the weights of deep encoders or subject-specific embeddings, it extracts ROI-level topological prior matrices (S and R) that are independent of specific model architectures, enabling decoupled cross-modal prior transfer.
  2. The dual-branch design and annealing mechanism are complementary in logic: It separates prior-free basic cross-modal learning and population semantic prior guidance into two independent branches. Combined with the annealing mechanism, the model regularizes the representation space with priors at the early training stage and gradually fits the real subject data in the later stage. This helps alleviate the suppression of individual heterogeneity imposed by strong priors.
Critical weaknesses
  1. The compared methods are outdated and lack representative state-of-the-art baselines. Among the baseline models compared in the paper, the unimodal baselines include many earlier classic models (e.g., BrainNetCNN 2017, GAT 2018, BrainGNN 2021), while the multimodal comparisons are mainly limited to MME-GCN (2022) and a small number of graph models. In the past two years, numerous advanced methods have emerged in self-supervised graph representation learning and multimodal fusion for brain connectomes, especially pretraining for missing medical modalities. The failure to comprehensively include recent strong baselines in the horizontal comparison makes it difficult to objectively reflect SCMT’s true relative level in the field of multimodal brain representation.

  2. Performance fluctuates sharply across datasets, and cross-task validation is insufficient (it is recommended to extend to tasks such as brain tumor classification). The cross-domain transfer effect is extremely weak: on same-source transfer ABIDE, the model achieves a significant gain (AUC improvement of about 0.045), but on cross-disease transfer ADNI, AUC only increases marginally from 0.6544 to 0.6556 (an improvement of only 0.0012), accuracy rises slightly by 0.6%, and the standard deviation is large. This large performance gap across datasets directly weakens the persuasiveness of the model’s claim to have “general cross-modal brain atlas transfer capability.” The downstream tasks are single: current validation is limited to binary classification tasks for neuropsychiatric/cognitive degenerative diseases (autism ASD and mild cognitive impairment MCI). It is recommended to extend validation to tasks with clearer clinical phenotypes and more pronounced histopathological differences (e.g., brain tumor subtype classification/grading) to further test the robustness of the semantic transfer mechanism on generalized neuroimaging tasks.

  3. The correctness and medical rigor of LLM-generated and LLM-encoded semantics cannot be guaranteed. The paper uses offline ChatGPT to generate functional and anatomical descriptions of each brain region in the AAL atlas, and uses frozen language model (Llama-Encoder) encodings as the source of priors. Risk of hallucination and medical accuracy: general large models are highly prone to hallucination or overly generalized (vague and imprecise) descriptions when generating specialized brain science and anatomical functional descriptions, and the paper has not undergone cross-audit by neuro-medical experts or clinical validation. Lack of causal evidence for the contribution of linguistic semantics: the ablation experiments do not prove the effectiveness of “textual semantics” itself. If the text is replaced with random Gaussian noise, one-hot brain-region IDs, or a statistical correlation matrix computed solely from source-domain FC, would the performance remain the same? In Fig. 5(b), the affinity matrix is significantly correlated with the source-domain mean FC (ρ = 0.396), raising the suspicion that the prior matrix essentially takes effect through source-domain connectome topological statistics rather than genuine “language understanding.”

Questions for authors

see weaknesses above

3 comments
AuthorsOpenReview ↗

Response to Reviewer jVaT: Recent Baselines, Cross-Disease Transfer, and Semantic Prior Reliability

Comment 1 — Recent Baselines and Comparative Evaluation

Reviewer Concern: The reviewer requests stronger recent baselines, particularly brain-graph pretraining and multimodal FC–SC fusion methods.

Response: We thank the reviewer for this suggestion. We have extended our comparisons to BrainGFM (ICLR 2026) and AFR-Net (ICML 2026), representing recent brain-graph pretraining and multimodal FC–SC fusion (Sec. 4.2 and Table 1). BrainGFM is evaluated both from scratch and by fine-tuning its officially released pretrained weights. The two settings use the same architecture and target-side tuning budget. AFR-Net is adapted from its official implementation, retaining adaptive flow routing and weighted SC inputs. All supplementary comparisons use the same target subjects, ROI correspondence, and five outer splits.

Table R1. Recent baseline comparisons on ABIDE (mean ± SD).

MethodACC (%)AUCF1 (%)
SCMT70.92 ± 5.890.6943 ± 0.064769.27 ± 6.74
BrainGFM (from scratch)68.73 ± 7.700.6446 ± 0.124764.73 ± 12.28
BrainGFM (official pretrained weights)70.80 ± 8.350.6737 ± 0.122469.23 ± 9.76
AFR-Net66.70 ± 7.250.5497 ± 0.088762.53 ± 11.94

Table R2. Recent baseline comparisons on ADNI (mean ± SD).

MethodACC (%)AUCF1 (%)
SCMT66.19 ± 3.060.6556 ± 0.033665.65 ± 3.27
BrainGFM (from scratch)57.98 ± 4.950.5516 ± 0.072954.75 ± 8.68
BrainGFM (official pretrained weights)62.63 ± 6.040.6015 ± 0.099760.19 ± 9.59
AFR-Net56.76 ± 4.670.5205 ± 0.088254.37 ± 7.05

Compared with pretrained BrainGFM, SCMT achieves higher mean AUC by 0.0206 on ABIDE and 0.0541 on ADNI; compared with AFR-Net, the differences are 0.1446 and 0.1351. SCMT retains the highest mean AUC among the listed configurations on both cohorts. The pretraining resources and input modalities differ across methods and will be explicitly documented.

The original manuscript also includes more recent methods—ALTER (2024), BrainNPT (2024), BrainMass (2024), Cross-GNN (2024), and MMBNF (2025)—in Section 4.2 and Table 1. We will make their coverage more visible and discuss missing-modality pretraining work such as BrainMVP (CVPR 2025) and BrainAnytime (MICCAI 2026), distinguishing volumetric-image settings from our FC–SC connectivity-graph setting. These additions broaden the context for evaluating SCMT against recent approaches.

AuthorsOpenReview ↗

2/3 Response to Reviewer jVaT: Recent Baselines, Cross-Disease Transfer, and Semantic Prior Reliability

Comment 2 — Cross-Disease Transferability and Task Scope

Reviewer Concern: The reviewer highlights the limited ADNI improvement and asks whether the transfer mechanism generalizes beyond ASD/MCI binary classification, including brain-tumor tasks.

Response: We appreciate this concern. We distinguish reuse of an atlas-level transfer mechanism across cohorts from obtaining equally large gains across different diseases. The two target evaluations represent different transfer settings.

Different source–target relationships. Source pretraining uses 655 ABIDE subjects, excluding sites represented in the paired ABIDE target cohort. The ABIDE evaluation therefore examines cross-site transfer within the same disease and classification task. ADNI uses the same ABIDE-derived prior for MCI classification, without ADNI-specific source pretraining, and additionally requires AAL116-to-AAL90 ROI correspondence (Sec. 4.1). Since the transferred relations are calibrated using source FC and diagnosis supervision, equal improvements across these settings are not assumed.

Table R3. Full SCMT versus SCMT-Lite across target cohorts (AUC, mean ± SD).

TargetSCMT-LiteFull SCMTΔAUC
ABIDE0.6496 ± 0.06390.6943 ± 0.0647+0.0447
ADNI0.6544 ± 0.03240.6556 ± 0.0336+0.0012

Cross-disease performance. On ABIDE, ACC and F1 improve by 4.25 and 5.30 percentage points. On ADNI, SCMT also achieves positive mean gains across all three metrics: +0.60 percentage points in ACC, +0.0012 in AUC, and +0.35 percentage points in F1 (Sec. 4.4). ADNI is a more challenging transfer setting: ABIDE-derived priors are applied to MCI classification without ADNI-specific source pretraining, across different clinical populations and connectivity patterns, with AAL116-to-AAL90 correspondence. Despite these challenges, SCMT reaches 0.6556 AUC on ADNI, exceeding BrainGFM fine-tuned from official pretrained weights (0.6015) and AFR-Net (0.5205). These comparisons demonstrate competitive mean performance in the cross-disease setting, while the smaller gain over SCMT-Lite suggests that source–target differences may affect the benefit of transferred priors.

Additional evidence from source-data scaling. Figure 3 and Appendix B.2 already examine the effect of source-pretraining scale with the ADNI target cohort and downstream settings fixed. Increasing the source pool from 25% to 100% raises ADNI AUC by 0.019, ACC by 2.1 percentage points, and F1 by 2.9 percentage points. This is a distinct source-scale comparison, not the Full-versus-Lite gain. It indicates that source-learning conditions remain relevant to ADNI performance even under cross-disease transfer.

Task scope. ASD and MCI provide two different clinical settings, but both are binary connectivity-based classification tasks. Brain-tumor subtyping and grading are valuable future evaluations; rigorous testing of this same mechanism would require paired fMRI/DTI connectomes with suitable pathological labels and validated ROI correspondence. We will present SCMT as a source-guided cross-modal framework evaluated across two distinct clinical cohorts, highlighting its positive mean improvements and competitive ADNI results while identifying brain-tumor tasks as a valuable direction for further validation.

AuthorsOpenReview ↗

3/3 Response to Reviewer jVaT: Recent Baselines, Cross-Disease Transfer, and Semantic Prior Reliability

Comment 3 — Semantic Reliability and Prior Specificity

Reviewer Concern: The reviewer questions the medical reliability of LLM-derived ROI descriptions and whether the prior's benefit reflects anatomical alignment, source-FC statistics, or language-derived information.

Response: We thank the reviewer for this important question. We clarify corpus provenance, the role of source-stage calibration, and additional controlled prior replacements.

(1) Corpus provenance and quality. SCMT reuses the fixed AAL ROI descriptions employed by BrainPrompt (MICCAI 2025) and SABER (ICME 2026) (App. E). Section 3.1 of the published BrainPrompt paper reports that domain experts reviewed the original ROI prompts for clinical relevance and correctness. This was an expert review in the source study, not a new independent clinical validation of SCMT. The descriptions specify ROI identity, anatomy, and general function, and are encoded offline using a frozen Llama-Encoder-1.0B. No patient-specific text is generated at training or inference time. We will provide the complete descriptions, resource provenance, and ROI correspondence in the revision.

(2) Interpretation of the transferred prior. The affinity prior is not simply raw text-embedding similarity: source FC reconstruction and diagnostic supervision calibrate the ROI representations before transfer. Therefore, its correlation with source connectivity is consistent with how it is constructed.

Table R4. Anatomical and connectivity properties of the transferred prior (Figure 5 audit).

AnalysisResult
Left–right homologous ROI pairs54
Mean homologous-pair weight increase over other pairs3.10%
Mean homologous-pair weight percentile92.6
Spearman ρ: visualized prior vs. source healthy-reference FC0.396

The relatively high homologous-pair weights support anatomical organization. Figure 5 visualizes a transformed affinity matrix, whereas target learning uses a standardized mapping of the cosine-affinity matrix S. A code audit gives ρ = 0.4227 between S and the same healthy-reference FC. We will distinguish the displayed transformation from the matrix actually deployed.

(3) Controlled prior replacement: anatomical alignment and prior provenance. We designed two complementary controls to distinguish whether the gains depend on correctly aligned ROI relations or can be reproduced by source-connectivity statistics.

Jointly permuted prior. We apply a common ROI permutation to both rows and columns: S_perm = P S Pᵀ. This preserves symmetry, all matrix entries, and the eigenvalue spectrum, but reassigns relational weights to different anatomical ROIs. Thus, it changes anatomical correspondence without changing the prior's global value distribution. We tested three independent permutations (instances 17, 29, and 43); their mean AUCs were 0.6549, 0.6557, and 0.6502, respectively. Their average (0.6536) is 0.0407 below full SCMT (0.6943), supporting the importance of matching transferred relations to the correct anatomical regions.

Source-FC statistical prior. To test whether source-connectivity statistics can reproduce the result, we replaced the text-derived affinity prior with the mean FC of source-training subjects (Table 2(b)). We kept the healthy reference R, target-side consistency objective, fusion mechanism, and training configuration unchanged. This control achieves 0.6786 ± 0.0604 AUC, compared with 0.6943 ± 0.0647 for full SCMT (a mean difference of 0.0157). Source FC statistics provide useful guidance, but this replacement does not match the full configuration's mean performance.

Table R5. Prior replacement and anatomical alignment on ABIDE (AUC, mean ± SD).

ConfigurationABIDE AUC
Full SCMT0.6943 ± 0.0647
Joint permutation (instance 17)0.6549 ± 0.0787
Joint permutation (instance 29)0.6557 ± 0.0572
Joint permutation (instance 43)0.6502 ± 0.0788
Source-FC statistical prior0.6786 ± 0.0604

Evidence for anatomically correct ROI alignment. The joint permutations retain the original relational weights, symmetry, element distribution, and eigenvalue spectrum; the key intervention is their reassignment to different anatomical ROIs. Across instances 17, 29, and 43, mean AUC decreases by 0.0394, 0.0386, and 0.0441, respectively, relative to full SCMT. This consistent reduction despite unchanged global matrix statistics provides direct controlled evidence that correct ROI–anatomical correspondence matters for SCMT's performance. Moreover, replacing the text-derived affinity prior with source-FC statistics yields a lower mean AUC (0.6786 versus 0.6943). Together, these results strengthen the case for transferring source-calibrated regional relations with their correct anatomical alignment. We will include the corpus provenance, matrix-processing definitions, and controlled results in the revision.

Reviewer_iUMWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCMT, a framework for transferring information from a larger fMRI-only source cohort to smaller paired fMRI–DTI cohorts. Source pretraining combines language-derived ROI descriptions with functional connectivity and diagnostic supervision to obtain an atlas-level affinity matrix and a healthy-control connectivity reference. These priors construct a subject-dependent auxiliary functional view, which regularizes a separate target encoder through an annealed consistency objective. Downstream classification combines frozen representations from the base and prior-regularized branches. Experiments use ABIDE I and ADNI under an explicitly transductive protocol, in which target reconstruction includes the unlabeled connectivity of subsequently held-out classification subjects. SCMT improves over SCMT-Lite on ABIDE, while improvements on ADNI are small. Additional analyses examine representation fusion, source-data scale, initialization sensitivity, computational overhead, and the structure of the transferred prior.

Critical strengths

1.A well-motivated atlas-level transfer interface. Transferring ROI relations rather than source encoder parameters provides a plausible way to connect unimodal source learning with a separately trained multimodal target model. The base branch also preserves a pathway that learns directly from observed FC–SC pairs, reducing dependence on the transferred prior. 2.Promising ABIDE results and appropriately qualified conclusions. Relative to SCMT-Lite, SCMT improves ABIDE accuracy by 4.25 percentage points, AUC by 0.0447, and Macro-F1 by 5.30 percentage points. These consistent improvements warrant further investigation. The paper also explicitly acknowledges its transductive setting, the limited ADNI gains, and the inability of the current ablations to isolate the contribution of language semantics.

Critical weaknesses

1.Metric inconsistencies and unclear comparison fairness. Several ADNI results in Table 1 report Macro-F1 above accuracy, which is inconsistent with standard binary classification metrics computed from the same predictions and aggregated identically. The metric implementation and aggregation require clarification. Moreover, it is unclear whether all baselines receive comparable source supervision and unlabeled target-data access, limiting the fairness of the comparisons. 2.The contribution of semantic transfer is not isolated. SCMT adds an encoder, source-derived priors, consistency regularization, and fusion, increasing parameters by 50.6%. Existing ablations do not distinguish semantic benefits from additional capacity or generic regularization. Capacity-matched dual-base and shuffled/non-language prior controls are needed, particularly given the marginal ADNI improvement of 0.0012 AUC.

Questions for authors

1.How do you explain the ADNI entries in Table 1 where the reported Macro-F1 exceeds accuracy? Please provide the metric definitions, aggregation procedure, and per-fold confusion matrices, together with corrected results if necessary. 2.Does SCMT retain its advantage over a capacity-matched pair of independently trained base encoders using the same fusion rule? What happens when the semantic prior is replaced by a jointly row-and-column-permuted prior or a prior derived solely from source FC statistics? 3.Which baselines receive the same source data, source labels, and unlabeled target connectivity as SCMT? Please also clarify how hyperparameters, fusion weights, and checkpoints are selected without using outer-test labels.

3 comments
AuthorsOpenReview ↗

1/3 Response to Reviewer iUMW: Metric Clarification, Controlled Experiments, and Evaluation Protocol

Comment 1 — Clarification of F1-score and Evaluation Metrics

Reviewer Concern: The reviewer questions why F1 exceeds accuracy in several ADNI results and requests clarification of metric definitions and aggregation.

Response:

We thank the reviewer for carefully examining the evaluation metrics. We clarify the F1-score definition and explain why F1 may exceed accuracy in some ADNI results.

Metric definition. The F1 values reported for the discussed baselines correspond to positive-class binary F1. The original table labeling was inaccurate, and we will correct the metric definition and table notes.

For binary classification, F1 = 2TP/(2TP + FP + FN), whereas ACC = (TP + TN)/(TP + TN + FP + FN). Unlike accuracy, binary F1 emphasizes precision and recall for the positive class. Therefore, F1 can exceed accuracy when predictions favor the positive class.

Table R1. Representative ADNI baseline results illustrating binary F1 and accuracy differences (mean ± SD).

MethodACC (%)AUCBinary F1 (%)SPE (%)
BrainGNN (DTI)53.08 ± 6.890.5476 ± 0.043064.27 ± 10.4011.87 ± 14.50
MME-GCN53.41 ± 5.820.5703 ± 0.059254.41 ± 9.2850.93 ± 14.30

For BrainGNN-DTI, specificity is only 11.87%, corresponding to a mean false-positive rate of 88.13%. This indicates a strong tendency toward positive-class predictions, which can produce relatively high positive-class F1 despite limited overall accuracy. Accordingly, BrainGNN-DTI achieves 64.27% F1 versus 53.08% ACC. MME-GCN similarly reports 54.41% F1 versus 53.41% ACC.

These observations are compatible with the standard binary F1 definition and do not indicate an inherent computational contradiction.

Revision. We will correct the F1 terminology, explicitly specify the positive-class convention, and clarify the aggregation procedure. Metrics are computed for each evaluation fold and summarized as mean ± SD across the evaluation runs. We will ensure that the revised tables consistently identify the F1 definition used for each method, allowing the results to be interpreted alongside accuracy, AUC, and class-specific performance. See Table 1 for the F1 heading.

AuthorsOpenReview ↗

2/3 Response to Reviewer iUMW: Metric Clarification, Controlled Experiments, and Evaluation Protocol

Comment 2 — Capacity-Matched Controls and Prior Effectiveness

Reviewer Concern: The reviewer asks whether SCMT benefits from additional capacity or semantic guidance, and requests capacity-matched, permuted-prior, and source-FC controls.

Response:

We thank the reviewer for this constructive suggestion. We conducted additional controlled experiments on ABIDE to distinguish the effects of model capacity, anatomical correspondence, and prior construction.

Capacity-matched comparison. We constructed a control with two independently trained base encoders (Table 2(b) and App. B.3), retaining the same fusion rule, structural representation, downstream classifier, and target-side training budget. Both configurations contain 31,058,690 parameters, ensuring an equal parameter budget.

Table R2. Capacity-matched comparison on ABIDE (mean ± SD).

ConfigurationACC (%)AUCF1 (%)
Full SCMT70.92 ± 5.890.6943 ± 0.064769.27 ± 6.74
Dual-base encoders66.57 ± 5.700.6441 ± 0.058264.22 ± 6.26

SCMT exceeds the capacity-matched control by 4.35 percentage points in ACC, 0.0502 in AUC, and 5.05 percentage points in F1. Thus, introducing an additional base encoder with the same fusion mechanism does not reproduce SCMT's mean performance, supporting the effectiveness of prior-guided representation learning beyond increased capacity alone.

Anatomical correspondence and prior replacement. We further evaluated two types of prior controls.

The jointly permuted prior is constructed as S_perm = P S Pᵀ, where P is an ROI permutation matrix. This transformation preserves the prior's symmetry, value distribution, and eigenvalue spectrum while disrupting its anatomical correspondence.

The source-FC statistical control replaces the text-derived relational prior with mean functional connectivity from source-training subjects. Both controls retain the healthy-connectivity reference R and target-side consistency objective.

Table R3. Effects of anatomical permutation and source-FC prior replacement on ABIDE (AUC, mean ± SD).

ConfigurationABIDE AUC
Full SCMT0.6943 ± 0.0647
Joint permutation (instance 17)0.6549 ± 0.0787
Joint permutation (instance 29)0.6557 ± 0.0572
Joint permutation (instance 43)0.6502 ± 0.0788
Source-FC statistical prior0.6786 ± 0.0604

The three jointly permuted priors achieve an average AUC of approximately 0.6536, 0.0407 below full SCMT. Since joint permutation preserves the matrix's global statistical properties while altering anatomical alignment, this reduction supports the importance of correctly aligned ROI relationships.

The source-FC statistical prior achieves 0.6786 AUC, whereas full SCMT reaches 0.6943, an improvement of 0.0157. This further demonstrates the higher mean performance of the proposed text-guided configuration relative to the statistical-prior alternative.

Summary. These experiments provide complementary evidence: capacity matching shows that additional encoder parameters alone do not reproduce SCMT's performance; joint permutation highlights the importance of anatomical correspondence; and source-FC replacement supports the effectiveness of the proposed prior construction. We will incorporate these controlled comparisons into the revised manuscript.

AuthorsOpenReview ↗

3/3 Response to Reviewer iUMW: Metric Clarification, Controlled Experiments, and Evaluation Protocol

Comment 3 — Baseline Fairness, Pretrained Comparisons, and Model Selection

Reviewer Concern: The reviewer asks which baselines receive comparable source supervision and unlabeled target data, and how model selection excludes outer-test labels.

Response:

We clarify baseline fairness, model selection, and additional pretrained comparisons.

(1) Data partitioning and model selection.

We adopt stratified five-fold outer cross-validation with an inner holdout validation set. Within each outer split, approximately 20% of subjects are reserved for testing, while 20% of the remaining 80% are allocated to inner validation. Thus, classifier training, inner validation, and outer testing account for approximately 64%, 16%, and 20% of the target cohort, respectively.

Table R4. Subject-level partitions for downstream classification.

DatasetTotalTrainingValidationOuter test
ABIDE8654–551417–18
ADNI247157–1584049–50

The downstream partitions are subject-disjoint, and each subject appears in exactly one outer-test fold within a five-fold evaluation run.

Importantly, outer-test labels are never used for model selection. Hyperparameters and fusion weights requiring tuning are selected exclusively using the inner-validation set. Classifier checkpoints are selected based on inner-validation AUC, with ties resolved by lower validation cross-entropy and then earlier training epoch. The selected checkpoint is subsequently evaluated on the outer-test fold, and all reported classification metrics are computed from its predictions.

Source-calibration checkpoints are selected using an independent source validation set, while target representation learning uses its reconstruction-based training objective without target diagnosis supervision.

(2) Source supervision and transductive target learning.

SCMT uses source-domain functional connectivity and diagnostic supervision to learn atlas-level regional relations and a healthy-connectivity reference. These ROI-aligned priors are transferred to a separately trained target model.

Target representation learning follows a label-blind transductive protocol (Sec. 4.1): it can access unlabeled FC–SC connectivity from the complete target cohort, including outer-test subjects, but their diagnosis labels are excluded from representation learning and model selection.

Thus, our evaluation measures held-out classification under transductive target representation learning rather than fully inductive generalization.

(3) Additional comparisons with pretrained and recent baselines.

To broaden the evaluation and address concerns about pretraining resources, we conducted additional experiments with BrainGFM (ICLR 2026), evaluating both training from scratch and fine-tuning from official pretrained weights (Sec. 4.2 and Table 1).

Importantly, BrainGFM was evaluated under two settings: training from scratch and fine-tuning using its officially released pretrained weights.

Table R5. SCMT versus BrainGFM under different pretraining settings (AUC, mean ± SD).

MethodABIDE AUCADNI AUC
SCMT0.6943 ± 0.06470.6556 ± 0.0336
BrainGFM (from scratch)0.6446 ± 0.12470.5516 ± 0.0729
BrainGFM (official pretrained weights)0.6737 ± 0.12240.6015 ± 0.0997

Fine-tuning pretrained BrainGFM improves mean AUC over training from scratch on both ABIDE (0.6737 vs. 0.6446) and ADNI (0.6015 vs. 0.5516), demonstrating the benefit of its pretrained initialization.

Nevertheless, SCMT achieves the highest mean AUC among the listed configurations. Compared with pretrained BrainGFM, SCMT improves mean AUC by 0.0206 on ABIDE and 0.0541 on ADNI.

Supplementary comparisons use the same target subjects and outer splits.

(4) Clarification of baseline information access.

We distinguish the information conditions across methods. BrainGFM uses functional imaging information and is evaluated both from scratch and with officially released pretrained weights, whereas SCMT uses paired FC–SC information and source connectivity with diagnostic supervision to construct transferable regional priors.

The two methods therefore do not receive identical source supervision or input modalities. We will explicitly document these differences in the revised manuscript.

Importantly, our controlled ablation experiments use the same target data, training budget, and fusion strategy as full SCMT. By varying model capacity or prior construction while keeping other settings consistent, these comparisons provide more direct evidence of the contributions of the proposed components.

Overall, these results support SCMT and clarify its evaluation conditions.

Reviewer_SuSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The manuscript proposed SCMT, a semantic-guided cross-modal transformer framework. The model is based on 4 stage approach that start with semantic functional pretraining, then imaging cross-modal reconstruction to the FC matrices, then additional reconstruction of the FC to the hypergraph, and finally downstream classification from graphs. The experiments show improved performance over baselines on the ADNI and ABIDE dataset.

Critical strengths
  • The experiments reported on two datasets.
  • The manuscript include ablation of the model components.
Critical weaknesses
  • The spread in the performance is very high in Table 1 and Table 2, the authors have to perform statistical testing to see whether the findings are significant. For example, Semantic-Guided Deviation-Aware Reconstruction Branch was proposed, however, "semantic branch alone gives only small gains" and "DNI AUC decreases from 0.6544 to 0.6471 when the base functional representation is replaced", while variance is 0.0343. So far, they are comparable, and there is no clear benefit in the component. Please clarify and support the significance of the findings.
  • It seems there is no a hold-out set and the checkpoints or hyperparameters were selected on the validation set. Hence there is possibility of the overfitting. I suggest to create a hold-out set, select above choices only on validation set, and also report generation gap between validation and hold-out set.

Minor:

  • Related work is very general and narrow, there is no literature on hypergraph frameworks that were applied to the brain data or other conditional approaches as semantic guided branch. Need to clarify, how the proposed modules are different, and based on this you need figure out relevant baselines.
  • There is no interpretability analysis and it is not clear how the proposed pushes brain analysis further.
  • It is difficult to read this paper (e.g., "The semantic constraint is gradually annealed, allowing subject-specific multimodal evidence to dominate the final representation")
Questions for authors
  • It is not clear what version and model of ChatGPT-5 was used, and how the description look like.
3 comments
AuthorsOpenReview ↗

1/3 Response to Reviewer SuSr: Robustness, Statistical Evidence, and Evaluation Protocol

Comment 1 — Robustness, Auxiliary-View Contribution, and Evaluation Protocol

Reviewer Concern: The reviewer questions the variability of the reported results, the contribution of the semantic-guided auxiliary view, and whether held-out evaluation and checkpoint selection adequately address overfitting.

Response: We thank the reviewer for highlighting the importance of robust interpretation. First, the reported “±” values are standard deviations (SD), not variances or standard errors. The experiments aggregate four initializations across the same five outer folds; these observations share subjects and should not be interpreted as 20 independent datasets.

Table R1. Original component ablation (AUC, mean ± SD) (Table 2(a)).

ConfigurationABIDEADNI
SCMT-Lite0.6496 ± 0.06390.6544 ± 0.0324
Semantic replacement0.6527 ± 0.05830.6471 ± 0.0343
Full SCMT0.6943 ± 0.06470.6556 ± 0.0336

The distinction between replacement and complementarity is essential. “Semantic replacement” substitutes the auxiliary functional representation for the base functional representation, while retaining the structural branch. Full SCMT instead preserves the base representation and integrates the auxiliary prior-guided view. Consequently, the lower ADNI result of semantic replacement (0.6471) does not contradict the full model's improvement (0.6556): the auxiliary view is intended to complement, not replace, subject-specific functional information.

On ABIDE, full SCMT improves AUC over Lite by 0.0447 and over semantic replacement by 0.0416. The five-fold mean full-minus-Lite AUC differences are positive under each of four initializations: +0.0536, +0.0533, +0.0078, and +0.0640. On the more challenging cross-disease ADNI transfer, the mean gains are smaller (+0.0012 over Lite; +0.0085 over replacement). These results establish the strongest and most consistent observed mean improvement on ABIDE while showing that the complete model retains competitive performance on ADNI.

Full SCMT achieved higher mean AUC than the semantic-prior replacement baseline on both ABIDE (+0.0417) and ADNI (+0.0085), with corresponding Holm-adjusted p-values of 0.024 and 0.048, respectively.

Taken together, these results provide strong evidence for SCMT’s effectiveness and robustness. Positive AUC gains across all four ABIDE initializations show that the improvement is not confined to a single random seed. The gains over semantic replacement on both datasets, together with the Holm-adjusted results, further support the contribution of semantic-guided learning. Notably, SCMT remains competitive on ADNI despite substantial cross-disease differences in populations, connectivity patterns, and atlas correspondence. These findings highlight the value of anatomically informed priors for multimodal brain learning and cross-disease transfer.

Held-out evaluation and checkpoint selection. SCMT uses stratified five-fold outer cross-validation with a separate inner validation split. In each outer fold, approximately 20% of target subjects form the held-out test set; 20% of the remaining 80% is reserved for inner validation. Thus, the approximate whole-cohort proportions are 64% training / 16% validation / 20% testing.

Table R2. Subject-disjoint classification splits per outer fold.

DatasetTotalTrainingInner validationOuter test
ABIDE8654–551417–18
ADNI247157–1584049–50

Each subject appears in exactly one outer-test fold per five-fold evaluation. The best classifier checkpoint is selected exclusively by inner-validation AUC, never by outer-test ACC or AUC. Hyperparameters and fusion weights requiring selection likewise use inner-validation information; outer-test diagnosis labels are reserved solely for final evaluation. The ACC, AUC, and F1 results for each fold are obtained from the same validation-selected checkpoint. Source-calibration checkpoints use a separate source validation set.

We also make the evaluation scope explicit: the target representation-learning stage is label-blind and transductive (§4.1). It may use unlabeled FC–SC connectivity from all target subjects, including those later held out for classification, but never their diagnosis labels for representation learning or model selection. Accordingly, the reported numbers assess held-out supervised classification within a transductive target-cohort protocol, rather than a fully inductive unseen-cohort setting.

The separation of training, validation-based selection, and held-out testing addresses the concern about test-label-driven model selection. To make possible overfitting measurable, we will additionally report the inner-validation versus outer-test metric gaps for the corresponding selected checkpoints; we do not infer the size of these gaps without computing them.

AuthorsOpenReview ↗

2/3 Response to Reviewer SuSr: Robustness, Statistical Evidence, and Evaluation Protocol

Comment 2 — Positioning, Recent Baselines, and Controlled Mechanism Tests

Reviewer Concern: The reviewer requests clearer distinctions from related hypergraph/semantic approaches and stronger evidence that improvements do not merely arise from larger capacity or generic source-connectivity statistics.

Response: We will strengthen the related-work comparison with HGST (Pattern Recognition 2026), multi-view FC–SC hypergraph fusion by Feng et al. (Information Fusion 2026), BrainPrompt (MICCAI 2025), and SABER (ICME 2026) (App. A.1 and A.3). Unlike approaches centered on a single modality, direct paired-modal fusion, or textual conditioning alone, SCMT addresses transfer from an fMRI-only source cohort to a small paired FC–SC target cohort. Source learning calibrates text-initialized ROI relations using source connectivity and diagnostic supervision, yielding regional affinity S and healthy reference R; the target model then integrates a prior-guided auxiliary functional view with a preserved data-driven base view and a decaying consistency objective.

We further evaluated recent models under the same target cohorts and five outer splits.

Table R3. Recent-method comparisons (AUC, mean ± SD).

MethodABIDEADNI
SCMT0.6943 ± 0.06470.6556 ± 0.0336
DHGFormer (AAL-adapted)0.6053 ± 0.12400.5567 ± 0.0960
BrainGFM (scratch)0.6446 ± 0.12470.5516 ± 0.0729
BrainGFM (official pretrained)0.6737 ± 0.12240.6015 ± 0.0997
AFR-Net0.5497 ± 0.08870.5205 ± 0.0882

SCMT obtains the highest mean AUC among these evaluated configurations, exceeding pretrained BrainGFM by 0.0206 on ABIDE and 0.0541 on ADNI (Table 1). We will document modality availability, atlas adaptation, and pretraining differences rather than portray all methods as identical-resource comparisons.

More importantly, a capacity-matched control directly tests whether an extra encoder alone explains the improvement. The full model and dual-base-encoder control each contain 31,058,690 parameters, with the same fusion, structural branch, classifier, and training budget.

Table R4. Capacity-matched comparison on ABIDE (mean ± SD).

ConfigurationParametersACC (%)AUCF1 (%)
Full SCMT31,058,69070.92 ± 5.890.6943 ± 0.064769.27 ± 6.74
Dual-base encoders31,058,69066.57 ± 5.700.6441 ± 0.058264.22 ± 6.26

With identical parameter counts, SCMT improves ACC by 4.35 percentage points, AUC by 0.0502, and F1 by 5.05 percentage points. This directly counters a capacity-only explanation. Additional prior-replacement controls further distinguish anatomical guidance from generic source statistics: joint ROI permutation preserves affinity weights and spectrum but disrupts anatomical assignment (three ABIDE AUCs: 0.6549, 0.6557, 0.6502), whereas replacing the regional prior with mean source-training FC yields 0.6786 ± 0.0604 (Table 2(b)), below full SCMT (0.6943 ± 0.0647). These controls support the importance of correct ROI alignment and the proposed source-calibrated regional prior.

AuthorsOpenReview ↗

3/3 Response to Reviewer SuSr: Robustness, Statistical Evidence, and Evaluation Protocol

Comment 3 — Interpretability, Annealing, and Description Provenance

Reviewer Concern: The reviewer requests more concrete interpretability evidence, a clearer explanation of annealing, and the exact origin and content of the ROI text descriptions.

Response: The manuscript already includes target ROI visualizations (Appendix D, Fig. 4) and an audit of the transferred prior (Fig. 5). We will explain their complementary roles: Fig. 4 visualizes regions highlighted in target tasks, whereas Fig. 5 evaluates the anatomical structure of the fixed transferred affinity, not clinically validated biomarkers.

Table R5. Anatomical organization of the transferred prior.

AuditHomologous-pair weight increaseMean homologous-pair percentileHolm-adjusted permutation p
Fig. 5 visualization matrix3.10%92.600.0002
Deployed affinity matrix S19.90%98.190.0002

The audit covers 54 predefined left–right homologous ROI pairs and uses 10,000 joint ROI-index permutations. Joint permutation preserves matrix values and spectral properties while disrupting ROI identity alignment. For deployed S, the observed mean homologous-pair percentile (98.19) is far above the permutation-null mean (approximately 50.01; central 95% range 44.18–56.18). The corrected p-value supports nonrandom anatomical organization of the prior; it is not a test of downstream classification significance. Fig. 5's Spearman correlation with source healthy-reference FC is ρ = 0.396, consistent with source-connectivity calibration. We will distinguish the visualization matrix from deployed S to avoid conflating their numerical properties.

The model's interpretable computational pathway is individual FC → deviation from healthy reference R → modulation by anatomical affinity S → auxiliary functional view → FC–SC consistency guidance. The prior therefore influences where individual connectivity deviations are emphasized, while the observed subject data still determine which deviations exist.

Annealing refers specifically to gradually reducing the target-side consistency-loss weight to zero (Sec. 3.3): prior-guided agreement shapes early optimization, whereas later optimization is driven by the paired FC–SC objectives. We will state this explicitly and clarify the corresponding module inputs and outputs. We will also review the wording throughout the manuscript and clarify any similar passages that may be difficult to follow.

Finally, SCMT reuses the fixed AAL116 ROI descriptions used in BrainPrompt (MICCAI 2025) and SABER (ICME 2026), with entries ordered by atlas (App. E). Section 3.1 of the BrainPrompt paper reports that domain experts reviewed the original ROI prompts for clinical relevance and correctness as part of that study. SCMT uses the same fixed descriptions for all subjects and encodes them offline with a frozen Llama-encoder-1.0B. Example descriptions include Precentral_L (“The left precentral gyrus, associated with motor control and planning”) and Insula_L (“The left insular cortex, critical for interoception, pain perception, and emotional awareness”). We will provide the full corpus, ROI ordering, encoder configuration, and fixed embeddings to support reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.