SCMT: Semantic-guided Cross-modal Transfer for Multimodal Brain Network Learning

Learning from functional and structural brain networks is constrained by the scarcity of paired neuroimaging data. Larger functional-only cohorts offer additional information, but differences in modalities and representations make direct transfer challenging. We propose SCMT, a semantic-guided cross-modal transfer framework that uses atlas-defined brain regions as a shared interface. SCMT calibrates language-derived regional descriptions with source functional connectivity and diagnosis supervision, then transfers regional affinities and a healthy-connectivity reference to multimodal learning. A dual-branch reconstruction model combines subject-specific structure–function relationships with an auxiliary functional view guided by the transferred priors. An annealed consistency objective progressively shifts emphasis from population guidance toward the observed paired data. Experiments on ABIDE I and ADNI show improved classification over the base reconstruction model under label-blind transductive adaptation. Fusion and source-scale analyses further characterize the role of transferred information. These findings support atlas-level transfer as an approach to learning multimodal brain representations from limited paired data. Code will be made publicly available upon acceptance.