UniBrain: Synergistic EEG-FMRI Modeling via Temporal-Spatial Constrained Representation

Electroencephalography (EEG) resolves millisecond-scale dynamics of neural activity, whereas functional magnetic resonance imaging (fMRI) localizes activity to distinct brain regions. A unified multimodal representation should draw its temporal structure from EEG and its spatial organization from fMRI. Existing fusion models map both modalities into a shared space and harmonize them with modality-agnostic objectives. Because EEG and fMRI lack a one-to-one token-level correspondence, such training blurs EEG-specific temporal fluctuations and fMRI-specific spatial organization. We propose UniBrain, a self-supervised foundation model in which each modality constrains the representation axis that it measures most reliably. Temporal Pattern Alignment (TPA) anchors latent trajectories to EEG fluctuation patterns through a differentiable monotonic alignment that tolerates unequal sequence lengths. Spatial Functional Connectivity Alignment (SFCA) anchors latent relational structure to functional connectivity derived from fMRI. Pretrained on simultaneously acquired EEG–fMRI recordings, UniBrain outperforms unimodal and multimodal foundation models on 4 paired and 6 unimodal benchmarks. Joint pretraining also improves decoding when only a single modality is available, showing that the two modalities refine a shared representation instead of coexisting as independent streams. Code will be released on GitHub.