- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes UniBrain, a self-supervised multimodal representation learning framework for jointly modeling EEG and fMRI signals. The motivation is that EEG and fMRI capture complementary aspects of brain activity: EEG provides high temporal resolution information, while fMRI provides spatial functional organization. The paper identifies a potential limitation of existing multimodal fusion approaches: modality-agnostic objectives may not explicitly preserve modality-specific structures, such as EEG temporal dynamics and fMRI spatial organization. To address this issue, UniBrain introduces two modality-specific representation constraints: Temporal Pattern Alignment (TPA), which aligns latent temporal trajectories with EEG temporal patterns through a differentiable monotonic alignment objective, and Spatial Functional Connectivity Alignment (SFCA), which aligns latent relational structures with fMRI-derived functional connectivity matrices. The model combines modality-specific encoders, a shared latent space refined by criss-cross Transformers, and reconstruction objectives for both modalities. The authors pretrain UniBrain on five simultaneously acquired EEG-fMRI datasets and evaluate it on paired EEG-fMRI benchmarks as well as EEG-only and fMRI-only downstream tasks. Experiments compare UniBrain with several existing unimodal and multimodal brain representation models and report improved performance across multiple downstream benchmarks with a relatively compact model size. Ablation studies are provided to analyze the contribution of TPA and SFCA.
Critical strengths
- Explicit modeling of EEG-fMRI complementary information.
A key strength of this work is that it explicitly considers the different characteristics of EEG and fMRI during multimodal representation learning. The proposed TPA and SFCA constraints encourage the learned representation to preserve EEG temporal dynamics and fMRI functional connectivity patterns, providing a modality-aware framework for EEG-fMRI modeling.
- Comprehensive evaluation and parameter efficiency.
The authors evaluate UniBrain against several recent unimodal and multimodal brain representation models across multiple datasets. The model achieves consistent improvements while maintaining a relatively compact parameter size. The ablation experiments further provide evidence that both TPA and SFCA contribute to the observed performance gains.
Critical weaknesses
- Limited evidence supporting the foundation model claim. Although UniBrain is presented as a brain foundation model, the current evaluation mainly focuses on performance improvements on several downstream benchmarks. The paper does not provide sufficient analysis of scaling behavior, representation quality, or transferability under larger and more diverse pretraining settings. Therefore, it remains unclear whether UniBrain should be considered a broadly generalizable foundation model or an effective pretrained representation model for EEG-fMRI tasks.
- The relationship to existing modality-aware representation learning approaches could be further clarified. The proposed TPA and SFCA constraints are well motivated and specifically designed for EEG-fMRI modeling. However, the underlying idea of preserving modality-specific information during multimodal representation learning is closely related to existing modality-aware and alignment-based approaches. The paper would benefit from a more detailed discussion of how UniBrain differs from prior methods and what unique insights are obtained from the proposed EEG-fMRI-specific formulation.
Questions for authors
- The reported improvements are mainly attributed to the proposed TPA and SFCA constraints. How sensitive are the results to the specific design choices and hyperparameters of these objectives? Have the authors investigated whether similar improvements can be achieved using alternative temporal or relational alignment strategies?
- The paper presents UniBrain as a generalizable multimodal brain representation model. Could the authors provide additional analysis beyond downstream classification performance, such as representation quality evaluation or transfer to unseen tasks/domains, to better support this claim? 3.The paper motivates the need for modality-specific constraints in EEG-fMRI representation learning. Could the authors further clarify how UniBrain differs from existing modality-aware or shared-private representation learning frameworks, and what specific challenges of EEG-fMRI are addressed by the proposed formulation?
4 comments
Response to Reviewer MWPj [2/n]
R3-2. Relationship to modality-aware and shared-private methods
Reviewer comment
The relationship to existing modality-aware representation learning approaches could be further clarified. The proposed TPA and SFCA constraints are well motivated and specifically designed for EEG-fMRI modeling. However, the underlying idea of preserving modality-specific information during multimodal representation learning is closely related to existing modality-aware and alignment-based approaches. The paper would benefit from a more detailed discussion of how UniBrain differs from prior methods and what unique insights are obtained from the proposed EEG-fMRI-specific formulation.
3.The paper motivates the need for modality-specific constraints in EEG-fMRI representation learning. Could the authors further clarify how UniBrain differs from existing modality-aware or shared-private representation learning frameworks, and what specific challenges of EEG-fMRI are addressed by the proposed formulation?
Response
We agree that exploiting modality-specific strengths has important precedents. In particular, MCSP explicitly distills spatial knowledge from fMRI to EEG and temporal/frequency knowledge in the reverse direction. Brain-OF also uses modality-specific spatial or temporal transformations for masked reconstruction.
Our distinction is the specific structure being supervised within one shared latent representation. MCSP aligns domain-wise projected embeddings through contrastive learning and distillation; UniBrain formulates temporal trajectory alignment to EEG features and relation-matrix alignment to fMRI features within the shared virtual-neuron-by-position representation. These address unequal latent sequence lengths and the absence of anatomical one-to-one correspondence between learned coordinates and ROIs. Soft-DTW and CKA are established tools, not claimed as new primitives.
Shared-private designs address how common and modality-specific information is allocated. Our constraints address which internal structures receive supervision and from which modality. These mechanisms can coexist; our claim concerns the proposed formulation and its evaluated benefit, not incompatibility with shared-private architectures.
The revised Introduction emphasizes this structural supervision, and section 2.2 now contrasts the supervision targets in MCSP and UniBrain.
Revision: Introduction (p. 2) and Section 2.2 (p. 3).
Response to Reviewer MWPj [3/n]
R3-3. Alternative objectives and hyperparameter sensitivity
Reviewer comment
The reported improvements are mainly attributed to the proposed TPA and SFCA constraints. How sensitive are the results to the specific design choices and hyperparameters of these objectives? Have the authors investigated whether similar improvements can be achieved using alternative temporal or relational alignment strategies?
Response
Defaults are λ_TPA=1, λ_SFCA=100, γ=1, and λ_f=1. Table 9 compares temporal objectives with SFCA disabled; Table 10 compares spatial objectives with TPA disabled. Reconstruction weights and all other settings remain fixed. TPA and SFCA exceed the strongest listed alternatives by 0.7–0.9 and 1.3–1.6 BACC percentage points, respectively.
Table 11 varies one parameter at a time: TPA weight 0.5/1/2, SFCA weight 50/100/200, and Soft-DTW temperature 0.5/1/2. The largest BACC drops are 0.7 points on NATView and 0.8 on ds002336, both with SFCA weight 50. Even these lowest scores (0.433/0.666) remain above the better single-constraint model on the same tasks (0.425/0.657 in Table 4), retaining gains of 0.8/0.9 points. Temperature changes have smaller effects, with maximum drops of 0.3 points on either benchmark. The benefit of using both constraints therefore persists throughout the tested neighborhood, although this is a local sensitivity result on two benchmarks. The four objective ablations explicitly set enabled/disabled weights while fixing all others, avoiding implicit changes from implementation mode defaults.
Revision: Section 3.5 (p. 5) specifies the default weights; Appendices A.5–A.6 (pp. 16–17) clarify the alternative-objective controls; Table 11 and the local-sensitivity paragraph (p. 17) report the parameter perturbations.
Table 9. Temporal alignment alternatives (BACC, mean ± standard deviation).
| Method | NATView | ds006040 | ds007216 | ds002336 |
|---|---|---|---|---|
| Uni_base | 0.398 ± 0.029 | 0.771 ± 0.033 | 0.407 ± 0.042 | 0.633 ± 0.036 |
| Uni_1stDiff | 0.403 ± 0.028 | 0.776 ± 0.032 | 0.411 ± 0.041 | 0.638 ± 0.035 |
| Uni_2ndDiff | 0.400 ± 0.029 | 0.773 ± 0.033 | 0.409 ± 0.041 | 0.636 ± 0.035 |
| Uni_PathSig | 0.407 ± 0.028 | 0.780 ± 0.032 | 0.414 ± 0.040 | 0.641 ± 0.034 |
| Uni_TPA | 0.416 ± 0.028** | 0.789 ± 0.032** | 0.423 ± 0.039** | 0.648 ± 0.033** |
* p ≤ 0.05; ** p ≤ 0.01.
Table 10. Spatial alignment alternatives (BACC, mean ± standard deviation).
| Method | NATView | ds006040 | ds007216 | ds002336 |
|---|---|---|---|---|
| Uni_base | 0.398 ± 0.029 | 0.771 ± 0.033 | 0.407 ± 0.042 | 0.633 ± 0.036 |
| Uni_partialCorr | 0.400 ± 0.028 | 0.774 ± 0.032 | 0.410 ± 0.041 | 0.636 ± 0.035 |
| Uni_GraphReg | 0.412 ± 0.027 | 0.784 ± 0.031 | 0.416 ± 0.041 | 0.642 ± 0.035 |
| Uni_SFCA | 0.425 ± 0.026* | 0.798 ± 0.030* | 0.432 ± 0.041** | 0.657 ± 0.035** |
* p ≤ 0.05; ** p ≤ 0.01.
Table 11. Local sensitivity to alignment hyperparameters (BACC). One parameter is varied at a time.
| Parameter varied | Actual value | NATView | ds002336 |
|---|---|---|---|
| TPA weight | 0.5 | 0.436 | 0.670 |
| TPA weight | 1 | 0.440 | 0.674 |
| TPA weight | 2 | 0.438 | 0.671 |
| SFCA weight | 50 | 0.433 | 0.666 |
| SFCA weight | 100 | 0.440 | 0.674 |
| SFCA weight | 200 | 0.437 | 0.670 |
| Soft-DTW temperature | 0.5 | 0.439 | 0.672 |
| Soft-DTW temperature | 1 | 0.440 | 0.674 |
| Soft-DTW temperature | 2 | 0.437 | 0.671 |
Response to Reviewer MWPj [4/n]
R3-4. Evaluation beyond classification and transfer scope
Reviewer comment
The paper presents UniBrain as a generalizable multimodal brain representation model. Could the authors provide additional analysis beyond downstream classification performance, such as representation quality evaluation or transfer to unseen tasks/domains, to better support this claim?
Response
Table 5 adds evidence beyond classification: both constraints improve EEG temporal PCC from 0.742 to 0.789 and fMRI spatial PCC from 0.960 to 0.997, while reducing MSE by 21.7% and 36.8%, respectively. The agreement between correlation and error metrics shows that the downstream gains are accompanied by improved signal reconstruction. Table 8 further shows that the alignment gains persist in EEG-only and fMRI-only transfer.
In A.7, freezing the backbone still trains the task head (two pre-normalized self-attention blocks, mean pooling, and a two-layer MLP) using supervised labels and validation model selection. These features therefore support supervised downstream decoding; this is not a zero-shot evaluation. Cross-site and cross-population transfer remains future work. A.9/A.10 document the experimental configuration and data-split protocols.
Revision: Table 5 (p. 8) and its analysis (p. 9), Table 8 and its analysis (p. 10), and Appendix A.7 (pp. 17–18), including the limitation on zero-shot and cross-site claims (p. 18).
Tables 5 and 8 (selected comparisons).
| Metric | Reconstruction only | Reconstruction + TPA + SFCA |
|---|---|---|
| EEG reconstruction MSE | 0.161 ± 0.011 | 0.126 ± 0.008 |
| fMRI reconstruction MSE | 0.057 ± 0.005 | 0.036 ± 0.003 |
| FACED BACC | 0.532 ± 0.013 | 0.565 ± 0.009 |
| HCP-Mot BACC | 0.691 ± 0.022 | 0.715 ± 0.018 |
Table 12 (UniBrain rows). Frozen-backbone evaluation and full fine-tuning; BACC (mean ± standard deviation).
| Setting | NATView | ds006040 | ds007216 | ds002336 |
|---|---|---|---|---|
| Frozen backbone | 0.312 ± 0.024 | 0.775 ± 0.028 | 0.312 ± 0.036 | 0.613 ± 0.031 |
| Full fine-tuning | 0.440 ± 0.027 | 0.816 ± 0.031 | 0.445 ± 0.040 | 0.674 ± 0.034 |
Response to Reviewer MWPj [1/n]
R3-1. Model scope and evidence for transfer
Reviewer comment
Response
We describe UniBrain as a self-supervised multimodal representation model. The added experiments strengthen the evidence within the reported tasks: Table 5 shows 21.7%/36.8% reductions in EEG/fMRI reconstruction MSE, and Table 8 shows 3.3/2.4-point BACC gains on EEG-only FACED/fMRI-only HCP-Mot over reconstruction-only pretraining. The alignment benefits therefore extend beyond paired-input classification to reconstruction and unimodal transfer. A.7 further compares supervised decoding with a frozen backbone and full adaptation; A.4 reports the five-corpus pretraining-scale diagnostic with fixed, session-disjoint ds006040 evaluation. Cross-site and cross-population validation remains future work. These results support the evaluated representation-learning setting rather than broad external generalization.
Revision: The abstract (p. 1) and Conclusion (p. 10) revise the model scope; Table 5 (p. 8), Table 8 (p. 10), Appendix A.4 (p. 16), and Appendix A.7 (pp. 17–18) provide the corresponding evaluations.
Tables 5 and 8 (selected comparisons).
Table 12 (UniBrain rows). Frozen-backbone evaluation and full fine-tuning; BACC (mean ± standard deviation).