acceptodds
Under review as a conference paper at ICLR 2027

UniBrain: Synergistic EEG-FMRI Modeling via Temporal-Spatial Constrained Representation

Abstract

Electroencephalography (EEG) resolves millisecond-scale dynamics of neural activity, whereas functional magnetic resonance imaging (fMRI) localizes activity to distinct brain regions. A unified multimodal representation should draw its temporal structure from EEG and its spatial organization from fMRI. Existing fusion models map both modalities into a shared space and harmonize them with modality-agnostic objectives. Because EEG and fMRI lack a one-to-one token-level correspondence, such training blurs EEG-specific temporal fluctuations and fMRI-specific spatial organization. We propose UniBrain, a self-supervised foundation model in which each modality constrains the representation axis that it measures most reliably. Temporal Pattern Alignment (TPA) anchors latent trajectories to EEG fluctuation patterns through a differentiable monotonic alignment that tolerates unequal sequence lengths. Spatial Functional Connectivity Alignment (SFCA) anchors latent relational structure to functional connectivity derived from fMRI. Pretrained on simultaneously acquired EEG–fMRI recordings, UniBrain outperforms unimodal and multimodal foundation models on 4 paired and 6 unimodal benchmarks. Joint pretraining also improves decoding when only a single modality is available, showing that the two modalities refine a shared representation instead of coexisting as independent streams. Code will be released on GitHub.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG-fMRI fusion, multimodal brain foundation model, temporal-spatial constraints

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MWPjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UniBrain, a self-supervised multimodal representation learning framework for jointly modeling EEG and fMRI signals. The motivation is that EEG and fMRI capture complementary aspects of brain activity: EEG provides high temporal resolution information, while fMRI provides spatial functional organization. The paper identifies a potential limitation of existing multimodal fusion approaches: modality-agnostic objectives may not explicitly preserve modality-specific structures, such as EEG temporal dynamics and fMRI spatial organization. To address this issue, UniBrain introduces two modality-specific representation constraints: Temporal Pattern Alignment (TPA), which aligns latent temporal trajectories with EEG temporal patterns through a differentiable monotonic alignment objective, and Spatial Functional Connectivity Alignment (SFCA), which aligns latent relational structures with fMRI-derived functional connectivity matrices. The model combines modality-specific encoders, a shared latent space refined by criss-cross Transformers, and reconstruction objectives for both modalities. The authors pretrain UniBrain on five simultaneously acquired EEG-fMRI datasets and evaluate it on paired EEG-fMRI benchmarks as well as EEG-only and fMRI-only downstream tasks. Experiments compare UniBrain with several existing unimodal and multimodal brain representation models and report improved performance across multiple downstream benchmarks with a relatively compact model size. Ablation studies are provided to analyze the contribution of TPA and SFCA.

Critical strengths
  1. Explicit modeling of EEG-fMRI complementary information.

A key strength of this work is that it explicitly considers the different characteristics of EEG and fMRI during multimodal representation learning. The proposed TPA and SFCA constraints encourage the learned representation to preserve EEG temporal dynamics and fMRI functional connectivity patterns, providing a modality-aware framework for EEG-fMRI modeling.

  1. Comprehensive evaluation and parameter efficiency.

The authors evaluate UniBrain against several recent unimodal and multimodal brain representation models across multiple datasets. The model achieves consistent improvements while maintaining a relatively compact parameter size. The ablation experiments further provide evidence that both TPA and SFCA contribute to the observed performance gains.

Critical weaknesses
  1. Limited evidence supporting the foundation model claim. Although UniBrain is presented as a brain foundation model, the current evaluation mainly focuses on performance improvements on several downstream benchmarks. The paper does not provide sufficient analysis of scaling behavior, representation quality, or transferability under larger and more diverse pretraining settings. Therefore, it remains unclear whether UniBrain should be considered a broadly generalizable foundation model or an effective pretrained representation model for EEG-fMRI tasks.
  2. The relationship to existing modality-aware representation learning approaches could be further clarified. The proposed TPA and SFCA constraints are well motivated and specifically designed for EEG-fMRI modeling. However, the underlying idea of preserving modality-specific information during multimodal representation learning is closely related to existing modality-aware and alignment-based approaches. The paper would benefit from a more detailed discussion of how UniBrain differs from prior methods and what unique insights are obtained from the proposed EEG-fMRI-specific formulation.
Questions for authors
  1. The reported improvements are mainly attributed to the proposed TPA and SFCA constraints. How sensitive are the results to the specific design choices and hyperparameters of these objectives? Have the authors investigated whether similar improvements can be achieved using alternative temporal or relational alignment strategies?
  2. The paper presents UniBrain as a generalizable multimodal brain representation model. Could the authors provide additional analysis beyond downstream classification performance, such as representation quality evaluation or transfer to unseen tasks/domains, to better support this claim? 3.The paper motivates the need for modality-specific constraints in EEG-fMRI representation learning. Could the authors further clarify how UniBrain differs from existing modality-aware or shared-private representation learning frameworks, and what specific challenges of EEG-fMRI are addressed by the proposed formulation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer MWPj [1/n]

R3-1. Model scope and evidence for transfer

Reviewer comment

Limited evidence supporting the foundation model claim. Although UniBrain is presented as a brain foundation model, the current evaluation mainly focuses on performance improvements on several downstream benchmarks. The paper does not provide sufficient analysis of scaling behavior, representation quality, or transferability under larger and more diverse pretraining settings. Therefore, it remains unclear whether UniBrain should be considered a broadly generalizable foundation model or an effective pretrained representation model for EEG-fMRI tasks.

Response

We describe UniBrain as a self-supervised multimodal representation model. The added experiments strengthen the evidence within the reported tasks: Table 5 shows 21.7%/36.8% reductions in EEG/fMRI reconstruction MSE, and Table 8 shows 3.3/2.4-point BACC gains on EEG-only FACED/fMRI-only HCP-Mot over reconstruction-only pretraining. The alignment benefits therefore extend beyond paired-input classification to reconstruction and unimodal transfer. A.7 further compares supervised decoding with a frozen backbone and full adaptation; A.4 reports the five-corpus pretraining-scale diagnostic with fixed, session-disjoint ds006040 evaluation. Cross-site and cross-population validation remains future work. These results support the evaluated representation-learning setting rather than broad external generalization.

Revision: The abstract (p. 1) and Conclusion (p. 10) revise the model scope; Table 5 (p. 8), Table 8 (p. 10), Appendix A.4 (p. 16), and Appendix A.7 (pp. 17–18) provide the corresponding evaluations.

Tables 5 and 8 (selected comparisons).

MetricReconstruction onlyReconstruction + TPA + SFCA
EEG reconstruction MSE0.161 ± 0.0110.126 ± 0.008
fMRI reconstruction MSE0.057 ± 0.0050.036 ± 0.003
FACED BACC0.532 ± 0.0130.565 ± 0.009
HCP-Mot BACC0.691 ± 0.0220.715 ± 0.018

Table 12 (UniBrain rows). Frozen-backbone evaluation and full fine-tuning; BACC (mean ± standard deviation).

SettingNATViewds006040ds007216ds002336
Frozen backbone0.312 ± 0.0240.775 ± 0.0280.312 ± 0.0360.613 ± 0.031
Full fine-tuning0.440 ± 0.0270.816 ± 0.0310.445 ± 0.0400.674 ± 0.034
AuthorsOpenReview ↗

Response to Reviewer MWPj [2/n]

R3-2. Relationship to modality-aware and shared-private methods

Reviewer comment

The relationship to existing modality-aware representation learning approaches could be further clarified. The proposed TPA and SFCA constraints are well motivated and specifically designed for EEG-fMRI modeling. However, the underlying idea of preserving modality-specific information during multimodal representation learning is closely related to existing modality-aware and alignment-based approaches. The paper would benefit from a more detailed discussion of how UniBrain differs from prior methods and what unique insights are obtained from the proposed EEG-fMRI-specific formulation.

3.The paper motivates the need for modality-specific constraints in EEG-fMRI representation learning. Could the authors further clarify how UniBrain differs from existing modality-aware or shared-private representation learning frameworks, and what specific challenges of EEG-fMRI are addressed by the proposed formulation?

Response

We agree that exploiting modality-specific strengths has important precedents. In particular, MCSP explicitly distills spatial knowledge from fMRI to EEG and temporal/frequency knowledge in the reverse direction. Brain-OF also uses modality-specific spatial or temporal transformations for masked reconstruction.

Our distinction is the specific structure being supervised within one shared latent representation. MCSP aligns domain-wise projected embeddings through contrastive learning and distillation; UniBrain formulates temporal trajectory alignment to EEG features and relation-matrix alignment to fMRI features within the shared virtual-neuron-by-position representation. These address unequal latent sequence lengths and the absence of anatomical one-to-one correspondence between learned coordinates and ROIs. Soft-DTW and CKA are established tools, not claimed as new primitives.

Shared-private designs address how common and modality-specific information is allocated. Our constraints address which internal structures receive supervision and from which modality. These mechanisms can coexist; our claim concerns the proposed formulation and its evaluated benefit, not incompatibility with shared-private architectures.

The revised Introduction emphasizes this structural supervision, and section 2.2 now contrasts the supervision targets in MCSP and UniBrain.

Revision: Introduction (p. 2) and Section 2.2 (p. 3).

AuthorsOpenReview ↗

Response to Reviewer MWPj [3/n]

R3-3. Alternative objectives and hyperparameter sensitivity

Reviewer comment

The reported improvements are mainly attributed to the proposed TPA and SFCA constraints. How sensitive are the results to the specific design choices and hyperparameters of these objectives? Have the authors investigated whether similar improvements can be achieved using alternative temporal or relational alignment strategies?

Response

Defaults are λ_TPA=1, λ_SFCA=100, γ=1, and λ_f=1. Table 9 compares temporal objectives with SFCA disabled; Table 10 compares spatial objectives with TPA disabled. Reconstruction weights and all other settings remain fixed. TPA and SFCA exceed the strongest listed alternatives by 0.7–0.9 and 1.3–1.6 BACC percentage points, respectively.

Table 11 varies one parameter at a time: TPA weight 0.5/1/2, SFCA weight 50/100/200, and Soft-DTW temperature 0.5/1/2. The largest BACC drops are 0.7 points on NATView and 0.8 on ds002336, both with SFCA weight 50. Even these lowest scores (0.433/0.666) remain above the better single-constraint model on the same tasks (0.425/0.657 in Table 4), retaining gains of 0.8/0.9 points. Temperature changes have smaller effects, with maximum drops of 0.3 points on either benchmark. The benefit of using both constraints therefore persists throughout the tested neighborhood, although this is a local sensitivity result on two benchmarks. The four objective ablations explicitly set enabled/disabled weights while fixing all others, avoiding implicit changes from implementation mode defaults.

Revision: Section 3.5 (p. 5) specifies the default weights; Appendices A.5–A.6 (pp. 16–17) clarify the alternative-objective controls; Table 11 and the local-sensitivity paragraph (p. 17) report the parameter perturbations.

Table 9. Temporal alignment alternatives (BACC, mean ± standard deviation).

MethodNATViewds006040ds007216ds002336
Uni_base0.398 ± 0.0290.771 ± 0.0330.407 ± 0.0420.633 ± 0.036
Uni_1stDiff0.403 ± 0.0280.776 ± 0.0320.411 ± 0.0410.638 ± 0.035
Uni_2ndDiff0.400 ± 0.0290.773 ± 0.0330.409 ± 0.0410.636 ± 0.035
Uni_PathSig0.407 ± 0.0280.780 ± 0.0320.414 ± 0.0400.641 ± 0.034
Uni_TPA0.416 ± 0.028**0.789 ± 0.032**0.423 ± 0.039**0.648 ± 0.033**

* p ≤ 0.05; ** p ≤ 0.01.

Table 10. Spatial alignment alternatives (BACC, mean ± standard deviation).

MethodNATViewds006040ds007216ds002336
Uni_base0.398 ± 0.0290.771 ± 0.0330.407 ± 0.0420.633 ± 0.036
Uni_partialCorr0.400 ± 0.0280.774 ± 0.0320.410 ± 0.0410.636 ± 0.035
Uni_GraphReg0.412 ± 0.0270.784 ± 0.0310.416 ± 0.0410.642 ± 0.035
Uni_SFCA0.425 ± 0.026*0.798 ± 0.030*0.432 ± 0.041**0.657 ± 0.035**

* p ≤ 0.05; ** p ≤ 0.01.

Table 11. Local sensitivity to alignment hyperparameters (BACC). One parameter is varied at a time.

Parameter variedActual valueNATViewds002336
TPA weight0.50.4360.670
TPA weight10.4400.674
TPA weight20.4380.671
SFCA weight500.4330.666
SFCA weight1000.4400.674
SFCA weight2000.4370.670
Soft-DTW temperature0.50.4390.672
Soft-DTW temperature10.4400.674
Soft-DTW temperature20.4370.671
AuthorsOpenReview ↗

Response to Reviewer MWPj [4/n]

R3-4. Evaluation beyond classification and transfer scope

Reviewer comment

The paper presents UniBrain as a generalizable multimodal brain representation model. Could the authors provide additional analysis beyond downstream classification performance, such as representation quality evaluation or transfer to unseen tasks/domains, to better support this claim?

Response

Table 5 adds evidence beyond classification: both constraints improve EEG temporal PCC from 0.742 to 0.789 and fMRI spatial PCC from 0.960 to 0.997, while reducing MSE by 21.7% and 36.8%, respectively. The agreement between correlation and error metrics shows that the downstream gains are accompanied by improved signal reconstruction. Table 8 further shows that the alignment gains persist in EEG-only and fMRI-only transfer.

In A.7, freezing the backbone still trains the task head (two pre-normalized self-attention blocks, mean pooling, and a two-layer MLP) using supervised labels and validation model selection. These features therefore support supervised downstream decoding; this is not a zero-shot evaluation. Cross-site and cross-population transfer remains future work. A.9/A.10 document the experimental configuration and data-split protocols.

Revision: Table 5 (p. 8) and its analysis (p. 9), Table 8 and its analysis (p. 10), and Appendix A.7 (pp. 17–18), including the limitation on zero-shot and cross-site claims (p. 18).

Tables 5 and 8 (selected comparisons).

MetricReconstruction onlyReconstruction + TPA + SFCA
EEG reconstruction MSE0.161 ± 0.0110.126 ± 0.008
fMRI reconstruction MSE0.057 ± 0.0050.036 ± 0.003
FACED BACC0.532 ± 0.0130.565 ± 0.009
HCP-Mot BACC0.691 ± 0.0220.715 ± 0.018

Table 12 (UniBrain rows). Frozen-backbone evaluation and full fine-tuning; BACC (mean ± standard deviation).

SettingNATViewds006040ds007216ds002336
Frozen backbone0.312 ± 0.0240.775 ± 0.0280.312 ± 0.0360.613 ± 0.031
Full fine-tuning0.440 ± 0.0270.816 ± 0.0310.445 ± 0.0400.674 ± 0.034
Reviewer_qjqROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes UniBrain, a self-supervised multimodal framework for joint EEG-fMRI representation learning. The framework aims to preserve the complementary temporal and spatial information of the two modalities in a shared representation. The paper uses Temporal Pattern Alignment with differentiable monotonic alignment to preserve EEG-derived temporal patterns while accommodating unequal sequence lengths, while Spatial Functional Connectivity Alignment encourages the shared representation to retain functional relationships derived from fMRI. UniBrain is pretrained on simultaneously acquired EEG-fMRI recordings and evaluated on four paired and six unimodal benchmarks. The reported results show improvements over unimodal and supervised multimodal baselines, with joint pretraining also benefiting downstream decoding when only one modality is available.

Critical strengths
  1. Using paired EEG-fMRI pretraining to improve single-modal downstream performance is a meaningful and well-motivated idea.
  2. The framework shows promising improvements in paired EEG-fMRI decoding, with higher balanced accuracy than the compared baselines across all four paired benchmarks.
Critical weaknesses
  1. Several important technical details are unclear, especially how the FC target for SFCA is constructed. How many TRs are used to estimate FC, given that Appendix A.1 describes each sample as corresponding to one TR? It is also unclear how the 1,000 ROI signals are mapped to the 32 virtual-neuron units for this objective. Is FC computed from the original ROI time series and then projected, or is the relation matrix computed directly from the compressed features? These details are essential for understanding and reproducing the method.
  2. The results in Tables 2 and 3 appear inconsistent. For joint pretraining with paired EEG–fMRI inputs, NATView scores are 0.440 and 0.431, respectively, while ds002336 scores are 0.674 and 0.661. These appear to correspond to the same overall setting (if I understand correctly), but the paper does not explain whether the training or evaluation setup differs.
  3. The reconstruction examples in Figure 2 and Appendix A.8 would benefit from quantitative evaluation. Although the examples are described as randomly selected, they provide limited evidence that TPA and SFCA consistently preserve temporal and spatial information. Metrics such as temporal correlation, reconstruction error, and FC recovery across the held-out test set would make this claim more convincing. The fMRI maps in Figure 2 also lack a colorbar, making the magnitude of the differences difficult to interpret.
  4. There are more recent baselines for EEG and fMRI FMs, but the paper didn't compare.
Questions for authors
  1. Table 1 lists PhysioNet-MI as having 1,983 subjects and six EEG channels, whereas the referenced dataset contains 109 subjects and 64-channel recordings. Could the authors clarify which data and channel subset were used, or whether the reported count is a table error?
  2. Have the authors evaluated the TPA and SFCA ablations from Table 4 on the single-modal downstream tasks in Section 5.4? This would help clarify whether these components contribute to the improvements in EEG-only and fMRI-only performance, beyond their benefits for paired-input condition.
7 comments
AuthorsOpenReview ↗

Response to Reviewer qjqR [1/n]

R2-1. Single-TR SFCA computation

Reviewer comment

Several important technical details are unclear, especially how the FC target for SFCA is constructed. How many TRs are used to estimate FC, given that Appendix A.1 describes each sample as corresponding to one TR? It is also unclear how the 1,000 ROI signals are mapped to the 32 virtual-neuron units for this objective. Is FC computed from the original ROI time series and then projected, or is the relation matrix computed directly from the compressed features? These details are essential for understanding and reproducing the method.

Response

Each sample contains one fMRI TR. Sections 3.2–3.4 and Appendix A.1 specify the path: 1,000 ROI values → 1,024 padded slots → spatial embedding, hierarchical convolution, and self-attention → Z_f with shape B×32×1×256 → A_f with shape 32×256 → G_f = A_f A_f^T / 256 → centered Gram comparison by CKA. The target is computed directly from compressed features, not from ROI time-series FC followed by projection. Its interpretation is latent feature-relation supervision; scan-level FC recovery was not measured.

Revision: Section 3.4, Eq. (9) (p. 5), and Appendix A.1 (p. 14), with the encoder definition in Section 3.2 (p. 4).

AuthorsOpenReview ↗

Response to Reviewer qjqR [2/n]

R2-2. Consistency between Tables 2 and 3

Reviewer comment

The results in Tables 2 and 3 appear inconsistent. For joint pretraining with paired EEG–fMRI inputs, NATView scores are 0.440 and 0.431, respectively, while ds002336 scores are 0.674 and 0.661. These appear to correspond to the same overall setting (if I understand correctly), but the paper does not explain whether the training or evaluation setup differs.

Response

These entries report the same setting. We apologize that Table 3 retained earlier-version results. Table 3 (section 5.2) now reports 0.440 ± 0.027 on NATView and 0.674 ± 0.034 on ds002336, matching Table 2 and Table 4.

Revision: Table 3 and its accompanying text in Section 5.2 (p. 8).

Table 3. Effect of joint pretraining and input modality (BACC, mean ± standard deviation).

SettingNATView EEGNATView fMRINATView EEG–fMRIds002336 EEGds002336 fMRIds002336 EEG–fMRI
EEG-only0.395 ± 0.027——0.615 ± 0.036——
fMRI-only—0.398 ± 0.024——0.634 ± 0.027—
UniBrain0.417 ± 0.0220.400 ± 0.0270.440 ± 0.0270.628 ± 0.0280.645 ± 0.0300.674 ± 0.034
AuthorsOpenReview ↗

Response to Reviewer qjqR [3/n]

R2-3. Quantitative reconstruction and fMRI colorbar

Reviewer comment

The reconstruction examples in Figure 2 and Appendix A.8 would benefit from quantitative evaluation. Although the examples are described as randomly selected, they provide limited evidence that TPA and SFCA consistently preserve temporal and spatial information. Metrics such as temporal correlation, reconstruction error, and FC recovery across the held-out test set would make this claim more convincing. The fMRI maps in Figure 2 also lack a colorbar, making the magnitude of the differences difficult to interpret.

Response

Table 5 provides quantitative support for the reconstruction examples. TPA alone reduces EEG MSE by 18.0% (0.161 to 0.132), versus 6.2% with SFCA alone (0.161 to 0.151); conversely, SFCA reduces fMRI MSE by 33.3% (0.057 to 0.038), versus 10.5% with TPA (0.057 to 0.051). The PCC results follow the same pattern: TPA gives the larger EEG improvement (0.742 to 0.773), while SFCA gives the larger fMRI improvement (0.960 to 0.990). Thus, the two constraints have different strengths in the measured temporal and spatial reconstruction metrics. Combining them further improves both modalities, reaching EEG PCC/MSE of 0.789/0.126 and fMRI PCC/MSE of 0.997/0.036: 21.7% and 36.8% lower MSE, respectively, than reconstruction alone. Neither modality is traded off to improve the other. These results evaluate signal fidelity rather than scan-level FC recovery.

Appendix A.9 describes paired-input self-reconstruction on held-out test participants, exclusion of padding, EEG PCC along time and fMRI PCC across valid ROIs, and aggregation by trial, participant, dataset, and run. Figure 2 presents reconstruction error with a common 0–1 display scale; Figure 5 displays signed activation with a common diverging scale from −1 to +1. Their captions and accompanying text distinguish the two quantities.

Revision: Table 5 (p. 8) and its analysis in Section 5.3 (p. 9); Figure 2 and its caption (p. 9); Figure 5 and Appendix A.8 (p. 18); the reconstruction-evaluation paragraph in Appendix A.9 (p. 19).

Table 5. Effect of pretraining objectives on EEG and fMRI reconstruction (mean ± standard deviation).

Pretraining objectiveEEG temporal PCC ↑EEG MSE ↓fMRI spatial PCC ↑fMRI MSE ↓
Reconstruction only0.742 ± 0.0180.161 ± 0.0110.960 ± 0.0080.057 ± 0.005
Reconstruction + TPA0.773 ± 0.0150.132 ± 0.0090.970 ± 0.0070.051 ± 0.004
Reconstruction + SFCA0.755 ± 0.0170.151 ± 0.0100.990 ± 0.0040.038 ± 0.003
Reconstruction + TPA + SFCA0.789 ± 0.0140.126 ± 0.0080.997 ± 0.0020.036 ± 0.003
Reviewer_6mPBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

UniBrain learns a shared EEG-fMRI representation through virtual-neuron tokens, a fusion projection and a criss-cross Transformer. Its reconstruction objectives are supplemented by soft sequence alignment and a spatial alignment loss. The release defines the latter as per-sample CKA between Gram matrices of 32 learned virtual-neuron features. The fMRI encoder compresses a single vector of 1,000 valid ROI values into these learned features. This makes the computation feasible for a single TR, while giving it a different meaning from temporal ROI functional connectivity. The paper reports paired and unimodal downstream evaluation, factorial modality comparisons and frozen-backbone tests, with paired balanced-accuracy gains of 2.0–3.4 percentage points on the four reported benchmarks.

Critical strengths
  1. A representation model that supports paired and missing-modality use

The same model can encode paired EEG-fMRI data and handle missing modalities. This is a useful design choice because paired acquisition is limited while downstream EEG-only and fMRI-only datasets are common. The release includes separate modality masks, reconstruction objectives and task-specific fine-tuning paths. It also provides a concrete mapping from spatially sorted, padded ROI inputs to the 32 virtual-neuron features (fmri_encoder.py, lines 225–258). The code therefore supports the architecture's basic feasibility and explains how the two modalities enter one shared representation.

  1. Several experiments address whether paired learning transfers useful information

The modality-factorial comparison in Table 3 separates multimodal pretraining from paired inputs at downstream inference. Frozen-backbone evaluation is particularly useful because an entirely trainable downstream model could recover performance without using much of the pretrained representation. Removing the two alignment objectives also tests their contribution. I appreciate that the ds007216 improvement is reported as nonsignificant. These are sensible controls for the representation-learning claim; their value depends on identifying the exact training corpora and checkpoints behind them.

Critical weaknesses
  1. The implemented spatial alignment does not yet support the claimed FC interpretation

The supplementary code resolves how a 32-by-32 target is computed. The fMRI encoder turns each TR's ROI values into 32 learned features, and _cka_loss uses the last fMRI latent token to form K_fmri = Z_f Z_f^T / D (unibrain.py, lines 433–439; alignment.py, lines 32–50). The centered Gram matrices are compared by CKA. This calculation is well defined for a single TR. The single-TR sample definition is therefore compatible with this implemented objective.

What remains unclear is the scientific meaning assigned to that target. The feature coordinates come from an encoder trained with the rest of the model; they are neither the original ROIs nor a temporal correlation matrix estimated from BOLD observations. Aligning them can be a reasonable regularizer, but Table 3's decoding gain does not demonstrate preservation of physiological FC. The manuscript should state this objective accurately and distinguish its feature-similarity claim from an empirical connectivity claim. If the latter is central, a separate comparison with scan-level ROI FC is needed. Otherwise the contribution is better assessed as multimodal feature alignment, for which the release now provides a reproducible definition.

  1. The released dataset configuration does not resolve the reported evaluation inconsistencies

Table 1 gives PhysioNet-MI as 1,983 subjects, 15,261 hours and six channels, while the official cited EEG Motor Movement/Imagery dataset has 109 volunteers and 64-channel recordings [1]. The supplied preprocessing reads a CBraMod-style LMDB, extracts subject identifiers from keys, and infers channel count from the arrays. It does not include the retained-record manifest needed to explain those totals. Likewise, Appendix A.4 uses up to 200,000 ds006040 pretraining items, although 27.90 hours at TR = 2 s give at most 50,220 unique TRs under the sample definition. The loader supports single-TR and multi-TR modes; the run-specific definition and unique-sample count are still needed.

There is also a protocol correspondence problem. The default pretraining YAML lists eight paired corpora, including ds002338, ds006040 and ds007216, whereas the paper describes five pretraining corpora with downstream datasets excluded. An example configuration can support other experiments, so its presence alone does not establish overlap in the reported runs. However, the release needs the actual configurations and subject manifests for each table and scaling figure. Subject-level splitting within one dataset does not by itself establish exclusion from earlier pretraining. Until these records are reconciled, the size of the transfer benefit and the interpretation of the scaling experiment remain difficult to assess.

Questions for authors
  1. Is the spatial objective in the paper exactly the latent Gram/CKA calculation in unibrain.py, or was ROI temporal FC used in a separate experiment? Please revise the definition of G_f accordingly. If the paper retains the physiological-connectivity claim, what direct FC-preservation analysis supports it beyond downstream decoding?

  2. Please provide the retained PhysioNet record manifest and explain the subject-key parsing used for the processed LMDB. What are the correct participant, duration and channel totals, and which reported tables change after correcting Table 1? For Appendix A.4, what exactly are the 200,000 items and how many distinct TRs do they contain?

  3. Which pretraining YAML and checkpoint were used for each downstream comparison and the scaling figure? Please show the corpus and participant assignments across pretraining, fine-tuning and testing. The release's default eight-corpus YAML needs to be distinguished from the five-corpus protocol stated in the manuscript.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6mPB [1/n]

R1-1. SFCA target and interpretation

Reviewer comment

What remains unclear is the scientific meaning assigned to that target. The feature coordinates come from an encoder trained with the rest of the model; they are neither the original ROIs nor a temporal correlation matrix estimated from BOLD observations. Aligning them can be a reasonable regularizer, but Table 3's decoding gain does not demonstrate preservation of physiological FC. The manuscript should state this objective accurately and distinguish its feature-similarity claim from an empirical connectivity claim. If the latter is central, a separate comparison with scan-level ROI FC is needed. Otherwise the contribution is better assessed as multimodal feature alignment, for which the release now provides a reproducible definition.

Is the spatial objective in the paper exactly the latent Gram/CKA calculation in unibrain.py, or was ROI temporal FC used in a separate experiment? Please revise the definition of G_f accordingly. If the paper retains the physiological-connectivity claim, what direct FC-preservation analysis supports it beyond downstream decoding?

Response

SFCA uses one fMRI TR, compressed fMRI features, and the Gram/CKA calculation identified by the reviewer. In sections 3.2–3.4, 1,000 ROI values are padded to 1,024 slots and encoded into Z_f of shape B×32×1×256. For each sample, A_f = Z_f[b, :, -1, :] has shape 32×256, and G_f = A_f A_f^T / 256; centered Gram matrices are compared using CKA. The scientific interpretation is supervision of relations among latent fMRI features. We do not estimate ROI time-series FC from one TR or directly validate scan-level FC preservation. Table 4 evaluates the contribution to paired decoding; Table 5 separately evaluates signal reconstruction fidelity.

Revision: Section 3.4, Eq. (9) (p. 5); Appendix A.1 (p. 14). The feature-relation interpretation is also revised in the abstract (p. 1), Introduction (p. 2), and Conclusion (p. 10).

Table 4. UniBrain component ablation for paired EEG–fMRI decoding (BACC, mean ± standard deviation).

MethodNATViewds006040ds007216ds002336
UniBrain_base0.398 ± 0.0290.771 ± 0.0330.407 ± 0.0420.633 ± 0.036
w/o TPA0.425 ± 0.0260.798 ± 0.0300.432 ± 0.0410.657 ± 0.035
w/o SFCA0.416 ± 0.0280.789 ± 0.0320.423 ± 0.0390.648 ± 0.033
UniBrain0.440 ± 0.027*0.816 ± 0.031*0.445 ± 0.040*0.674 ± 0.034*

* p ≤ 0.05; ** p ≤ 0.01.

AuthorsOpenReview ↗

Response to Reviewer 6mPB [2/n]

R1-2. PhysioNet statistics and scaling trials

Reviewer comment

Please provide the retained PhysioNet record manifest and explain the subject-key parsing used for the processed LMDB. What are the correct participant, duration and channel totals, and which reported tables change after correcting Table 1? For Appendix A.4, what exactly are the 200,000 items and how many distinct TRs do they contain?

Response

Table 1 incorrectly used statistics from the 2018 PhysioNet/Computing in Cardiology Challenge. The corrected PhysioNet-MI entry is 109 participants and 64 channels, with 9,837 four-second source samples totaling 10.93 hours (9,837 × 4 / 3,600), rather than the duration of the full raw database. Training uses S001–S070 (70 participants), validation S071–S089 (19), and testing S090–S109 (20). Inputs and predictions are unchanged: Table 1 is updated and Table 7 is unchanged. Appendix A.10 describes the participant splits and source-sample counts.

For scaling, one trial is one fMRI TR and its concurrent EEG segment; a task label is not required for self-supervised pretraining. The 200,000 units are distinct synchronized acquisition units from the five-corpus training pool, while ds006040 is the fixed downstream GradCPT correct/incorrect task. Fine-tuning uses 600 trials; testing uses 200 trials from different sessions, repeated five times with the same downstream samples at every scale. Appendix A.10 defines the unique acquisition units and the session-disjoint evaluation protocol. The curve increases from 61.2% at 100 trials to 79.4% at 10,000 and above 81.5% at 200,000, with diminishing gains at larger scales.

Revision: Table 1 (p. 6) and the PhysioNet-MI description in Section 4.2 (p. 7); Figure 4 caption (p. 15), Appendix A.4 (p. 16), and Appendix A.10 (p. 19).

AuthorsOpenReview ↗

Response to Reviewer 6mPB [3/n]

R1-3. Pretraining configuration and dataset splits

Reviewer comment

There is also a protocol correspondence problem. The default pretraining YAML lists eight paired corpora, including ds002338, ds006040 and ds007216, whereas the paper describes five pretraining corpora with downstream datasets excluded. An example configuration can support other experiments, so its presence alone does not establish overlap in the reported runs. However, the release needs the actual configurations and subject manifests for each table and scaling figure. Subject-level splitting within one dataset does not by itself establish exclusion from earlier pretraining. Until these records are reconciled, the size of the transfer benefit and the interpretation of the scaling experiment remain difficult to assess.

Which pretraining YAML and checkpoint were used for each downstream comparison and the scaling figure? Please show the corpus and participant assignments across pretraining, fine-tuning and testing. The release's default eight-corpus YAML needs to be distinguished from the five-corpus protocol stated in the manuscript.

Response

The main pretraining protocol uses 94c5t, ds002338, ds003768, ds006033, and CineBrain. NATView, ds002336, ds006040, and ds007216 are excluded from both main pretraining and pretraining model selection. The five-corpus configuration specifies 100 pretraining epochs, learning rate 2×10⁻⁴, batch size 64 per device and 256 globally, one TR, 32 virtual units, feature dimension 256, and uni-space length 10. Fine-tuning specifies 20 epochs, learning rate 10⁻⁴, batch size 8 per device, and effective batch size 64; these settings are synchronized with sections 4.1–4.2 and Appendix A.2.

The eight-corpus YAML illustrates the range of datasets supported by the code. The formal code release will synchronize the five-corpus configuration above rather than retain the eight-corpus demonstration as the reported experiment configuration. Appendix A.9 documents the experimental configuration and result correspondence; Appendix A.10 describes participant splits, session separation, and distinct TR-unit counting. The session-disjoint scaling diagnostic is specified separately in A.4/A.10.

Revision: Sections 4.1–4.2 (pp. 6–7); Appendices A.1–A.2 (p. 14), A.3 (p. 15), A.4 (p. 16), A.9 (pp. 18–19), and A.10 (p. 19).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global rebuttal

We thank the reviewers for their constructive feedback. We revised the manuscript to clarify what UniBrain learns in its shared EEG–fMRI representation and to test those claims with additional controls and stronger baselines. Section, table, and appendix references below refer to the revised manuscript. Page numbers follow the printed page numbers in the revised manuscript PDF.

  1. Method and scope. We describe UniBrain as a self-supervised multimodal representation model and distinguish its supervision of temporal trajectories and spatial relations within one shared representation from prior modality-aware approaches. We clarify the single-TR ROI encoding and the direct Gram/CKA objective on compressed fMRI features. We use “frozen-backbone evaluation” for the experiment that trains the task head, and identify cross-site transfer as future work.

  2. New experiments. Tables 2, 6 and 7 add REVE-Base and fMRI-LM-B; UniBrain remains close to REVE-Base on FACED (0.565 vs. 0.564 BACC) and exceeds fMRI-LM-B on HCP-Mot (0.715 vs. 0.696). Table 5 adds reconstruction PCC and MSE; with both constraints, EEG and fMRI MSE fall from 0.161 to 0.126 and from 0.057 to 0.036. Figure 2 labels the fMRI reconstruction-error panels and their shared 0–1 scale. In Table 8, both constraints together achieve the highest BACC and weighted-F1 means on FACED and HCP-Mot, with BACC gains of 3.3 and 2.4 percentage points and weighted-F1 gains of 2.3 and 1.3 points, respectively. Tables 9–10 compare alternative objectives; the new Table 11 shows maximum drops of 0.7 and 0.8 BACC points on NATView and ds002336 across the tested perturbations.

  3. Data and reporting. Table 1 corrects the PhysioNet-MI entry to 109 participants, 10.93 hours and 64 channels. Table 3 corrects the paired results to match Tables 2 and 4. Sections 4.1–4.2 and Appendix A.1 identify the five pretraining corpora, exclude downstream benchmarks from main pretraining, and describe subject-level splitting. The eight-corpus YAML is an example of supported datasets; the release will follow the five-corpus specification in A.9, with manifest requirements in A.10. Appendix A.4 defines distinct synchronized TR units from the five-corpus training pool; ds006040 GradCPT correct/incorrect classification uses fixed session-disjoint downstream samples. A.9/A.10 describe the experimental configuration, result correspondence, and data splits.

Dear Reviewers 6mPB, qjqR, and MWPj,

Thank you again for your constructive reviews. We briefly summarize (1) UniBrain's contribution to the BCI field and (2) what was added during rebuttal.

1. Contributions

  • Modality-specific supervision for EEG–fMRI fusion. Existing fusion models harmonize the two modalities with modality-agnostic objectives, blurring EEG-specific temporal fluctuations and fMRI-specific spatial organization. UniBrain lets each modality constrain the axis it measures most reliably: TPA anchors latent temporal trajectories to EEG features via differentiable monotonic alignment tolerant of unequal sequence lengths, and SFCA anchors latent relational structure to fMRI-derived functional connectivity.
  • Paired multimodal pretraining. UniBrain is pretrained self-supervised on simultaneously acquired EEG–fMRI recordings from five public datasets, achieving the best balanced accuracy on four paired decoding benchmarks.
  • Benefits unimodal decoding. Joint pretraining improves decoding even when only one modality is available downstream — relevant for real-world BCI settings where paired recording is infeasible.

2. Added during rebuttal

  • Stronger baselines: REVE-Base and fMRI-LM-B (Tables 2, 6, 7); UniBrain keeps gains of +2.2–2.5 BACC points on paired benchmarks and +1.9 on HCP-Mot.
  • Quantitative reconstruction: Table 5 shows 21.7%/36.8% MSE reductions for EEG/fMRI.
  • Unimodal ablations: Table 8 (+3.3/+2.4 BACC on FACED/HCP-Mot).
  • Clarified SFCA definition (Sections 3.2–3.4, Eq. 9): Gram/CKA supervises relations among latent fMRI features; the connectivity claim is reworded.
  • Corrected statistics: PhysioNet-MI in Table 1 and Table 3 results now match Tables 2 and 4.
  • Protocol transparency: Appendices A.9–A.10 document the five-corpus configuration, splits, and session-disjoint evaluation.
  • Robustness and scope: alternative objectives (Tables 9–10), sensitivity (Table 11), frozen-backbone evaluation (Table 12), and a revised scope as a self-supervised multimodal representation model.

We sincerely appreciate your time. As the discussion period is closing, we would be grateful if you could kindly share your final evaluation when convenient — we remain happy to address any remaining questions promptly.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.