acceptodds
Under review as a conference paper at ICLR 2027

Bridging Electrophysiology and Hemodynamics: Adapting EEG Foundation Models for EEG-fNIRS Representation Learning

Abstract

Electroencephalography (EEG) has long served as a fundamental modality for decoding brain activity. Existing EEG foundation models learn transferable neural representations from large-scale electrophysiological recordings, but their unimodal design prevents them from leveraging the complementary hemodynamic information measured by functional near-infrared spectroscopy (fNIRS). Extending these models to multimodal EEG-fNIRS data is non-trivial, as the signals differ sharply in sensor layouts, temporal resolution, and physiological origins. To overcome these challenges, we propose EF-GO, a modular framework that adapts pretrained EEG foundation models for multimodal EEG-fNIRS representation learning without retraining the original EEG backbone. EF-GO first learns fNIRS-specific representations via self-supervised masked reconstruction, then employs Region-prior-guided Graph Aggregation to map heterogeneous EEG and fNIRS channel features into a shared, region-aware graph space. To align their distinct temporal dynamics, EF-GO incorporates a time-aware Neural Ordinary Differential Equation (Neural ODE) objective to capture lagged correspondences between electrophysiological and hemodynamic representations. Experiments across six downstream tasks with two pretrained EEG backbones show that EF-GO outperformed unimodal EEG baselines across most metrics and delivered performance competitive with task-specific EEG-fNIRS models. These results demonstrate the potential of adapting pretrained EEG models for multimodal neurophysiological representation learning.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS), Foundation model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_reVMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EF-GO, a framework for adapting pretrained EEG foundation models to multimodal EEG-fNIRS representation learning. It combines self-supervised fNIRS pretraining, region-prior-guided graph aggregation, and time-aware Neural ODE alignment. Experiments cover six downstream tasks with two EEG backbones. The problem is well motivated, and the results suggest benefits from multimodal adaptation. However, the current experiments do not sufficiently establish the specific contribution of continuous-time, delay-aware alignment or clarify the conditions under which adaptation is beneficial.

Critical strengths
  1. Adapting pretrained EEG models to paired EEG-fNIRS data addresses a meaningful representation-learning problem involving spatial and temporal heterogeneity.
  2. The evaluation includes multiple task categories and two pretrained backbones, with improvements over the corresponding EEG-only models on several tasks.
  3. The ODE module is used during cross-modal post-training and excluded from downstream inference, avoiding ODE integration in the inference path.
Critical weaknesses
  1. The specific benefit of continuous-time modeling is not isolated. Table 3 shows that adding ODE-based alignment can improve performance, but removing the entire alignment module does not establish whether continuous-time dynamics are responsible for the improvement. The gains could also reflect the additional cross-modal alignment objective or transformation capacity. The task-specific multimodal baselines do not provide a controlled comparison within the same pretrained-backbone adaptation setup. A non-ODE alignment model using the same paired data, alignment targets, and comparable capacity and training budget would help establish the contribution of the proposed formulation. This is a central concern because continuous-time neurovascular modeling is a principal methodological claim. 
  2. The temporal-weighting ablation does not clearly establish the contribution of learned delay preferences. Table 3 does not specify the replacement weights used for “w/o time-aware,” and Figure 4(b) lacks a numerical vertical scale. Appendix F does report that the effective sigmoid weights change approximately from 0.7311 to 0.7291, which provides useful context. However, this does not clarify the ablation configuration or quantify the individual delay-dependent profiles. Since Equation (13) does not normalize by the sum of the learned weights, replacing them with a different constant may also change the overall alignment-loss scale. A fixed-weight comparison with a matched overall scale is needed to distinguish the contribution of relative delay preferences. The reported peak-shaped curves alone do not establish that these preferences account for the downstream gains. 
  3. The backbone-dependent degradation is not adequately investigated. On hBCI-MI, Table 2 shows that EF-GO(C) reduces CBraMod’s balanced accuracy from 0.6377 to 0.5922 and AUC-ROC from 0.7124 to 0.6236. Section 4.1 acknowledges that the backbones have different strengths, but does not explain this degradation. Table 3 only provides EF-GO(L) ablations on Cognitive-WG and hBCI-MA, leaving the affected backbone and task unexamined. This does not negate the improvements elsewhere, but it limits the evidence for the framework’s applicability across backbones. A targeted analysis would help identify whether the degradation arises from fusion, alignment, or downstream adaptation. 
  4. Robustness to the selection of held-out subjects remains unclear. Section 4 uses three validation subjects and three test subjects per dataset. Appendix D reports five random seeds, but does not clarify whether the subject partitions change across these runs. It is therefore unclear whether the reported variability captures sensitivity to held-out-subject selection or only training randomness. Clarifying the split procedure and reporting results across multiple subject partitions would strengthen the subject-independent generalization claims.
Questions for authors
  1. How does ODE alignment compare with a non-ODE alignment model under matched training conditions?
  2. What exact weights and normalization are used in “w/o time-aware,” and does the improvement persist when the overall loss scale is matched?
  3. Can targeted EF-GO(C) ablations explain the hBCI-MI degradation?
  4. Do the five random seeds change the held-out subjects, or only the training randomness?
2 comments

Thank you for identifying the controls needed to evaluate our alignment method and for raising questions about temporal weighting, the performance drop with CBraMod, and held-out subject selection.

Q1. Comparison with non-ODE alignment

We added three non-ODE variants that use MSE, a cosine-based loss, or a contrastive loss to directly align paired EEG and fNIRS embeddings after graph aggregation. Graph–ODE achieved the highest mean score on all three metrics on hBCI-MA and on balanced accuracy and ROC-AUC on Cognitive-WG. Cosine-based alignment achieved the highest mean PR-AUC on Cognitive-WG. These comparisons quantify the downstream performance of Graph–ODE relative to direct alignment. Isolating the contribution of continuous-time modeling would additionally require a discrete alignment model with matched temporal targets and comparable transformation capacity.

TaskAlignment methodBalanced accuracyPR-AUCROC-AUC
hBCI-MAContrastive learning0.6899 ± 0.01130.7376 ± 0.01460.7553 ± 0.0110
Cosine-based alignment0.6875 ± 0.01280.7411 ± 0.01710.7592 ± 0.0141
MSE0.6842 ± 0.00790.7411 ± 0.01340.7558 ± 0.0086
Graph–ODE0.6969 ± 0.00560.7472 ± 0.01290.7638 ± 0.0060
Cognitive-WGContrastive learning0.7256 ± 0.03760.7815 ± 0.04200.8145 ± 0.0336
Cosine-based alignment0.7200 ± 0.02890.7849 ± 0.04190.8125 ± 0.0303
MSE0.7345 ± 0.02840.7744 ± 0.02940.8049 ± 0.0264
Graph–ODE0.7389 ± 0.01000.7826 ± 0.02310.8167 ± 0.0163

Q2. Temporal weight replacements and alignment-loss scale

In the original w/o time-aware ablation, we set every time-step parameter to 1 before applying the sigmoid in Equation (13), yielding a uniform effective weight of approximately 0.7311. We expanded the ablation to include two uniform settings with pre-sigmoid values of 1 and 10, a fixed profile derived from a hemodynamic response function (HRF), and a random profile. The learned weights achieved the highest mean scores on all three metrics on both tasks.

As for Figure 4(b) we will add a numerical vertical axis to show the magnitude of the learned weight differences.

TaskTemporal weightingBalanced accuracyPR-AUCROC-AUC
hBCI-MAHRF-based0.6887 ± 0.01070.7384 ± 0.00330.7589 ± 0.0027
Random0.6710 ± 0.00800.7317 ± 0.00710.7435 ± 0.0042
Uniform, pre-sigmoid value of 100.6895 ± 0.01470.7451 ± 0.02880.7564 ± 0.0177
Uniform, pre-sigmoid value of 10.6895 ± 0.00730.7457 ± 0.00810.7616 ± 0.0059
Learned weights (EF-GO)0.6969 ± 0.00560.7472 ± 0.01290.7638 ± 0.0060
Cognitive-WGHRF-based0.7200 ± 0.02860.7453 ± 0.03640.7848 ± 0.0304
Random0.7100 ± 0.02970.7690 ± 0.02580.7877 ± 0.0307
Uniform, pre-sigmoid value of 100.7244 ± 0.01030.7689 ± 0.02820.8063 ± 0.0211
Uniform, pre-sigmoid value of 10.7256 ± 0.02830.7688 ± 0.01290.8076 ± 0.0171
Learned weights (EF-GO)0.7389 ± 0.01000.7826 ± 0.02310.8167 ± 0.0163

Q3. Performance degradation with CBraMod on hBCI-MI

The effect of multimodal adaptation varies across both tasks and backbones. There are inherent differences in the contributions of EEG and fNIRS signals across various tasks, such as cognitive, emotional, and motor imagery tasks. At the same time, due to differences in pre-training datasets and pre-training methods, different backbone models inherently exhibit performance variations. For example, EF-GO(C) achieves higher balanced accuracy than EF-GO(L) on both REFED tasks, where the fNIRS-only branch also performs well in the LaBraM ablations.

To investigate the hBCI-MI results, we measured the effective rank of model weights and representations before and after downstream fine-tuning. The mean effective rank of EEG backbone weights increased slightly, while the effective rank of fNIRS branch weights and representations were essentially unchanged. This suggests that the model's performance did not degrade as a result of fine-tuning. However, the effective rank of the EEG representations showed a significant decline, which may indicate that the EEG signals in this dataset are more sensitive to this backbone, and that the EEG representations may have experienced a collapse in representation. This phenomenon may be the cause of the performance decline.

QuantityBefore fine-tuningAfter fine-tuningRelative change
EEG backbone weights, mean effective rank91.8309792.00325+0.19%
fNIRS branch weights, mean effective rank70.4299870.42979−0.0003%
EEG representations, effective rank89.8120069.78312−22.30%
fNIRS representations, effective rank3.214443.21482+0.0119%

Q4. Cross-subject cross-validation

In the experiments, the subject partitions were fixed across all five random seeds. The seeds varied the training randomness, so the reported variability reflects training variability within a fixed split.

To assess performance across different held-out subject groups, we added five-fold subject-level cross-validation on Cognitive-WG dataset. Each fold held out a distinct group of subjects for testing. Validation subjects were selected from the remaining subjects, and the rest were used for training. Both EF-GO variants achieved higher mean scores than all three task-specific EEG–fNIRS models on all reported metrics. EF-GO(L) achieved the highest mean balanced accuracy, PR-AUC, and ROC-AUC among the evaluated methods. It is worth noting that task-specific EEG-fNIRS models all exhibited a decline in performance under this experimental setup, whereas the Foundation Models remained highly stable under these conditions, which to some extent demonstrates the Foundation Model's robustness to subject-to-subject variability.

MethodBalanced accuracyPR-AUCROC-AUC
FGA-Net0.6340±0.04980.6521±0.02690.6715±0.0408
M2NN0.6522±0.02190.7010±0.02610.7296±0.0295
STA-Net0.6114±0.05060.6517±0.06480.6654±0.0459
LaBraM0.7022±0.02190.7372±0.05200.7664±0.0423
CBraMod0.6807±0.04840.7233±0.05060.7480±0.0498
EF-GO(L)0.7386±0.02560.7900±0.03380.8139±0.0291
EF-GO(C)0.6949±0.04160.7155±0.04960.7532±0.0395
Reviewer_vp7yOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a relevant and important problem - how to adapt pretrained EEG foundation model to multimodal EEG-fNIRS learning. The paper mainly addresses the challenge that EEG and fNIRS exhibit different temporal dynamics, spatial mismathc, and different physiological processes. The paper proposes an integrated pipeline for this model adaptation, which combines self-supervised fNIRS pre-training, EEG-fNIRS spatial-temporal alignment via region-prior-guided graph aggregation and temporal ODE, and downstream finetuning.

Critical strengths
  1. Undoubtedly, the problem studied in this paper is practially important and useful.
  2. The proposed framework shows empirical improvements over both small-scale EEG-fNIRS models and EEG foundation models.
  3. The paper is overall clearly written and easy to follow.
Critical weaknesses
  1. The individual technical components in the proposed pipleline are mostly from established methods.
  2. As described in Page 7, the baselines of those small-scale models are multi-modal combining both EEG and fNIRS inputs. However, the other two baselines of large-scale foundation models, LaBraM and CBraMod, are EEG-only models. The empirical improvement of the proposed method over these two models indeed confirm the meric of adding the second data modality (fNIRS). However, it does not show that the proposed method is architecturally superior than a straightforward data fusion. The paper also does not clearly explain if the two standalone foundation models are properly finetuned. For the REFED emotion-recognition tasks, the paper reports very low performance for the two large-scale foundation models. The paper needs to clearly document the experiment setup of these baselines.
  3. Based on Table 1, the input duration of 5 out of 6 datasets is as low as 2s, 1s, or 3s. However, Figure 4 (b) shows that the learned temporal weighting profiles across ODE trajectory steps reach peak at around 5 seconds. It is not clear why the neural ODE can capture the hemodynamic delays of 5s on these datasets.
Questions for authors
  1. Were the large-scale foundation models LaBraM and CBraMod were fine-tuned, frozen, or partially runed?
  2. Can you add a baseline where LaBraM and CBraMod were used with simple data fusion of EEG + fNIRS.
  3. Can you add a table clearly documenting the inputs of each baseline and the proposed method (EEG and/or fNIRS)
  4. There seems to be mismatch in the reported values between Table 2 and 8.
2 comments

Thank you for raising questions about the contribution of EF-GO, the baseline settings, and the interpretation of the temporal weights. We have added a multimodal foundation model baseline and rechecked the aggregate results in Table 8.

W1. Methodological contribution

EF-GO introduces a cross-modal post-training framework that enables pretrained EEG foundation models to learn from paired fNIRS recordings while accounting for differences in sensor layout and response timing. We evaluate this framework with two pretrained EEG backbones, LaBraM and CBraMod.

For spatial alignment, region-prior-guided graph aggregation maps EEG and fNIRS features from different sensor layouts into a common set of anatomically defined regions. This gives the two modalities a shared spatial reference for alignment. For temporal alignment, a neural ODE models a continuous trajectory in latent space, and learned weights determine how much different time offsets contribute to the alignment objective. Together, these mechanisms let the model align representations by brain region and across time offsets.

The framework also provides a way to investigate why multimodal gains vary across tasks. Spatial correspondences and temporal offsets can be varied separately to test which regions provide complementary cross-modal information and which offsets benefit downstream decoding. The same approach could help adapt pretrained EEG models to EEG–fNIRS datasets with different sensor layouts and to other paired brain signals for which spatial correspondences and differences in response timing can be defined. Evaluating those settings would help establish where the approach transfers.

W2. Comparison with a multimodal foundation model

To demonstrate that EF-GO's performance is not solely attributable to the incorporation of fNIRS data, we introduced a multimodal baseline model, baseline-PhysioOmni. Across six downstream tasks, EF-GO continued to outperform PhysioOmni on nearly all metrics.

TaskMethodBalanced accuracyPR-AUC / Weighted F1ROC-AUC / Cohen's Kappa
Cognitive-WGPhysioOmni0.6534 ± 0.01810.7354 ± 0.03430.7416 ± 0.0252
EF-GO(L)0.7389 ± 0.01000.7826 ± 0.02310.8167 ± 0.0163
EF-GO(C)0.6955 ± 0.02030.7880 ± 0.01060.7770 ± 0.0114
Cognitive-nBackPhysioOmni0.3869 ± 0.01180.4024 ± 0.01610.0951 ± 0.0162
EF-GO(L)0.4137 ± 0.01130.4314 ± 0.01280.1458 ± 0.0201
EF-GO(C)0.3948 ± 0.01190.4152 ± 0.01950.1134 ± 0.0177
REFED-ValencePhysioOmni0.4478 ± 0.01460.4778 ± 0.01180.1738 ± 0.0323
EF-GO(L)0.4211 ± 0.01720.4451 ± 0.01630.1170 ± 0.0202
EF-GO(C)0.4871 ± 0.01050.4830 ± 0.03430.1901 ± 0.0217
REFED-ArousalPhysioOmni0.4537 ± 0.03510.4358 ± 0.03960.1728 ± 0.0600
EF-GO(L)0.4342 ± 0.02050.4431 ± 0.02070.1559 ± 0.0331
EF-GO(C)0.4623 ± 0.01250.4206 ± 0.02300.1770 ± 0.0174
hBCI-MAPhysioOmni0.6424 ± 0.02230.6925 ± 0.03160.7080 ± 0.0241
EF-GO(L)0.6969 ± 0.00560.7472 ± 0.01290.7638 ± 0.0060
EF-GO(C)0.6958 ± 0.01230.7784 ± 0.01040.7668 ± 0.0098
hBCI-MIPhysioOmni0.5420 ± 0.01610.5630 ± 0.01930.5618 ± 0.0214
EF-GO(L)0.6557 ± 0.02230.7255 ± 0.03130.7130 ± 0.0312
EF-GO(C)0.5922 ± 0.02100.6273 ± 0.03390.6236 ± 0.0280

W3. Temporal weights

The temporal weights in Figure 4(b) were learned during the spatiotemporal representation alignment (post-training) phase of EEG–fNIRS, followed by downstream fine-tuning. In contrast, the durations listed in Table 1 describe the input windows used for downstream classification. Therefore, the peak around 5 seconds reflects the preferences learned during the cross-modal alignment phase.

Q1. Fine-tuning of LaBraM and CBraMod

We initialize LaBraM and CBraMod from their publicly released pretrained checkpoints. In EF-GO, we freeze the EEG backbone during alignment post-training to reduce the training cost and provide a stable EEG representation space for aligning the fNIRS features. During downstream fine-tuning, all backbone parameters are trainable, including for the standalone EEG-only baselines. This follows the full fine-tuning protocol of LaBraM and CBraMod.

Q2. Simple EEG–fNIRS fusion

Variants 3 and 4 in Table 3 implement two simple fusion baselines. Variant 3 directly concatenates the EEG and fNIRS branch embeddings. Variant 4 applies region-guided graph aggregation to each branch before concatenating the resulting embeddings. Both variants omit ODE-based alignment. The full EF-GO(L) model achieves higher mean scores than both fusion baselines on all three reported metrics for Cognitive-WG and hBCI-MA. We will revise the variant labels and descriptions to make these comparisons explicit.

VariantWG balanced accuracyWG PR-AUCWG ROC-AUCMA balanced accuracyMA PR-AUCMA ROC-AUC
Direct concatenation0.6767±0.01660.7490±0.03040.7594±0.02890.6770±0.01590.7320±0.02630.7467±0.0185
Concatenation after graph agg.0.6934±0.02060.7417±0.01900.7716±0.01040.6928±0.00530.7365±0.00690.7527±0.0048
Full EF-GO(L)0.7389±0.01000.7826±0.02310.8167±0.01630.6969±0.00560.7472±0.01290.7638±0.0060

Q3. Input modalities

The inputs used in our evaluation are listed below. We will add this input-modality column to Table 2.

MethodInputs
FGA-NetEEG + fNIRS
M2NNEEG + fNIRS
STA-NetEEG + fNIRS
LaBraMEEG
CBraModEEG
PhysioOmniEEG + fNIRS
EF-GO(L)EEG + fNIRS
EF-GO(C)EEG + fNIRS

Q4. Discrepancy between Tables 2 and 8

Thank you for flagging this discrepancy. Table 2 reports results for each task, while Table 8 aggregates balanced accuracy across all six tasks and aggregates the other metrics over the tasks for which they are reported. On rechecking Table 8, we found an error in the aggregation of PR-AUC and ROC-AUC for the binary classification tasks. The corrected summary is provided below. We will update Table 8 in the revised version.

MethodBalanced accuracy, all tasksPR-AUC, binaryROC-AUC, binaryWeighted F1 (3-class)Cohen's Kappa (3-class)
FGANet0.5186±0.12260.6984±0.04260.6980±0.02870.4112±0.05850.1052±0.0420
M2NN0.5155±0.12200.6672±0.09750.6673±0.09300.4237±0.01630.1187±0.0142
STA-Net0.5250±0.13670.7551±0.06050.7538±0.04000.4045±0.05970.0898±0.0534
LaBraM0.5044±0.14470.7074±0.02450.7174±0.01850.3800±0.02860.0475±0.0457
CBraMod0.5101±0.13330.7174±0.03080.7233±0.02500.3877±0.02850.0799±0.0699
PhysioOmni0.5210±0.10050.6636±0.07330.6705±0.07810.4387±0.03080.1472±0.0369
EF-GO(L)0.5601±0.13930.7518±0.02350.7645±0.04230.4399±0.00600.1396±0.0165
EF-GO(C)0.5546±0.11530.7312±0.07360.7225±0.07000.4396±0.03080.1602±0.0335

We have updated the PDF with all of the above information. Thank you again for your review.

Reviewer_FS1tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces EF-GO, a framework that adapts pretrained EEG foundation models for joint EEG-fNIRS representation learning. It first pre-trains an fNIRS encoder through self-supervised masked reconstruction, then uses region-prior-guided graph aggregation to map features from different sensor layouts into a shared, region-aware representation space. They also proposes a time-aware neural ODE aligns EEG and fNIRS representations across temporal offsets to account for their different temporal dynamics. The resulting representations are jointly used for downstream classification. Experiments across six classification tasks with two pretrained EEG backbones show improvements over the corresponding EEG-only baselines on most reported metrics and competitive performance against task-specific multimodal models.

Critical strengths
  1. The latent-space visualizations suggest that the ODE module helps bring EEG and fNIRS representations closer together. Quantatively, the framework also shows promising downstream performance, with improvements over several task-specific multimodal baselines.
  2. The overall design is well motivated. The graph aggregation module addresses differences in sensor layouts, while the ODE module accounts for temporal delays between EEG and fNIRS.
Critical weaknesses
  1. The improvements over EEG-only baselines in Table 2 reflect both the addition of fNIRS information and the proposed adaptation method. The multimodal baselines and fusion ablations help address this, but the ablations cover only two tasks with one backbone. Also, the multimodal baselines are described as models trained from scratch, whereas EF-GO uses pretrained EEG foundation models. This makes it difficult to separate the benefits of EEG pretraining from those of the proposed Graph-ODE design.
  2. In introduction, the paper describes advanced performance against EEG-only model as one of the major contirbution. However, I feel this contribution could be framed more carefully. Improving over EEG-only baselines is useful, but adding fNIRS provides extra information and at the mean time requires paired recordings, which are harder to obtain. These improvements alone therefore do not establish the advantage of the proposed adaptation method.
  3. It would be helpful to compare the proposed alignment method with simpler alternatives, such as directly aligning embeddings using MSE, cosine similarity, or a contrastive loss.
Questions for authors
  1. Could the authors provide some insight into why multimodal learning helps more on some tasks than others? For example, several multimodal methods outperform the EEG-only foundation models on Cognitive-WG, while some fall behind them on hBCI-MI. EF-GO(C) also performs worse than its EEG-only backbone on MI. What might explain these differences?
  2. Given the use of masked reconstruction during fNIRs pretraining and EEG FMs, have the authors evaluated robustness to noisy inputs, missing channels, or masked signal segments? In particular, it would be interesting to see: does ODE-based alignment improve downstream robustness compared with direct fusion under the same conditions?
5 comments

Thank you for highlighting the motivation for spatial and temporal alignment and for asking us to clarify what drives the performance gains. We have added a multimodal foundation model baseline, extended the ablations, evaluated performance with fewer downstream training subjects, and compared Graph–ODE alignment with simpler alternatives.

W1. Multimodal baseline and ablation coverage

To demonstrate that EF-GO's performance is not solely attributable to the incorporation of fNIRS data, we introduced a multimodal baseline model, baseline-PhysioOmni. Across six downstream tasks, EF-GO continued to outperform PhysioOmni on nearly all metrics. Additionally, we extended our ablation experiments to all six datasets, illustrating the impact of different components and their designs on model performance.

TaskMethodBalanced accuracyPR-AUC / Weighted F1ROC-AUC / Cohen's Kappa
Cognitive-WGPhysioOmni0.6534 ± 0.01810.7354 ± 0.03430.7416 ± 0.0252
EF-GO(L)0.7389 ± 0.01000.7826 ± 0.02310.8167 ± 0.0163
EF-GO(C)0.6955 ± 0.02030.7880 ± 0.01060.7770 ± 0.0114
Cognitive-nBackPhysioOmni0.3869 ± 0.01180.4024 ± 0.01610.0951 ± 0.0162
EF-GO(L)0.4137 ± 0.01130.4314 ± 0.01280.1458 ± 0.0201
EF-GO(C)0.3948 ± 0.01190.4152 ± 0.01950.1134 ± 0.0177
REFED-ValencePhysioOmni0.4478 ± 0.01460.4778 ± 0.01180.1738 ± 0.0323
EF-GO(L)0.4211 ± 0.01720.4451 ± 0.01630.1170 ± 0.0202
EF-GO(C)0.4871 ± 0.01050.4830 ± 0.03430.1901 ± 0.0217
REFED-ArousalPhysioOmni0.4537 ± 0.03510.4358 ± 0.03960.1728 ± 0.0600
EF-GO(L)0.4342 ± 0.02050.4431 ± 0.02070.1559 ± 0.0331
EF-GO(C)0.4623 ± 0.01250.4206 ± 0.02300.1770 ± 0.0174
hBCI-MAPhysioOmni0.6424 ± 0.02230.6925 ± 0.03160.7080 ± 0.0241
EF-GO(L)0.6969 ± 0.00560.7472 ± 0.01290.7638 ± 0.0060
EF-GO(C)0.6958 ± 0.01230.7784 ± 0.01040.7668 ± 0.0098
hBCI-MIPhysioOmni0.5420 ± 0.01610.5630 ± 0.01930.5618 ± 0.0214
EF-GO(L)0.6557 ± 0.02230.7255 ± 0.03130.7130 ± 0.0312
EF-GO(C)0.5922 ± 0.02100.6273 ± 0.03390.6236 ± 0.0280
DatasetEEG FMfNIRS branchGraph Agg.ODE AlignmentBalanced accuracyPR-AUC / Weighted F1ROC-AUC / Cohen's Kappa
MI✓×××0.6441±0.00820.7023±0.01230.6977±0.0124
×✓✓×0.5000±0.00000.5003±0.00070.4996±0.0006
✓✓××0.5857±0.02180.6429±0.02870.6361±0.0303
✓✓✓×0.6189±0.02130.6832±0.02130.6706±0.0197
✓w/o warm-up✓✓0.6106±0.01060.6672±0.02310.6571±0.0252
✓✓w/o region prior✓0.5939±0.03140.6574±0.03870.6458±0.0433
✓✓✓w/o time-aware0.6121±0.01590.6671±0.02890.6592±0.0268
✓✓✓✓0.6557±0.02230.7255±0.03130.7130±0.0312
nback✓×××0.3971±0.00860.4161±0.01650.1117±0.0176
×✓✓×0.3328±0.00300.2726±0.0137-0.001±0.0056
✓✓××0.4039±0.01560.4310±0.01530.1251±0.0254
✓✓✓×0.4023±0.01250.4207±0.01070.1215±0.0214
✓w/o warm-up✓✓0.4049±0.01050.4293±0.01480.1232±0.0184
✓✓w/o region prior✓0.3983±0.01430.4151±0.01580.1133±0.0219
✓✓✓w/o time-aware0.3972±0.01080.4172±0.01010.1090±0.0155
✓✓✓✓0.4137±0.01130.4314±0.01280.1458±0.0201
Valence✓×××0.3421±0.00780.3776±0.01910.0085±0.0162
×✓✓×0.3823±0.01400.4208±0.02180.0539±0.0232
✓✓××0.4055±0.01760.4343±0.01710.0908±0.0236
✓✓✓×0.3900±0.00910.4099±0.01870.0537±0.0144
✓w/o warm-up✓✓0.4032±0.0100.4268±0.02100.0864±0.0220
✓✓w/o region prior✓0.4011±0.01520.4346±0.00550.0853±0.0154
✓✓✓w/o time-aware0.4301±0.01830.4395±0.03170.1110±0.0258
✓✓✓✓0.4211±0.01720.4451±0.01630.1170±0.0202
Arousal✓×××0.3448±0.01170.3462±0.01120.0224±0.0173
×✓✓×0.4097±0.00820.4135±0.00820.1217±0.0136
✓✓××0.4115±0.01610.4212±0.01890.1375±0.0270
✓✓✓×0.3987±0.00870.4046±0.01040.1286±0.0167
✓w/o warm-up✓✓0.4257±0.01860.4362±0.01980.1408±0.0279
✓✓w/o region prior✓0.4333±0.02260.4397±0.02270.1606±0.0343
✓✓✓w/o time-aware0.4283±0.02380.4440±0.01890.1436±0.0352
✓✓✓✓0.4342±0.02050.4431±0.02070.1559±0.0331

W2. Scope of the EEG-only comparison and performance with limited paired data

We will revise the description of our contributions to make it more rigorous and avoid overclaiming.

Additionally, although paired EEG–fNIRS data are required for fine-tuning, we also conducted experiments demonstrating its low dependence on fine-tuning data. We included comparative experiments using a small number of samples. When fine-tuned using data from only 1 to 3 participants, EF-GO achieved the best performance compared to task-specific EEG–fNIRS models.

Fine-tuning subjects1 subject2 subjects3 subjects
WG datasetBalanced accuracyPR-AUCROC-AUCBalanced accuracyPR-AUCROC-AUCBalanced accuracyPR-AUCROC-AUC
FGA-Net0.52000.54360.55050.51110.56710.56450.59780.61850.6416
M2NN0.51560.52150.53580.54440.55450.55520.57670.62360.6407
STA-Net0.46560.48620.46860.50000.53840.50110.56560.60820.5982
EF-GO(L)0.56660.63110.61480.62220.67530.66920.62560.71930.7029

W3. Comparison with simpler alignment objectives

We compared Graph–ODE alignment with three direct alignment baselines using MSE, a cosine-based loss, and a contrastive loss. Each baseline loss was applied directly to paired EEG and fNIRS embeddings after graph aggregation. Graph–ODE achieved the highest mean score on all three metrics on hBCI-MA, and on balanced accuracy and ROC-AUC on Cognitive-WG. Cosine-based alignment achieved the highest mean PR-AUC on Cognitive-WG. These results show how the choice of alignment method affects downstream classification on the two evaluated tasks.

TaskAlignment methodBalanced accuracyPR-AUCROC-AUC
hBCI-MAContrastive learning0.6899 ± 0.01130.7376 ± 0.01460.7553 ± 0.0110
Cosine-based alignment0.6875 ± 0.01280.7411 ± 0.01710.7592 ± 0.0141
MSE0.6842 ± 0.00790.7411 ± 0.01340.7558 ± 0.0086
Graph–ODE0.6969 ± 0.00560.7472 ± 0.01290.7638 ± 0.0060
Cognitive-WGContrastive learning0.7256 ± 0.03760.7815 ± 0.04200.8145 ± 0.0336
Cosine-based alignment0.7200 ± 0.02890.7849 ± 0.04190.8125 ± 0.0303
MSE0.7345 ± 0.02840.7744 ± 0.02940.8049 ± 0.0264
Graph–ODE0.7389 ± 0.01000.7826 ± 0.02310.8167 ± 0.0163

Q1. Why does multimodal learning help more on some tasks?

The effect of multimodal adaptation varies across both tasks and backbones. There are inherent differences in the contributions of EEG and fNIRS signals across various tasks, such as cognitive, emotional, and motor imagery tasks. At the same time, due to differences in pre-training datasets and pre-training methods, different backbone models inherently exhibit performance variations. For example, EF-GO(C) achieves higher balanced accuracy than EF-GO(L) on both REFED tasks, where the fNIRS-only branch also performs well in the LaBraM ablations.

To investigate the hBCI-MI results, we measured the effective rank of model weights and representations before and after downstream fine-tuning. The mean effective rank of EEG backbone weights increased slightly, while the effective rank of fNIRS branch weights and representations were essentially unchanged. This suggests that the model's performance did not degrade as a result of fine-tuning. However, the effective rank of the EEG representations showed a significant decline, which may indicate that the EEG signals in this dataset are more sensitive to this backbone, and that the EEG representations may have experienced a collapse in representation. This phenomenon may be the cause of the performance decline.

QuantityBefore fine-tuningAfter fine-tuningRelative change
EEG backbone weights, mean effective rank91.8309792.00325+0.19%
fNIRS branch weights, mean effective rank70.4299870.42979−0.0003%
EEG representations, effective rank89.8120069.78312−22.30%
fNIRS representations, effective rank3.214443.21482+0.0119%

Q2. Robustness to noisy or incomplete inputs

We are evaluating robustness to input noise by comparing downstream performance on the same held-out examples before and after noise is added. We will report the results before the rebuttal period closes.

We will upload the revised version later.

AuthorsOpenReview ↗

Additional response

Q2. Robustness to noisy or incomplete inputs

We tested the model's robustness to noise on the Cognitive-WG dataset using three scenarios: 30% Gaussian noise (based on the sample standard deviation), 30% random patch masking, and 30% random channel missing. Under all three noise scenarios, the proposed EF-FO method still achieved optimal performance. In particular, under patch masking and channel missing conditions, the performance degradation of EF-GO was significantly less than that of the task-specific EEG-fNIRS model, which aligns with our original intent in training the foundation model.

NoisyMethodBalanced accuracyPR-AUCROC-AUC
Clean DataEF-GO(L)0.7389 ± 0.01000.7826 ± 0.02310.8167 ± 0.0163
EF-GO(C)0.6955 ± 0.02030.7880 ± 0.01060.7770 ± 0.0114
30% Gaussian noisyFGA-Net0.6097 ± 0.02340.6277 ± 0.02520.6103 ± 0.0178
M2NN0.5950 ± 0.04130.6264 ± 0.06560.6533 ± 0.0331
STA-Net0.5911 ± 0.03440.6810 ± 0.05310.6690 ± 0.0308
LaBraM0.5945 ± 0.05080.6332 ± 0.03030.6568 ± 0.0218
CBraMod0.6089 ± 0.01030.6244 ± 0.00950.6577 ± 0.014
EF-GO(L)0.6244 ± 0.01470.6817 ± 0.01880.6891 ± 0.0125
EF-GO(C)0.6367 ± 0.02400.6659 ± 0.03030.6929 ± 0.0227
30% patch maskFGA-Net0.6033 ± 0.03980.6214 ± 0.02620.6114 ± 0.0360
M2NN0.5639 ± 0.03940.6318 ± 0.03310.6480 ± 0.0334
STA-Net0.5544 ± 0.03030.6548 ± 0.05540.6308 ± 0.0380
LaBraM0.6344 ± 0.03150.6680 ± 0.03310.6985 ± 0.0297
CBraMod0.6122 ± 0.00220.6302 ± 0.01710.6572 ± 0.0191
EF-GO(L)0.7045 ± 0.01080.7687 ± 0.02730.7847 ± 0.0150
EF-GO(C)0.6422 ± 0.03590.6821 ± 0.02880.7045 ± 0.0247
30% channel missingFGA-Net0.5478 ± 0.05780.5583 ± 0.08120.5519 ± 0.0866
M2NN0.6072 ± 0.04000.6178 ± 0.03110.6402 ± 0.0292
STA-Net0.5511 ± 0.02530.6710 ± 0.06860.6502 ± 0.0637
LaBraM0.6300 ± 0.02290.6555 ± 0.02570.6939 ± 0.0186
CBraMod0.6222 ± 0.03000.6486 ± 0.01910.6782 ± 0.0212
EF-GO(L)0.7056 ± 0.02140.7608 ± 0.02730.7796 ± 0.0167
EF-GO(C)0.6600 ± 0.01420.6976 ± 0.03370.7191 ± 0.0217

W1. Multimodal baseline and ablation coverage

Supplement: The table below presents the ablation experiment results for the EF-GO(C) model, which uses CBraMod as its EEG backbone, on the Cognitive-WG and hBCI-MA datasets:

TaskEEG FMfNIRS branchGraph Agg.ODE AlignmentBalanced accuracyAUC-PRAUC-ROC
Cognitive-WG✓×××0.6389±0.02520.6777±0.02840.6996±0.0271
×✓✓×0.6233±0.02570.6596±0.04030.6756±0.0335
✓✓××0.6622±0.01630.7064±0.02130.7259±0.0177
✓✓✓×0.6811±0.01220.7552±0.01990.7599±0.0212
✓w/o warm-up✓✓0.6678±0.01810.6953±0.02910.7465±0.0287
✓✓w/o region prior✓0.6422±0.03270.6902±0.02340.7266±0.0242
✓✓✓w/o time-aware0.6544±0.02790.7107±0.04290.7464±0.0386
✓✓✓✓0.6955±0.02030.7880±0.01060.7770±0.0114
hBCI-MA✓×××0.6431±0.01060.7527±0.01670.7579±0.0187
×✓✓×0.5695±0.02070.5981±0.03500.6002±0.0280
✓✓××0.6449±0.01030.7121±0.02070.7167±0.0184
✓✓✓×0.6658±0.01450.7686±0.01740.7402±0.0170
✓w/o warm-up✓✓0.6509±0.00730.7187±0.01990.7289±0.0106
✓✓w/o region prior✓0.6637±0.01210.7428±0.03280.7490±0.0220
✓✓✓w/o time-aware0.6559±0.00690.7430±0.01430.7388±0.0080
✓✓✓✓0.6958±0.01230.7784±0.01040.7668±0.0098

We updated the PDF with all of the above information. Thank you again for your review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.