acceptodds
Under review as a conference paper at ICLR 2027

CoLaP: Coupled Latent Prediction for a Synchronized EEG–fNIRS Foundation Model

Abstract

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) capture the neuronal electrical activity and cortical hemodynamic changes of the same underlying brain activity, respectively. Existing brain signal foundation models rely almost exclusively on EEG as input, while synchronous EEG-fNIRS modeling methods are mostly designed for a single task on a fixed dataset, with the model architecture bound to the signal channel configuration and therefore difficult to adapt to datasets with different electrode and optode layouts. To address this problem, this paper proposes the Coupled Latent Prediction (CoLaP) foundation model. The pretraining of CoLaP consists of two objectives, intra-modality masked latent prediction and cross-modality temporal latent prediction. The former infers the latent representation of masked positions from visible context within each modality, and the latter predicts the latent representation of adjacent time windows on the cross-modality fused representation, so that the learned EEG-fNIRS representations preserve the structure of each modality while encoding the coupling between the two modalities. CoLaP is pretrained on 96,305 ten-second synchronized windows from 16 public datasets and evaluated on 7 tasks across 4 downstream datasets. The results show that CoLaP ranks among the top three on 6 of the 7 downstream tasks and outperforms all baselines on mental arithmetic, valence, and arousal recognition. Adaptation protocol experiments show that pretraining yields an average improvement of 7.74% over training from scratch. Modality ablation shows that the coupling information between EEG and fNIRS learned during pretraining transfers to different downstream tasks. For reproducibility, the code and pretrained weights are submitted anonymously with the manuscript.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, fNIRS, brain signal foundation model, self-supervised pretraining, cross-modality representation learning, latent prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WngtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoLaP, a self-supervised pre-training framework for synchronous EEG–fNIRS data. The model employs modality-specific encoders and is trained using a dual objective: intra-modal masked latent prediction and cross-modal, cross-temporal-window latent prediction. The latter integrates EEG and fNIRS data via joint self-attention to predict the representation of the subsequent time window, as generated by an EMA (exponential moving average) target encoder. The training corpus aggregates 16 publicly available synchronous datasets, comprising 96,305 ten-second windows; the authors explicitly state that the pre-training data does not overlap with the four downstream datasets or the subjects included therein. Across seven task categories within these four downstream datasets, CoLaP ranks in the top three for six of them, achieving the best balanced accuracy—among the methods compared by the authors—on mental arithmetic (MA) tasks and the REFED valence/arousal assessment.

Critical strengths
  1. Addressing an empirical gap in self-supervised pre-training for synchronized EEG–fNIRS across datasets. By utilizing 16 publicly available datasets—featuring varying numbers of electrode/optode channels—along with four held-out datasets (unseen during pre-training) and seven downstream tasks, this study provides a robust validation framework that extends beyond single-task fusion models. The cross-dataset and cross-subject partitioning strategy effectively mitigates the risk of inflated performance metrics caused by data leakage.

  2. Benchmarking against both EEG-only models and modality-matched EEG–fNIRS baselines. The evaluation includes models such as EF-Net, TSMMF, and STA-Net, with results reported as mean values ​​alongside standard deviations derived from three independent random seeds. Comparisons across "Full" (multimodal), "EEG-only," and "fNIRS-only" configurations show that performance (MA) improves from 56.31% (EEG-only) to 66.93% (multimodal), and WG increases from 58.43% to 65.00%—demonstrating that multimodal inputs provide complementary information beneficial for the tasks.

Critical weaknesses
  1. The paper lacks a critical ablation study that isolates the two self-supervised objectives; consequently, the causal claim that the model has "learned cross-modal coupling" is somewhat overstretching. Existing experiments primarily involve removing one of the input modalities rather than comparing different configurations while maintaining dual-modal inputs—such as using only intra-modal masked objectives, only cross-temporal prediction, a combination of both, or alternative fusion strategies (e.g., omitting joint self-attention or using simple concatenation). The fact that the full model outperforms single-modal variants merely demonstrates that the second modality is useful; it does not, by itself, prove that the proposed cross-modal temporal latent prediction is essential.

  2. The performance advantage is limited in both magnitude and consistency across tasks. For the MA task, the model achieves a balanced accuracy of 66.93% compared to 66.37% for the best-performing baseline (LaBraM)—a margin of only about 0.56 percentage points, which is smaller than the reported standard deviations. On the WG task, the model’s 65.00% is slightly lower than the 65.09% achieved by TSMMF. For the NB task, the 38.27% result falls below the 40.84% ​​achieved by an EEG-only baseline, as well as several other baselines. Finally, on the ENTER task, the model’s 51.85% is lower than LaBraM’s 57.97%.

Questions for authors
  1. While strictly maintaining identical dual-modal inputs, encoder parameter counts, and training budgets, please evaluate the impact of removing each of the following components: masked latent prediction, cross-modal temporal prediction, joint attention, and the objective based on adjacent time windows. What are the resulting changes in downstream performance for each ablation?

  2. Do the key competitive advantages (particularly on the MA and REFED-Valence/Arousal datasets) remain consistent when evaluated across multiple random seeds (using fixed subject splits) and through paired significance tests or confidence intervals? Furthermore, how should the non-monotonic performance variations observed across different pre-training budgets be interpreted?

  3. How are the practical synchronization errors between EEG and fNIRS—arising from variations across subjects and hardware—as well as the hemodynamic lag of fNIRS relative to EEG, addressed? If the temporal axes of the two modalities are intentionally misaligned, does the proposed cross-modal temporal prediction mechanism remain effective?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Wngt

Thank you for the thorough and encouraging review. We greatly appreciate your recognition of the robust validation framework of this work, of the effective suppression of data leakage through cross-dataset and cross-subject splits, and of our benchmarking against both EEG-only models and modality-matched EEG-fNIRS baselines. We also recognize two shortcomings: the causal claim of learning cross-modal coupling is not isolated, and the performance advantage is limited in both magnitude and cross-task consistency, which we address one by one below.

Response to Weakness 1

Thank you for the suggestion. In the original manuscript's experimental design, comparing the full model with single-modality variants by removing one input modality only establishes that the second modality is useful, and it is insufficient to show that our proposed cross-modal temporal latent-space prediction is indispensable. To remedy this, we added a set of component-wise ablations that remove the masked latent-space prediction, the cross-modal temporal prediction, the joint attention, and the adjacent-window objective one at a time while keeping the bimodal input, the encoder parameter count, and the training budget unchanged, with the detailed results in the first response below. These experiments consistently show that removing the cross-modal temporal objective or the joint attention causes a clear drop in downstream performance, most pronounced on MA, indicating that the model has learned cross-modal coupling rather than merely using two modalities. Accordingly, we position our contribution as the semantic alignment jointly established by the cross-modal temporal latent-space prediction objective and the joint self-attention.

Response to Weakness 2

We explain the limited magnitude and cross-task consistency of the gains from two aspects. First, collecting synchronized EEG-fNIRS data is far more difficult than collecting a single modality: the pretraining corpus of this work consists of 16 public synchronized datasets, totaling 96,305 ten-second windows and about 268 hours. For comparison, large-scale EEG-only foundation models can rely on much larger corpora: LaBraM is pretrained on about 2,500 hours of EEG and CBraMod on about 9,000 hours of the Temple University EEG corpus, roughly ten and thirty times ours, respectively. Given a corpus more than ten times larger and the requirement of synchronized acquisition of two modalities, CoLaP is on par with these models on MA and achieves better and significant per-subject results on MI and on the REFED valence and arousal tasks. Second, our model is a foundation model that transfers to seven downstream tasks after a single pretraining, with no need to design a task-specific architecture from scratch for each task. Taking TSMMF as an example, such task-specific models are designed for a specific task and dataset and trained on that task, so their representations are hard to transfer to other tasks, whereas our channel-agnostic architecture and statistical pooling allow the same representation to transfer across heterogeneous datasets. The limited performance advantage on some tasks is therefore mainly constrained by the scale of synchronized data rather than by a lack of generality in the method. The contribution of this work is, to the best of our knowledge, the first cross-dataset synchronized EEG-fNIRS foundation model and its channel-agnostic architecture, a contribution that does not depend on the accuracy ranking on any single task; we have revised the abstract and introduction to position the contribution accordingly, away from the highest accuracy.

AuthorsOpenReview ↗

Response to Question 1

Under strictly the same bimodal input, encoder parameter count, and training budget, we removed these four components one at a time and fine-tuned each variant on the seven downstream tasks under the same protocol. The cross-modal temporal prediction you mention is our cross-modal temporal latent-space prediction task, in which each modality's temporal prediction branch predicts the latent representation of the adjacent window from the fused representation of the joint self-attention, and the joint attention you mention is our joint self-attention module. Accordingly, the four variants are denoted the no-temporal-objective, no-mask-objective, no-joint-attention, and no-adjacent-window variants, where the no-adjacent-window variant changes the objective from predicting the next window to predicting the current window.

The table below reports the balanced accuracy of the full model and the four variants on the seven tasks, in percentage.

TaskFull modelNo temporalNo maskNo joint attentionNo adj. window
MA67.8661.0168.2462.8661.75
MI55.5658.8057.9659.5457.78
WG63.6159.9157.4859.7556.51
NB40.2339.7140.6440.7441.05
REFED valence49.1548.7148.7747.4449.05
REFED arousal46.8346.5248.8445.8748.16
ENTER53.1552.5852.8153.7952.36

We describe the contribution of each component in turn below.

First, after removing the cross-modal temporal prediction, MA drops from 67.86% to 61.01%, a decrease of 6.85 percentage points, the largest drop among the four ablations, and this drop has a consistent direction across all seeds with decreases ranging from 3.59 to 11.12 percentage points. This shows that the cross-modal temporal latent-space prediction task is the main source of the cross-modal coupling learned by the model. We should note honestly that this objective does not bring gains on every task: MI rises slightly by 3.24 percentage points after the temporal objective is removed.

Second, after removing the joint attention, MA drops to 62.86%, a decrease of 5.00 percentage points. Third, after removing the adjacent-window objective, MA drops to 61.75%, a decrease of 6.11 percentage points. Removing either component causes a clear drop, showing that the joint attention and the adjacent-window objective are both components of the cross-modal coupling, and removing either weakens the coupling.

Fourth, after removing the masked latent-space prediction, MA rises by 0.38 percentage points while WG drops by 6.13 percentage points. This shows that the intra-modal mask objective and the cross-modal temporal objective play their roles on different tasks: the mask objective mainly supports tasks such as WG that rely on within-modality structure, and it contributes little on tasks such as MA that rely mainly on cross-modal coupling.

In addition, on ENTER the changes of all four ablations are within 1 percentage point, consistent with the observation that this task relies mainly on EEG, which further indicates that the contribution of cross-modal coupling is concentrated on tasks where the two modalities genuinely bring complementary information.

Taken together, these four points show that this set of ablations isolates the cross-modal coupling itself rather than merely the gain from using two modalities. We position our contribution as the semantic alignment jointly established by the cross-modal temporal latent-space prediction objective and the joint self-attention, together with the channel-agnostic architecture and the 16-dataset corpus.

AuthorsOpenReview ↗

Response to Question 2

We address the two parts separately. For cross-seed robustness, we added a leave-one-subject-out evaluation that pairs CoLaP and LaBraM on exactly the same subjects, and we report paired t-tests and Wilcoxon signed-rank tests, with the results in the table below, in balanced accuracy, in percentage points.

TaskCoLaPLaBraMDifferencePaired t-test (p)Wilcoxon (p)
MI (29 subjects)55.8650.98+4.882.749 (0.010)0.009
REFED valence (32 subjects)47.9743.37+4.604.678 (<0.001)<0.001
REFED arousal (32 subjects)45.8939.20+6.694.606 (<0.001)<0.001
MA (29 subjects)64.7567.11-2.36-1.174 (0.250)0.294

The key competitive advantage on MI and the two emotion tasks is consistently preserved across subjects. On MI, CoLaP is on average 4.88 percentage points higher, with a paired t-test p = 0.010 and a Wilcoxon test p = 0.009. On REFED valence, CoLaP is on average 4.60 percentage points higher, with both the paired t-test and the Wilcoxon test giving p-values below 0.001. On REFED arousal, CoLaP is on average 6.69 percentage points higher, and both tests are significant. On MA, the per-subject difference is not significant, with CoLaP on average 2.36 percentage points lower and a paired t-test p-value of 0.250. Therefore the key competitive advantage is consistently preserved across subjects on MI and on the REFED valence and arousal tasks.

Regarding the non-monotonic changes across pretraining budgets, the pattern in the paper is task-dependent rather than a single trend, and we report this pattern honestly in the paper, as shown in Table 10. Between 16, 40, and 80 epochs, MA rises monotonically from 59.62% through 62.39% to 66.93%, and WG rises from 53.10% through 56.64% to 65.00%. NB moves in the opposite direction, from 41.15% down to 38.27%; REFED valence plateaus after 16 epochs and then declines slightly, and REFED arousal first rises and then declines slightly. MI and ENTER stay roughly flat. For the tasks that keep rising, the discriminative signal benefits as the representation is progressively refined, whereas for the tasks that plateau or decline, the representation is already close to its ceiling at 16 or 40 epochs, and for them the later representation is slightly less useful than the earlier, more general representation.

Reviewer_354NOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoLaP combines two modality-specific criss-cross encoders with intra-modality masked latent prediction and cross-modal temporal latent prediction. The first objective predicts masked representations supplied by an EMA encoder; the second fuses EEG and fNIRS representations through joint attention and predicts representations of the next time window. Channel-independent tokenization and statistical pooling allow the model to accept different channel counts. CoLaP is pretrained on 96,305 ten-second windows from 16 synchronized datasets and evaluated on seven classification tasks across four downstream datasets. The paper reports modality ablations, frozen probing, partial/full fine-tuning, and pretraining-budget comparisons, with the highest balanced accuracy on mental arithmetic and the two REFED tasks.

Critical strengths
  1. A useful effort toward reusable synchronized EEG-fNIRS representations. The paper consolidates 16 public synchronized datasets and develops a model that accepts heterogeneous channel counts. This addresses a practical obstacle in multimodal brain-signal learning, where acquisition configurations vary considerably. The corpus aggregation and reusable architecture could provide a useful basis for subsequent research beyond individual task-specific fusion models.
  2. Clear presentation with informative adaptation experiments. The figures and tables communicate the method and results effectively. In addition to the main comparison, the authors evaluate linear and nonlinear probing, partial/full fine-tuning, unimodal variants, and different pretraining budgets. These experiments provide meaningful evidence that the pretrained representations are useful on several evaluated tasks and expose limitations, including the N-back degradation. This makes the empirical contribution more informative than a comparison of final fine-tuned scores alone.
Critical weaknesses
  1. The specific motivation and empirical advantage of coupled latent prediction are insufficiently established. The introduction argues that latent prediction may avoid fitting irrelevant details in raw signals. This is a reasonable general motivation, but it does not establish why the proposed objectives are particularly suitable for EEG-fNIRS or preferable to reconstruction, contrastive alignment, or hybrid objectives. Related approaches already provide relevant alternatives: EEG2Rep studies masked latent prediction, while EFRM combines masked reconstruction with cross-modal contrastive learning. CoLaP includes comparisons with methods using different objectives, but their architectures, pretraining data, and input configurations also differ. Consequently, these comparisons do not isolate the effect of the proposed objective. More importantly, Table 3 changes both pretraining modality availability and downstream inputs. Its results therefore cannot distinguish a benefit learned through coupling during pretraining from the benefit of having two modalities at inference. The temporal predictors also retain access to their own modality’s current representation, including the residual pathway in Eq. (11), so within-modality predictability remains an alternative explanation for successful forecasting. The paper lacks the decisive controls: retaining both downstream inputs while removing the temporal objective, and comparing against independently pretrained modality encoders followed by the same downstream fusion. Since coupled latent prediction is the central methodological contribution, this attribution gap substantially weakens the case for acceptance.
  2. The breadth and robustness of transfer are insufficiently demonstrated for the foundation-model claim. All seven tasks come from four synchronized EEG-fNIRS datasets, with some targets sharing the same recordings, such as REFED valence and arousal. The cross-subject splits and frozen-probing results provide meaningful transfer evidence within this setting. However, the unimodal variants in Table 3 are separately pretrained models evaluated on the same paired-data collections. They do not establish that the jointly pretrained checkpoint transfers effectively to independent EEG-only or fNIRS-only datasets, or remains useful when a modality is unavailable. This distinction matters for whether multimodal pretraining produces broadly reusable modality representations. A model specialized for synchronized recordings can still be valuable, but the scope of its generality should be supported or explicitly restricted. The small held-out subject sets further limit the strength of the generalization evidence. Table 7 uses only two test subjects per split for the cognitive dataset, three for EEG-NIRS BCI and REFED, and five for ENTER. The three seeds repartition subjects, so the reported variability reflects changes in evaluation subjects as well as any training randomness. Some advantages are modest, including 0.56 percentage points over LaBraM on MA, while CoLaP underperforms its EEG-only variant, scratch training, and linear probing on N-back. Subject-level paired results and broader subject coverage would help establish whether the reported improvements are reliable.
Questions for authors
  1. Clarification and targeted evidence concerning objective choice. Are there controlled comparisons that keep the dual encoders, paired corpus, tokenization, and training budget comparable while replacing latent prediction with reconstruction or reconstruction plus contrastive alignment? In particular, what happens when the temporal objective is removed but both modalities and the downstream fusion architecture are retained? These results would directly affect my assessment of the methodological contribution.
  2. Targeted evidence concerning coupling attribution. Does CoLaP outperform independently pretrained EEG and fNIRS encoders combined through the same downstream fusion model? Alternatively, what evidence shows that the temporal predictors materially depend on the other modality, rather than primarily using within-modality temporal structure? This is a request to substantiate the current coupling claim.
  3. Clarification of transfer scope. Have the jointly pretrained encoders, without separate unimodal pretraining, been evaluated on independent EEG-only or fNIRS-only datasets? If so, please report the adaptation protocol and missing-modality handling. If not, is the intended claim specifically limited to synchronized EEG-fNIRS transfer? This distinction would affect my assessment of the breadth of the contribution.
  4. Clarification of evaluation robustness. Can the authors provide per-subject paired performance differences for the leading comparisons, particularly MA and the REFED tasks, and clarify whether model-training randomness was varied independently of the subject partitions? Such evidence would help determine whether the small reported advantages are stable across subjects.
7 comments
AuthorsOpenReview ↗

Response to Reviewer 354N

Thank you for the careful and detailed feedback. We especially appreciate your recognition of the reusable synchronized EEG-fNIRS representations we build, and of the value of the corpus integration and the channel-agnostic architecture. You also point out two shortcomings: first, the contribution of cross-modal temporal latent-space prediction is not isolated on its own, and second, the breadth and robustness of transfer are insufficiently demonstrated.

Response to Weakness 1 (Part 1)

We thank you for pointing out that the contribution of cross-modal temporal latent-space prediction is not demonstrated on its own. The original submission did not adequately establish the specific motivation and empirical advantage of cross-modal temporal latent-space prediction, and the existing comparisons mix the objective, the architecture, and the data together, so they cannot isolate the effect of the objective alone. We fully follow your suggestion and add decisive controls, which we describe in turn below.

First, the cross-modal temporal latent-space prediction auxiliary task suits synchronized EEG-fNIRS data, which is determined by the structure of the data itself. EEG and fNIRS are two physical views of the same neural state, and neurovascular coupling makes the two views correlated at the physiological and signal levels, so a model that uses both must map the two views into the same shared latent representation, which is exactly the semantic alignment we aim to learn. The joint-embedding predictive architecture is designed for such a multi-view structure: it predicts representations in a shared latent space rather than reconstructing the raw signal. Reconstruction forces the model to recover raw-signal details that are irrelevant to the downstream cognitive state, such as high-frequency EEG noise and the slow drift and optical artifacts of fNIRS, and this reconstruction cost is larger in the bimodal setting than in the single-modality setting. Contrastive alignment depends on negative-sample construction and is prone to representation collapse. The EEG2Rep model you mentioned performs masked prediction in the latent space, but it is a single-EEG-modality model that predicts only within the modality and cannot establish semantic alignment between EEG and fNIRS. Similarly, EFRM combines masked reconstruction with cross-modal contrastive learning, but its contrastive coupling merely pulls paired views together coarsely and does not exploit the synchronization that predictive objectives rely on, while its masked reconstruction inherits the drawback of recovering irrelevant details. Neither therefore provides prediction-based cross-modal temporal latent-space prediction, and this is exactly where CoLaP differs.

Second, to isolate the contribution of the objective on its own, we added the within-model ablation in Table 11, which changes only the pretraining objective while keeping the architecture, data, split, and training budget unchanged. The issue you raised about Table 5, namely that the pretraining modality and the downstream input are changed simultaneously, is resolved by the no-temporal-objective variant, which keeps the downstream bimodal input and both encoders and removes only the cross-modal temporal latent-space prediction objective. The comparison between the full model and this variant therefore isolates exactly the benefit of cross-modal temporal latent-space prediction learned during pretraining, and it also serves as the control against independently pretrained encoders combined with the same downstream fusion. The no-mask-objective variant keeps the temporal objective and removes the mask objective, and is used to examine the contribution of the mask objective on its own.

Response to Weakness 1 (Part 2)

Third, regarding the concern that the residual connection in Equation (11) still lets the predictor read the information of its own modality, we added a no-residual control that removes this residual connection so the temporal predictor can read only the representation produced by the joint attention. The no-residual ablation shows that, under the same protocol, the no-residual variant achieves 66.75% on MA, differing from the full model's 67.86% by 1.11 percentage points, far smaller than the 5.00-percentage-point drop caused by the no-joint-attention variant. This shows that the temporal predictor indeed reads the information produced by the joint attention, that the within-modality residual path contributes only a small part of the gain, and that the cross-modal path is the main source, so this gain is a genuine cross-modal contribution. The no-joint-attention variant removes the joint attention and keeps only the within-modality path with the residual connection, so any drop from the full model to this variant is a cross-modal contribution; its downstream result is 62.86% on MA, a drop of 5.00 percentage points relative to the full model's 67.86%. The table below summarizes the balanced accuracy of the full model, the no-residual variant, and the no-joint-attention variant on seven downstream tasks, in percentage.

TaskFull modelNo residualNo joint attention
MA67.8666.7562.86
MI55.5658.5259.54
WG63.6161.1559.75
NB40.2342.2840.74
REFED valence49.1548.8947.44
REFED arousal46.8344.2045.87
ENTER53.1551.6953.79

The diagnostics in Table 12 additionally and independently show that the offset and strength of the joint attention are produced entirely by the temporal objective and disappear when the temporal objective is removed; the within-modality residual connection alone cannot produce this result.

In summary, comparing the full model with the no-temporal-objective variant, MA drops from 67.86% to 61.01%, a decrease of 6.85 percentage points, and this drop has a consistent direction across all seeds, whereas removing the mask objective leaves MA nearly unchanged at 68.24%. The benefit of cross-modal temporal latent-space prediction is now distinguished from two alternative explanations: one that the gain comes from within the modality, and the other that it comes from having both modalities at the downstream stage.

Response to Weakness 2

Regarding transfer scope, all seven tasks come from four synchronized EEG-fNIRS datasets, and the REFED valence and arousal tasks share recordings, so we do not claim breadth beyond the scope of synchronized EEG-fNIRS data. The upper bound of this scope comes from the availability of the data itself rather than from any deliberate narrowing on our part. To the best of our knowledge, the 16 public synchronized EEG-fNIRS datasets used for pretraining and the four downstream datasets are already the largest publicly available synchronized EEG-fNIRS corpus. To further broaden the coverage, we also requested authorization from the authors or platforms of several synchronized EEG-fNIRS datasets not included in this work, such as the video-emotion fNIRS-EEG database FEAD, the music-emotion AfNIRS-EEG emotion dataset FEEL, and the restricted-access emotion dataset MER-PS of the MER 2026 challenge, but none of these requests had been answered at the time of writing, so they could not be included. The single-modality variants in Table 5 are pretrained separately on the same paired data and can only show that the extra modality available at inference provides complementary information; they do not prove that the model can transfer to independent EEG-only or fNIRS-only datasets. We make no such claim, and we explain in our response to Question 3 why such an evaluation cannot test the ability the model is meant to learn. The claim of this work is the semantic alignment learned from synchronized EEG-fNIRS data, which is stated in the abstract and introduction; we do not claim generalization to independently collected single-modality datasets.

Regarding robustness, we acknowledge and address three shortcomings. First, the number of held-out test subjects is small: the cognitive dataset has only 2, the EEG-NIRS BCI and REFED datasets have 3 each, and the ENTER dataset has 5, which reflects the number of subjects that synchronized EEG-fNIRS datasets can provide. To address the resulting statistical weakness, we added a leave-one-subject-out (LOSO) evaluation that pairs CoLaP and LaBraM on all subjects one by one, with sample sizes covering 29 subjects in MA and 32 in each REFED task, as described in our response to Question 4. Second, the reported random seeds re-split the subjects, so the result variation contains both subject selection and training randomness; the LOSO evaluation fixes the split so that each subject is tested exactly once, so the paired difference reflects only the model behavior rather than a favorable split, with the paired-test statistics in our response to Question 4. The contribution of this work is, to the best of our knowledge, the first cross-dataset synchronized EEG-fNIRS foundation model and its channel-agnostic architecture.

Reviewer_ReraOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduced a model to learn a joint representation of EEG and fNIRS

Critical strengths

The results of the work are very strong, and they show that it performs and generalizes very well. Also, ablation studies show empirical evidence for this joint model against single-modality models.

Critical weaknesses

The algorithmic contribution of the work is very minor for a top-tier AI venue.

Questions for authors

1- Have you tried to bookmarks that only includes EEG or fNIRS? Does multimodality add extra information, or does it uncover something that is not possible with a single modality? 2- Have you tried pretrained time series models (next-token prediction) and CLIP loss?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Rera

Thank you for the detailed and specific comments. We greatly appreciate your recognition of the robustness and generalization of our results, and of the ablation evidence showing the joint model over the single-modality models. We take equally seriously the concern that the algorithmic contribution is too incremental for a top-tier AI venue, and we address it first, in our response to Comment 1 below.

Regarding how we position the algorithmic contribution, we first describe the state of the field. Synchronized EEG and fNIRS have previously been used mainly for supervised fusion on a single dataset, where the model depends on the number of electrodes and optodes and the montage of the training data and cannot transfer directly to other datasets or tasks. Brain foundation models instead learn general-purpose representations through self-supervised pretraining on large-scale unlabeled signals, thereby enabling cross-dataset and cross-task transfer. Almost all existing brain foundation models are built around a single EEG modality; for example, LaBraM, EEGPT, CBraMod, and LUNA each model only within their own modality. In the EEG-fNIRS multimodal direction, EFRM was previously the only model, and because most of its training data are asynchronously collected, it can only learn coarse-grained contrastive alignment between the two modalities and cannot establish semantic alignment that truly reflects the synchronization relationship. Semantic alignment of synchronized EEG and fNIRS therefore remains an open gap.

CoLaP fills this gap. It is trained on fully synchronized data and learns the semantic alignment between the two modalities through a joint self-attention mechanism. To the best of our knowledge, the cross-modal temporal latent-space prediction task supported by this mechanism is first proposed in this work.

We state our contributions in two points. First, CoLaP is, to the best of our knowledge, the first synchronized EEG-fNIRS foundation model across heterogeneous datasets. It is self-supervised pretrained on 96,305 ten-second windows from 16 public synchronized datasets to learn general-purpose representations of synchronized EEG and fNIRS. Because the datasets differ in channel count, CoLaP adopts a channel-agnostic architecture with per-channel tokenization and statistical pooling, which enables cross-dataset and cross-task transfer. Second, CoLaP is a JEPA-style foundation model that combines cross-modal temporal latent-space prediction with intra-modal masked latent-space prediction, and its joint self-attention mechanism learns the semantic alignment between the two modalities.

We also wish to note that reproducibility, evaluation breadth, and statistical rigor are themselves scientific contributions, which, together with the channel-agnostic architecture, the JEPA-style objectives, and the cross-dataset synchronized EEG-fNIRS corpus, constitute the contributions of this work.

AuthorsOpenReview ↗

Response to Question 1

There are two sub-questions here: whether we trained EEG-only or fNIRS-only backbones, and whether the multimodal setting brings information that a single modality cannot provide.

For the first sub-question, under exactly the same encoder, tokenization, pretraining budget, and downstream protocol as the full model, we separately pretrained EEG-only and fNIRS-only variants, each retaining only one modality's input and loss, so the difference among the three stems only from the input modality. The balanced accuracy on the seven tasks is shown below, with full results in Table 5 of the paper.

TaskCoLaPEEG-onlyfNIRS-only
MI56.7654.2653.80
MA66.9356.3159.95
NB38.2740.8435.08
WG65.0058.4355.85
REFED valence48.3644.1346.43
REFED arousal46.3442.4538.85
ENTER51.8549.0235.62

The full model surpasses both single-modality backbones on six of the seven tasks, and this is not driven by a single modality alone. On MA, the full model achieves 66.93%, which is 10.62 percentage points higher than the EEG-only 56.31% and 6.98 percentage points higher than the fNIRS-only 59.95%, so neither single modality reaches the full model's level. On WG, the full model reaches 65.00%, higher than the EEG-only 58.43% and the fNIRS-only 55.85%. On REFED arousal, the full model reaches 46.34%, higher than the EEG-only 42.45% and the fNIRS-only 38.85%. On these tasks both single modalities are clearly surpassed, indicating that the gains come from the joint participation of both modalities. On ENTER, the fNIRS-only model achieves only 35.62%, far below the EEG-only 49.02% and the full model 51.85%, indicating that this task relies mainly on EEG. We also note that on MA and REFED valence, the fNIRS-only model reaches 59.95% and 46.43%, higher than the EEG-only 56.31% and 44.13%, indicating that fNIRS provides complementary information beyond EEG on emotion-related tasks.

The ablation in Table 11 further localizes the source of this gain. Keeping the bimodal input, the encoder parameter count, and the training budget unchanged and removing only the cross-modal temporal latent-space prediction task, the balanced accuracy on the key tasks is shown below, with the full seven-task ablation in Table 11 of the paper.

TaskFull modelNo temporal objective
MA67.8661.01
REFED arousal46.8346.52

MA drops by 6.85 percentage points, whereas REFED arousal stays nearly unchanged, down only 0.31 percentage points, indicating that the gain comes mainly from cross-modal temporal latent-space prediction itself. This task drives the joint self-attention to align the EEG and fNIRS views to the same neural state and thus learn the semantic alignment between the two modalities, rather than merely concatenating the two modalities. This table uses the same three seeds (7, 42, 2026) as Table 5, and the full model's MA of 67.86% differs from Table 5's 66.93% by 0.93 percentage points, which falls within the range of seed variation.

For the second sub-question, the additional information brought by the multimodal setting comes from the two modalities' different ways of observing the same neural event. EEG records cortical electrical activity at the millisecond scale with high temporal resolution but limited spatial localization, because volume conduction spreads the activity of a single neural source to multiple scalp electrodes and the signal is easily contaminated by muscle activity and motion artifacts. fNIRS records the hemodynamic response of oxygenated and deoxygenated hemoglobin at the second scale with lower temporal resolution but more localized spatial coverage, reflecting cortical activation and metabolic demand. These two signals are not independent of each other; neurovascular coupling links fast electrical firing with the slow vascular response, making them two complementary views of the same underlying neural state. EEG alone records only fast electrical activity and cannot obtain the metabolic information carried by the hemodynamic response, while fNIRS alone records only the slow vascular response and cannot obtain millisecond-scale electrical activity. This correspondence between the two views is exactly the information that neither single modality can obtain alone, and this is where the multimodal setting reveals extra information relative to a single modality.

AuthorsOpenReview ↗

Response to Question 2

Regarding the contrastive CLIP loss. The contrastive baselines in the original submission confirm that contrastive alignment is inferior to predictive objectives. EFRM combines masked reconstruction with cross-modal contrastive alignment, and BIOT is a general contrastive biosignal model; both are evaluated as EEG-fNIRS models with the same inputs, windows, and subject splits as CoLaP, with results in Tables 2 and 4 of the paper. The balanced accuracy on three representative tasks is shown below.

TaskCoLaP (predictive)EFRM (contrastive alignment)BIOT (contrastive)
MA66.9356.2454.15
REFED valence48.3644.6044.85
REFED arousal46.3435.0940.36

CoLaP outperforms both contrastive baselines on all three tasks. On MA, CoLaP achieves 66.93%, which is 10.69 percentage points higher than EFRM's 56.24% and 12.78 percentage points higher than BIOT's 54.15%. On REFED valence and arousal, CoLaP again surpasses both baselines with 48.36% and 46.34%, respectively, where EFRM reaches only 35.09% on arousal. The consistent results of the published contrastive baselines show that contrastive alignment is inferior to predictive objectives, which is exactly why we adopt JEPA-style latent-space prediction.

Regarding autoregressive next-token prediction. We searched for published autoregressive EEG foundation models and found no baseline that can be compared fairly under our research setting. The autoregressive model most directly relevant to this question, EEGPT, has not released code and cannot be reproduced. NeuroLM provides runnable code and public weights, but its design goal is to align EEG with language rather than to align two neural modalities; its text-alignment tokenizer and language-oriented instruction tuning address a different research question, and the model is still EEG-only, so it can neither use fNIRS nor learn the semantic alignment between EEG and fNIRS that this work studies. There is therefore no fairly comparable autoregressive next-token baseline.

In future work, we plan to explore brain foundation models based on autoregressive next-token prediction, and we will add a comparison in this direction once a baseline that can be fairly compared under our setting becomes available.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response (Part 1)

Dear Area Chair and Reviewers,

Re: Response to Reviewers for ICLR 2027 Submission ID 59433, titled 'CoLaP: Coupled Latent Prediction for a Synchronized EEG–fNIRS Foundation Model'

We thank the reviewers and the area chair for their careful reading and for their specific, actionable comments. In response, we conducted a set of additional experiments, several of which were computationally expensive. We have revised the manuscript accordingly and address each comment in turn below.

1. Motivation. Why we conduct this study now.

Brain foundation models are almost exclusively built around a single modality. Most existing work pursues higher accuracy on single-modality EEG benchmarks, with far less exploration of the multimodal EEG-fNIRS direction. EEG records cortical electrical activity at the millisecond scale, whereas fNIRS records hemodynamic responses at the second scale. The two are complementary in temporal scale and physiological mechanism through neurovascular coupling, and they constitute two physical views of the same neural activity. For a model to exploit both, it must align the two views in a shared latent space according to the same neural state, that is, learn the semantic alignment between the two modalities. Such alignment presupposes synchronized acquisition, because only by knowing which EEG segment and which fNIRS segment originate from the same state can a model learn their true correspondence.

However, most existing EEG foundation models rely on generative reconstruction or contrastive learning. Generative reconstruction forces the model to recover details of the raw signal that are often irrelevant to the downstream cognitive state. Contrastive learning depends on negative-sample construction and is prone to representation collapse. To the best of our knowledge, EFRM, the only prior EEG-fNIRS foundation model, was trained mostly on asynchronously collected data, so it can only learn coarse-grained contrastive alignment and cannot establish semantic alignment between the two modalities. The semantic alignment of synchronized EEG and fNIRS therefore remains an open gap.

To this end, we propose the first foundation model trained entirely on synchronized EEG-fNIRS data. It learns general-purpose representations of synchronized EEG and fNIRS through self-supervised pretraining and learns the semantic alignment between the two modalities through a joint self-attention mechanism. To address the key questions raised by the reviewers, namely whether the hemodynamic modality helps, whether cross-modal temporal learning is genuinely cross-modal, and which component of the pretraining objective plays the key role, we conducted controlled ablation studies and provide evidence for each in our responses.

2. Contribution I. The first synchronized EEG-fNIRS foundation model across heterogeneous datasets, with a channel-agnostic architecture.

CoLaP is a 53.2-million-parameter foundation model trained on 16 public synchronized EEG-fNIRS datasets, covering 96,305 ten-second windows with heterogeneous channel counts. Per-channel tokenization combined with statistical pooling lets a single model accept any number of electrodes and optodes without retraining, which allows it to be trained and transferred directly across the 16 datasets with different montages. Models that embed a fixed channel order, such as the standard 10-20 montage, lack this capability. Synchronized EEG and fNIRS were previously used mainly for supervised fusion on a single dataset, where the network structure and weights depend on the number of electrodes and optodes and the montage of the training data, so changing datasets or tasks requires reconfiguration and training from scratch. CoLaP instead performs self-supervised pretraining on large-scale unlabeled synchronized signals, and the learned general-purpose representations are independent of any specific downstream task, enabling cross-dataset and cross-task transfer.

(Continued in the next comment.)

AuthorsOpenReview ↗

Overall Response (Part 2)

3. Contribution II. We first propose the cross-modal temporal latent-space prediction task, which, combined with intra-modal masked latent-space prediction, forms a JEPA-style foundation model.

This contribution consists of two latent-space prediction tasks completed within a single joint-embedding predictive architecture (JEPA). Existing EEG and EEG-fNIRS foundation models are mostly built on masked reconstruction, contrastive learning, or next-token prediction, whereas CoLaP adopts latent-space prediction. The first task is intra-modal masked latent-space prediction, in which each modality predicts the latent representations of its own masked positions using the output of an exponential moving average (EMA) encoder as the target, consistent with existing JEPA-style single-modality models. The second task is cross-modal temporal latent-space prediction, in which the latent representations of the two modalities are first coupled through joint self-attention and a temporal predictor then predicts the latent representation of the next window for each modality. To the best of our knowledge, this cross-modal temporal latent-space prediction task is first proposed in this work, and its joint self-attention mechanism exploits the synchronized paired information to learn the semantic alignment between the two modalities, which intra-modal prediction alone cannot provide.

4. New experiments.

We have added three categories of experiments and incorporated the corresponding results into the revised manuscript. First, a set of component-wise ablation studies that remove each self-supervised objective one at a time while keeping the bimodal input, the encoder parameter count, and the training budget unchanged; the results are reported in the new Tables 11 and 12 in the main text. Second, a set of analyses on evaluation robustness. The first is a per-subject paired significance test using leave-one-subject-out evaluation, which pairs CoLaP and LaBraM on exactly the same subjects; the results for the four tasks of the MI, MA, and REFED valence and arousal datasets are presented in our response to Reviewer 2 Question 4 and Reviewer 3 Question 2. The second is per-seed robustness of the ablation, with per-seed results for the full model and the no-temporal-objective variant on MA reported in the new Table 13 in the appendix. Third, a misalignment sensitivity study that examines whether the model remains robust when the fNIRS timeline is deliberately shifted relative to EEG, with results in the new Table 14 in the appendix. The code and pretrained weights have been submitted anonymously; we will release the full repository, including code and pretrained weights, after double-blind review to support transparent verification and reproduction.

Warm regards,

Authors of Submission 59433

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.