- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CoLaP, a self-supervised pre-training framework for synchronous EEG–fNIRS data. The model employs modality-specific encoders and is trained using a dual objective: intra-modal masked latent prediction and cross-modal, cross-temporal-window latent prediction. The latter integrates EEG and fNIRS data via joint self-attention to predict the representation of the subsequent time window, as generated by an EMA (exponential moving average) target encoder. The training corpus aggregates 16 publicly available synchronous datasets, comprising 96,305 ten-second windows; the authors explicitly state that the pre-training data does not overlap with the four downstream datasets or the subjects included therein. Across seven task categories within these four downstream datasets, CoLaP ranks in the top three for six of them, achieving the best balanced accuracy—among the methods compared by the authors—on mental arithmetic (MA) tasks and the REFED valence/arousal assessment.
Critical strengths
-
Addressing an empirical gap in self-supervised pre-training for synchronized EEG–fNIRS across datasets. By utilizing 16 publicly available datasets—featuring varying numbers of electrode/optode channels—along with four held-out datasets (unseen during pre-training) and seven downstream tasks, this study provides a robust validation framework that extends beyond single-task fusion models. The cross-dataset and cross-subject partitioning strategy effectively mitigates the risk of inflated performance metrics caused by data leakage.
-
Benchmarking against both EEG-only models and modality-matched EEG–fNIRS baselines. The evaluation includes models such as EF-Net, TSMMF, and STA-Net, with results reported as mean values alongside standard deviations derived from three independent random seeds. Comparisons across "Full" (multimodal), "EEG-only," and "fNIRS-only" configurations show that performance (MA) improves from 56.31% (EEG-only) to 66.93% (multimodal), and WG increases from 58.43% to 65.00%—demonstrating that multimodal inputs provide complementary information beneficial for the tasks.
Critical weaknesses
-
The paper lacks a critical ablation study that isolates the two self-supervised objectives; consequently, the causal claim that the model has "learned cross-modal coupling" is somewhat overstretching. Existing experiments primarily involve removing one of the input modalities rather than comparing different configurations while maintaining dual-modal inputs—such as using only intra-modal masked objectives, only cross-temporal prediction, a combination of both, or alternative fusion strategies (e.g., omitting joint self-attention or using simple concatenation). The fact that the full model outperforms single-modal variants merely demonstrates that the second modality is useful; it does not, by itself, prove that the proposed cross-modal temporal latent prediction is essential.
-
The performance advantage is limited in both magnitude and consistency across tasks. For the MA task, the model achieves a balanced accuracy of 66.93% compared to 66.37% for the best-performing baseline (LaBraM)—a margin of only about 0.56 percentage points, which is smaller than the reported standard deviations. On the WG task, the model’s 65.00% is slightly lower than the 65.09% achieved by TSMMF. For the NB task, the 38.27% result falls below the 40.84% achieved by an EEG-only baseline, as well as several other baselines. Finally, on the ENTER task, the model’s 51.85% is lower than LaBraM’s 57.97%.
Questions for authors
-
While strictly maintaining identical dual-modal inputs, encoder parameter counts, and training budgets, please evaluate the impact of removing each of the following components: masked latent prediction, cross-modal temporal prediction, joint attention, and the objective based on adjacent time windows. What are the resulting changes in downstream performance for each ablation?
-
Do the key competitive advantages (particularly on the MA and REFED-Valence/Arousal datasets) remain consistent when evaluated across multiple random seeds (using fixed subject splits) and through paired significance tests or confidence intervals? Furthermore, how should the non-monotonic performance variations observed across different pre-training budgets be interpreted?
-
How are the practical synchronization errors between EEG and fNIRS—arising from variations across subjects and hardware—as well as the hemodynamic lag of fNIRS relative to EEG, addressed? If the temporal axes of the two modalities are intentionally misaligned, does the proposed cross-modal temporal prediction mechanism remain effective?
5 comments
Response to Question 1
Under strictly the same bimodal input, encoder parameter count, and training budget, we removed these four components one at a time and fine-tuned each variant on the seven downstream tasks under the same protocol. The cross-modal temporal prediction you mention is our cross-modal temporal latent-space prediction task, in which each modality's temporal prediction branch predicts the latent representation of the adjacent window from the fused representation of the joint self-attention, and the joint attention you mention is our joint self-attention module. Accordingly, the four variants are denoted the no-temporal-objective, no-mask-objective, no-joint-attention, and no-adjacent-window variants, where the no-adjacent-window variant changes the objective from predicting the next window to predicting the current window.
The table below reports the balanced accuracy of the full model and the four variants on the seven tasks, in percentage.
| Task | Full model | No temporal | No mask | No joint attention | No adj. window |
|---|---|---|---|---|---|
| MA | 67.86 | 61.01 | 68.24 | 62.86 | 61.75 |
| MI | 55.56 | 58.80 | 57.96 | 59.54 | 57.78 |
| WG | 63.61 | 59.91 | 57.48 | 59.75 | 56.51 |
| NB | 40.23 | 39.71 | 40.64 | 40.74 | 41.05 |
| REFED valence | 49.15 | 48.71 | 48.77 | 47.44 | 49.05 |
| REFED arousal | 46.83 | 46.52 | 48.84 | 45.87 | 48.16 |
| ENTER | 53.15 | 52.58 | 52.81 | 53.79 | 52.36 |
We describe the contribution of each component in turn below.
First, after removing the cross-modal temporal prediction, MA drops from 67.86% to 61.01%, a decrease of 6.85 percentage points, the largest drop among the four ablations, and this drop has a consistent direction across all seeds with decreases ranging from 3.59 to 11.12 percentage points. This shows that the cross-modal temporal latent-space prediction task is the main source of the cross-modal coupling learned by the model. We should note honestly that this objective does not bring gains on every task: MI rises slightly by 3.24 percentage points after the temporal objective is removed.
Second, after removing the joint attention, MA drops to 62.86%, a decrease of 5.00 percentage points. Third, after removing the adjacent-window objective, MA drops to 61.75%, a decrease of 6.11 percentage points. Removing either component causes a clear drop, showing that the joint attention and the adjacent-window objective are both components of the cross-modal coupling, and removing either weakens the coupling.
Fourth, after removing the masked latent-space prediction, MA rises by 0.38 percentage points while WG drops by 6.13 percentage points. This shows that the intra-modal mask objective and the cross-modal temporal objective play their roles on different tasks: the mask objective mainly supports tasks such as WG that rely on within-modality structure, and it contributes little on tasks such as MA that rely mainly on cross-modal coupling.
In addition, on ENTER the changes of all four ablations are within 1 percentage point, consistent with the observation that this task relies mainly on EEG, which further indicates that the contribution of cross-modal coupling is concentrated on tasks where the two modalities genuinely bring complementary information.
Taken together, these four points show that this set of ablations isolates the cross-modal coupling itself rather than merely the gain from using two modalities. We position our contribution as the semantic alignment jointly established by the cross-modal temporal latent-space prediction objective and the joint self-attention, together with the channel-agnostic architecture and the 16-dataset corpus.
Response to Question 2
We address the two parts separately. For cross-seed robustness, we added a leave-one-subject-out evaluation that pairs CoLaP and LaBraM on exactly the same subjects, and we report paired t-tests and Wilcoxon signed-rank tests, with the results in the table below, in balanced accuracy, in percentage points.
| Task | CoLaP | LaBraM | Difference | Paired t-test (p) | Wilcoxon (p) |
|---|---|---|---|---|---|
| MI (29 subjects) | 55.86 | 50.98 | +4.88 | 2.749 (0.010) | 0.009 |
| REFED valence (32 subjects) | 47.97 | 43.37 | +4.60 | 4.678 (<0.001) | <0.001 |
| REFED arousal (32 subjects) | 45.89 | 39.20 | +6.69 | 4.606 (<0.001) | <0.001 |
| MA (29 subjects) | 64.75 | 67.11 | -2.36 | -1.174 (0.250) | 0.294 |
The key competitive advantage on MI and the two emotion tasks is consistently preserved across subjects. On MI, CoLaP is on average 4.88 percentage points higher, with a paired t-test p = 0.010 and a Wilcoxon test p = 0.009. On REFED valence, CoLaP is on average 4.60 percentage points higher, with both the paired t-test and the Wilcoxon test giving p-values below 0.001. On REFED arousal, CoLaP is on average 6.69 percentage points higher, and both tests are significant. On MA, the per-subject difference is not significant, with CoLaP on average 2.36 percentage points lower and a paired t-test p-value of 0.250. Therefore the key competitive advantage is consistently preserved across subjects on MI and on the REFED valence and arousal tasks.
Regarding the non-monotonic changes across pretraining budgets, the pattern in the paper is task-dependent rather than a single trend, and we report this pattern honestly in the paper, as shown in Table 10. Between 16, 40, and 80 epochs, MA rises monotonically from 59.62% through 62.39% to 66.93%, and WG rises from 53.10% through 56.64% to 65.00%. NB moves in the opposite direction, from 41.15% down to 38.27%; REFED valence plateaus after 16 epochs and then declines slightly, and REFED arousal first rises and then declines slightly. MI and ENTER stay roughly flat. For the tasks that keep rising, the discriminative signal benefits as the representation is progressively refined, whereas for the tasks that plateau or decline, the representation is already close to its ceiling at 16 or 40 epochs, and for them the later representation is slightly less useful than the earlier, more general representation.
Response to Reviewer Wngt
Thank you for the thorough and encouraging review. We greatly appreciate your recognition of the robust validation framework of this work, of the effective suppression of data leakage through cross-dataset and cross-subject splits, and of our benchmarking against both EEG-only models and modality-matched EEG-fNIRS baselines. We also recognize two shortcomings: the causal claim of learning cross-modal coupling is not isolated, and the performance advantage is limited in both magnitude and cross-task consistency, which we address one by one below.
Response to Weakness 1
Thank you for the suggestion. In the original manuscript's experimental design, comparing the full model with single-modality variants by removing one input modality only establishes that the second modality is useful, and it is insufficient to show that our proposed cross-modal temporal latent-space prediction is indispensable. To remedy this, we added a set of component-wise ablations that remove the masked latent-space prediction, the cross-modal temporal prediction, the joint attention, and the adjacent-window objective one at a time while keeping the bimodal input, the encoder parameter count, and the training budget unchanged, with the detailed results in the first response below. These experiments consistently show that removing the cross-modal temporal objective or the joint attention causes a clear drop in downstream performance, most pronounced on MA, indicating that the model has learned cross-modal coupling rather than merely using two modalities. Accordingly, we position our contribution as the semantic alignment jointly established by the cross-modal temporal latent-space prediction objective and the joint self-attention.
Response to Weakness 2
We explain the limited magnitude and cross-task consistency of the gains from two aspects. First, collecting synchronized EEG-fNIRS data is far more difficult than collecting a single modality: the pretraining corpus of this work consists of 16 public synchronized datasets, totaling 96,305 ten-second windows and about 268 hours. For comparison, large-scale EEG-only foundation models can rely on much larger corpora: LaBraM is pretrained on about 2,500 hours of EEG and CBraMod on about 9,000 hours of the Temple University EEG corpus, roughly ten and thirty times ours, respectively. Given a corpus more than ten times larger and the requirement of synchronized acquisition of two modalities, CoLaP is on par with these models on MA and achieves better and significant per-subject results on MI and on the REFED valence and arousal tasks. Second, our model is a foundation model that transfers to seven downstream tasks after a single pretraining, with no need to design a task-specific architecture from scratch for each task. Taking TSMMF as an example, such task-specific models are designed for a specific task and dataset and trained on that task, so their representations are hard to transfer to other tasks, whereas our channel-agnostic architecture and statistical pooling allow the same representation to transfer across heterogeneous datasets. The limited performance advantage on some tasks is therefore mainly constrained by the scale of synchronized data rather than by a lack of generality in the method. The contribution of this work is, to the best of our knowledge, the first cross-dataset synchronized EEG-fNIRS foundation model and its channel-agnostic architecture, a contribution that does not depend on the accuracy ranking on any single task; we have revised the abstract and introduction to position the contribution accordingly, away from the highest accuracy.