- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoCoT-EEG, a contrastive-pretrained EEG foundation model that combines multiscale temporal convolutions with a Transformer encoder. Motivated by the limitations of conventional patch-first tokenization and masked reconstruction, the authors adopt a convolution-first strategy to better capture continuous temporal dependencies in EEG signals.
Critical strengths
-
The motivation is clear. This paper raises a relevant question about the widely adopted patch-first tokenization and masked reconstruction paradigm in current EEG foundation models. The proposed convolution-first approach, which extracts features before patchification, is well motivated by the continuous and multiscale nature of EEG signals.
-
The paper evaluates the proposed method on 11 EEG benchmarks covering diverse downstream tasks. The results are competitive with recent EEG foundation models.
Critical weaknesses
-
The overall technical novelty is relatively limited, as both multiscale convolutions and contrastive pretraining have been widely explored in prior work. The main architectural contribution is essentially reversing the order of convolution and patchification. While this is an intuitive design choice, the results in Tables 2 and 8 show that its benefits are generally modest or inconsistent, despite notable improvements on a few datasets. In my view, the empirical gains are not sufficiently compelling to justify such a relatively incremental architectural modification.
-
Since the main contribution lies in changing the order of convolution and patchification, I would be interested to see whether this design choice can also improve existing EEG foundation models, rather than only the proposed architecture. For example, applying the convolution-first strategy to established models such as CBraMod or LaBraM would help assess whether the observed benefits are generalizable. Demonstrating consistent improvements across different architectures would substantially strengthen the contribution of this work.
Questions for authors
See in Weakness . 2
3 comments
I thank the authors for their response and additional experiments. I understand the computational constraints during the rebuttal period. However, the current experiments on CBraMod and CSBrain do not provide sufficient evidence that changing the tokenization order leads to consistent improvements.
Given these computational constraints, a more feasible and rigorous alternative would be to pretrain CBraMod and CSBrain on smaller datasets, keeping their original architectures and pretraining objectives unchanged while modifying only the order of convolution and patchification. Such controlled experiments would more clearly isolate the contribution of the proposed method. However, the authors' current experiments do not evaluate the proposed modification under pretraining, which is particularly important for EEG foundation models, especially given the substantial performance gap between pretrained and from-scratch CSBrain. More concerningly, even in the from-scratch setting, the effect of tokenization order is not properly isolated: the authors retain the original tokenizers for the Patch-first baselines but replace them with the proposed CoCoT tokenizer for the Conv-first variants. This introduces additional architectural differences beyond tokenization order, making the comparison insufficiently controlled and the observed improvements difficult to attribute to the claimed contribution.
Since my primary concerns regarding methodological novelty and generalizability remain unresolved, I do not find the current experimental evidence sufficiently rigorous or convincing to revise my assessment. I therefore maintain my initial rating.
We appreciate your continued engagement and the opportunity to clarify our overall contribution.
To clarify: our contribution on the architectural side is a multiscale convolutional tokenizer that extracts continuous features before aggregating them into tokens. Within this design, the manuscript separately examines tokenization order using matched truncated kernel banks, and the additional benefit of extending the kernel bank (Section 4.3.1, Table 2) which is only enabled by the reversed tokenization order.
As a result, our contribution on this matter is the tokenizer module itself, and not merely the reversed order of convolutional embedding. The experiment above demonstrates that simply importing our tokenizer to existing architectures can result in substantial improvements when trained from scratch, including +2.6% on FACED, +4% on BCI-IV and +8% on SEED-VIG for CBraMod compared to the published result of the pretrained CBraMod (see Table 1 in main paper).
Thus, the tokenizer itself—which includes the multiscale convolutions enabled by switching the order—is the contribution, which we show above can be flexibly introduced to other architectures.
Furthermore, we did not intend to claim that conv-before-patch is better in every single architecture, nor is it feasible (or valuable, for that matter) to run such an experiment. Nevertheless, we acknowledge that some of our wording maybe have suggested a broader claim about reversing the operations of arbitrary existing tokenizers, and we will clarify this scope.
But to further test your specific hypothesis, we extend the Native versus Conv-first comparison provided in our initial response and have added an intermediate Patch-first configuration for both CBraMod and CSBrain. All configurations retain each model’s Transformer backbone and spectral and positional encoding paths and use the same supervised training protocol. The expanded table therefore compares the native tokenizer, Patch-first tokenizer, and Conv-first tokenizer. The Conv-first tokenizer still achieves the highest mean performance on all five evaluated tasks for both backbones.
| Backbone | Task | Native | Patch-first tokenizer | Conv-first tokenizer |
|---|---|---|---|---|
| CBraMod | FACED | 52.63 ± 2.54 | 52.91 ± 1.14 | 57.69 ± 0.75 |
| PHYSIO | 59.52 ± 2.20 | 57.92 ± 1.42 | 60.07 ± 0.56 | |
| BCI-IV | 46.03 ± 2.22 | 51.32 ± 7.15 | 55.21 ± 10.62 | |
| SHU-MI | 58.73 ± 2.57 | 58.33 ± 4.62 | 60.65 ± 1.81 | |
| SEED-VIG | 54.57 ± 3.20 | 58.45 ± 1.66 | 63.88 ± 1.41 | |
| CSBrain | FACED | 44.81 ± 1.29 | 48.20 ± 1.28 | 52.26 ± 0.85 |
| PHYSIO | 58.00 ± 1.96 | 56.69 ± 1.50 | 58.49 ± 1.31 | |
| BCI-IV | 48.55 ± 1.32 | 50.30 ± 4.03 | 56.55 ± 6.31 | |
| SHU-MI | 59.08 ± 1.62 | 60.69 ± 1.71 | 61.28 ± 2.83 | |
| SEED-VIG | 43.76 ± 8.52 | 52.11 ± 6.46 | 54.47 ± 3.37 |
Furthermore, within the constrained rebuttal compute budget, we prioritized an additional small-scale pretrained comparison for CBraMod. We pretrained the Native, Patch-first and Conv-first variants for 25,000 steps under a common MoCo protocol, using the same corpus, augmentation recipe, and optimization settings, followed by finetuning on the same five tasks with four seeds. That setting preserves the observations made in the from-scratch setup: the full Conv-first tokenizer achieves higher mean performance than the truncated Patch-first variant on all five tasks after pretraining.
| Backbone | Task | Native | Patch-first tokenizer | Conv-first tokenizer |
|---|---|---|---|---|
| CBraMod | FACED | 54.83 ± 0.87 | 56.21 ± 1.30 | 58.11 ± 0.58 |
| PHYSIO | 62.49 ± 0.64 | 62.27 ± 0.23 | 62.90 ± 0.73 | |
| BCI-IV | 46.94 ± 6.81 | 51.87 ± 0.80 | 55.13 ± 6.43 | |
| SHU-MI | 60.39 ± 1.51 | 62.11 ± 0.76 | 62.31 ± 1.23 | |
| SEED-VIG | 62.60 ± 1.58 | 62.85 ± 2.08 | 63.86 ± 1.54 |
These results extend the observed advantage of the full tokenizer over its truncated Patch-first alternative and the Native tokenizer to pretrained CBraMod. Together with the matched architectural ablations within CoCoT, these experiments support the effectiveness of the proposed tokenizer and its transfer to additional encoder backbones.
Finally, we again emphasize that our main contribution is much more than simply the order of operation within the tokenizer. We hope the reviewer will also consider these architectural results in the context of the paper’s central contribution: a systematic, architecture-matched investigation of contrastive versus reconstruction-based pretraining for EEG foundation models, supported by extensive downstream evaluation and analyses of the learned representations. If you still have concerns regarding methodological novelty and generalizability on those broader contributions, we are happy to clarify further.
Thank you for your concise, constructive feedback and a clear request for how to strengthen the contribution of our work. We would like to clarify that our main contribution does not lie in changing the order of convolution and patchification (Weakness 2), but rather the systematic investigation of pretraining objective while controlling for model architecture in a new proposed model. Nevertheless, we test the effect of the conv-first tokenizer on CBraMod and CSBrain below and find significant performance boosts in the from-scratch (no pretraining) setting, thus providing evidence for a potentially generalizable design principle and further strengthening our architectural contribution.
1. Clarifying contributions beyond novel architecture
First, while the proposed architecture is intuitive and “well motivated”, our architectural contribution is not only to introduce the model, but to systematically understand its impact through ablation and interpretability studies (e.g., Fig. 2), in addition to showing performance gains across extensive benchmark tasks. More broadly, the main contribution of the paper is our systematic comparison between reconstruction and contrastive objectives under architecture-matched settings.
As the reviewer points out in Table 2, Conv-1st indeed introduces small (but consistent) performance benefits over Patch-1st (on par or better on 10/11 tasks). But the more striking improvement comes from contrastive pretraining (MoCo) compared to MAE, which can be seen in the same Table.
As two recent review papers found (Fig. 1A), while contrastive pretraining has been explored in prior works such as BENDR and BIOT, reconstruction pretraining represents an overwhelming majority of the approaches in the last 5 years (~90%). Moreover, as a consequence of adopting MAE pretraining, almost all papers directly inherit a patch-first modeling approach, which makes direct comparison of design contributions more challenging. To study them separately, we make a further contribution here by proposing essentially a second novel and competitive architecture in mask-aware, sparse-conv CoCoT as a control, which enabled the systematic study of tokenization order and pretraining objective.
Thus, we want to highlight our investigation of both model architecture and pretraining objective, especially in systematically disentangling these two factors, as a key contribution, which is to our best knowledge the first paper to do so at scale. We therefore very much welcome any feedback from the reviewer regarding the novelty, extensiveness, and depth of these contributions relative to the current literature.
2. Conv-first modification for existing model architectures
Nevertheless, we agree that demonstrating consistent improvements across different architectures would substantially strengthen the contribution of this work. Thus, we test the effect of our Conv-1st tokenizer on modified variants of CBraMod and CSBrain in the from-scratch setting, removing the confound of reproduce heterogeneous pretraining recipes and datasets while bypassing compute resource limitations during the rebuttal period.
Compared to the native configuration of CBraMod and CSBrain, which both employ within-patch convolutions after tokenization, we reverse this order by swapping in the same convolution-first tokenizer of CoCoT (which also enables cross-patch, long convolutional kernels).
In summary, we find that the conv-first tokenizer significantly boosts both CBraMod and CSBrain on several tasks, including +5-7% on FACED and around +10% on SEED-VIG (Table 1 below). Thus, we conclude that the advantages of the conv-first recipe may be generalizable beyond our CoCoT architecture, and we thank the reviewer for this suggestion.
Additional details
Table 1: from-scratch performance of modified Conv-1st baselines Results are balanced accuracy except for SEED-VIG, where we report Pearson correlation ×100. Values are mean ± 95% Student-t CI (4 seeds).