acceptodds
Under review as a conference paper at ICLR 2027

CoEEG: Benchmarking Collaborative Learning in EEG Decoding

Abstract

Electroencephalography (EEG) decoding is constrained by between-subject variability, scarce per-user labeled data, and privacy restrictions that scatter records across clinics and devices. In this work, we build CoEEG to jointly compare model families and collaborative learning protocols on three EEG paradigms, Motor Imagery (MI), SSVEP, and P300. Our benchmark covers three model families, traditional machine learning with paradigm-specific priors, deep models trained from scratch, and EEG foundation models. We compare ten widely used models under four protocols that span non-collaborative and collaborative training. We find that collaboration helps EEG decoding, but not uniformly. On SSVEP, nine of ten models benefit, consistent with a decision boundary that is largely shared across subjects. On P300, every model benefits only when each client keeps its classification head private. On MI, the gains concentrate in models that perform poorly on their own. Across the six federated cells the from-scratch family has the higher family mean, though a foundation model posts the single best score in two of them. The sharing boundary follows a measured factor, how far representations align, and an inferred one, whether the decision boundary is shared. A private head splits motor imagery and never lifts the weakest clients, who lose 20.7 percentage points (pp) on SSVEP against 9.3 pp for the strongest. Once parameter count, memory, and communication are included, lightweight models are the better performance–cost choice in most settings, and the largest foundation model transmits up to 378.73 GiB. Because the value of collaborative learning depends jointly on the (1) paradigm, (2) model, and (3) sharing boundary, CoEEG supports holistic evaluation and model selection for different EEG decoding tasks.

Primary area:
datasets and benchmarks
Keywords:
EEG decoding, brain-computer interfaces, federated learning, personalized federated learning, EEG foundation models, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8ySpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces CoEEG, a benchmark evaluating EEG decoding across three paradigms (Motor Imagery, SSVEP, P300), three model families (traditional ML, DL from scratch, and EEG foundation models), and four training protocols (Local, Centralized, FedAvg, FedPer) in a subject-as-client federated setting. The study highlights that collaboration benefits depend jointly on the task paradigm, model type, and sharing boundary, while lightweight models generally offer the best performance-cost trade-off.

Critical strengths
  1. Comprehensive Multi-Dimensional Evaluation: Integrates paradigms, model architectures, and federation protocols into a unified framework.
  2. Valuable Cost-Performance Insight: Demonstrates that lightweight models often outperform resource-heavy foundation models in federated settings, providing practical deployment guidance.
  3. Rigorous Experimental Design: Utilizes temporal train-test splits within clients and fixed-final-round evaluations to prevent data leakage.
Critical weaknesses
  1. Forcing a uniform large learning rate (10−310^{-3}) and full-parameter training without early stopping disadvantages pre-trained large models, making them prone to severe overfitting on small-scale clients.
  2. Allowing models to use their own default resampling rates, filter bands, and alignment strategies mixes preprocessing advantages with actual model capability.
  3. The benchmark relies on small, static settings (9–10 clients, single-session, 100% full participation), failing to capture real-world challenges like user scale, dropouts, and cross-device heterogeneity.
  4. Lacks comparisons against mainstream client-drift mitigation algorithms (e.g., FedProx, SCAFFOLD) and parameter-efficient federated fine-tuning strategies.
Questions for authors
  1. How would the foundation models perform if tuned with lower learning rates or parameter-efficient methods (like LoRA) instead of uniform full-parameter training?
  2. Can you isolate how much performance variation stems from differing model-level preprocessing steps (like Euclidean alignment or filter bands) rather than the model architectures themselves?
Reviewer_N6kyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper attempts at benchmarking collaborative learning in the federated learning sense for EEG decoding. The authors employ several model families and collaboration protocols grid to benchmark this scenario. As a result, the authors find four takeaways, providing insights to researchers in the relevant fields.

Critical strengths

The deliverables, represented by the four messages, seem quite insightful and helpful. Many of the results, such that the from-scratch family does better than the pretrained ones, follow quite closely with existing FM works.

Critical weaknesses

I'm not sure what is the novelty of this paper. Although benchmarking is well-defined, but whether the community requires this benchmark is uncertain. Also, it's unlikely that prospective researchers would require this benchmark to evaluate their system. Overall, the scheme is pretty narrow, on federated learning + EEG decoding. The reason underlying this doubt is perhaps due to missing background on federated learning across EEG domains. In the intro, the authors suggest three challenges: subject-level / acquisition-level variabilities, and privacy issues. It is not so obvious how the proposed benchmark help address those challenges.

The data itself is derived from existing databases. Hence, this paper does not seem to target aspects that prior models were not able to.

Questions for authors
  • How do you envision that fellow researchers would leverage this benchmark for evaluating their models, in which scenarios, and why?
  • How does collaborative have implications for subject / acquisition level variabilities?
Reviewer_scaVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work explores federated learning to address a key challenge in EEG representation learning: variability in recordings across subjects, devices, and sites. Using the subject as a client is an interesting way to frame the problem. Primarily, the paper presents CoEEG, which is a benchmark to compare model families and collaborative learning protocols on three EEG paradigms. The authors conducted experiments to examine these effects under four well-defined research questions, which is commendable.

Critical strengths
  • The formulation of the subject as a client for EEG collaborative learning is commendable.
  • Also, the evaluation is well structured around four important research questions.
Critical weaknesses
  • I am not convinced by the experimental setup, particularly the use of temporally separated training and test splits from the same subjects. This evaluation does not demonstrate whether the model generalizes to unseen patients or across sites. Consequently, the practical applicability of the proposed remains unclear. For instance, how would the model be deployed for a new patient without prior training data? Additionally, the benchmark only covers 4 EEG tasks, making it difficult to determine whether the reported observations generalize across different applications. A broader range of EEG tasks and evaluation of cross-patient and cross-site generalization may improve the rigor and quality of the benchmark.

  • Additionally, with regard to the observations and findings, the manuscript does not provide the reason why they occur. Additionally, some of the findings are vague. For example, the deep learning models trained from scratch perform better than large foundation models in the datasets used for experiments. This could have emerged due to limited data used in the study (240 trials for training and 48 for testing). The performance of foundation models could have been better with more subjects.

Questions for authors
  1. How does the performance transfer to unseen patients during inference? What will be the practical use case of the proposed EEG-as-client approach?

  2. It is an interesting finding that personalization effects vary between tasks. What is the reason behind this observation? This is important, as it could give insights into which EEG events or patterns have less variability between subjects.

  3. How does the performance vary with the number of subjects/clients used in collaborative learning?

  4. How are the best model checkpoints selected without a validation set?

  5. Seems like Figure 4 got cropped

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.