- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces CoEEG, a benchmark evaluating EEG decoding across three paradigms (Motor Imagery, SSVEP, P300), three model families (traditional ML, DL from scratch, and EEG foundation models), and four training protocols (Local, Centralized, FedAvg, FedPer) in a subject-as-client federated setting. The study highlights that collaboration benefits depend jointly on the task paradigm, model type, and sharing boundary, while lightweight models generally offer the best performance-cost trade-off.
Critical strengths
- Comprehensive Multi-Dimensional Evaluation: Integrates paradigms, model architectures, and federation protocols into a unified framework.
- Valuable Cost-Performance Insight: Demonstrates that lightweight models often outperform resource-heavy foundation models in federated settings, providing practical deployment guidance.
- Rigorous Experimental Design: Utilizes temporal train-test splits within clients and fixed-final-round evaluations to prevent data leakage.
Critical weaknesses
- Forcing a uniform large learning rate () and full-parameter training without early stopping disadvantages pre-trained large models, making them prone to severe overfitting on small-scale clients.
- Allowing models to use their own default resampling rates, filter bands, and alignment strategies mixes preprocessing advantages with actual model capability.
- The benchmark relies on small, static settings (9–10 clients, single-session, 100% full participation), failing to capture real-world challenges like user scale, dropouts, and cross-device heterogeneity.
- Lacks comparisons against mainstream client-drift mitigation algorithms (e.g., FedProx, SCAFFOLD) and parameter-efficient federated fine-tuning strategies.
Questions for authors
- How would the foundation models perform if tuned with lower learning rates or parameter-efficient methods (like LoRA) instead of uniform full-parameter training?
- Can you isolate how much performance variation stems from differing model-level preprocessing steps (like Euclidean alignment or filter bands) rather than the model architectures themselves?