ShortEEG-Bench: A Comprehensive Benchmark for EEG Foundation Models under Short-Window Protocols

EEG Foundation Models (EEG-FMs) have demonstrated promising neural decoding capabilities across diverse tasks and subjects. However, current evaluations mainly rely on long temporal windows (4–30s), which may obscure fine-grained temporal dynamics and limit the applicability of these models to low-latency brain–computer interfaces. Therefore, we introduce ShortEEG-Bench, a unified benchmark for systematically evaluating EEG-FMs under short-window protocols (1/8-1/2 of the existing long windows). We evaluate 10 models on 11 datasets, covering cross-model comparisons, temporal analysis and multi-task training. Our results reveal several key findings. First, EEG-FMs maintain competitive performance with shortened inputs, and task-specific models do not consistently dominate, where 36% short-window majority voting outperforming long-window predictions. Second, short-window analysis reveals substantial temporal fluctuations hidden by long-window aggregation, including higher accuracy in early post-cue intervals followed by a gradual decline in specific tasks. Third, in-domain multi-task learning yields positive transfer for 57% of the evaluated EEG-FMs, whereas cross-domain training more often induces negative transfer. Overall, ShortEEG-Bench reveals that short-window evaluation preserves competitive decoding performance while exposing temporal dynamics and transfer behaviors masked by long-window aggregation, enabling more temporally resolved and diagnostically meaningful EEG-FM evaluation.