acceptodds
Under review as a conference paper at ICLR 2027

ShortEEG-Bench: A Comprehensive Benchmark for EEG Foundation Models under Short-Window Protocols

Abstract

EEG Foundation Models (EEG-FMs) have demonstrated promising neural decoding capabilities across diverse tasks and subjects. However, current evaluations mainly rely on long temporal windows (4–30s), which may obscure fine-grained temporal dynamics and limit the applicability of these models to low-latency brain–computer interfaces. Therefore, we introduce ShortEEG-Bench, a unified benchmark for systematically evaluating EEG-FMs under short-window protocols (1/8-1/2 of the existing long windows). We evaluate 10 models on 11 datasets, covering cross-model comparisons, temporal analysis and multi-task training. Our results reveal several key findings. First, EEG-FMs maintain competitive performance with shortened inputs, and task-specific models do not consistently dominate, where 36% short-window majority voting outperforming long-window predictions. Second, short-window analysis reveals substantial temporal fluctuations hidden by long-window aggregation, including higher accuracy in early post-cue intervals followed by a gradual decline in specific tasks. Third, in-domain multi-task learning yields positive transfer for 57% of the evaluated EEG-FMs, whereas cross-domain training more often induces negative transfer. Overall, ShortEEG-Bench reveals that short-window evaluation preserves competitive decoding performance while exposing temporal dynamics and transfer behaviors masked by long-window aggregation, enabling more temporally resolved and diagnostically meaningful EEG-FM evaluation.

Primary area:
datasets and benchmarks
Keywords:
EEG Foundation Models, Short-Window Evaluation, Benchmarking, Multi-Task Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eFdVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

ShortEEG-Bench evaluates EEG foundation models under short input windows (0.5–15 s, i.e., 1/8–1/2 of conventional 4–30 s windows), motivated by low-latency BCI and by the fact that long-window aggregation hides temporal dynamics. It benchmarks 10 models (7 EEG-FMs: LaBraM, EEGPT, BENDR, BIOT, CBraMod, REVE, CSBrain; 3 conventional DNNs: EEGNet, EEG Conformer, ADFCNN) on 11 datasets across 5 task families, under full fine-tuning / LoRA / linear probing, single-dataset vs in-domain vs cross-domain multi-dataset training, and single-window vs majority-vote inference. Main findings: short-window decoding remains competitive (36% of voting configurations match or exceed long-window predictions); majority voting recovers evidence once single windows exceed chance; early post-cue windows carry most motor-imagery information and 0.5-s windows expose sharper temporal fluctuations; in-domain multi-task training yields positive transfer for 57% of EEG-FMs while cross-domain training often causes negative transfer; and short-window EEG-FMs can rescue subjects with weak long-window baselines.

Critical strengths
  • A useful community resource: the window-length axis is genuinely under-examined in EEG-FM evaluation, and the benchmark's breadth (11 datasets × 5 task families × 3 adaptation strategies × 3 training regimes) under one protocol with promised code, checkpoints, and split definitions would save the field substantial redundant effort.
  • The evaluation design separates two scientifically distinct questions — whether a local segment is decodable at all versus how much evidence is recoverable by temporal aggregation — and adds temporal-position and subject-level analyses that expose variation hidden by aggregate scores, which is where the benchmark delivers its most interesting observations.
Critical weaknesses
  • Short windows inherit their parent segment's label, so window-level performance mixes signal decodability with label validity of the segment (e.g., 2-s windows of a 30-s sleep stage; sub-second windows of a 4-s MI trial). The paper does not separate these (e.g., via event-aligned or per-window oracle analyses), and the headline findings are reported as counts and means ("36%", "57%") without confidence intervals, seed variance, or significance tests.
  • The incremental insight over recent EEG-FM benchmarks rests entirely on the window-length axis, and several findings are expected a priori (fine-tuning > LoRA > linear probing; early post-cue MI dominance; voting helps once above chance). The long-window comparison protocol — whether long-window baselines receive the same training budget and tuning — is not fully specified in the main text, which matters for the "short windows compete with long windows" claim.
Questions for authors
  • How much of the short-window degradation reflects label mismatch within parent segments rather than lost signal? A simple control (e.g., restricting to segments with stable labels, or aligning windows to events) would disentangle the two.
  • Are long-window baselines trained with identical compute, epochs, and hyperparameter tuning, and can you report seed-level variance or CIs for the main findings?
  • For the "57% of EEG-FMs show positive in-domain transfer" claim, what is the denominator of counted cases, and is the pattern reliable under resampling?
6 comments
AuthorsOpenReview ↗

1. Label Validity vs. Decodability

We are grateful to the reviewer for the insightful and constructive feedback. All comments have been carefully considered and addressed individually. Corresponding corrections and additional information, including revised and newly added tables, have been incorporated into the revised Supplementary.

Weakness / Question. Short windows inherit the parent segment’s label, so window-level performance mixes signal decodability with label validity (e.g., 2-s windows of a 30-s sleep stage; sub-second windows of a 4-s MI trial). How much short-window degradation reflects label mismatch rather than lost signal, and could a stable-label or event-aligned control disentangle the two?

Response. We agree that short-window degradation can reflect both reduced temporal information and local label–state mismatch. We organize the answer as: (1) limitations of existing datasets, (2) analysis under our protocol, and (3) future controls.

1. Limitations of existing datasets

Public EEG datasets provide segment- or trial-level labels from clinical scoring or cognitive paradigms—e.g., one label per 30 s sleep epoch (HMC, Sleep-EDF), ~200 s emotion trial (SEED, SEED-V), ~60 s workload trial, or recording-level abnormality (TUAB). Table 1 of the main text shows that prior deep-learning and EEG-FM work already uses short windows (few seconds to sub-second) far below this granularity, so short windows inherit segment-level labels by construction. ShortEEG-Bench does the same: each short window inherits its parent recording/trial/epoch label.

This gap reflects neuroscience and measurement constraints, not only engineering choice:

  • Slow dynamics. Sleep, sustained emotion, workload, and clinical abnormalities are macroscopic states integrated over seconds–minutes; a short window often captures local transients or background rather than a stationary instantaneous class.
  • Scoring time scales. AASM 30 s PSG, clinical review, and post-trial reports need a long integration window; dense few-second annotation is costly and hard to keep inter-rater reliable.

Thus two confounders coexist under inherited labels: information loss (less evidence than the parent span) and local label–state mismatch (transition/baseline/background windows that still carry the parent label).

2. Analysis under the current protocol

We retain EEG-FM-Bench task definitions and label boundaries (Xiong et al., 2026) for comparability. The core research question of this work is: while keeping the original clinical/cognitive task label definitions unchanged, how does shortening the EEG input duration affect model decoding ability, and whether temporal aggregation can compensate for short-window information loss. Every short window stays inside one native annotated interval and inherits only that label—no cross-boundary mixing—so long- and short-window models share the same prediction target. This controls one error source but does not create instantaneous physiological ground truth.

Inherited labels act as weak supervision. Comparing single-window decoding with hard majority voting quantifies how local heterogeneity affects stability and whether voting recovers the segment-level target. Voting gains show better segment-level prediction; they do no tell whether each failed window erred from lost signal or local mismatch. The two mechanisms remain non-identifiable without fine-grained annotations—we state this explicitly.

Event alignment is already used for MI and TUEV. For other tasks, an oracle/stable-label control needs new annotations. We therefore interpret the benchmark as segment-level task prediction from short-window input, not instantaneous-state recognition.

3. Future directions

To better disentangle the two mechanisms we plan: (i) stable-epoch filtering (e.g., drop sleep epochs next to stage changes) as a proxy sensitivity check—still using the 30-s label; (ii) fine-grained short-window or event-level oracles where feasible; (iii) event-aligned designs beyond MI/TUEV for currently coarse-label tasks. These are not in the current results.

Revision. Expanded dataset profiles and protocols for annotation span, event alignment, boundary-constrained extraction, and aggregation (Appendix A.1–A.2; Fig. 5 for within-interval positions). The discussion states that information loss and local label mismatch cannot be uniquely separated with present data, and that stable-epoch/oracle analyses remain future controls.

AuthorsOpenReview ↗

2. Incremental Insight Beyond Recent EEG-FM Benchmarks

Weakness. The incremental insight over recent EEG-FM benchmarks rests entirely on the window-length axis, and several findings are expected a priori (fine-tuning > LoRA > linear probing; early post-cue MI dominance; voting helps once above chance).

Response. We agree that qualitative expectations alone are not sufficient novelty claims. Our incremental contribution is not “short windows only,” but a more comprehensive evaluation than prior EEG-FM benchmarks: under one EEG-FM-Bench-aligned protocol we jointly assess (i) short-window decoding, (ii) majority-voting performance, (iii) within-trial window order / temporal position, and (iv) different multi-task training regimes (ST, MT-INN, MT-INT(−c), MT-INT(+c)), across seven FMs, three adaptations, and eleven datasets. Continuity with EEG-FM-Bench keeps splits and preprocessing comparable; the added axes expose effects masked by long-window aggregation. Key conclusions from the main text and Supplement follow.

Short-window decoding remains viable, but degradation is task- and model-dependent (RQ1). EEG-FMs often outperform conventional DNNs under short inputs and obtain the best result on most datasets (Fig. 1c). Absolute performance stays meaningful (e.g., REVE 41.1% bAcc on BCIC-IV-2a at 1 s; CSBrain 75.4% on TUSL at 2 s), yet shortening hurts all datasets with widely varying magnitude. Full fine-tuning incurs the smallest average drop, then LoRA, then linear probing; backbone robustness also differs (BIOT/EEGPT smaller average drops; absolute SW accuracy need not track drop size).

Voting recovers evidence under conditions, not unconditionally (RQ2). Majority voting generally preserves or improves SW accuracy across Full/LoRA/Linear (Fig. 3a–c), often closing the gap to—or surpassing—published long-window references (Fig. 1d; ~36% of matched full-FT Vote>LW configs). Gains are dataset-dependent (larger on TUSL 2 s and HMC 5 s; consistent on MI). Aggregating four or six windows yields ~+2.6 pp average bAcc with fewer ineffective votes than two-window settings. Critically, voting helps mainly when the base SW model is above chance; near-chance bases (e.g., BENDR on SEED-V; collapsed 0.5-s REVE) gain little or nothing—voting is a variance-reduction amplifier, not a substitute for missing temporal context (Supplement SW/Vote/LW core findings). On sleep, extending SW from 5 s to 15 s helps, and 15-s voting can rival native long windows (e.g., ~71.0% vs. 71.8% on HMC).

Window order matters: early post-cue intervals dominate on MI (RQ3 / Fig. 4–5). On BCIC-IV-2a and PhysioNet-MI, early post-cue windows generally achieve the highest decoding accuracy; 0.5-s windows localize better but fluctuate more than 1-s windows. The temporal pattern holds across Full/LoRA/Linear, indicating signal structure rather than a single adaptation artifact—variation that long-window averages hide.

Multi-task training helps selectively and can hurt (RQ4). In-domain MT-INN is generally more favorable and consolidates homogeneous representations; cross-domain MT-INT(−c) more often induces negative transfer on unaligned backbones. Adding clinical data (MT-INT(+c)) is asymmetric: it can boost clinical slowing (e.g., EEGPT on TUSL up to ~79% SW / 80.7% Vote under full FT) while penalizing non-clinical cognitive/affective tasks. Effects depend jointly on dataset composition, backbone, and adaptation; majority voting cannot reliably offset adverse joint-training effects. Abstract-level summary: in-domain multi-task yields positive mean transfer for 57% (4/7) of FMs under the stated counting rule, whereas cross-domain training more often hurts.

We therefore treat “FT>LoRA>linear,” “early MI windows,” and “voting helps” as sanity-check hypotheses, and present ShortEEG-Bench as a controlled map of when short windows, voting, temporal position, and multi-task regimes help or fail—not as rediscovering those qualitative expectations.

AuthorsOpenReview ↗

3. Long-Window Training Budgets

Weakness / Question. Are long-window baselines trained with identical compute, epochs, and hyperparameter tuning? The protocol is not fully specified for the “short windows compete with long windows” claim.

Response. The primary goal of this study is to systematically evaluate how foundation-model representations degrade under short time windows, and how multi-window majority voting and multi-dataset joint training improve decisions under limited temporal information. To avoid incomparable long-window baselines across papers with inconsistent training protocols, we adopt the published single-task, full fine-tuning EEG-FM-Bench results (Xiong et al., 2026, ICML) as the long-window FM reference. That benchmark standardizes long-window training within a unified framework; its compute budget, epochs, and hyperparameters follow its publicly documented settings and were not retrained with our short-window optimizers. We will release ShortEEG-Bench models, configurations, and related artifacts for broader EEG research.

Short-window runs reuse the same EEG-FM-Bench construction/preprocessing (montage, filtering, resampling, unit scaling) and subject partitions before SW segmentation. Differing SW hyperparameters are documented beside LW entries in Appendix A.3 and Supplementary Tables 3–4 (source, model, duration, adaptation, budget). “Short windows compete with long windows” is thus a comparison to a standardized published reference, not identical-compute equivalence. Voting is more tightly controlled: same SW checkpoint and test intervals before vs. after aggregation.

Revision. Appendix A.3; SW/LW captions label published LW values as external references.

Reviewer_k3egOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ShortEEG-Bench, a benchmark for evaluating EEG foundation models under shortened temporal input windows. It evaluates seven EEG-FMs and three conventional deep learning models across 11 datasets spanning five task families. The benchmark examines performance degradation under reduced temporal context, majority-voting aggregation, temporal-position sensitivity, multi-task learning, and subject-level variability. The results characterize task and model dependent performance changes, showing that short-window decoding can remain competitive, while voting and multi-task training offer varying benefits across settings.

Critical strengths

S1. Extensive Experimental Coverage

The paper presents a substantial experimental evaluation across diverse EEG tasks, foundation models, and training configurations. The breadth of experiments provides useful reference results for benchmarking EEG-FMs.

S2. Targeted Short-Window Evaluation Framework

This benchmark systematically examines EEG decoding under different input durations, incorporating majority voting, temporal-position analysis, and subject-level comparisons. These complementary perspectives extend conventional performance benchmarking and help characterize model behavior under reduced temporal context.

Critical weaknesses

W1. Limited Depth of Analysis

Although the research question is meaningful, the analysis remains largely descriptive, focusing on performance variations across models, tasks, and window configurations. More detailed analyses of window-length sensitivity, temporal-position effects, and model-specific temporal dependencies would provide stronger, more generalizable insights.

W2. Insufficient Specification of Model-Specific Input Adaptation

The paper does not adequately describe how patching, padding, resampling, and channel mapping are handled for different models and window lengths. These choices may substantially affect short-window performance, making it difficult to distinguish limitations of pretrained representations from input compatibility and implementation effects.

W3. Limitations of Inherited Labels for Fine-Grained EEG Decoding

All short windows inherit the labels of their original longer segments. However, local physiological states may differ from segment-level labels, particularly in sleep staging, emotion recognition, and clinical EEG tasks. Consequently, the benchmark primarily measures short-window prediction of coarse labels rather than accurate recognition of local physiological states, limiting the interpretation of its results for fine-grained decoding and real-time monitoring.

W4. Incomplete Coverage of Low-Latency BCI Paradigms

Although short-window decoding and rapid decision-making are important motivations, the benchmark does not include representative stimulus-driven BCI paradigms such as P300 and SSVEP, relying primarily on motor imagery for BCI evaluation. This limits its coverage of time-sensitive BCI applications.

W5. Inconsistent Evaluation Metric Definitions and Window Segmentation

Section 3.3 states that Macro-F1 is used, whereas multiple supplementary tables report Weighted F1. These metrics can differ substantially under class imbalance. The authors should clarify the actual metric and ensure consistent reporting throughout the paper.

Supplementary Table 2 and Appendix A.2 report inconsistent numbers of sub-windows. For example, TUSL with 2-second windows uses 9 sub-windows in Table 2 but 5 non-overlapping windows in Appendix A.2; HMC and Sleep-EDF similarly report 11 versus 6 for 5-second windows. These differences suggest potentially different overlap settings. Since window overlap and prediction correlation directly affect majority voting, the inconsistencies undermine protocol clarity and reproducibility.

Questions for authors

Q1 (W2). Could the authors provide model-specific details of patching, padding, resampling, and channel mapping for each short-window configuration, particularly when input durations are shorter than the models' native patch lengths?

Q2 (W3). How do the authors account for potential inconsistencies between inherited segment-level labels and the actual physiological states within short windows?

Q3 (W5). Could the authors clarify the discrepancy in sub-window counts between Supplementary Table 2 and Appendix A.2, and specify the actual segmentation strides and overlap ratios used for majority voting? Were the reported F1 scores computed using macro-averaging or support-weighted averaging?

Q4 (W4). Could the authors conduct experiments on P300 and SSVEP paradigms, given the emphasis on short-window decoding and low-latency BCI applications?

4 comments
AuthorsOpenReview ↗

1. Limited Depth of Analysis

We sincerely appreciate the reviewer’s constructive comments and suggestions. We have provided detailed point-by-point responses and have accordingly revised and supplemented the tables and relevant content in the updated Supplementary.

Response. To move beyond descriptive comparison, the revision adds five complementary analyses that separate temporal duration, temporal position, architecture-specific input dependence, aggregation behavior, and statistical uncertainty. All five retain the canonical EEG-FM-Bench task definitions, subject-disjoint splits, and core preprocessing.

Revision. We expanded Supplementary “Benchmark Results” with window-length comparisons and model-wise short-window/voting/long-window tables; added the TUSL tie-rule sensitivity analysis to “Datasets and Benchmark Protocols”; added “Training Configurations and Dataset Processing” for model-specific temporal units; added “Uncertainty of Short-Window Results”; and revised main-text Figure 5 to compare cue-relative temporal positions.

Added analysisControlled comparisonQuestion addressedLocation
Window-length sensitivity0.5 vs. 1 s (MI); 2 vs. 5 s (TUSL); 5 vs. 15 s (HMC, Sleep-EDF)Whether more temporal context helps uniformlySupp. “Benchmark Results”
Temporal-position effectsFour contiguous 1-s cue-relative windows; same checkpoints and test subjectsWhether information is uniform within a trialFig. 5; Supp. “Benchmarking Protocols”
Model-specific temporal dependencePatch length, stride, valid-patch formula, resampling, channel mapping, boundary handlingWhether gaps reflect native input compatibilitySupp. “Training Configurations…”
Statistical reliabilitySubject-level bootstrap; training-seed Student-t intervalsStability across subjects and training randomnessSupp. “Uncertainty…”

These additions show that temporal dependence is task- and model-specific, not a monotonic “longer is always better” effect. For example, EEGPT rises from 66.3% to 73.9% single-window bAcc when TUSL input increases from 2 to 5 s, while magnitude and even direction differ for other models. On HMC and Sleep-EDF, 15-s inputs generally improve single-window sleep staging over 5-s inputs, consistent with the need for longer context. Conversely, the cue-relative analysis shows that early and late windows need not be equally informative even at fixed duration. We therefore revise the discussion to emphasize task morphology, within-trial position, and model input construction rather than treating window length as a single global factor.

AuthorsOpenReview ↗

2. Model-Specific Input Adaptation

Response. The new Appendix A.3 reports, for every model, input sampling rate, channel adaptation, patch length, stride, and incomplete-trailing-sample rules. A companion dataset table reports native/cached rates, channel order, and filtering, making dataset→model conversion traceable.

Revision. Added “Training Configurations and Dataset Processing,” with tables for model-specific training/input configurations and dataset-level channel selection and sampling rates.

ModelInput rateChannel adaptationTemporal constructionBoundary handling
BENDR256 HzMontage-aware mappingConvolutional encoderConfigured conv. padding
BIOT200 HzDynamic channel routingSTFT: 200-sample frame, 100-sample hopTrim to complete 200-sample blocks
CBraMod200 HzDataset montage retainedP=N/200P=N/200 when divisibleExact 200-sample reshape; no extra padding
CSBrain200 HzChannels sorted by brain regionP=⌊N/200⌋P=\lfloor N/200\rfloorDrop trailing incomplete block
EEGPT256 HzMap to channel IDsP=1+⌊(N−64)/32⌋P=1+\lfloor(N-64)/32\rfloorKeep complete receptive fields
LaBraM200 HzMap to channel IDsP=⌊N/200⌋P=\lfloor N/200\rfloorDrop trailing incomplete block
REVE200 HzMap to spatial positionsP=1+⌊(N−200)/180⌋P=1+\lfloor(N-200)/180\rfloorKeep complete 200-sample patches (stride 180)
EEGNet256 HzDataset channels directlyConvolutional temporal featuresLength handled by network layers
Conformer256 HzDataset channels directlyConvolutional patch embeddingLength handled by embedding layers
ADFCNN256→250 HzDataset channels retainedConvolutional featuresCrop + linear resample to 250 Hz

Here NN is the number of input samples at the model rate and PP is the number of temporal units per channel before spatial/channel fusion. Patch-count formulas apply only when at least one complete patch is present.

ModelBatchEpochsMax LRWeight decayLR schedule
BENDR256201.00×10−41.00\times10^{-4}0.01Cosine
BIOT192201.73×10−41.73\times10^{-4}0.01Cosine
CBraMod192208.66×10−48.66\times10^{-4}0.01Cosine
CSBrain64201.00×10−41.00\times10^{-4}0.01Cosine
EEGPT96208.70×10−48.70\times10^{-4}0.01One-cycle
LaBraM96201.39×10−31.39\times10^{-3}0.05Cosine
REVE64123.39×10−43.39\times10^{-4}0.01Plateau
EEGNet256502.80×10−32.80\times10^{-3}0.01Cosine
Conformer128202.00×10−32.00\times10^{-3}0.01Cosine
ADFCNN (MI)64202.00×10−32.00\times10^{-3}0.075Cosine
DatasetNative HzCache HzShort windowMontageChannels
BCIC-IV-2a250200 / 2561 s10–2022 EEG
PhysioNet-MI160200 / 2561 s10–1064 EEG
SEED / SEED-V / SEED-VII1000200 / 2562 s10–1062 EEG
EEGMat500200 / 2562 s10–2019 EEG
TUAB250–512200 / 2565 sTCP-AR23 ref.
TUEV256200 / 2562 sTCP-AR21 ref.
TUSL250200 / 2562 / 5 sTCP-AR / TCP-LEMontage-dependent
HMC256200 / 2565 / 15 sAASM minimalF4–M1, C4–M1, O2–M1, C3–M2
Sleep-EDF100200 / 2565 / 15 sSleep 2-chFpz–Cz, Pz–Oz

Truncation and padding are not interchangeable: dropping an incomplete trailing patch does not make a sub-native-patch input valid, and we do not assume a common zero-padding rule. For standard windows above, each architecture uses its own stated rule.

AuthorsOpenReview ↗

3. Inherited Segment-Level Labels

Question. How do the authors account for inconsistencies between inherited segment-level labels and physiological states inside short windows?

Response. We agree that inherited labels create a genuine weak-supervision problem. Public EEG datasets typically provide segment- or trial-level annotations based on clinical scoring guidelines or cognitive paradigms—for example, one label per 30 s epoch in sleep staging (HMC, Sleep-EDF), one label per ~200 s video trial in emotion recognition (SEED, SEED-V), one label per ~60 s trial in stress/workload monitoring, and recording-level labels in clinical abnormality detection (TUAB). As shown in Table 1 of the main text, existing deep-learning and EEG foundation-model studies on these tasks commonly use short windows (from a few seconds down to sub-second) that are much shorter than the native annotation granularity, and thus similarly face short windows inheriting segment-level labels.

From a neuroscience and experimental-measurement perspective, the absence of fine-grained short-window labels reflects practical and physiological constraints:

  • Slow dynamics and temporal integration of macroscopic brain states. Sleep-stage transitions, sustained emotion induction, mental workload, and clinical abnormalities are macroscopic physiological or pathological states integrated over seconds to minutes across multi-scale neural activity (rhythmic modulation, micro-arousals, sleep spindles, sporadic abnormal discharges, etc.). EEG within a short window often reflects local transient features or background fluctuations rather than a standalone, stationary instantaneous class state.
  • Time-scale constraints of clinical scoring and subjective reporting. Manual PSG sleep staging (AASM 30 s epoch rule), clinical EEG review, and post-trial subjective reports in emotion/workload paradigms all require a sufficiently long integration window or trial completion for reliable judgment. Independent window-by-window annotation at sub-second or few-second scales is prohibitively costly and hard to keep inter-rater reliable.

Our study asks how shortening EEG input duration affects decoding while keeping the original clinical/cognitive task and label definitions unchanged, and whether temporal aggregation can compensate for short-window information loss. Accordingly, following EEG-FM-Bench task definitions and label boundaries (Xiong et al., 2026), all short windows are extracted within a single native annotated interval and inherit that interval’s label without crossing label boundaries—avoiding cross-boundary mixing and ensuring long- and short-window models are compared under the same prediction target. This does not create instantaneous physiological ground truth.

Interpreting single-window limits and voting gains as weak supervision. Inheriting segment-level labels introduces a challenge analogous to weak supervision or local label–state mismatch: some windows may fall into transitions, baseline periods, or non-specific background. This is why we evaluate both single-window decoding and hard majority voting across multiple short windows: the gap between single-window and aggregated performance helps quantify how short-window physiological heterogeneity affects decision stability, and tests whether multi-window voting can suppress local transient noise to recover the segment-level target. Without independent fine-grained annotations, short-input degradation cannot be uniquely decomposed into (i) loss of temporal information and (ii) local label–state mismatch; we now acknowledge this identifiability limitation explicitly.

Revision. Appendix A.1 describes annotation units and task definitions per dataset; Appendix A.2 specifies short-window segmentation and voting; Figure 5 examines decoding across temporal positions within a task interval rather than assuming equal information in every short window. In the revision we further distinguish non-overlapping window counts from actual voting-window boundaries in Supplementary Table 2 and Appendix A.2, and clarify in the discussion that the benchmark evaluates predicting segment-level task labels from short-window inputs—not independent instantaneous physiological labels.

AuthorsOpenReview ↗

4. Metrics and Window Segmentation

Question. Clarify the sub-window count discrepancy between Supplementary Table 2 and Appendix A.2; specify voting strides/overlap; state whether F1 is macro- or support-weighted.

Response. We thank the reviewer for pointing out both ambiguities. We address window counts / voting protocols and F1 averaging separately below.

Window Counts and Voting Protocols

Our study uses two window-sampling strategies: non-overlapping segmentation covering all tasks, and sliding-window sampling focused on tasks with less distinct state transitions, including sleep staging and slowing events. Non-overlapping segmentation is used both to construct short-window training and evaluation samples and for majority voting: within the same evaluation interval, non-overlapping short windows are fed into the model separately and their class predictions are aggregated. Sliding-window sampling extracts overlapping short windows with a stride smaller than the window length, increasing the number of prediction positions participating in majority voting within the same interval. Therefore, the two strategies differ in window positions, stride, and overlap ratio, rather than in whether majority voting is applied.

Specifically, for each 30 s interval in HMC and Sleep-EDF, the non-overlapping setting contains 6 windows of 5 s or 2 windows of 15 s (6 or 2 votes); with a half-window stride, the strides are 2.5 s and 7.5 s (both 50% overlap), corresponding to 11 or 3 votes. For each 10 s interval in TUSL, the non-overlapping setting contains 5 windows of 2 s or 2 windows of 5 s (5 or 2 votes); the original 2 s voting evaluation further uses a 1 s stride with 50% overlap to obtain 9 votes, whereas the original 5 s voting setting retains a 5 s stride with zero overlap, remaining at 2 votes. Main TUSL 5-s votes start at 0 and 5 s; a separate supplementary analysis adds 2.5 s (three votes).

DatasetWindow / decisionDisjoint cutsDisjoint stride / overlapMain votesVote stride / overlap
TUSL2 / 10 s52 s / 0%91 s / 50%
TUSL5 / 10 s25 s / 0%25 s / 0%
HMC5 / 30 s65 s / 0%112.5 s / 50%
HMC15 / 30 s215 s / 0%37.5 s / 50%
Sleep-EDF5 / 30 s65 s / 0%112.5 s / 50%
Sleep-EDF15 / 30 s215 s / 0%37.5 s / 50%

Appendix A.2 originally gave the window counts under non-overlapping segmentation, while Supplementary Table 2 listed the vote counts actually used in the main experiments, without sufficiently explaining the relationship between the two sampling strategies. The revised protocol table and Appendix A.2 clearly distinguish non-overlapping window counts from main-experiment vote counts, and list their respective strides and overlap ratios to avoid conflating different sampling settings into a single protocol.

F1 Averaging

EEG tasks often exhibit imbalanced class distributions—for instance, different sleep stages or clinical events differ substantially in sample counts within the test set. We therefore report both bAcc and F1 to evaluate models from two complementary perspectives: class-balanced recognition ability and overall performance under the empirical sample distribution.

The F1 scores reported in our paper use support-weighted averaging (Weighted F1). Specifically, we first compute the F1 score for each class individually and then take a weighted average according to the true sample proportion (support) of each class in the evaluation set, whereas Macro-F1 assigns equal weight to all classes. Using Weighted F1 summarizes precision and recall while accounting for actual class prevalence; meanwhile, bAcc takes an unweighted average of per-class recall, giving minority classes equal weight. The two metrics thus complement each other.

The captions of the result tables in the original Appendix already specify Weighted F1, and all F1 values in the tables come from this calculation. The mention of “Macro-F1” in the “Evaluation metrics” paragraph of the main text was a naming typo rather than a difference in calculation between the main text and the Appendix. We have corrected this term to Weighted F1 in the main text and added the support-weighted definition so that the main-text description, Appendix captions, and evaluation code are fully aligned.

Reviewer_4LdsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ShortEEG-Bench, a benchmark for evaluating EEG foundation models with reduced temporal context. It covers eleven datasets spanning motor imagery, mental workload, emotion recognition, sleep staging, and clinical EEG classification, with seven foundation models and three conventional neural-network baselines. The evaluation considers short-window durations, full fine-tuning, linear probing, LoRA, single-dataset and multi-dataset training, and aggregation of short-window predictions by majority voting. The reported experiments suggest that pretrained models can retain useful decoding performance with shortened inputs, voting often improves individual-window predictions, early post-cue intervals are particularly informative in the evaluated motor-imagery datasets, and multi-dataset transfer depends strongly on the model and dataset composition. Subject-level analyses additionally examine performance for users with weak conventional baselines. The contribution is primarily an empirical evaluation framework and characterization of existing models, rather than a new decoding method.

Critical strengths
  1. A consequential evaluation question with broad empirical coverage. The paper systematically examines how EEG foundation models behave when temporal context is reduced, covering eleven datasets, multiple task families, and several adaptation strategies. This is useful because performance on conventional long inputs does not establish suitability for earlier decisions. The benchmark could become a valuable resource for selecting models and adaptation strategies under constrained observation budgets.
  2. Analysis beyond a single leaderboard. The distinction between individual-window prediction and temporal aggregation, together with temporal-position and subject-level analyses, exposes behavior hidden by dataset-level scores. Figure 5, for example, shows substantial variation across post-cue intervals in both motor-imagery datasets. These analyses could inform practical evaluation protocols, provided their interpretation and statistical reliability are established.
Critical weaknesses
  1. The central duration and voting comparisons are not sufficiently auditable or demonstrably controlled. Supplementary Tables 4–5 identify the long-window foundation-model results as “published,” but the submission does not adequately establish that these results match the current experiments in subject partitions, preprocessing, checkpoint selection, and optimization. Consequently, the reported short-versus-long differences cannot yet be attributed specifically to input duration. There are also concrete reporting conflicts: Supplementary Table 2 lists 11/3 voting windows for the two sleep settings and 9/2 for TUSL, whereas Section A.2 specifies 6/2 and 5/2, respectively. These alternatives imply different overlap or aggregation protocols. Furthermore, Supplementary Table 15 reports identical short- and long-window EEGNet/Conformer scores for HMC within each duration setting, including identical per-subject values in Tables 11 and 14. This requires explanation before the HMC subject-level comparison can support the stated conclusion. These issues directly affect the benchmark’s principal contribution.
  2. The evidence does not sufficiently establish the reliability or attribution of the reported findings. Several datasets have very small held-out subject populations: one for SEED, two for BCIC-IV-2a and SEED-V, and three for TUSL. No adequate uncertainty estimates across subject partitions and training repetitions are provided, despite conclusions about model rankings, positive transfer, and user robustness. Supplementary Table 15 also selects foundation-model settings by their highest subject-mean performance, without establishing an independent validation-based selection procedure; comparisons using the best score among several settings can further overstate gains. Essential implementation details—including model-specific preprocessing, short-input handling, optimization, LoRA configuration, and multi-dataset sampling and loss weighting—are insufficiently documented. Finally, comparing short-window foundation models against long-window conventional models changes both model family and duration, so it does not isolate whether shortening itself improves user performance. The early post-cue peaks are interesting, but do not by themselves distinguish motor-imagery information from cue-evoked responses or preprocessing effects.
Questions for authors
  1. Long-window controls: What is the exact source and experimental configuration of every long-window baseline used in Figures 1–3 and Supplementary Tables 4–5? Were subject partitions, preprocessing, adaptation strategies, and model-selection rules identical to the short-window experiments? Why are the HMC short- and long-window conventional-model results identical in Supplementary Tables 11, 14, and 15?
  2. Implemented segmentation and voting: Which window counts were actually used for HMC, Sleep-EDF, and TUSL: those in Supplementary Table 2 or those in Section A.2? Please specify offsets, overlap, and handling of the remaining second in SEED-VII. How sensitive are the two-window results to the lowest-class-index tie-breaking rule?
  3. Reproducibility and transfer controls: Please specify model-specific preprocessing and short-input handling, checkpoint versions, optimization and LoRA settings, and the exact dataset pools, sampling rules, loss weights, and training budgets for each multi-dataset configuration. Were these comparisons controlled for target-dataset exposure? Was overlap between pretraining data and downstream evaluation subjects or recordings audited?
  4. Statistical reliability and selection: How many training repetitions and subject partitions support the reported results? Were the three foundation-model settings in the subject-level analysis selected using validation data or held-out test performance? Please provide uncertainty estimates that account for subject-level dependence, and the precise counting and aggregation rules behind the abstract’s 36% and 57% figures.
  5. Temporal interpretation: What exact cue-relative intervals and filtering procedures were used for Figure 5? Does the early-window advantage persist when excluding the immediate cue response and using preprocessing restricted to information available at prediction time? For TUEV, does the benchmark evaluate classification around a known event timestamp, or detection without access to that timestamp?
5 comments
AuthorsOpenReview ↗

1. Long-window controls

We thank the reviewer for the valuable comments. We have addressed each comment point by point and made corresponding corrections and additions to the tables and relevant content in the revised Supplementary.

Question. The source and configuration of each long window; whether they were consistent with those of the short-window setting; and why the conventional results for HMC were identical across the short- and long-window settings in Supplementary Tables 11, 14, and 15.

Response. The primary goal of this study is to systematically evaluate how foundation-model representations degrade under short time windows, and how multi-window majority voting and multi-dataset joint training improve decisions under limited temporal information. To avoid incomparable long-window baselines across papers with inconsistent training protocols, the long-window foundation-model values in Figures 1–3 and Supplementary Tables 4–5 are not new repetitions from this study: they are the published EEG-FM-Bench single-task, full fine-tuning, average-pooling results (Xiong et al., 2026, ICML). That benchmark standardizes long-window training within a unified framework; its compute budget, epochs, and hyperparameters follow its publicly documented settings. We retain them as standardized external references. The revised Supplement identifies, for each cited entry, the source, model, input duration, and adaptation setting.

Long- and short-window settings were well aligned. Consequently, long-window values should be read as external performance references, not as a fully paired compute-matched experiment. By contrast, the voting comparison is internally controlled: the same checkpoint and test intervals are used before and after aggregation, so the measured voting gain is not caused by a change in weights, split, or adaptation.

Our own short-window repetition protocol is as follows. Each dataset uses one fixed subject-disjoint partition; seeds are independent training runs, not new splits.

Training regimeAdaptationSeedsRuns
Single-task (11 datasets)FM: Full / Linear / LoRA; DM: Full42–4611*(8*5*3 + 2*5)=1430
Within-task multi-datasetFull / Linear / LoRA42, 43, 444*(8*3*3)=288
Cross-task multi-datasetFull / Linear / LoRA42, 43, 448*3*3=72

We re-audited the aggregation used for Supplementary Tables 11, 14, and 15 and found that the EEGNet and Conformer rows labeled as long-window results mistakenly pointed to the short-window result entries. This duplicated both the overall and per-subject values. We have replaced those entries with results from the corresponding 30-s long-window training and evaluation and recomputed all affected HMC subject-level comparisons.

For verification, the revised Appendix reports 5-s and 15-s per-subject results in the “HMC (5 s)” and “HMC (15 s)” blocks and their summaries in the “Subject-mean bAcc” table. The accompanying HMC discussion has also been updated using the corrected 30-s EEGNet and Conformer values.

The correction changes the numerical comparison but not the main conclusion. On HMC, the best short-window foundation model increases from 64.4% subject-mean bAcc at 5 s to 70.5% at 15 s. The 15-s result is close to the corrected 30-s Conformer result of 70.1% and exceeds the corrected 30-s EEGNet result of 67.9%. Thus, very short inputs lose context needed for sleep staging, whereas a moderately longer short window can approach long-window conventional-model performance.

HMC settingBest short-window FM configurationSubject-mean bAccDiff. from 30-s EEGNet (67.9)Diff. from 30-s Conformer (70.1)
5-s short windowBIOT, full fine-tuning64.4−3.5−5.7
15-s short windowREVE, LoRA70.5+2.6+0.4
30-s long-window conventional baselineEEGNet67.90.0−2.2
30-s long-window conventional baselineConformer70.1+2.20.0
AuthorsOpenReview ↗

2. Implemented segmentation and voting

Question. Actual voting-window counts for HMC, Sleep-EDF, and TUSL; SEED-VII offsets, overlap, and the remaining second; sensitivity of two-window results to lowest-class-index tie-breaking.

Response. The discrepancy arose because Appendix A.2 reported non-overlapping stored short-window cuts (“Disjoint cuts”), whereas Supplementary Table 2 reported the windows used by sliding-window majority voting (“Votes”). These are two different sampling settings inside the same labeled decision interval. Parent intervals and label boundaries retain the EEG-FM-Bench protocol; stride and overlap are our controlled extensions. The main voting experiments use the Votes column.

For the main TUSL 5-s setting, the two votes start at 0 s and 5 s. A separate supplementary experiment adds a window at 2.5 s (three votes) and is reported separately. Hard-vote ties in the main protocol are resolved by the lowest class index. We now report interval length, window length, stride, overlap, and vote count together so each count maps to one unambiguous protocol.

DatasetShort / decision (s)Disjoint cutsVote stride (s)OverlapVotes
HMC / Sleep-EDF5 / 3062.550%11
HMC / Sleep-EDF15 / 3027.550%3
TUSL2 / 1051.050%9
TUSL5 / 1025.00%2

For SEED-VII, segmentation is defined relative to each annotated video clip, not fixed 15-s blocks. Starting at clip onset, we extract non-overlapping 2-s windows at offsets 0, 2, 4, … s; each window inherits the clip label and may not cross a clip/label boundary. A trailing fragment shorter than 2 s is discarded. For majority voting, seven consecutive 2-s windows from the same clip are grouped (offsets 0, 2, 4, 6, 8, 10, 12 s), jointly covering the half-open interval [0, 14) s. Incomplete trailing groups are excluded. Seven windows were chosen to approximate the scale of the published 15-s long-window reference. We do not first cut 15-s blocks and then carve seven 2-s windows from each; therefore there is no “remaining 1 s” inside a fixed 15-s interval that requires special handling.

As for the two-window hard voting necessarily, we have revised Appendix A.2 and Supplementary Table 2 to explicitly distinguish and report two voting protocols side by side: non-overlapping voting (S=TSWS=T_{\mathrm{SW}}, 0% overlap) and sliding-window voting (S=TSW/2S=T_{\mathrm{SW}}/2, 50% overlap), where SS denotes the stride and TSWT_{\mathrm{SW}} denotes the short-window duration. Both protocols retain the parent decision intervals and label boundaries defined by EEG-FM-Bench, while stride and overlap are controlled extensions introduced in our study. For each configuration, we now report the decision interval duration, window duration, stride, overlap ratio, and vote count, ensuring that each count corresponds to an unambiguously defined protocol.

Specifically, the main-text experiments use the window counts listed in the “Main-text votes” column below (11, 3, 9, and 2, respectively). The first three configurations use 50%-overlapping sliding-window voting, whereas the TUSL 5-s configuration uses non-overlapping voting.

DatasetWindow / interval (s)Disjoint cuts / non-overlapping votesMain-text votesSliding stride (s)Sliding overlapSliding votes
HMC / Sleep-EDF5 / 306112.550%11
HMC / Sleep-EDF15 / 30237.550%3
TUSL2 / 10591.050%9
TUSL5 / 10222.550%3

In the revised supplementary material, Supplementary Table 7 consistently reports non-overlapping majority-voting results (6, 2, 5, and 2 votes), while Appendix B.3 (Supplementary Tables 8–10) provides side-by-side comparisons of non-overlapping and half-stride sliding-window voting (50% overlap) across HMC, Sleep-EDF, and TUSL. These comparisons include the TUSL 5-s configuration with a 2.5-s stride and three votes (window start times: 0, 2.5, and 5 s), contrasted with its two-vote non-overlapping counterpart (0 and 5 s). Across all protocols, hard-voting ties are resolved deterministically by selecting the smallest class index among the tied classes.

These revisions clarify the distinction between segmentation and voting counts, ensure consistency across the main text and supplementary material, and facilitate experimental reproducibility.

AuthorsOpenReview ↗

3. Reproducibility and transfer controls

Question. Please specify model-specific preprocessing and short-input handling, checkpoint versions, optimization and LoRA settings, and the exact dataset pools, sampling rules, loss weights, and training budgets for each multi-dataset configuration. Were these comparisons controlled for target-dataset exposure? Was overlap between pretraining data and downstream evaluation subjects or recordings audited?

Response. Different EEG foundation models have different native input interfaces, and multi-dataset training further introduces differences in channel montages, task labels, and dataset sizes. While keeping each dataset’s task definition and evaluation split consistent, we use model-specific input adaptation and separately evaluate Full fine-tuning, Linear probing, and LoRA across training regimes.

Appendix A.1 (“Individual Dataset Profiles”) and A.2 (“Benchmarking Protocols”) already describe each dataset’s task, annotation unit, train/val/test splits, and short-window construction. The revision adds Appendix A.3 (“Training Configurations and Dataset Processing”) with two aligned tables: (i) a model configuration table (input sampling rate, channel adaptation, patch length/stride, trailing-incomplete handling, batch size, epochs, learning rate, weight decay, scheduler); (ii) a dataset processing table (native/cached sampling rates, channel selection and ordering, short-window durations). Dataset-level preprocessing selects the ordered montage, filters the signal, and resamples to a 200 Hz / 256 Hz cache; model adapters then apply architecture-specific channel identity mapping, brain-region ordering, or spatial position mapping, keep complete temporal blocks, and truncate trailing incomplete segments. Truncation is not padding: dropping an incomplete trailing patch does not make an input shorter than one native patch valid, and we do not assume a shared zero-padding rule across models.

ModelHzTemporal constructionBoundary / short-input handling
BENDR256Convolutional encoderConfigured convolutional padding
BIOT200STFT: 200-sample frame, 100-sample hopTrim to complete 200-sample blocks
CBraMod200P=N/200P=N/200 when divisibleExact 200-sample reshape; no extra padding
CSBrain / LaBraM200P=⌊N/200⌋P=\lfloor N/200\rfloorDrop trailing incomplete 200-sample block
EEGPT256P=1+⌊(N−64)/32⌋P=1+\lfloor(N-64)/32\rfloorKeep complete receptive fields (len 64, stride 32)
REVE200P=1+⌊(N−200)/180⌋P=1+\lfloor(N-200)/180\rfloorKeep complete 200-sample patches (stride 180)
EEGNet / Conformer256Conv features / patch embeddingTemporal length handled by network layers
ADFCNN256→250Convolutional featuresCrop and linearly resample to 250 Hz

Full fine-tuning updates backbone and classification head; Linear probing freezes the backbone and trains the readout; LoRA adapts the backbone via low-rank updates. We use the official released pretrained weights of the seven foundation models; reproduction requires matching the specific checkpoint file, model structural configuration, and downstream adaptation settings. Model-specific budgets appear in the A.3 model table (e.g., EEGPT 96 / 20 / 8.70×10−48.70\times10^{-4} / 0.01 / One-cycle; REVE 64 / 12 / 3.39×10−43.39\times10^{-4} / 0.01 / Plateau). Training code, fine-tuning parameters, and weights will be released.

Multi-dataset experiments are organized by task relationship and summarized with adaptations and seeds in the training-regime overview at the start of Appendix A:

Regime / poolIncluded datasetsAdaptationsSeeds
Single-dataset (ST)Each of 11 datasets separatelyFull / Linear / LoRA42–46
Within-task: MIBCIC-IV-2a, PhysioNet-MIFull / Linear / LoRA42–44
Within-task: emotionSEED, SEED-V, SEED-VIIFull / Linear / LoRA42–44
Within-task: sleepHMC, Sleep-EDFFull / Linear / LoRA42–44
Within-task: clinicalTUAB, TUEV, TUSLFull / Linear / LoRA42–44
Cross-task MT-4dsBCIC-IV-2a, SEED-V, HMC, EEGMatFull / Linear / LoRA42–44
Cross-task MT-allBCIC-IV-2a, SEED-V, HMC, EEGMat, TUSLFull / Linear / LoRA42–44

Comparisons use these explicitly defined training pools and target test splits. We also checked the publicly disclosed pretraining data sources of the seven official checkpoints to clarify data scope and model provenance. Downstream data are organized under a subject-disjoint principle: different recordings and sessions from the same subject remain in the same split, preventing leakage via adjacent short windows or across sessions. Appendix A.2 lists per-dataset subject counts and held-out IDs (e.g., BCIC-IV-2a 1–5 / 6–7 / 8–9; PhysioNet-MI 1–69 / 70–88 / 89–109); revised A.3 further summarizes training configurations beside these splits.

Comments on the paper

1 comment

We sincerely thank all reviewers for their thoughtful, constructive, and comprehensive feedback, as well as for recognizing the importance of short-window EEG decoding, the extensive experimental coverage of ShortEEG-Bench, and the value of our complementary analyses from temporal aggregation, temporal-position sensitivity, and subject-level perspectives. We greatly appreciate the reviewers’ detailed comments and insightful suggestions, which help us improve the methodological rigor, reproducibility, clarity, and scientific insights of our work. Below, we provide point-by-point responses to address the reviewers’ concerns and clarify the relevant experimental details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.