- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
ShortEEG-Bench evaluates EEG foundation models under short input windows (0.5–15 s, i.e., 1/8–1/2 of conventional 4–30 s windows), motivated by low-latency BCI and by the fact that long-window aggregation hides temporal dynamics. It benchmarks 10 models (7 EEG-FMs: LaBraM, EEGPT, BENDR, BIOT, CBraMod, REVE, CSBrain; 3 conventional DNNs: EEGNet, EEG Conformer, ADFCNN) on 11 datasets across 5 task families, under full fine-tuning / LoRA / linear probing, single-dataset vs in-domain vs cross-domain multi-dataset training, and single-window vs majority-vote inference. Main findings: short-window decoding remains competitive (36% of voting configurations match or exceed long-window predictions); majority voting recovers evidence once single windows exceed chance; early post-cue windows carry most motor-imagery information and 0.5-s windows expose sharper temporal fluctuations; in-domain multi-task training yields positive transfer for 57% of EEG-FMs while cross-domain training often causes negative transfer; and short-window EEG-FMs can rescue subjects with weak long-window baselines.
Critical strengths
- A useful community resource: the window-length axis is genuinely under-examined in EEG-FM evaluation, and the benchmark's breadth (11 datasets × 5 task families × 3 adaptation strategies × 3 training regimes) under one protocol with promised code, checkpoints, and split definitions would save the field substantial redundant effort.
- The evaluation design separates two scientifically distinct questions — whether a local segment is decodable at all versus how much evidence is recoverable by temporal aggregation — and adds temporal-position and subject-level analyses that expose variation hidden by aggregate scores, which is where the benchmark delivers its most interesting observations.
Critical weaknesses
- Short windows inherit their parent segment's label, so window-level performance mixes signal decodability with label validity of the segment (e.g., 2-s windows of a 30-s sleep stage; sub-second windows of a 4-s MI trial). The paper does not separate these (e.g., via event-aligned or per-window oracle analyses), and the headline findings are reported as counts and means ("36%", "57%") without confidence intervals, seed variance, or significance tests.
- The incremental insight over recent EEG-FM benchmarks rests entirely on the window-length axis, and several findings are expected a priori (fine-tuning > LoRA > linear probing; early post-cue MI dominance; voting helps once above chance). The long-window comparison protocol — whether long-window baselines receive the same training budget and tuning — is not fully specified in the main text, which matters for the "short windows compete with long windows" claim.
Questions for authors
- How much of the short-window degradation reflects label mismatch within parent segments rather than lost signal? A simple control (e.g., restricting to segments with stable labels, or aligning windows to events) would disentangle the two.
- Are long-window baselines trained with identical compute, epochs, and hyperparameter tuning, and can you report seed-level variance or CIs for the main findings?
- For the "57% of EEG-FMs show positive in-domain transfer" claim, what is the denominator of counted cases, and is the pattern reliable under resampling?
6 comments
2. Incremental Insight Beyond Recent EEG-FM Benchmarks
Weakness. The incremental insight over recent EEG-FM benchmarks rests entirely on the window-length axis, and several findings are expected a priori (fine-tuning > LoRA > linear probing; early post-cue MI dominance; voting helps once above chance).
Response. We agree that qualitative expectations alone are not sufficient novelty claims. Our incremental contribution is not “short windows only,” but a more comprehensive evaluation than prior EEG-FM benchmarks: under one EEG-FM-Bench-aligned protocol we jointly assess (i) short-window decoding, (ii) majority-voting performance, (iii) within-trial window order / temporal position, and (iv) different multi-task training regimes (ST, MT-INN, MT-INT(−c), MT-INT(+c)), across seven FMs, three adaptations, and eleven datasets. Continuity with EEG-FM-Bench keeps splits and preprocessing comparable; the added axes expose effects masked by long-window aggregation. Key conclusions from the main text and Supplement follow.
Short-window decoding remains viable, but degradation is task- and model-dependent (RQ1). EEG-FMs often outperform conventional DNNs under short inputs and obtain the best result on most datasets (Fig. 1c). Absolute performance stays meaningful (e.g., REVE 41.1% bAcc on BCIC-IV-2a at 1 s; CSBrain 75.4% on TUSL at 2 s), yet shortening hurts all datasets with widely varying magnitude. Full fine-tuning incurs the smallest average drop, then LoRA, then linear probing; backbone robustness also differs (BIOT/EEGPT smaller average drops; absolute SW accuracy need not track drop size).
Voting recovers evidence under conditions, not unconditionally (RQ2). Majority voting generally preserves or improves SW accuracy across Full/LoRA/Linear (Fig. 3a–c), often closing the gap to—or surpassing—published long-window references (Fig. 1d; ~36% of matched full-FT Vote>LW configs). Gains are dataset-dependent (larger on TUSL 2 s and HMC 5 s; consistent on MI). Aggregating four or six windows yields ~+2.6 pp average bAcc with fewer ineffective votes than two-window settings. Critically, voting helps mainly when the base SW model is above chance; near-chance bases (e.g., BENDR on SEED-V; collapsed 0.5-s REVE) gain little or nothing—voting is a variance-reduction amplifier, not a substitute for missing temporal context (Supplement SW/Vote/LW core findings). On sleep, extending SW from 5 s to 15 s helps, and 15-s voting can rival native long windows (e.g., ~71.0% vs. 71.8% on HMC).
Window order matters: early post-cue intervals dominate on MI (RQ3 / Fig. 4–5). On BCIC-IV-2a and PhysioNet-MI, early post-cue windows generally achieve the highest decoding accuracy; 0.5-s windows localize better but fluctuate more than 1-s windows. The temporal pattern holds across Full/LoRA/Linear, indicating signal structure rather than a single adaptation artifact—variation that long-window averages hide.
Multi-task training helps selectively and can hurt (RQ4). In-domain MT-INN is generally more favorable and consolidates homogeneous representations; cross-domain MT-INT(−c) more often induces negative transfer on unaligned backbones. Adding clinical data (MT-INT(+c)) is asymmetric: it can boost clinical slowing (e.g., EEGPT on TUSL up to ~79% SW / 80.7% Vote under full FT) while penalizing non-clinical cognitive/affective tasks. Effects depend jointly on dataset composition, backbone, and adaptation; majority voting cannot reliably offset adverse joint-training effects. Abstract-level summary: in-domain multi-task yields positive mean transfer for 57% (4/7) of FMs under the stated counting rule, whereas cross-domain training more often hurts.
We therefore treat “FT>LoRA>linear,” “early MI windows,” and “voting helps” as sanity-check hypotheses, and present ShortEEG-Bench as a controlled map of when short windows, voting, temporal position, and multi-task regimes help or fail—not as rediscovering those qualitative expectations.
3. Long-Window Training Budgets
Weakness / Question. Are long-window baselines trained with identical compute, epochs, and hyperparameter tuning? The protocol is not fully specified for the “short windows compete with long windows” claim.
Response. The primary goal of this study is to systematically evaluate how foundation-model representations degrade under short time windows, and how multi-window majority voting and multi-dataset joint training improve decisions under limited temporal information. To avoid incomparable long-window baselines across papers with inconsistent training protocols, we adopt the published single-task, full fine-tuning EEG-FM-Bench results (Xiong et al., 2026, ICML) as the long-window FM reference. That benchmark standardizes long-window training within a unified framework; its compute budget, epochs, and hyperparameters follow its publicly documented settings and were not retrained with our short-window optimizers. We will release ShortEEG-Bench models, configurations, and related artifacts for broader EEG research.
Short-window runs reuse the same EEG-FM-Bench construction/preprocessing (montage, filtering, resampling, unit scaling) and subject partitions before SW segmentation. Differing SW hyperparameters are documented beside LW entries in Appendix A.3 and Supplementary Tables 3–4 (source, model, duration, adaptation, budget). “Short windows compete with long windows” is thus a comparison to a standardized published reference, not identical-compute equivalence. Voting is more tightly controlled: same SW checkpoint and test intervals before vs. after aggregation.
Revision. Appendix A.3; SW/LW captions label published LW values as external references.
1. Label Validity vs. Decodability
We are grateful to the reviewer for the insightful and constructive feedback. All comments have been carefully considered and addressed individually. Corresponding corrections and additional information, including revised and newly added tables, have been incorporated into the revised Supplementary.
Weakness / Question. Short windows inherit the parent segment’s label, so window-level performance mixes signal decodability with label validity (e.g., 2-s windows of a 30-s sleep stage; sub-second windows of a 4-s MI trial). How much short-window degradation reflects label mismatch rather than lost signal, and could a stable-label or event-aligned control disentangle the two?
Response. We agree that short-window degradation can reflect both reduced temporal information and local label–state mismatch. We organize the answer as: (1) limitations of existing datasets, (2) analysis under our protocol, and (3) future controls.
1. Limitations of existing datasets
Public EEG datasets provide segment- or trial-level labels from clinical scoring or cognitive paradigms—e.g., one label per 30 s sleep epoch (HMC, Sleep-EDF), ~200 s emotion trial (SEED, SEED-V), ~60 s workload trial, or recording-level abnormality (TUAB). Table 1 of the main text shows that prior deep-learning and EEG-FM work already uses short windows (few seconds to sub-second) far below this granularity, so short windows inherit segment-level labels by construction. ShortEEG-Bench does the same: each short window inherits its parent recording/trial/epoch label.
This gap reflects neuroscience and measurement constraints, not only engineering choice:
Thus two confounders coexist under inherited labels: information loss (less evidence than the parent span) and local label–state mismatch (transition/baseline/background windows that still carry the parent label).
2. Analysis under the current protocol
We retain EEG-FM-Bench task definitions and label boundaries (Xiong et al., 2026) for comparability. The core research question of this work is: while keeping the original clinical/cognitive task label definitions unchanged, how does shortening the EEG input duration affect model decoding ability, and whether temporal aggregation can compensate for short-window information loss. Every short window stays inside one native annotated interval and inherits only that label—no cross-boundary mixing—so long- and short-window models share the same prediction target. This controls one error source but does not create instantaneous physiological ground truth.
Inherited labels act as weak supervision. Comparing single-window decoding with hard majority voting quantifies how local heterogeneity affects stability and whether voting recovers the segment-level target. Voting gains show better segment-level prediction; they do no tell whether each failed window erred from lost signal or local mismatch. The two mechanisms remain non-identifiable without fine-grained annotations—we state this explicitly.
Event alignment is already used for MI and TUEV. For other tasks, an oracle/stable-label control needs new annotations. We therefore interpret the benchmark as segment-level task prediction from short-window input, not instantaneous-state recognition.
3. Future directions
To better disentangle the two mechanisms we plan: (i) stable-epoch filtering (e.g., drop sleep epochs next to stage changes) as a proxy sensitivity check—still using the 30-s label; (ii) fine-grained short-window or event-level oracles where feasible; (iii) event-aligned designs beyond MI/TUEV for currently coarse-label tasks. These are not in the current results.
Revision. Expanded dataset profiles and protocols for annotation span, event alignment, boundary-constrained extraction, and aggregation (Appendix A.1–A.2; Fig. 5 for within-interval positions). The discussion states that information loss and local label mismatch cannot be uniquely separated with present data, and that stable-epoch/oracle analyses remain future controls.