acceptodds
Under review as a conference paper at ICLR 2027

Neural State Prediction: Obstructing Shortcut Learning in EEG Foundation Models

Abstract

EEG foundation models increasingly use masked prediction to learn from unlabeled recordings, but optimizing this objective does not ensure transferable neural representations. A central challenge is that stable positional cues and local correlations can make masked regions predictable without integrating distributed neural context. To reduce this reliance on low-information prediction paths, we introduce Neural State Prediction (NSP), a latent-predictive framework that constrains both the prediction target and the available context. NSP uses a Target Encoder updated by an exponential moving average (EMA) to define latent supervision. Identity residualization removes additive effects associated with channel identity and relative time from the targets, while topology-separated context excludes their immediate spatial and temporal neighborhood from the visible input. We pretrain NSP on 2.2 million EEG segments from TUEG and evaluate it across 30 downstream datasets spanning clinical diagnosis, sleep staging, emotion recognition, motor imagery, event-related potentials, cognitive-state decoding, and language retrieval. Under full-parameter multi-task fine-tuning on EEG-FM-Bench, NSP achieves 63.94 macro balanced accuracy across 14 datasets, exceeding the strongest evaluated baseline by 2.35 percentage points. Controlled component ablations assess the contribution of each mechanism, while matched context controls and held-out interventions characterize the role of context geometry, signal content, and positional information. Jointly designing latent targets and their context offers a promising direction for EEG foundation models that learn from distributed signal structure. Our code is available at https://anonymous.4open.science/r/Neural-State-Prediction-7289/anonymous.4open.science/r/Neural-State-Prediction-7289/.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG Foundation Model, Latent Predictive Learning, JEPA

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_H5RBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Neural State Prediction (NSP), a self-supervised latent-predictive framework for EEG foundation models designed to eliminate low-information shortcuts during pretraining. It addresses two key shortcuts: position-conditioned prediction and local spatial-temporal interpolation. It introduces Identity Residualization (ResID) to subtract marginal channel/time means from target latents, topology-separated context and a cascaded spatial-temporal transformer (CST). NSP is pretrained on 2.2M clinical EEG segments from TUEG and evaluated across 30 downstream datasets.

Critical strengths
  • NSP effectively targets volume conduction and positional identity leaks in JEPA-style architectures.
  • Extensive benchmarking: NSP was evaluated across 30 datasets spanning 4 adaptation protocols (Full FT, Multi-Task FT, Frozen Backbone, LoRA).
Critical weaknesses
  • Methodological limitations in presenting, evaluating, and conceptualizing the central problem.

    • Although obstructing shortcut learning is the central premise and title of the paper, the direct evaluation confirming this claim is relegated entirely to the Appendix (Figure 17 and App. H.1) rather than featured in the main body.
    • Unconvincing diagnostic setup. The anti-shortcut diagnostic is evaluated on an extremely small sample of only 256 or 512 segments on a single checkpoint without reporting standard deviations, confidence intervals, or multi-seed trials while the model was pretrained on 2.2M segments. This setup raises critical concerns regarding sampling bias and lack of statistical power.
    • Hidden dependency on moment regularization.The performance of NSP heavily depends on moment regularization; removing it causes an important in frozen transfer. Crucially, the authors never evaluated Figure 17 without moment regularization. This masks whether moment regularization acts as the vital crutch preventing representation collapse, improperly attributing performance gains solely to ResID rather than the complete loss formulation. This raises a fundamental question: Is moment regularization the primary driver preventing collapse, rather than the proposed anti-shortcut techniques?
  • Mathematical limits of identity residualization. ResID is an additive first-order least-squares projection and thus mathematically incapable of removing non-linear channel-time interactions. This directly connects to the evaluation limitation: evaluating leakage solely with a Linear Probe measures linear decodability, which may offer an overly optimistic view of shortcut removal. Non-linear positional shortcuts could still persist undetected without non-linear probes.

  • Topology guard is heuristic. The spatial guard of masking Ns=2N_s = 2 nearest channels (App. B.3, Table 4) is a discrete heuristic calibrated for 19-channel montages used during pretraining. This fixed discrete rule lacks physical scaling when transferring to high-density EEG downstream datasets.

  • Minor: Performance degradation on fine-grained & ERP tasks

Questions for authors
  • What are the results of Figure 17 on a significantly larger, representative sample (e.g., 50000+ segments) with multi-seed error bars and with non-linear probes?
  • What are the results of Figure 17 without regularization to evaluate the true anti-shortcut effect of ResID?
  • How effectively does the fixed Ns=2N_s = 2 spatial guard scale to high-density downstream montages (e.g., 60–64 channels), and what are the results when using an adaptive or physical-distance-based spatial guard?
Reviewer_5uVFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Neral State Prediction (NSP) is a new JEPA style EEG foundation model. Its main contribution comes from the design of the context and target used, motivated by avoiding shortcuts in the representation learning from (1) local interpolation and (2) position condition prediction. The solution proposed for the shortcut (1) is to remove from context the neighbor channels and time points from the target window. The solution proposed for the shortcut (2) is to apply identity residualization on the target latent. The method is validated using two recent EEG foundation model benchmarks, NeralBench and EEG-FM-bench, spanning 30 datasets in total.

Critical strengths
  • The quantitative validation is extensive with the number of datasets and regimes tested, while benefiting from a consistent evaluation protocol with previous methods from using recent benchmarks.
  • The method motivation is clear and the solution proposed targets the potential shortcut that a joint-embedding predictive model could face in EEG.
Critical weaknesses
  • The identification and solution to local interpolation and position condition prediction would benefit from more analysis.
  • The qualitative analysis would benefit from more depth.
Questions for authors

Despite the method's very strong quantitative evaluation and good motivation, I would lean for a weak rejection of the paper at the current state due to the limited analysis of the qualitative results. If the analysis is strengthened, I am willing to increase the score.

Questions ordered by importance:

  • Q1: Did you test different spatiotemporal guards (different size across both dimensions)? Did you ablate this component?
  • Q2: For Figure 7 results:
    • Could you provide more analysis and explain what could be the takeaways from those visualizations? Could the analysis be matched to know physiological established results? Could those features match with some EEG biomarkers, for example?
    • Would it be possible to compare NSP embedding space with other baselines? Similarly, is it possible to compare the other visualization across baselines?
    • For the t-SNE visualization in Figure 7, have you tried a linear dimensionality reduction method like PCA to see if the embedding space is linearly separable or if its structure lies on a non-linear manifold?
  • Q3:
    • Could you provide some explanations for this observation:

"Frozen transfer is generally strongest at the shorter budget, whereas full fine-tuning is maintained or improves on several tasks." (line 431).

  • Similarly, for this observation:

"TUAB and TUEV change by less than 1.5 percentage points between 10% and 100% labels" (line 447).

  • Are those observations shared or previously observed on other baselines?

  • Did you study the label efficiency on additional datasets? Are TUAB and TUEV label efficiency results a common behavior across other datasets or are they outliers?

  • Q4: In the experimental setup subsection you are describing the preprocessing of the pretraining data. Is this preprocessing based on standard preprocessing used in other referenced models or is there some key differences that could give insights into future methods that pretrain on EEG data? Additionally, could you comment and clarify this sentence:

"Non-overlapping 6-s windows undergo quality checks after normalization".

  • Q5 Could you develop more about the limitations of the method in the conclusion?
  • Q6 Could you declare the total training time took to pretrain/finetune your model?
  • Q7: Could you explain what the "late target states" are (line 183)? Could you motivate this design?

Additional feedback not affecting the decision assessment:

  • The figures would benefit from adjustments.
    • Some font sizes are too small for a printed version of the paper (e.g., legends in Figures 1 and 4, most of the text in Figure 2, Lpred\mathcal L_{pred} formula in figure 3).
    • Figure 7 has a lot of white space and could benefit from using it (e.g., increasing the size of the inter-channel correlations subplot).
  • In figure 3, could you explain why ACPE and electrode coordinates enter the tokenize arrow? From the text description, it seems that the electrode coordinates are a feature used to construct ACPE and not a disjoint element used for tokenization.
  • In table 2, the different columns have little margin space. One suggestion to improve this could be to use a smaller font for the standard deviation. It would help to gain space and enable more spacing between columns.
Reviewer_jGREOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper proposes Neural State Prediction (NSP), a self-supervised framework designed to reduce shortcut learning in EEG foundation models. The method combines EMA-based latent prediction with identity residualization and topology-separated context masking. The former removes additive channel- and time-dependent components from prediction targets, while the latter excludes nearby spatial and temporal information to encourage learning from distributed EEG context. Experiments span 30 downstream datasets across clinical, BCI, cognitive, and language-related tasks. NSP achieves competitive or improved performance over existing EEG foundation models, with component ablations and input interventions providing additional evidence for the proposed design.
Critical strengths
  1. The method is well motivated. The paper identifies two plausible shortcuts in masked EEG prediction and addresses them through complementary target and context designs. This provides a coherent methodological contribution rather than simply combining existing pretraining objectives.
  2. The experimental validation is comprehensive. NSP achieves 73.14 macro BAC on NeuralBench, compared with 70.33 for LaBraM, and improves the strongest baseline by 2.35% on EEG-FM-Bench under multi-task full fine-tuning. Table 3 further supports the contributions of identity residualization and topology-separated context through matched comparisons.
Critical weaknesses
  1. Comparisons with closely related methods are incomplete. EEG2Rep [1] and other EEG-JEPA approaches also investigate informative masking and latent prediction, but are not directly evaluated. This makes it difficult to assess NSP's advantages over the most relevant alternatives.
  2. The evidence for shortcut reduction remains indirect. Table 3 demonstrates transfer gains, while Appendix H examines input sensitivity and positional information. These analyses support the proposed design but do not directly quantify the reduction in reliance on local interpolation or positional shortcuts. [1] Mohammadi Foumani, N., Mackellar, G., Ghane, S., Irtza, S., Nguyen, N., & Salehi, M. (2024, August). Eeg2rep: enhancing self-supervised eeg representation through informative masked inputs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 5544-5555).
Questions for authors
  1. The paper discusses EEG2Rep and other EEG-JEPA methods but does not directly compare their performance with NSP. How does NSP differ empirically from these closely related approaches under comparable pretraining and transfer settings?
  2. Table 3 and Appendix H provide evidence of improved transfer and altered shortcut sensitivity. Can the authors further clarify which findings specifically support reduced reliance on positional or local interpolation shortcuts, rather than the general benefit of more constrained latent prediction?
1 comment

We thank the reviewer for the positive assessment and the questions about closely related methods and shortcut reduction. We provide the empirical comparisons below, followed by diagnostics and matched component controls that address the two concerns separately.

Q1. Empirical and methodological comparison with EEG2Rep and EEG-JEPA

We agree that comparisons with closely related latent-prediction methods are needed. The available results show higher aggregate means for NSP, with task-dependent exceptions.

(1) EEG-JEPA: frozen multi-task transfer on EEG-FM-Bench. We compare BAC (%) on the same 14 benchmark tasks. EEG-JEPA results are taken from its reported frozen-transfer evaluation, and NSP results are provided in Appendix E.2.

DatasetEEG-JEPANSPΔ (pp)
TUAB78.3081.80+3.50
TUEV54.9552.90−2.05
HMC65.7072.97+7.27
Siena78.1169.93−8.18
ADFTD46.2054.10+7.90
PhysioMI29.8036.90+7.10
BCIC-2a39.1536.50−2.65
Mimul-1138.2040.47+2.27
SEED62.2068.93+6.73
SEED-V30.6038.77+8.17
SEED-VII23.4024.23+0.83
Workload67.4075.77+8.37
TUSL76.7067.07−9.63
Things-EEG-250.4850.23−0.25
Macro52.94 ± 0.3055.04+2.10

NSP has a higher mean on 9/14 tasks and a 2.10 pp higher macro BAC. It scores lower on TUEV, Siena, BCIC-2a, TUSL, and Things-EEG-2. This comparison is included in the revised Appendix E.2.9.

(2) EEG2Rep: single-task full fine-tuning.

MethodTUAB BACTUAB AUROCTUEV BAC
EEG2Rep80.52 ± 2.2288.43 ± 3.0952.95 ± 1.58
NSP83.83 ± 0.5089.40 ± 1.0073.17 ± 1.70
Δ (pp)+3.31+0.97+20.22

NSP reports higher means on all three metrics. Differences in pretraining, preprocessing, and data splits limit attribution of these gaps to the method itself. The matched controls in Q2 provide the component-level evidence.

(3) Specific methodological contribution. EEG2Rep uses semantic subsequence preserving masking; EEG-JEPA uses structured electrode–temporal masking and multi-depth latent targets. NSP jointly constrains target content and visible context. Identity residualization subtracts additive channel and relative-time means from Target states. Topology-separated context excludes the two nearest spatial neighbors and adjacent temporal patches around targets. These operations respectively reduce additive positional effects and restrict immediate-neighbor information that could support local interpolation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.