acceptodds
Withdrawn from ICLR 2027

NEAR-Bench: Benchmarking Predictive Performance and EEG Input Dependence in EEG-to-Text Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Advances in electroencephalography (EEG) representation learning and pretrained language models have driven research on decoding text from noninvasive brain signals. However, differences in data splits and evaluation tasks hinder direct comparison across studies; even under identical evaluation conditions, higher text scores do not necessarily indicate greater use of EEG corresponding to the target text. We introduce NEAR-Bench, an evaluation and diagnostic benchmark that jointly assesses text prediction performance and dependence on EEG input. Built on ZuCo 1.0 and 2.0, NEAR-Bench standardizes data splits, candidate text identities, and reporting rules, jointly holding out participants and texts during downstream training. For sequence decoders, we fix the trained model, replace only its EEG input, and compare predictions on the same observations. We also evaluate candidate identification separately from text generation without candidate sentences or reference prefixes. Results show that better text prediction does not necessarily accompany stronger EEG input dependence: some sequence models outperform an independent language-model reference, yet their prediction scores and generation behavior change little when EEG values are zeroed or replaced with other real recordings. NEAR-Bench provides a common basis for comparing model performance under explicit conditions, testing the contribution of correctly paired EEG input, and interpreting progress in EEG-to-text.

Primary area:
datasets and benchmarks
Keywords:
EEG-to-text decoding, benchmarking, EEG input dependence, cross-subject generalization, cross-text generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sKfeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces NEAR-Bench, an evaluation framework for EEG-to-text decoding that jointly assesses predictive performance and dependence on correctly paired EEG signals. The proposed protocol combines participant–text holdout, standardized candidate identification, and input-replacement diagnostics. The main finding is that several EEG-to-text models exhibit negligible performance differences when EEG signals are replaced with zero or unrelated recordings.

Critical strengths

The joint participant–text holdout protocol is a reasonable attempt to control potential data leakage and improve evaluation consistency.

Critical weaknesses
  1. The central finding that EEG-to-text models can achieve similar performance when EEG inputs are replaced or corrupted has already been extensively discussed by Jo et al. (2024), Are EEG-to-Text Models Working?. Subsequent works, including GLIM and SemKey, have also investigated evaluation reliability and signal dependence.
  2. The benchmark lacks convincing validation of its ability to detect genuine EEG-dependent information. A diagnostic benchmark should demonstrate that its evaluation procedure can detect meaningful signal dependence when such dependence is present. However, the real-data micro-fitting experiment in Appendix C.2 fails its own predefined positive-control criteria. The model achieves 77.61% accuracy with real EEG but 78.00% with zeroed EEG. The artificial-signal experiment establishes that the implementation can learn strong synthetic class codes, but this does not establish sensitivity to weak, physiologically plausible EEG–language relationships. Consequently, the near-zero input-dependent gains are difficult to interpret. The results demonstrate the behavior of the selected implementations, but do not validate the proposed benchmark as a reliable general diagnostic of EEG information utilization.
Questions for authors

See weaknesses.

Reviewer_ZKoJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces NEAR-Bench, a benchmark for evaluating EEG-to-text systems along two distinct dimensions: predictive performance and dependence on EEG that is correctly paired with the target text. The benchmark is constructed from ZuCo 1.0/2.0 and uses joint participant-text holdout, leakage-controlled text grouping, shared candidate sets, and separate evaluation of candidate identification and prefix-free/candidate-free generation. For sequence models, the authors perform same-checkpoint interventions in which the EEG input is zeroed, replaced by noise, permuted, or replaced by another real EEG recording while holding the model, target, and candidate set fixed. The benchmark is evaluated on BT, CSCL, CE-only and an auxiliary frozen-representation track. The main empirical finding is that predictive advantages relative to an independent LM baseline do not necessarily correspond to greater dependence on correctly paired EEG: real-minus-control differences are generally very small, and generated outputs often remain unchanged under EEG replacement. The paper additionally provides extensive diagnostics for data leakage, optimisation, input sensitivity, uncertainty, and reproducibility.

Critical strengths
  1. The benchmark isolates an important confound in EEG-to-text evaluation. The distinction between absolute text-prediction performance and same-checkpoint dependence on correctly paired EEG is conceptually important. In particular, the experiments demonstrate cases where a trained model improves substantially over an independent LM reference while showing almost no real-versus-zero/donor gain. This provides a useful diagnostic against attributing all downstream performance improvements to neural information.

  2. The evaluation protocol is unusually careful and reproducible. The joint participant-text holdout, explicit grouping of duplicate/near-duplicate texts before splitting, shared candidate identities, matched candidate pools, paired interventions and participant/text-aware uncertainty analyses address several weaknesses of prior EEG-to-text evaluations. The extensive appendices and implementation checks also make the benchmark substantially more auditable than typical EEG-to-text evaluations.

Critical weaknesses
  1. The sensitivity of the proposed EEG-dependence diagnostic to weak but genuine real-EEG information is not established. The artificial-signal experiment shows that the implementation can exploit sufficiently strong synthetic signals, but the real-data micro-fit does not pass the authors' own paired-input positive-control criterion: despite high training-set Top-1 accuracy, zeroing EEG produces essentially the same accuracy. Consequently, near-zero real-minus-control effects admit at least two interpretations: the evaluated models fail to use sentence-specific EEG information, or the modelling/evaluation pipeline is insufficiently sensitive to the weak information available in real EEG. The manuscript acknowledges that sensitivity to weak real signals is uncalibrated, but this limitation materially constrains how broadly the negative empirical findings can be interpreted.

  2. The empirical model coverage is narrower than the benchmark's intended scope, and the sequence models exhibit severe held-out optimisation/generalisation failure. BT, CSCL and CE-only share the same BART-large-based sequence architecture, while the frozen-representation experiments use a substantially different conditional evaluation setup. Moreover, every main CSCL and CE-only run selects the first cross-entropy epoch and subsequently shows sharply worsening validation NLL. Thus the experiments convincingly diagnose these implementations, but provide weaker evidence that near-zero paired-input gains characterise EEG-to-text approaches more generally.

Questions for authors
  1. The artificial-signal experiment demonstrates learnability for sufficiently strong injected signals, but real-data micro-fitting achieves similar real and zero-input accuracy even when evaluated on the observations used for fitting. Could you clarify how NEAR-Bench distinguishes lack of usable EEG-text information from insufficient sensitivity of the evaluated model/intervention pipeline to weak real EEG information? Is it possible to calibrate the minimum paired EEG effect that the benchmark can reliably recover, for example by injecting progressively weaker structured signals into real EEG features?

  2. All 25 main CE-only and 25 main CSCL runs select cross-entropy epoch 1, while validation NLL deteriorates substantially thereafter. Have you tested whether stronger regularisation, partial language-model freezing, more frequent validation/early stopping, or another optimisation strategy produces a model with materially better joint-holdout generalisation? If so, do the real-minus-control conclusions persist?

  3. How robust are the core conclusions to models that differ fundamentally from the shared EEG-Transformer + BART-large sequence architecture? The frozen-representation track helps, but because it uses a different reference and adaptation objective, it does not directly provide another same-checkpoint sequence-decoding test. Could the authors clarify the intended scope of the empirical conclusion relative to the scope of the benchmark?

Reviewer_wEXUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NEAR-Bench, a diagnostic benchmark intended to distinguish text-prediction performance from dependence on correctly paired EEG in natural-reading EEG-to-text decoding. It constructs 25 joint participant–text holdout test cells from ZuCo 1.0/2.0, standardizes eight-way candidate identities and scoring, and compares the same trained sequence decoder under paired, zeroed, and real-donor EEG. A separate frozen-representation track examines EEG scores conditional on non-EEG covariates. The main finding is that improvements over an independently trained language-model reference do not necessarily translate into gains from correctly paired EEG values.

Critical strengths
  • The paired-input design tests the right scientific question. Jointly holding out participants and texts, fixing candidate identities, and replacing EEG at the same checkpoint distinguish successful text prediction from improvements caused by the recording paired with the target. The real-EEG donor control, matched on effective length and drawn from a different text cluster, is substantially more informative than Gaussian noise alone. The paper also explicitly distinguishes candidate identification from generation without a reference prefix (Sections 3–4, Figure 2).

  • The diagnostic reporting is unusually careful. The authors report paired Top-1 and continuous scores, changes in candidate rankings, output agreement, participant and crossed participant–text uncertainty estimates, and learning diagnostics. In particular, the paper does not equate a confidence interval containing zero with proof of equivalence, nor does it interpret output invariance as a direct measure of semantic accuracy (Sections 4.4–5.4).

Critical weaknesses
  1. The empirical scope is too narrow for the proposed benchmark to substantiate its broader claim about progress in EEG-to-text. All three main sequence variants (BT, CSCL and CE-only) share essentially the same GD840 encoder-to-BART architecture; their differences are largely in optimization and contrastive training. The sequence track does not evaluate a substantially different contemporary decoder family, such as one based on semantic retrieval, discrete neural codes, or alternative language-model conditioning. The frozen CBraMod/LaBraM experiments are not an adequate substitute for testing modern generative decoders: they use a different prediction objective, and Appendix A.1 explicitly states that the anatomical correspondence assumed for their input channels has not been independently validated. Thus, the evidence establishes a problem for the implemented decoder family rather than showing that NEAR-Bench can discriminate genuine neural-language gains across current model classes.
  2. The benchmark's ability to detect realistic, weak EEG information remains uncalibrated, making the near-zero paired gains difficult to interpret. In the primary Pool-S/MEAN setting, BT, CSCL and CE-only attain approximately 11.28%, 11.61% and 11.71% Top-1, all below the 12.5% uniform reference (Table 1). Their paired-input gains are effectively zero. More importantly, CSCL and CE-only select epoch 1 as the best cross-entropy checkpoint in all 25 cells, while the real-EEG 64-observation micro-fit yields 77.6% versus 78.0% with zero EEG (Section 5.4). The injected-signal experiment only reaches 100% accuracy at relatively strong synthetic-signal levels and is not a calibrated positive control for weak semantic signals. Since the crossed holdout also leaves most participant–text cells unused during each training run, these findings may reflect models that have failed to learn the EEG task under the available supervision, rather than a generally sensitive test of meaningful paired-EEG contributions. The generation analysis further limits the interpretability of the results. Figure 4 evaluates output diversity and exact agreement under EEG replacement, but does not assess whether EEG-dependent output changes improve semantic fidelity to the reference text. Consequently, the benchmark cannot distinguish meaningful EEG-driven reconstruction from arbitrary output variation. Reference-grounded semantic evaluation under real, shuffled, and zero-EEG conditions would be needed to establish this distinction.
Questions for authors
  1. Can the authors demonstrate that the paired-donor evaluation remains informative for a competitive sequence decoder with a substantially different architecture or conditioning mechanism? What evidence shows that the benchmark can distinguish a genuine EEG-dependent gain from failure to learn under the joint participant–text holdout?
  2. Can the authors calibrate the smallest detectable paired-input advantage with weak, plausible injected signals, and report reference-grounded semantic changes for generated sentences under paired versus donor EEG?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.