- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces NEAR-Bench, an evaluation framework for EEG-to-text decoding that jointly assesses predictive performance and dependence on correctly paired EEG signals. The proposed protocol combines participant–text holdout, standardized candidate identification, and input-replacement diagnostics. The main finding is that several EEG-to-text models exhibit negligible performance differences when EEG signals are replaced with zero or unrelated recordings.
Critical strengths
The joint participant–text holdout protocol is a reasonable attempt to control potential data leakage and improve evaluation consistency.
Critical weaknesses
- The central finding that EEG-to-text models can achieve similar performance when EEG inputs are replaced or corrupted has already been extensively discussed by Jo et al. (2024), Are EEG-to-Text Models Working?. Subsequent works, including GLIM and SemKey, have also investigated evaluation reliability and signal dependence.
- The benchmark lacks convincing validation of its ability to detect genuine EEG-dependent information. A diagnostic benchmark should demonstrate that its evaluation procedure can detect meaningful signal dependence when such dependence is present. However, the real-data micro-fitting experiment in Appendix C.2 fails its own predefined positive-control criteria. The model achieves 77.61% accuracy with real EEG but 78.00% with zeroed EEG. The artificial-signal experiment establishes that the implementation can learn strong synthetic class codes, but this does not establish sensitivity to weak, physiologically plausible EEG–language relationships. Consequently, the near-zero input-dependent gains are difficult to interpret. The results demonstrate the behavior of the selected implementations, but do not validate the proposed benchmark as a reliable general diagnostic of EEG information utilization.
Questions for authors
See weaknesses.