- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EEG-to-text systems produce fluent output, but prior work shows performance often survives when EEG is replaced by noise, suggesting the prior from a strong language model rather than the neural signal does the work, specifically in typical settings with a closed pool of candidate sentences. The paper studies how many bits per token recorded evidence actually adds beyond a frozen LM.
Decoding is framed as a channel-capacity measurement. An evidence encoder (fixation-locked EEG band power) produces an embedding that that tilts a frozen LM prior (GPT-2, Qwen). An InfoNCE-style contrastive score yields a per-token lower bound on mutual information between evidence and token given context. The estimator is calibrated on synthetic channels of known capacity and tested against structured nulls that preserve non-linguistic structure while breaking text alignment. A structured pipeline (steps E0–E4) drives the evaluation on three different data sets.
The authors find that their framework recovers information from text biases and gaze, but EEG itself yields 0 bits/token, consistently across several different configurations. Further experiments show that the EEG signal still contains structural information (e.g., identifying tasks/sessions), which do not transfer to the sentence identification.
Critical strengths
- The paper tackles an important, timely problem as it quantifies a fundamental flaw in EEG-to-text evaluation that fluency metrics (BLEU, BERTScore) cannot detect, and quantifies existing skepticism in the field.
- Through frozen prior, exact zero anchor, exact partition function, structured nulls, synthetic calibration, pre-specified hypotheses, gated pipeline, text-disjoint splits, provenance logs, and released code, the authors go beyond a simple real-vs-noise comparison with .
- The reported positive control shows that another biosignal in gaze provides a positive information rate and that EEG contains decodable information beyond the text prediction, showing that the negative result is not just the result of a broken pipeline.
- The authors do their best (but see also weaknesses) to triangulate the results through cross-corpus (ZuCo, COFETT, OneStop), cross-language, and cross-subject evaluation, and all results point in a similar direction.
- The paper provides transparent reporting, honest disclosure of limitations, and a conservative interpretation, which is important for a negative main result (but also see below).
- The paper is well-structured and written in a human-understandable fashion, with some leakage of AI jargon.
Critical weaknesses
Weaknesses:
- The paper mentions multiple times "pre-registered" hypotheses, but those do not make an appearance. This concerns me deeply. A paper, specifically one which carries a negative result, should be structured along hypotheses which drive the expectation of the analysis. If there are pre-registed hypotheses, they need to be disclosed and made an integral part of the paper, irrespectively of the observed outcome.
- Calibration (E1) shows the estimator saturates near ~0.1 bits/token fails to detect synthetic channels below roughly 0.25–0.5 bits/token. Gaze is reported at 0.107, but EEG is framed as "0.000" although this is not a tight bound on EEG mutual information beyond the established detectability threshold. The text acknowledges that correctly, but the abstract's "0.000 bits/token" does not. Is also unclear why the positive gaze result is treated differently when lying under the detection threshold and warrants a discussion of how reliable the estimates from the calibration are for real data (or can there be a gaze-specific calibration with a tighter threshold?).
- The results are by design limited to the specific chosen representation, using only fixation-locked, word-level band power, which says nothing about raw waveforms, source-localized signals, temporal dynamics, or learned EEG encoders. The per-token likelihood-ratio framework does not rule out that EEG might carry information about higher-level structure (sentence meaning, discourse). The authors acknowledge this limitation properly, but it limits the scope. Ideally, the paper would at least outline how subsequent analysis could be conducted to extend the scope while staying as comparable as possible to the current pipeline. In that sense, the reproduction analysis in appendix H might actually deserve a more prominent role, as it provides some level of pipeline-independence. Why did the main analysis not rely on an established pipeline in the first place?
- Minor: Figure 2 is not scaled properly
Questions for authors
see above.
4 comments
We would be glad to hear whether our response and the revised PDF address your concerns, and we are happy to clarify anything before the discussion period ends.
Thank you for your honest and critical re-analysis and addressing the "pre-registration". I adjusted my score to reflect this.
Thank you for your careful review and for reconsidering your score.
Author's response to Reviewer's comment
W1. Pre-registered hypotheses
You are right, and we thank you for raising it. "Pre-registered" overstated what we did. We committed the hypotheses H1–H5 locally on 8 August 2026, four days before the first access to the test split on 12 August, but without third-party registration, so the revised paper calls them pre-specified. We should also have been open about test-split use during development: earlier versions accessed it at least 11 times between 12 August and 17 September and computed E4 results. One change followed a test run: on 15 August we found that the N∗ predictor used validation bits instead of E2 bits, contrary to its documented rule, so we corrected it and reran E4. The bits/token results (E2) use validation data only and are unaffected, but the revised paper no longer calls the E4 results at N∗ a clean held-out test. The revised Appendix J gives the full timestamped access log (Table 18), and Table 2 now lists every hypothesis with its outcome:
H5 was not decidable at N∗ because the smallest margin 133 pools can resolve is 6.2 points; at N = 64, 15 of 30 controls are formally equivalent to chance.
W2. The abstract's 0.000, and why gaze and EEG are treated differently
We agree that the abstract overstated the EEG result. The revised abstract now states that no reliable token-level information was detected in the band-power representation, and reports 0.38 bits/token as the capacity at which 9 of 10 synthetic seeds are detected, not as an upper bound on real EEG. The different treatment is deliberate: our estimate is a lower bound, so a significant positive value (gaze) establishes at least that much information, whereas a non-detection (EEG) shows only that no information was detected. Gaze is robust across five retrained models (0.265 ± 0.025 bits/token) and GPT-2 sizes from 124M to 1.5B (0.17–0.30). Following your suggestion, we ran a gaze-matched calibration (Appendix N, Table 25):
All seeds are detected from a true capacity of 0.18 bits/token at 2,000 sentences, and the single real-size seed from 0.07. A bound of 0.271 corresponds to a true capacity of 0.74–1.47 bits/token at 2,000 sentences and above 0.37 at the real size, so the gaze value is, if anything, an underestimate.
W3. Scope of the representation and established pipelines
We tested four further representations with the same pipeline (Appendix O, Table 27). The BrainTranslator encoder carries a negligible 3.0×10−7 bits/token. LaBraM and CBraMod give 0.055 and 0.068, but structure-only controls reach 0.056 and 0.110, so neither carries token information beyond sentence structure. ERP features give 0.072, a third of it from which words were fixated; ERP + gaze exceeds gaze alone (0.342 vs. 0.267), but the intervals overlap and ocular artifacts may remain, so we treat ERP as a lead requiring ocular-corrected validation. Source-localized analysis would need head models that ZuCo lacks; the revised limitations state this, and that sentence-level meaning lies outside a token-level measure. Appendix O now gives a protocol for new extractors (same prior, splits, nulls and gain selection, with structure-only and injected controls), and Section 5.6 now discusses Appendix H: established pipelines give unchanged scores with real, shuffled or noise EEG, because the language model does the work, which is exactly what our measure separates.
W4. Figure 2
Agreed. The new Figure 2 uses a linear scale with b = 0 as its own point, shows every seed with the means and the exact capacity blocal, and plots detection rates with Clopper–Pearson intervals, including the ten-seed analysis; Figure 7 adds every seed's 95% interval, and Figure 4 is redrawn the same way.
Thank you for holding the paper to a high standard; we welcome further suggestions.