- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
None
Critical strengths
None
Critical weaknesses
- The abstract is not readable.
- Figure creation confusion.
- EXPERIMENTAL issues.
Questions for authors
- The abstract is not readable. Hard to read and understand.
- The AI use statement does not include figure creation; was Figure 1 produced using AI tools?
- The EXPERIMENTAL SETUP is limited to support the claimed contribution.
- The RESULTS analysis, the description is very poor.
- Table 1 does not support the claimed statement.
- No github, the implementation details description is very bad in EXPERIMENTAL SETUP.
4 comments
Q4. The results analysis is poorly described.
We agree it needed a clearer structure. §5 now leads with the main result and then breaks it down:
- SNS leads on meaning-level retrieval (Table 1). The margin holds under four independent evaluation encoders, with distractors from the same reading task, and across random splits (Figure 2a; Tables 6 and 9).
- A new per-subject figure (Figure 2b) shows that SNS is ahead of SemKey for nearly every subject.
§5 then explains why the word-overlap metrics behave differently, and §6 (Figure 3) measures how much of each metric survives when the EEG is destroyed. We also added example outputs (best, typical and worst cases) beside those of SemKey, GLIM and our own system on shuffled EEG (Tables 20–21).
Q5. Table 1 does not support the claimed statement.
We have rewritten the paper to state exactly what Table 1 supports. Together with the controls in Figure 3a, it supports two claims:
| method | 24-way, real EEG | information-free control | BLEU-1 |
|---|---|---|---|
| EEG2Text | 4.32 | 4.40 | 33.14 |
| GLIM | 12.58 | 5.17 | 21.89 |
| SemKey | 13.60 | 6.98 | 20.68 |
| SNS | 18.22 | 4.27 | 20.35 |
- SNS leads meaning-level retrieval, 18.22 against 13.60 (95% CI of the difference +4.01 [+2.50, +5.58]).
- Word-overlap metrics can be won without using the EEG: EEG2Text has the best BLEU-1, yet its real-EEG retrieval never beats its own control.
We agree the submission's claim to be "best on nine of the ten metrics that score meaning" was loose, since it counted several closely related retrieval scores separately. We removed that count from the abstract, text and Table 1, and state only the two claims above; their confidence intervals are in Table 15. If you had a different statement in mind, we would be glad to address it.
Q6. No code; the implementation details are poorly described.
The anonymous code is at https://anonymous.4open.science/r/eeg2text_sns-78BD. It contains the evaluation harness, all five methods, the frozen splits, every stored model output (so every table can be re-scored on a CPU), and step-by-step reproduction instructions, including how to obtain ZuCo. In the paper, the details are now grouped by topic: §4 and Table 14 (setup, baselines, metrics, statistics, compute), Appendix A (EEG front-end), Appendix B (model, training and the exact synthesis prompt), and Appendix C (how each method's test trials are selected and the fairness checks).
We hope these revisions address your concerns, and would be glad to clarify anything further.
I will maintain my original rating.
Thank you for your response. Your comments help us improve the quality of the paper, including our analysis and experimental setup, readability of abstract, AI use statement, and the release of repository.
Thank you for your review. The three Weaknesses correspond to the six Questions, so we answer them through the Questions:
Q1–Q3 are answered here and Q4–Q6 in part 2. We mark the changes in the revised PDF in blue.
Q1. The abstract is hard to read.
We acknowledge that the abstract can and should be improved. The OpenReview abstract field lost several numbers and symbols in rendering (e.g. "34%" and the arrow in "EEG→CLIP"). That field is now fixed. We also rewrote the abstract itself in clear and structured language, one idea per sentence. We thank the reviewer for raising this issue and hope this version of abstract reads better:
Q2. Was Figure 1 produced with AI tools?
Yes. Figure 1 was generated with an AI tool, and the front-end schematic (Figure 5) was composed with an AI tool from panels our scripts generate from real EEG. Thank you for catching this, and we apologize: the AI-use statement in the submission did not describe how these figures were made. The revised statement now does so. Both figures are schematics that report no results; the numbers drawn in Figure 1 are illustrative, and the channel and dimension labels in Figure 5 are the actual specifications. Every data figure, table and number in the paper is produced by our code from stored results. The revised statement also lists each other use of AI tools: aiding and polishing the writing, drafting sections, finding related work, and research ideation and execution with coding assistants, all under author direction and review.
Q3. The experimental setup is too limited to support the claims.
We have strengthened it and made it explicit in a new setup table (Table 14, referenced from §4):
The new experiments directly test the claims: the margin holds under four encoders our pipeline never uses, on every converged random split and for 28 of 30 readers. They also expose a limitation we now state plainly: the model does not yet transfer to readers unseen in training (§8).
Continued in part 2.