- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper evaluates text retrieval based on EEG signals. It references previous issues with relevant retrieval-based work. A few models are compared on a single, five-subject dataset. Its primary point is that better aggregated retrieval performance does not guarantee a model is better on every test subject. While its conclusion on the importance of reporting subject-level effects and variability across seeds is appreciated, it is also already widely recognized as a general phenomenon across science and statistics, and the phenomenon itself is only evaluated and established here with an absolute minimal level of effort.
Critical strengths
None
Critical weaknesses
- The paper does not position itself with respect to the literature and does not set out to solve a question or problem that goes beyond a simple ablation. Its contribution is very minimal.
Questions for authors
This work is so far below the bar that I don't foresee a possibility of a productive discussion and a potential recommendation for acceptance.