- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents EEG2MM-Bench, which evaluates EEG-based retrieval and reconstruction across images, videos, and text using 11 existing datasets. It compares three EEG foundation models with task-specific architectures. For reconstruction, the authors evaluate both the original systems and a shared downstream pipeline, aiming to separate encoder performance from the effects of the generator. They also test reconstruction after replacing EEG inputs with Gaussian noise. The results suggest that foundation models perform well on retrieval, but their advantages do not consistently carry over to reconstruction.
Critical strengths
-
The most useful part of this work is the distinction between evaluating a complete reconstruction system and evaluating its EEG encoder under a common downstream pipeline.
-
The coverage of both retrieval and reconstruction is also valuable. Retrieval is easier to evaluate, but good retrieval performance alone does not establish that an encoder supports content reconstruction. Including both tasks makes the benchmark more informative than either evaluation in isolation.
Critical weaknesses
-
My main problem is that I couldn't figure out which protocol was actually used for the reported results. Section 3.3 and Table C.3 disagree in several places: -image reconstruction: contrastive Stage I in 3.3, MSE + InfoNCE in C.3; -video: contrastive in 3.3, MSE in C.3; -text: CE + embedding regression in 3.3, but C.3 describes symmetric contrastive alignment with cross-attention pooling, 8 rewritten variants per trial, and a different generator for the Chinese datasets. These are differences in objectives and supervision, not minor details. Since the controlled encoder comparison is the main contribution, this needs to be addressed.
-
The chance levels also don't line up with the splits. Table C.1 says 20 held-out stimuli for Grootswagers2019, but the Dummy Top-1/Top-5 in Table D.1 (2.50/12.50%) corresponds to 40 candidates. For NOD-EEG, 200 held-out classes with class-level averaging doesn't obviously give 0.01/0.04%. Maybe the gallery is defined at the image level, but the paper doesn't say. It is also unclear whether test trials are averaged (Table C.1) or kept as separate queries.
-
Metrics are reported inconsistently. The text describes BLEU-1–4 and ROUGE-1 for text, but Table D.4 reports BLEU-1/2, N-way, C.Recall, Dist-1/2, H.Ent and FD, and C.Recall, H.Ent and FD are never defined. PSNR and Hue-pcc appear for video without definitions.
-
On the noise control, replacing EEG with N(0,1) removes stimulus information, but it also pushes the input out of distribution, so a performance drop doesn't separate "sensitive to stimulus content" from "sensitive to input statistics.
-
Finally, video models are trained per subject while image/text seem to be trained across subjects, which makes the cross-modality conclusions about foundation models harder to read. Everything is also single-seed with no error bars, and a lot of the differences near chance are small.
Questions for authors
-
For each dataset, what is the gallery size and the target definition, and are test queries single trials or averages? How does this explain the Grootswagers2019 and NOD-EEG Dummy numbers?
-
Please define the undefined text/video metrics, the Dummy reconstruction baselines, and the relative-change formula in Fig. 6.
-
Could you add a shuffled-EEG control and multiple seeds for at least the main encoder comparisons?
3 comments
Author Response (Part 2/2)
5. Subject-training scope (W5—subject scope)
The current training recipes specify per-subject video training and joint training over the included subjects for Unified text. The local EEGVid Original runner also restricts evaluation to designated subject tracks.
Shared training scope supports an encoder comparison within a verified dataset–task configuration. Across modalities, training scope, target space, and generator conditioning all change. Our cross-task interpretation concerns utility under the evaluated configurations.
6. Bootstrap intervals and training seeds (W5, Q3—training seeds)
Appendix D now includes seven bootstrap tables using 10,000 subject-level resamples, seed 2000, and 95% percentile intervals. For example, the retrieval table reports LaBraM Things EEG Top@1 of 60.120%
. The reconstruction captions identify unavailable intervals and exploratory EMD estimates with fewer than seven subjects. These intervals capture uncertainty across evaluated subjects for fixed trained models. We could not complete training runs with additional seeds within the rebuttal period, so variability across training runs remains unassessed.
References
[1] Hyejeong Jo, Yiqian Yang, Juhyeok Han, Yiqun Duan, Hui Xiong, and Won Hee Lee. Evaluating EEG-to-text models through noise-based performance analysis. Scientific Reports, 16:350, 2026.
Thanks for the detailed response and the updated draft. The revisions clear up the confusion I had around the training protocols, split definitions, and metrics. Adding the subject-level bootstrap CIs also helps a lot. I'm still not convinced by the argument against using a shuffled-EEG control—shuffling keeps the marginal EEG distribution intact while breaking the pairing, which is the standard way to test if the model actually uses stimulus-specific information. Because these core experimental limitations are unchanged, I will keep my original rating.
Author Response (Part 1/2)
Thank you for the review. We have uploaded the revised manuscript and appendices as part of our revision and included the source code in the Supplementary Material.
1. Implemented objectives and manuscript inconsistencies W1
We apologize for this protocol-reporting error. The implemented settings are given below.
The image Stage I head is a linear layer. English text uses Flan-T5-Large and Chinese text uses mT0-Large. Original sentences supervise alignment, and rewritten variants supervise the decoder during training.
Section 3.3, Appendix C, and the code now consistently specify these training objectives, the linear projection head, and the loss-averaging conventions.
2. Gallery size, target identity, query averaging, and Dummy W2,Q1
The 20 held-out Grootswagers2019 stimuli describe reconstruction. Retrieval uses a 140/20/40 image split, so its test-target pool contains 40 images. NOD-EEG retrieval uses 11,154 images from 200 held-out classes, as documented in the separate retrieval-split table in Appendix C. The evaluated gallery sizes are 40 and 11154, respectively.
Predicted embeddings are averaged by subject–target pair. The global gallery averages target embeddings by target ID. Cosine ranking follows L2 normalization, and subject accuracies receive equal weights. Gallery size G counts distinct target IDs.
Our retrieval Dummy ignores EEG and uses the mean training-and-validation target embedding as a fixed query. It ranks gallery targets by cosine similarity and applies the same ranking to every test input. Top@K is computed from the paired target IDs within each subject and then averaged equally across subjects. Thus, the reported Dummy score is an empirical hit rate determined by the fixed ranking and evaluated target coverage, rather than a theoretical score assigned from gallery size G. We therefore report its actual evaluation results.
3. Metric definitions, reconstruction Dummy, and figure formula W3,Q2
Appendix B defines the requested metrics, and the main-text list now follows the reported columns. Text BLEU-1/2 uses eight rewritten references. C.RECALL, H.ENT, FD, PSNR, and Hue-pcc have explicit computational definitions. H.ENT measures sentence-prefix entropy, and Hue-pcc measures hue cosine similarity.
Reconstruction Dummy uses fixed training-derived outputs, comprising the mean image, a training-frame-average video, and the training sentence closest to the mean sentence embedding.
For dataset d and model m, the revised clean–noisy caption reports
$\Delta_{d,m}=\frac{1}{|\mathcal M_{d,m}|} \sum_{k\in\mathcal M_{d,m}} \left(\widetilde s^{\mathrm{clean}}{d,m,k}-\widetilde s^{\mathrm{noisy}}{d,m,k}\right).$
Here, d, m, and k index datasets, models, and metrics; Md,m is the metric set and ∣Md,m∣ its size. s denotes normalized scores, clean/noisy indicate the input condition, and Δd,m is the mean clean-minus-noisy difference.
Scores share higher-is-better orientation and common clean/noisy bounds. PSNR, Hue-pcc, H.ENT, and FD use empirical clean-score min–max bounds, with theoretical bounds for the other metrics.
4. Gaussian inputs and shuffled EEG (W4, Q3—shuffling)
Following Jo et al. [1], we use same-shaped N(0,1) inputs as a simple, reproducible negative control for reconstruction without recorded EEG values, keeping the trained model and remaining inference pipeline fixed.
Test-time shuffling of EEG across stimulus targets breaks input–target correspondence, analogous at the pairing level to training-time label corruption. Its effects combine responses to unrelated stimuli, cross-stimulus similarity, and generative priors, making interpretation difficult. It therefore cannot serve as a standalone baseline for establishing whether the model has learned stimulus-specific EEG information.
Gaussian replacement changes the input distribution. Accordingly, our Methods and Discussion interpret the clean–noisy difference as replacement sensitivity; a drop alone does not establish stimulus-specific decoding. Dummy provides a separate, fixed-output reference derived from training targets.