acceptodds
Withdrawn from ICLR 2027

EEG2MM-Bench: A Unified Benchmark for EEG-to-Multimodal Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

EEG-to-multimodal decoding aims to retrieve or reconstruct images, videos, and text from electroencephalography (EEG) signals. However, the transferability of pretrained EEG representations to these tasks remains unclear, while inconsistent evaluation protocols and strong generative priors complicate comparisons. We introduce EEG2MM-Bench, a unified benchmark comprising 11 existing datasets with 468 hours of EEG recordings. We evaluate 17 task-specific architectures and three EEG foundation models across multimodal retrieval and reconstruction under shared data splits and consistent evaluation protocols. Retrieval uses a common alignment and ranking protocol. Reconstruction combines original-system evaluations with unified modality-specific pipelines, comparing EEG encoders under consistent downstream architectures and training procedures with frozen pretrained generators. Gaussian-noise input controls further assess reconstruction sensitivity to EEG replacement. By jointly evaluating stimulus matching and content recovery, EEG2MM-Bench provides a common framework for characterizing the capabilities and limitations of pretrained and task-specific EEG representations across multimodal decoding tasks.

Primary area:
datasets and benchmarks
Keywords:
Electroencephalography, EEG Foundation Models, Multimodal Decoding, Stimulus Retrieval, Stimulus Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gYBxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EEG2MM-Bench, which evaluates EEG-based retrieval and reconstruction across images, videos, and text using 11 existing datasets. It compares three EEG foundation models with task-specific architectures. For reconstruction, the authors evaluate both the original systems and a shared downstream pipeline, aiming to separate encoder performance from the effects of the generator. They also test reconstruction after replacing EEG inputs with Gaussian noise. The results suggest that foundation models perform well on retrieval, but their advantages do not consistently carry over to reconstruction.

Critical strengths
  1. The most useful part of this work is the distinction between evaluating a complete reconstruction system and evaluating its EEG encoder under a common downstream pipeline.

  2. The coverage of both retrieval and reconstruction is also valuable. Retrieval is easier to evaluate, but good retrieval performance alone does not establish that an encoder supports content reconstruction. Including both tasks makes the benchmark more informative than either evaluation in isolation.

Critical weaknesses
  1. My main problem is that I couldn't figure out which protocol was actually used for the reported results. Section 3.3 and Table C.3 disagree in several places: -image reconstruction: contrastive Stage I in 3.3, MSE + InfoNCE in C.3; -video: contrastive in 3.3, MSE in C.3; -text: CE + embedding regression in 3.3, but C.3 describes symmetric contrastive alignment with cross-attention pooling, 8 rewritten variants per trial, and a different generator for the Chinese datasets. These are differences in objectives and supervision, not minor details. Since the controlled encoder comparison is the main contribution, this needs to be addressed.

  2. The chance levels also don't line up with the splits. Table C.1 says 20 held-out stimuli for Grootswagers2019, but the Dummy Top-1/Top-5 in Table D.1 (2.50/12.50%) corresponds to 40 candidates. For NOD-EEG, 200 held-out classes with class-level averaging doesn't obviously give 0.01/0.04%. Maybe the gallery is defined at the image level, but the paper doesn't say. It is also unclear whether test trials are averaged (Table C.1) or kept as separate queries.

  3. Metrics are reported inconsistently. The text describes BLEU-1–4 and ROUGE-1 for text, but Table D.4 reports BLEU-1/2, N-way, C.Recall, Dist-1/2, H.Ent and FD, and C.Recall, H.Ent and FD are never defined. PSNR and Hue-pcc appear for video without definitions.

  4. On the noise control, replacing EEG with N(0,1) removes stimulus information, but it also pushes the input out of distribution, so a performance drop doesn't separate "sensitive to stimulus content" from "sensitive to input statistics.

  5. Finally, video models are trained per subject while image/text seem to be trained across subjects, which makes the cross-modality conclusions about foundation models harder to read. Everything is also single-seed with no error bars, and a lot of the differences near chance are small.

Questions for authors
  1. For each dataset, what is the gallery size and the target definition, and are test queries single trials or averages? How does this explain the Grootswagers2019 and NOD-EEG Dummy numbers?

  2. Please define the undefined text/video metrics, the Dummy reconstruction baselines, and the relative-change formula in Fig. 6.

  3. Could you add a shuffled-EEG control and multiple seeds for at least the main encoder comparisons?

3 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

Thank you for the review. We have uploaded the revised manuscript and appendices as part of our revision and included the source code in the Supplementary Material.

1. Implemented objectives and manuscript inconsistencies W1W1

We apologize for this protocol-reporting error. The implemented settings are given below.

StageImplemented objective and supervision
Image Stage IMSE + bidirectional CLIP loss, toward frozen CLIP image embeddings
Image Stage IIMSE, with the EEG encoder and projection head frozen and a linear adapter trained
Video Stage IMSE toward frozen CLIP text embeddings
Video Stage IIMSE, with the EEG encoder and projection head frozen and a per-token linear adapter trained
TextEqually weighted LM cross entropy and bidirectional CLIP loss; LM cross entropy is averaged over valid target tokens

The image Stage I head is a linear layer. English text uses Flan-T5-Large and Chinese text uses mT0-Large. Original sentences supervise alignment, and rewritten variants supervise the decoder during training.

Section 3.3, Appendix C, and the code now consistently specify these training objectives, the linear projection head, and the loss-averaging conventions.

2. Gallery size, target identity, query averaging, and Dummy W2,Q1W2, Q1

The 20 held-out Grootswagers2019 stimuli describe reconstruction. Retrieval uses a 140/20/40 image split, so its test-target pool contains 40 images. NOD-EEG retrieval uses 11,154 images from 200 held-out classes, as documented in the separate retrieval-split table in Appendix C. The evaluated gallery sizes are 40 and 11154, respectively.

Predicted embeddings are averaged by subject–target pair. The global gallery averages target embeddings by target ID. Cosine ranking follows L2 normalization, and subject accuracies receive equal weights. Gallery size GG counts distinct target IDs.

Our retrieval Dummy ignores EEG and uses the mean training-and-validation target embedding as a fixed query. It ranks gallery targets by cosine similarity and applies the same ranking to every test input. Top@K is computed from the paired target IDs within each subject and then averaged equally across subjects. Thus, the reported Dummy score is an empirical hit rate determined by the fixed ranking and evaluated target coverage, rather than a theoretical score assigned from gallery size GG. We therefore report its actual evaluation results.

3. Metric definitions, reconstruction Dummy, and figure formula W3,Q2W3, Q2

Appendix B defines the requested metrics, and the main-text list now follows the reported columns. Text BLEU-1/2 uses eight rewritten references. C.RECALL, H.ENT, FD, PSNR, and Hue-pcc have explicit computational definitions. H.ENT measures sentence-prefix entropy, and Hue-pcc measures hue cosine similarity.

Reconstruction Dummy uses fixed training-derived outputs, comprising the mean image, a training-frame-average video, and the training sentence closest to the mean sentence embedding.

For dataset dd and model mm, the revised clean–noisy caption reports

$\Delta_{d,m}=\frac{1}{|\mathcal M_{d,m}|} \sum_{k\in\mathcal M_{d,m}} \left(\widetilde s^{\mathrm{clean}}{d,m,k}-\widetilde s^{\mathrm{noisy}}{d,m,k}\right).$

Here, dd, mm, and kk index datasets, models, and metrics; Md,m\mathcal M_{d,m} is the metric set and ∣Md,m∣|\mathcal M_{d,m}| its size. s~\widetilde s denotes normalized scores, clean/noisy indicate the input condition, and Δd,m\Delta_{d,m} is the mean clean-minus-noisy difference.

Scores share higher-is-better orientation and common clean/noisy bounds. PSNR, Hue-pcc, H.ENT, and FD use empirical clean-score min–max bounds, with theoretical bounds for the other metrics.

4. Gaussian inputs and shuffled EEG (W4, Q3—shuffling)

Following Jo et al. [1], we use same-shaped N(0,1)\mathcal N(0,1) inputs as a simple, reproducible negative control for reconstruction without recorded EEG values, keeping the trained model and remaining inference pipeline fixed.

Test-time shuffling of EEG across stimulus targets breaks input–target correspondence, analogous at the pairing level to training-time label corruption. Its effects combine responses to unrelated stimuli, cross-stimulus similarity, and generative priors, making interpretation difficult. It therefore cannot serve as a standalone baseline for establishing whether the model has learned stimulus-specific EEG information.

Gaussian replacement changes the input distribution. Accordingly, our Methods and Discussion interpret the clean–noisy difference as replacement sensitivity; a drop alone does not establish stimulus-specific decoding. Dummy provides a separate, fixed-output reference derived from training targets.

AuthorsOpenReview ↗

Author Response (Part 2/2)

5. Subject-training scope (W5—subject scope)

The current training recipes specify per-subject video training and joint training over the included subjects for Unified text. The local EEGVid Original runner also restricts evaluation to designated subject tracks.

Shared training scope supports an encoder comparison within a verified dataset–task configuration. Across modalities, training scope, target space, and generator conditioning all change. Our cross-task interpretation concerns utility under the evaluated configurations.

6. Bootstrap intervals and training seeds (W5, Q3—training seeds)

Appendix D now includes seven bootstrap tables using 10,000 subject-level resamples, seed 2000, and 95% percentile intervals. For example, the retrieval table reports LaBraM Things EEG Top@1 of 60.120%

564˙00,640˙5056\.400, 64\.050

. The reconstruction captions identify unavailable intervals and exploratory EMD estimates with fewer than seven subjects. These intervals capture uncertainty across evaluated subjects for fixed trained models. We could not complete training runs with additional seeds within the rebuttal period, so variability across training runs remains unassessed.

References

[1] Hyejeong Jo, Yiqian Yang, Juhyeok Han, Yiqun Duan, Hui Xiong, and Won Hee Lee. Evaluating EEG-to-text models through noise-based performance analysis. Scientific Reports, 1611:350, 2026.

Thanks for the detailed response and the updated draft. The revisions clear up the confusion I had around the training protocols, split definitions, and metrics. Adding the subject-level bootstrap CIs also helps a lot. I'm still not convinced by the argument against using a shuffled-EEG control—shuffling keeps the marginal EEG distribution intact while breaking the pairing, which is the standard way to test if the model actually uses stimulus-specific information. Because these core experimental limitations are unchanged, I will keep my original rating.

Reviewer_9fJ8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper introduces EEG2MM-Bench, a unified benchmark for EEG-to-image, video, and text retrieval and reconstruction. It integrates 11 existing datasets with 468 hours of EEG recordings and evaluates 17 task-specific architectures and three EEG foundation models. Retrieval is evaluated under a common protocol, while reconstruction uses both original model pipelines and unified modality-specific pipelines. Gaussian-noise controls are included to assess sensitivity to EEG inputs. The main findings are that foundation models perform competitively in retrieval but not consistently in reconstruction, model rankings depend on downstream pipelines, and reconstruction metrics do not always favor real EEG over noise inputs.
Critical strengths
  • The benchmark covers retrieval and reconstruction across three output modalities under a common evaluation framework. The Original versus Unified comparison reveals a concrete ranking reversal between ENIGMA and ATM, showing the influence of downstream pipelines on model comparisons.
Critical weaknesses
  1. The main findings add limited insight beyond prior work. EEG-FM-Bench [1] has already shown that pretraining scale alone does not explain transfer performance and that adaptation strategies influence model effectiveness. Yang et al. [2] further demonstrated that EEG foundation models do not consistently outperform simpler decoders, with performance varying substantially across downstream tasks. The current conclusions largely reinforce these findings. Although the Original versus Unified comparison is useful, I do not see a broader conclusion that substantially extends our understanding of EEG representation transfer.
  2. The retrieval–reconstruction comparison does not isolate EEG information recovery. Retrieval uses DINOv2, V-JEPA2, or GPT-2 representations, whereas reconstruction uses different supervision targets and pretrained generators. In particular, video reconstruction predicts BLIP-derived caption embeddings without direct supervision of video dynamics, which are instead supplied by the generator's motion prior. Consequently, the observed retrieval–reconstruction differences cannot be attributed solely to EEG representation quality, and the video results do not establish recovery of EEG-dependent temporal information. [1] Xiong, W., Li, J., Li, J., Zhu, K., & Jiang, C. (2026). EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation and Diagnostic Analyses of EEG Foundation Models. International Conference on Machine Learning (ICML 2026). [2] Yang, L., Sun, Q., Li, A., & Van Hulle, M. M. (2026). Are EEG foundation models worth it? comparative evaluation with traditional decoders in diverse BCI tasks. In The Fourteenth International Conference on Learning Representations.
Questions for authors
  1. EEG-FM-Bench [1] and Yang et al. [2] have already examined the limitations of EEG foundation model transfer across different tasks and evaluation settings. What specific new insight does EEG2MM-Bench establish beyond these findings?
  2. The ENIGMA–ATM ranking reverses between Original and Unified reconstruction. Which downstream components account for this reversal, and is the same pattern observed across other models and datasets?
  3. The video reconstruction pipeline uses caption-level supervision without directly modeling the target video's temporal dynamics. What evidence shows that its reconstruction results capture EEG-dependent motion information rather than caption semantics and the generator's prior?
2 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

Thank you for the review. We have uploaded the revised manuscript and appendices as part of our revision and included the source code in the Supplementary Material.

1. Insights beyond previous transfer benchmarks W1,Q1W1, Q1

Our motivation is to test whether findings from conventional EEG transfer benchmarks extend to multimodal content decoding. EEG foundation models are pretrained on heterogeneous recordings whose distributions and objectives need not match those of stimulus-driven multimodal decoding. EEG-FM-Bench [1] evaluates classification, while Yang et al. [2] also examine limited regression, including speech-envelope reconstruction; neither jointly evaluates retrieval and generative reconstruction across image, video, and text. Their conclusions therefore require direct validation in this setting.

Our evaluation identifies three relationships not examined by those benchmarks: retrieval performance does not guarantee reconstruction utility; complete-system rankings can differ from encoder rankings under shared downstream conditions; and reconstruction scores can reward input-independent outputs. For example, LUNA achieves 7.830% retrieval Top@1 on ZuCo 2, yet its reconstruction two-way identification is 0.514 versus Dummy's 0.500. The ATM/ENIGMA CLIP ranking also reverses between Original and Unified. These findings support evaluating multimodal transfer directly, separating retrieval from reconstruction and encoder utility from complete-system performance, with input controls (Sections 4–5).

2. Attribution of retrieval–reconstruction differences W2W2

Unified controls the downstream architecture, target condition, and training recipe within each dataset–task configuration. It therefore measures an encoder's usefulness under that particular adaptation procedure.

Across retrieval and reconstruction, target spaces and objectives change. We examine whether a model's relative advantage persists across those tasks. The supported conclusion is that a retrieval advantage does not guarantee an advantage in the evaluated reconstruction pipeline. Encoder–adapter compatibility and generator conditioning contribute to that outcome, so the benchmark provides no unique causal attribution to intrinsic EEG information recovery. Section 5 states this limitation.

3. Components and ranking reversals Q2Q2

We designed the Unified pipelines specifically for this benchmark to compare EEG encoders within a common modality-specific framework. Task-specific models use identical dataset-specific preprocessing and splits. For images, the downstream pipeline uses a common linear adapter architecture and frozen SDXL-Turbo/IP-Adapter, excluding method-specific diffusion adapters and low-level image guidance. Our contribution is this controlled evaluation design, which enables more comparable assessments of task-specific and foundation encoders under shared downstream conditions.

On Things EEG, Original ENIGMA exceeds ATM in CLIP two-way identification 0\.853 vs\. 0\.624, whereas Unified ATM-Enc. exceeds ENIGMA-Enc. 0\.739 vs\. 0\.702. Related context-dependent rankings appear in the Alljoined-1.6M benchmark [3], Table 1: ATM-S exceeds ENIGMA in retrieval Top@1 on THINGS-EEG2 301˙530\.15% vs\. 27\.60%, but the ordering reverses on Alljoined-1.6M 05˙00\.50% vs\. 6\.00%. That result concerns a change of dataset; our comparison examines complete systems versus encoders evaluated within a shared reconstruction pipeline.

Components and training objectives change jointly between Original and Unified, so we do not attribute the reversal to any individual module. Neither ATM nor ENIGMA leads across all four image datasets and seven metrics; we report these as descriptive rankings under the specified evaluation conditions.

4. EEG-dependent video dynamics W2,Q3W2, Q3

The current Unified video experiment establishes performance under semantic conditioning. Its caption-level target provides no explicit framewise motion supervision.

The original EEG2Video method uses overlapping EEG windows, a Seq2Seq module supervised by framewise VAE latents, and a dynamic-aware noise-adding process. Our shared pipeline omits those dynamic-specific branches and uses a frozen AnimateDiff generator. A caption may describe an action or temporal relation, while generated motion also depends on the generator prior. Establishing EEG-dependent motion trajectories would require a dedicated temporal or matched-semantic motion control. Such evidence is currently unavailable.

AuthorsOpenReview ↗

Author Response (Part 2/2)

References

[1] Wei Xiong, Jiangtong Li, Jie Li, Kun Zhu, and Changjun Jiang. EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation and Diagnostic Analyses of EEG Foundation Models. In ICML, 2026.

[2] Liuyin Yang, Qiang Sun, Ang Li, and Marc M. Van Hulle. Are EEG Foundation Models Worth It? Comparative Evaluation with Traditional Decoders in Diverse BCI Tasks. In ICLR, 2026.

[3] Jonathan Xu, Ugo Bruzadin Nunes, Wangshu Jiang, Samuel Ryther, Jordan Pringle, Paul S. Scotti, Arnaud Delorme, and Reese Kneeland. Alljoined-1.6M: A Million-Trial EEG-Image Dataset for Evaluating Affordable Brain-Computer Interfaces. arXiv preprint arXiv:2508.18571v2, 2025.

Reviewer_oEHYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the difficulty of comparing results in EEG-to-image, video and text decoding, since papers use different splits, generators and metrics, and strong generative models can hide how little the output actually depends on the input EEG. It introduces a benchmark, EEG2MM-Bench, with 11 public datasets covering retrieval and reconstruction for images, video and text. Using it, models are evaluated both with their original full pipeline and in a unified mode (only the EEG encoder is swapped into a shared pipeline), together with a control where the EEG is replaced by Gaussian noise at test time. This benchmark is used to compare 3 EEG foundation models (LaBraM, REVE and LUNA) against 17 task-specific architectures. The main findings of the paper are that foundation models are competitive in retrieval but not consistently in reconstruction, that retrieval rank does not guarantee reconstruction quality, and that system and encoder rankings can differ.

Critical strengths

The most valuable part of this paper is the evaluation design. Running every model both in its original pipeline and in a unified one, where only the EEG encoder changes, separates how good a full system is from how good its EEG encoder is. This is not just a formality, as it flips the ranking of ATM and ENIGMA, which is exactly the kind of confusion that makes results in this field so hard to compare today. The splits are also mostly done well (Things EEG is split by concept, SEED-DV by block with unique clips, and the text datasets by sentence), so the test stimuli are not seen during training. Where I could check, the numbers also look credible: the THINGS-EEG2 retrieval results agree with NeuralBench’s independent numbers on the same protocol.

I also appreciate the honesty of the controls. The noise and Dummy comparisons show that text decoding and pixel-level image metrics barely depend on the EEG at all (Fig. 6, Tables D.2 and D.4). Many papers in this area would just not report this. A benchmark that runs this check by default could help the field stop overclaiming, which alone makes it useful.

Critical weaknesses

My main concern is that several conclusions of the paper rest on results that do not reflect the EEG at all. Some outputs completely ignore the input. LUNA’s image reconstructions are at chance on three datasets, with exactly the same clean and noisy scores, even though the same encoder reaches 43.4% retrieval Top-1 on Things EEG. REVE produces empty or constant text, with a BLEU-1 of 0 on two datasets. These look like failed training runs, but the paper interprets them as findings about the EEG representations. On top of that, on four datasets (Grootswagers2025, Chisco, TMNRED and NOD-EEG) every model is within about one point of the Dummy baseline in retrieval, so they can't really rank anything. And all results come from a single seed, with no confidence intervals.

This matters since two of the headline claims depend on these cases. “A foundation model achieves the highest retrieval Top-1 on all datasets” counts wins among models that are all at chance, and “retrieval does not translate into reconstruction” rests largely on the broken LUNA runs. I would recommend diagnosing the degenerate runs and retraining or excluding them, adding bootstrap confidence intervals and a test against chance (which needs no retraining), using a few seeds, and leaving chance-level datasets out of the rankings.

The second problem is reproducibility, which for a benchmark is not a minor issue. Nothing is released: no code, no split files, no configs, no anonymous link and no supplementary material. The written protocol doesn't fill the gap either, since the main text and the appendix disagree on basic details, such as whether test trials are averaged or kept as separate queries, and which training losses are used for images, video and text. The text pipeline also relies on “8 rewritten variants” per trial, and where these come from is never explained.

For a benchmark paper, the released and reproducible artifact is precisely the contribution. Without it, no one can evaluate a new model on EEG2MM-Bench or check the numbers reported here. I would recommend releasing the anonymized code, split files and evaluation scripts, making the main text and the appendix agree, and defining every metric as well as the Dummy baseline.

Other minor points: several metrics in the tables (e.g., N-WAY, C.RECALL, H.ENT, Hue-pcc) are never defined, and some promised ones (ROUGE, BLEU-3/4, VideoMAE) are missing. The test-set size per dataset is never reported, so chance levels and confidence intervals cannot be computed. MindCine is listed among the 17 models but has no results. The relative change in Fig. 6 blows up when scores are close to zero, so absolute differences would be easier to interpret. Finally, the paper needs a proofreading pass (there is a garbled sentence in the results section, and the template Acknowledgments text was left in).

Questions for authors
  1. [Clarification] Release. Will the code, split files and evaluation scripts be released, anonymously, during the review period? This would raise my assessment substantially.

  2. [Clarification] Degenerate runs. Are the LUNA image reconstructions and the REVE text outputs constant across inputs? Training curves or a measure of output diversity would show if these are simply training failures.

  3. [Clarification] Smit2025 split. According to the dataset paper, every block contains all 360 stimuli (90 videos in 4 orientations), so holding out one block seems to test on videos that were already seen in training. Could the authors re-split by video or at least label this as a seen-stimulus setting? It would also explain why EEGVid looks so strong on this dataset and nowhere else.

  4. [Clarification] Protocol. Were the test queries averaged or single-trial, and which losses were actually used, the ones in Section 3.3 or the ones in the appendix? How were the “8 rewritten variants” produced? Reporting the test-set size for each dataset would also help.

  5. [Clarification] Unified pipelines. Each unified pipeline is borrowed from one of the competitors (ATM for images, EEG2Video for video, GLIM for text), which gives that method’s encoder a home advantage. Do the ATM/ENIGMA reversal and GLIM’s lead in text hold with a neutral pipeline? Randomly initialized and frozen versions of the foundation models would also show if pretraining is really what helps.

  6. [Suggestion] Mismatched-EEG control. Gaussian noise does not look like EEG, so a drop in performance may only mean that the model is confused by unusual input. Pairing each stimulus with another trial’s real EEG at test time would be a cleaner control in my opinion.

  7. [Suggestion] NeuralBench. A direct comparison with NeuralBench on the three shared datasets would help, and the REVE results on THINGS-EEG2 should be marked, since REVE was pretrained on that dataset.

3 comments
AuthorsOpenReview ↗

Author Response (Part 1/3)

Thank you for the review. We have uploaded the revised manuscript and appendices as part of our revision and included the source code in the Supplementary Material.

1. Degenerate runs, retrieval rankings, and uncertainty W1–2,Q2W1–2, Q2

The previous LUNA image and REVE text results included failed training runs. We apologize for interpreting those runs as evidence about representation quality. We reran the affected experiments. Appendix D contains the updated scores and subject-level bootstrap summaries. The selected clean-input estimates and intervals below are quoted together from the bootstrap tables, whose captions specify 10,000 subject-level resamples, seed 2000, and 95% percentile intervals. Additional training-seed runs could not be supplied within the rebuttal budget.

Image datasetLUNA CLIP identification [95% CI]Text datasetREVE MTV BLEU-1 [95% CI]
Things EEG0.572
05˙49,06˙000\.549, 0\.600

|ZuCo 1|0.305

03˙00,03˙100\.300, 0\.310

| |NOD-EEG|0.500

05˙00,05˙000\.500, 0\.500

|ZuCo 2|0.323

03˙18,03˙280\.318, 0\.328

| |Grootswagers2019|0.533

05˙09,05˙680\.509, 0\.568

|Chisco|0.087

00˙85,00˙890\.085, 0\.089

| |Grootswagers2025|0.500

05˙00,05˙000\.500, 0\.500

|TMNRED|0.139

01˙34,01˙430\.134, 0\.143

|

LUNA's image identification exceeds the fixed-output reference of 0.500 on Things EEG and Grootswagers2019 and remains at that value on the other two datasets. REVE has nonzero lexical overlap on all four text datasets, while several identification results remain weak.

We agree that small absolute separation from Dummy warrants caution in interpreting retrieval rankings. The maximum absolute differences between an EEG-based model and Dummy are as follows, in percentage points. Section 4 now advises against relying on these four dataset–protocol combinations for retrieval evaluation.

DatasetTop@1 difference (percentage points)Top@5 difference (percentage points)
Grootswagers20250.461.08
Chisco0.110.35
TMNRED1.505.83
NOD-EEG0.100.42

2. Reproducibility, averaging, objectives, and rewritten variants W3–4,Q1,Q4W3–4, Q1, Q4

Please see the Supplementary Material for the benchmark code. We specify the configurations mentioned in the review here:

Retrieval averages predicted embeddings within each subject–target group. The gallery averages target embeddings by target ID over the test rows. Both are L2-normalized before cosine ranking. Accuracy is computed per subject and then averaged with equal subject weights. Dataset-specific averaging of raw EEG is a separate preprocessing step documented in the reconstruction split table.

The implemented reconstruction objectives are image Stage I with MSE and bidirectional CLIP loss, image Stage II with MSE, both video stages with MSE, and text with equally weighted cross entropy and bidirectional CLIP loss. These settings are now consistent across the main text, appendix, and code.

Appendix C documents the eight variants and prompts. Six English variants use Llama-3.1-70B-Instruct and two use Flan-T5-Large. Chinese variants use Qwen2.5-72B-Instruct and mT0-Large. Variants supervise the teacher-forced decoder during training, while the original sentence supplies the alignment target. Test generation is autoregressive. BLEU-1/2 uses the eight variants as references. Other reference-based text metrics use the original stimulus sentence.

3. Reported metrics and Dummy definitions W4–5W4–5

Appendix B now defines the reported metrics, including C.RECALL, DIST-1/2, H.ENT, FD, PSNR, and Hue-pcc. H.ENT measures the entropy of two-token sentence prefixes. Hue-pcc computes hue cosine similarity. Video N-way evaluation accepts the reference classifier's top-three classes. VideoMAE supplies the video classifier. The main-text metric list follows the result tables, with BLEU-3/4 and ROUGE removed from the list of reported metrics.

Section 3.3 defines each input-independent Dummy. Retrieval uses a fixed query equal to the mean training-and-validation target embedding. Images use the pixelwise mean training image. Videos use a fixed output constructed from training-frame pixel averages. Text uses the training sentence closest in cosine similarity to the mean training-sentence embedding.

AuthorsOpenReview ↗

Author Response (Part 2/3)

4. Test-set size and gallery definition W5,Q4W5, Q4

Appendix C now separates retrieval splits from reconstruction splits. Its documented retrieval test-target pools are given below. NOD-EEG uses image-level targets, so 200 classes do not imply a 200-candidate retrieval task. Dummy uses a fixed ranking induced by the mean training-and-validation target embedding; its measured accuracy cannot generally be inferred from gallery size alone, because it also depends on each subject's target coverage.

DatasetDocumented retrieval test-target pool
Things EEG200 concepts
NOD-EEG11,154 images from 200 classes
Grootswagers201940 images
Grootswagers202540 images
SEED-DV280 clips from 8 concepts
Smit202572 clips from 18 original videos
EMD102 videos
ZuCo 1208 sentences
ZuCo 2133 sentences
Chisco1,314 sentences
TMNRED52 sentences

5. MindCine and model count W5W5

MindCine has been removed from the evaluated roster. The final list contains 16 task-specific architectures and three foundation models, as stated in the current Abstract, Introduction and Conclusion.

6. Clean–noisy visualization W5W5

The revised caption defines the display as the mean difference between normalized clean and noisy scores. For dataset dd and model mm, this is

$\Delta_{d,m}=\frac{1}{|\mathcal M_{d,m}|} \sum_{k\in\mathcal M_{d,m}} \left(\widetilde s^{\mathrm{clean}}{d,m,k}-\widetilde s^{\mathrm{noisy}}{d,m,k}\right).$

Here, dd, mm, and kk index datasets, models, and metrics; Md,m\mathcal M_{d,m} is the metric set and ∣Md,m∣|\mathcal M_{d,m}| its size. s~\widetilde s denotes normalized scores, clean/noisy indicate the input condition, and Δd,m\Delta_{d,m} is the mean clean-minus-noisy difference.

All scores are oriented so that higher is better. PSNR, Hue-pcc, H.ENT, and FD use within-dataset empirical bounds from clean scores. The other metrics use theoretical bounds. Clean and noisy values share the bounds.

7. Proofreading W5W5

We corrected the garbled sentence in the results section and removed the template Acknowledgments text.

8. Smit2025 seen-stimulus reconstruction Q3Q3

The reconstruction experiment uses a held-out block containing previously presented video targets. Appendix C now labels it block-disjoint and seen-target. The retrieval experiment separately groups variants by original video and holds out video identities.

For Smit2025 reconstruction, the fixed training-frame Dummy exceeds the evaluated EEG methods on all seven reported metrics. Its Video-40-way, Frame-40-way, SSIM, and PSNR are 0.720, 0.786, 0.768, and 17.202. We recommend video-disjoint splits when the objective is unseen-video generalization.

9. Shared-pipeline bias and pretraining controls Q5Q5

The shared image pipeline uses semantic embedding alignment, a linear adapter, and frozen SDXL-Turbo with IP-Adapter. ATM's full system additionally contains a diffusion adapter and low-level image guidance. The shared video pipeline retains semantic conditioning and omits the source system's frame-latent predictor and explicit dynamic predictor. For text, Unified disables GLIM's task-, dataset-, and subject-specific domain prompts by using null prompt IDs, while retaining its querying aligners and joint LM/contrastive objective. These simplifications reduce the source-specific machinery retained in Unified.

The ordering depends on the metric. On Things EEG, Original ENIGMA exceeds ATM in CLIP identification, 0.853 versus 0.624, while ATM exceeds ENIGMA in SSIM, 0.371 versus 0.260. Unified ATM-Enc. exceeds ENIGMA-Enc. in CLIP, 0.739 versus 0.702, while ENIGMA-Enc. exceeds ATM-Enc. in SSIM, 0.225 versus 0.219. On ZuCo 1, GLIM-Enc. reaches 0.711 two-way identification, compared with LUNA's 0.599, while REVE has higher MTV BLEU-1 than GLIM-Enc., 0.305 versus 0.296. These metric-dependent rankings suggest that the home advantage is limited under the relatively neutral Unified pipelines evaluated here.

AuthorsOpenReview ↗

Author Response (Part 3/3)

10. Gaussian replacement and mismatched EEG Q6Q6

We chose Gaussian replacement as a simple, reproducible negative control, following the recommendation of Jo et al. [1] for dimension-matched noise inputs and non-teacher-forced evaluation. We replace EEG values with independent samples from N(0,1)\mathcal N(0,1), keeping the trained model and remaining inference pipeline fixed.

Test-time mismatched EEG breaks input–target correspondence, analogous at the pairing level to training-time label corruption. Its effects combine responses to unrelated stimuli, cross-stimulus similarity, and generative priors, making interpretation difficult. It therefore cannot serve as a standalone baseline for establishing whether the model has learned stimulus-specific EEG information.

Gaussian replacement changes the input distribution. Accordingly, we interpret the clean–noisy difference as replacement sensitivity; a drop alone does not establish stimulus-specific decoding. Dummy provides a separate, fixed-output reference derived from training targets.

11. NeuralBench and REVE pretraining overlap Q7Q7

The three datasets shared with NeuralBench are THINGS-EEG2, ZuCo 1, and SEED-DV. NeuralBench Figure 4 reports per-subject Top-5 accuracy with the standard error of the mean across three training seeds. Its best-performing foundation models are LaBraM for images and REVE for sentences and videos. Our corresponding results are shown below.

DatasetLaBraM Top@5REVE Top@5LUNA Top@5
Things EEG89.500%81.830%78.280%
ZuCo 17.330%13.770%14.270%
SEED-DV2.850%4.800%2.460%

The leading models agree across benchmarks for image and video retrieval but differ for sentence retrieval. On ZuCo 1, LUNA's lead over REVE is small. REVE's pretraining overlap with Things EEG is noted in the retrieval and reconstruction figure descriptions.

References

[1] Hyejeong Jo, Yiqian Yang, Juhyeok Han, Yiqun Duan, Hui Xiong, and Won Hee Lee. Evaluating EEG-to-text models through noise-based performance analysis. Scientific Reports, 1611:350, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.