- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies multi-entity visual decoding from EEG signals. Existing EEG-to-video work largely assumes a single dominant semantic target per video; the authors instead ask (RQ1) whether multiple co-occurring entity semantics are decodable from EEG, and (RQ2) whether such neural representations support video reconstruction of multi-entity scenes.
The paper introduces EntiNeuro, a controlled dataset of 624 videos spanning eight predefined cross-category compositions over six entity categories (person, dog, bird, flower, car, ball), with synchronized 62-channel EEG and eye-tracking recordings from 20 participants across three sessions. Built on this data, EntiBench evaluates entity recognition (six-class multi-label classification) and video reconstruction (semantic, frame, video, and multi-entity fidelity metrics plus a five-rater human study).
The proposed method, EntiRecon, conditions a frozen pretrained video diffusion model hierarchically: a coarse branch decodes entity composition and multi-label entity presence from EEG to form a semantic anchor (a soft context and a hard entity-level prompt), while a fine branch fuses eye-tracking-guided EEG representations that are injected into intermediate diffusion layers via gated cross-attention.
Validation shows entity recognition above chance under both session-average (Macro AP ~0.40 vs 0.33 chance) and cross-session settings, with early viewing windows and theta/alpha bands most informative. EntiRecon reports the best multi-entity fidelity (Entity F1 0.468 vs 0.322 for the strongest baseline, ECE 0.53 vs 0.835) and favorable human ratings, with ablations supporting the complementary roles of coarse and fine conditioning.
Critical strengths
-
The problem formulation is valuable. The paper asks whether multiple co-occurring entity semantics are simultaneously encoded in EEG responses, a question that existing single-target EEG-to-video work does not address. The temporal and spectral analyses (early-viewing decoding saturation around 400–800 ms, consistent theta/alpha band advantage across all six entity categories) constitute a genuine empirical finding about when and in which bands multi-entity semantics are represented, beyond the reconstruction task itself.
-
The dataset and benchmark provide community value. EntiNeuro's controlled cross-category compositions with synchronized EEG and eye-tracking enable systematic study of a setting that existing datasets (SEED-DV, CineBrain) do not support, and the entity-level evaluation protocol (Entity F1, Entity Count Error) exposes failure modes that aggregate semantic metrics miss. These resources could support follow-up work even if the proposed method itself is superseded.
Critical weaknesses
-
The empirical evidence is weak and rests on a self-constructed, narrow setting. Entity recognition is only marginally above chance in the cross-session setting (Macro AP 0.372 vs 0.333 chance; Table 1), and even session-averaged decoding remains modest (Macro AP ~0.40). All reconstruction claims are established exclusively on synthetic stimuli over six entity categories and eight predefined compositions, generated by a video model and filtered by human raters. Whether the reported decodability transfers to naturalistic multi-entity scenes is untested, and the authors' own three-entity generalization experiments (Appendix G) remain limited. Consequently, both RQ1 and RQ2 are answered only within a closed, six-category world, and the demonstrated gains may reflect overfitting to a synthetic stimulus distribution rather than general neural decoding ability.
-
The method contains little new mechanism. EntiRecon's components are largely standard: a convolutional EEG encoder with query resampling, multi-label classification heads, cross-attention-based multimodal fusion, and gated injection of learned tokens into a frozen pretrained video diffusion model. The coarse semantic bottleneck essentially reduces the task to predicting entity labels from EEG and then performing text-conditioned video generation, with the reconstruction quality heavily dependent on the frozen generative prior rather than on neural conditioning; the fine branch's contribution over the coarse-only variant is limited on most metrics (e.g., 2-way-I 0.815 to 0.830, SSIM 0.211 to 0.222; Table 3). Since the dataset construction is the main novel element but is itself narrow (weakness 1), the paper offers limited methodological advance for ICLR.
Questions for authors
-
Can the authors provide any evidence that the decoded multi-entity representations are not specific to the synthetic stimulus distribution? If decodability collapses outside the six-category closed world, the central claim of the paper would need to be restated accordingly.
-
The reconstructions rely on a frozen pretrained video diffusion model, and the coarse branch essentially supplies entity labels as a text prompt. Could the authors quantify how much of the reported fidelity is attributable to EEG-derived conditioning rather than to the generative prior? The current shuffled-EEG ablation (Table 3) is binary and does not indicate the magnitude of this contribution.
5 comments
(2/2) Response to Reviewer JSRf
W2/Q2: Quantifying the contribution of EEG conditioning.
For the matched-versus-shuffled comparisons, we use the same trained EEG-only Coarse+Fine model and frozen Tora backbone, changing EEG correspondence only at inference.
| Coarse EEG | Fine EEG | 2-way-I ↑ | 2-way-V ↑ | ViFi ↑ | FVD ↓ | Entity F1 ↑ |
|---|---|---|---|---|---|---|
| Matched | None | 0.815 | 0.907 | 0.317 | 2774.27 | 0.427 |
| Matched | Matched | 0.824 | 0.910 | 0.328 | 2219.83 | 0.454 |
| Matched | Global shuffle | 0.800 | 0.891 | 0.290 | 2511.77 | 0.415 |
| Global shuffle | Matched | 0.779 | 0.868 | 0.269 | 2613.18 | 0.353 |
| Global shuffle | Global shuffle | 0.735 | 0.837 | 0.221 | 2300.05 | 0.305 |
| Matched | Within-composition shuffle | 0.810 | 0.905 | 0.305 | 2397.71 | 0.430 |
To further examine the contribution of fine conditioning beyond coarse entity prompts, we evaluate retrieval among the 11 ground-truth videos sharing its entity composition:
| Coarse EEG | Fine EEG | Within-composition R@1 (%) ↑ | Within-composition R@5 (%) ↑ |
|---|---|---|---|
| Matched | None | 10.61 | 52.53 |
| Matched | Matched | 24.24 | 68.69 |
| Matched | Globally shuffled | 8.08 | 47.47 |
| Matched | Shuffled within composition | 9.63 | 50.50 |
This controlled comparison supports the fine branch’s use of stimulus-specific information beyond the entity composition supplied by the coarse branch.
Finally, fine conditioning improves all five reported metrics in each of three split–seed runs. The following paired gains are averaged over all participants within each run and summarized as mean ± SD across runs.
| Comparison | 2-way-V gain ↑ | ViFi gain ↑ | Entity F1 gain ↑ | ECE reduction ↑ | FVD reduction ↑ |
|---|---|---|---|---|---|
| Coarse+Fine vs. Coarse Only | +0.0040 ± 0.0017 | +0.0113 ± 0.0015 | +0.0267 ± 0.0100 | +0.0467 ± 0.0216 | +554.44 ± 53.57 |
W2:Methodological contribution.
Existing methods such as EEG2Video and MindCine predict continuous text embeddings from EEG to condition generation, with EEG2Video predicting a representation. Learning such high-dimensional mappings from limited, noisy EEG data can increase the risk of overfitting. To address this challenge, we propose a coarse-to-fine approach. Our methodological contribution lies in coupling explicit entity-semantic prediction with coarse-guided fine EEG conditioning: the decoded semantics provide a generation prompt, while the fine branch injects stimulus-specific EEG information to refine reconstruction beyond entity-level prompts.
Under the same Tora backbone and EEG-only inputs, Coarse+Fine outperforms direct Text Alignment and all three adapted baselines across all five reported metrics. These results support the effectiveness of our hierarchical conditioning design under a controlled generator and matched input modalities. Together with the branch-specific controls reported above, they demonstrate the benefits of combining explicit entity-semantic anchors with fine EEG conditioning for multi-entity reconstruction.
| Method | 2-way-I ↑ | 2-way-V ↑ | ViFi ↑ | FVD ↓ | ECE ↓ |
|---|---|---|---|---|---|
| Text Alignment | 0.744 ± 0.03 | 0.820 ± 0.02 | 0.184 ± 0.02 | 2910.88 ± 88.87 | 0.855 ± 0.03 |
| EEG2Video + Tora | 0.755 ± 0.02 | 0.820 ± 0.03 | 0.221 ± 0.03 | 2604.46 ± 144.87 | 0.810 ± 0.03 |
| MindCine + Tora | 0.784 ± 0.02 | 0.838 ± 0.02 | 0.258 ± 0.03 | 2718.26 ± 153.09 | 0.768 ± 0.04 |
| DynaMind + Tora | 0.747 ± 0.01 | 0.782 ± 0.01 | 0.100 ± 0.01 | 2875.93 ± 51.91 | 0.809 ± 0.03 |
| Coarse+Fine (ours, EEG-only) | 0.824 ± 0.03 | 0.910 ± 0.02 | 0.328 ± 0.02 | 2219.83 ± 77.63 | 0.561 ± 0.03 |
Reference:
[1] Girdhar R, Ramanan D. Cater: A diagnostic dataset for compositional actions and temporal reasoning[J]. arXiv preprint arXiv:1910.04744, 2019.
[2] Yi K, Gan C, Li Y, et al. Clevrer: Collision events for video representation and reasoning[J]. arXiv preprint arXiv:1910.01442, 2019.
We appreciate your thoughtful review and the opportunity to discuss our work. With the discussion period nearing its end, we would welcome your thoughts on our response and any remaining concerns you would like us to address.
(1/2) Response to Reviewer JSRf
Dear JSRf,
We sincerely thank you for your devoted review. We have listed your concerns point by point and provided our responses. If there is anything we have not addressed sufficiently, please let us know, we would be very happy to have a further discussion with you.
EntiNeuro investigates joint multi-entity EEG decoding and its use in video reconstruction. Guided by prior SEED-DV findings, our pre-collection reproduction experiments showed uneven category-wise decoding despite training on all 40 categories. We therefore selected six categories with comparatively stronger decoding performance to test whether their semantics can be jointly recovered when they co-occur. six balanced two-entity compositions support systematic evaluation, while two additional three-entity compositions are reserved for exploratory transfer testing.
Researchers often use controlled stimuli to study visual perception. For example, CATER[1] and CLEVRER[2] use synthetic videos of moving and interacting geometric objects to study compositional and temporal reasoning. We adopt this approach to specify entity categories and interaction scenarios, balance compositions, and standardize stimulus duration within each setting. Candidate videos were independently evaluated by three raters for entity suitability and interaction naturalness before selection.
To our knowledge, no public natural-video EEG benchmark is specifically designed for controlled multi-entity decoding. As acknowledged in the manuscript’s Limitations section, generalization to naturalistic scenes, larger entity counts, and more complex interactions is a current limitation and an important direction for future work. Nevertheless, EntiNeuro provides synchronized neural recordings and a controlled benchmark for systematically investigating multi-entity semantic decoding and reconstruction from EEG.
To provide further evidence, we added label-permutation controls under both unconstrained Top-2 and six-composition constrained decoding. Our encoder significantly exceeds the corresponding uniform-guessing baseline and matched label-shuffle control in both evaluation protocols. Since Set Accuracy requires both entities to be correctly identified, these gains directly support our conclusion that multi-entity semantic information can be decoded from EEG signals
∗ and † indicate performance above the uniform-guessing baseline (one-sample t-test) and label-shuffle control (permutation test), respectively, matched to each setting and evaluation protocol. Both tests are one-sided with p<0.05.