acceptodds
Under review as a conference paper at ICLR 2027

EntiNeuro: Multi-Entity Visual Decoding and Video Reconstruction from EEG Signals

Abstract

Natural visual scenes often contain multiple semantically distinct entities whose information must be represented simultaneously during perception. However, existing EEG-based visual decoding studies largely focus on a single dominant semantic target or overall scene content, and available EEG–video datasets do not explicitly control multi-entity compositions. Consequently, whether multiple entity semantics can be decoded from EEG and further support video reconstruction remains largely unexplored. To address this gap, we introduce EntiNeuro, a multi-entity visual decoding dataset with 624 dynamic videos across eight predefined cross-category compositions and synchronized EEG and eye-tracking recordings from 20 participants over three sessions. Based on EntiNeuro, we establish EntiBench for evaluating entity recognition and video reconstruction in multi-entity scenes. We further propose EntiRecon, a hierarchical EEG-based reconstruction framework that combines coarse entity semantics with fine-grained EEG conditioning using eye tracking as auxiliary attentional guidance. Experiments consistently show that multi-entity semantic information can be decoded from EEG. EntiRecon further improves semantic and multi-entity fidelity over existing EEG-to-video reconstruction methods. These results demonstrate the potential of EEG representations for reconstructing dynamic scenes containing multiple semantic entities.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain Visual Decoding, EEG, Eye Tracking, Multi-Entity Visual Decoding, Video Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JSRfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-entity visual decoding from EEG signals. Existing EEG-to-video work largely assumes a single dominant semantic target per video; the authors instead ask (RQ1) whether multiple co-occurring entity semantics are decodable from EEG, and (RQ2) whether such neural representations support video reconstruction of multi-entity scenes.

The paper introduces EntiNeuro, a controlled dataset of 624 videos spanning eight predefined cross-category compositions over six entity categories (person, dog, bird, flower, car, ball), with synchronized 62-channel EEG and eye-tracking recordings from 20 participants across three sessions. Built on this data, EntiBench evaluates entity recognition (six-class multi-label classification) and video reconstruction (semantic, frame, video, and multi-entity fidelity metrics plus a five-rater human study).

The proposed method, EntiRecon, conditions a frozen pretrained video diffusion model hierarchically: a coarse branch decodes entity composition and multi-label entity presence from EEG to form a semantic anchor (a soft context and a hard entity-level prompt), while a fine branch fuses eye-tracking-guided EEG representations that are injected into intermediate diffusion layers via gated cross-attention.

Validation shows entity recognition above chance under both session-average (Macro AP ~0.40 vs 0.33 chance) and cross-session settings, with early viewing windows and theta/alpha bands most informative. EntiRecon reports the best multi-entity fidelity (Entity F1 0.468 vs 0.322 for the strongest baseline, ECE 0.53 vs 0.835) and favorable human ratings, with ablations supporting the complementary roles of coarse and fine conditioning.

Critical strengths
  1. The problem formulation is valuable. The paper asks whether multiple co-occurring entity semantics are simultaneously encoded in EEG responses, a question that existing single-target EEG-to-video work does not address. The temporal and spectral analyses (early-viewing decoding saturation around 400–800 ms, consistent theta/alpha band advantage across all six entity categories) constitute a genuine empirical finding about when and in which bands multi-entity semantics are represented, beyond the reconstruction task itself.

  2. The dataset and benchmark provide community value. EntiNeuro's controlled cross-category compositions with synchronized EEG and eye-tracking enable systematic study of a setting that existing datasets (SEED-DV, CineBrain) do not support, and the entity-level evaluation protocol (Entity F1, Entity Count Error) exposes failure modes that aggregate semantic metrics miss. These resources could support follow-up work even if the proposed method itself is superseded.

Critical weaknesses
  1. The empirical evidence is weak and rests on a self-constructed, narrow setting. Entity recognition is only marginally above chance in the cross-session setting (Macro AP 0.372 vs 0.333 chance; Table 1), and even session-averaged decoding remains modest (Macro AP ~0.40). All reconstruction claims are established exclusively on synthetic stimuli over six entity categories and eight predefined compositions, generated by a video model and filtered by human raters. Whether the reported decodability transfers to naturalistic multi-entity scenes is untested, and the authors' own three-entity generalization experiments (Appendix G) remain limited. Consequently, both RQ1 and RQ2 are answered only within a closed, six-category world, and the demonstrated gains may reflect overfitting to a synthetic stimulus distribution rather than general neural decoding ability.

  2. The method contains little new mechanism. EntiRecon's components are largely standard: a convolutional EEG encoder with query resampling, multi-label classification heads, cross-attention-based multimodal fusion, and gated injection of learned tokens into a frozen pretrained video diffusion model. The coarse semantic bottleneck essentially reduces the task to predicting entity labels from EEG and then performing text-conditioned video generation, with the reconstruction quality heavily dependent on the frozen generative prior rather than on neural conditioning; the fine branch's contribution over the coarse-only variant is limited on most metrics (e.g., 2-way-I 0.815 to 0.830, SSIM 0.211 to 0.222; Table 3). Since the dataset construction is the main novel element but is itself narrow (weakness 1), the paper offers limited methodological advance for ICLR.

Questions for authors
  1. Can the authors provide any evidence that the decoded multi-entity representations are not specific to the synthetic stimulus distribution? If decodability collapses outside the six-category closed world, the central claim of the paper would need to be restated accordingly.

  2. The reconstructions rely on a frozen pretrained video diffusion model, and the coarse branch essentially supplies entity labels as a text prompt. Could the authors quantify how much of the reported fidelity is attributable to EEG-derived conditioning rather than to the generative prior? The current shuffled-EEG ablation (Table 3) is binary and does not indicate the magnitude of this contribution.

5 comments
AuthorsOpenReview ↗

(1/2) Response to Reviewer JSRf

Dear JSRf,

We sincerely thank you for your devoted review. We have listed your concerns point by point and provided our responses. If there is anything we have not addressed sufficiently, please let us know, we would be very happy to have a further discussion with you.

W1/Q1: Controlled synthetic stimuli and generalization.

EntiNeuro investigates joint multi-entity EEG decoding and its use in video reconstruction. Guided by prior SEED-DV findings, our pre-collection reproduction experiments showed uneven category-wise decoding despite training on all 40 categories. We therefore selected six categories with comparatively stronger decoding performance to test whether their semantics can be jointly recovered when they co-occur. six balanced two-entity compositions support systematic evaluation, while two additional three-entity compositions are reserved for exploratory transfer testing.

W1: Controlled synthetic stimuli and generalization.

Researchers often use controlled stimuli to study visual perception. For example, CATER[1] and CLEVRER[2] use synthetic videos of moving and interacting geometric objects to study compositional and temporal reasoning. We adopt this approach to specify entity categories and interaction scenarios, balance compositions, and standardize stimulus duration within each setting. Candidate videos were independently evaluated by three raters for entity suitability and interaction naturalness before selection.

To our knowledge, no public natural-video EEG benchmark is specifically designed for controlled multi-entity decoding. As acknowledged in the manuscript’s Limitations section, generalization to naturalistic scenes, larger entity counts, and more complex interactions is a current limitation and an important direction for future work. Nevertheless, EntiNeuro provides synchronized neural recordings and a controlled benchmark for systematically investigating multi-entity semantic decoding and reconstruction from EEG.

W1. Evidence for multi-entity EEG decoding.

To provide further evidence, we added label-permutation controls under both unconstrained Top-2 and six-composition constrained decoding. Our encoder significantly exceeds the corresponding uniform-guessing baseline and matched label-shuffle control in both evaluation protocols. Since Set Accuracy requires both entities to be correctly identified, these gains directly support our conclusion that multi-entity semantic information can be decoded from EEG signals

MethodSettingSession-average: Set Accuracy ↑Session-average: Jaccard ↑Cross-session: Set Accuracy ↑Cross-session: Jaccard ↑
Uniform guessingAll 15 combinations0.0670.2440.0670.244
Label-shuffleUnconstrained Top-2 decoding0.140 ± 0.0050.252 ± 0.0030.137 ± 0.0070.250 ± 0.003
Our encoderUnconstrained Top-2 decoding0.173 ± 0.037$^{*\dagger}$0.311 ± 0.031$^{*\dagger}$0.166 ± 0.025$^{*\dagger}$0.300 ± 0.021$^{*\dagger}$
Uniform guessingSix valid combinations0.1670.2780.1670.278
Label-shuffleSix-combination constrained decoding0.166 ± 0.0030.278 ± 0.0030.167 ± 0.0020.278 ± 0.002
Our encoderSix-combination constrained decoding0.227 ± 0.028$^{*\dagger}$0.344 ± 0.037$^{*\dagger}$0.222 ± 0.029$^{*\dagger}$0.338 ± 0.034$^{*\dagger}$

∗^{*} and †^{\dagger} indicate performance above the uniform-guessing baseline (one-sample t-test) and label-shuffle control (permutation test), respectively, matched to each setting and evaluation protocol. Both tests are one-sided with p<0.05p<0.05.

AuthorsOpenReview ↗

(2/2) Response to Reviewer JSRf

W2/Q2: Quantifying the contribution of EEG conditioning.

For the matched-versus-shuffled comparisons, we use the same trained EEG-only Coarse+Fine model and frozen Tora backbone, changing EEG correspondence only at inference.

Coarse EEGFine EEG2-way-I ↑2-way-V ↑ViFi ↑FVD ↓Entity F1 ↑
MatchedNone0.8150.9070.3172774.270.427
MatchedMatched0.8240.9100.3282219.830.454
MatchedGlobal shuffle0.8000.8910.2902511.770.415
Global shuffleMatched0.7790.8680.2692613.180.353
Global shuffleGlobal shuffle0.7350.8370.2212300.050.305
MatchedWithin-composition shuffle0.8100.9050.3052397.710.430

To further examine the contribution of fine conditioning beyond coarse entity prompts, we evaluate retrieval among the 11 ground-truth videos sharing its entity composition:

Coarse EEGFine EEGWithin-composition R@1 (%) ↑Within-composition R@5 (%) ↑
MatchedNone10.6152.53
MatchedMatched24.2468.69
MatchedGlobally shuffled8.0847.47
MatchedShuffled within composition9.6350.50

This controlled comparison supports the fine branch’s use of stimulus-specific information beyond the entity composition supplied by the coarse branch.

Finally, fine conditioning improves all five reported metrics in each of three split–seed runs. The following paired gains are averaged over all participants within each run and summarized as mean ± SD across runs.

Comparison2-way-V gain ↑ViFi gain ↑Entity F1 gain ↑ECE reduction ↑FVD reduction ↑
Coarse+Fine vs. Coarse Only+0.0040 ± 0.0017+0.0113 ± 0.0015+0.0267 ± 0.0100+0.0467 ± 0.0216+554.44 ± 53.57

W2:Methodological contribution.

Existing methods such as EEG2Video and MindCine predict continuous text embeddings from EEG to condition generation, with EEG2Video predicting a (77×768)(77\times768) representation. Learning such high-dimensional mappings from limited, noisy EEG data can increase the risk of overfitting. To address this challenge, we propose a coarse-to-fine approach. Our methodological contribution lies in coupling explicit entity-semantic prediction with coarse-guided fine EEG conditioning: the decoded semantics provide a generation prompt, while the fine branch injects stimulus-specific EEG information to refine reconstruction beyond entity-level prompts.

Under the same Tora backbone and EEG-only inputs, Coarse+Fine outperforms direct Text Alignment and all three adapted baselines across all five reported metrics. These results support the effectiveness of our hierarchical conditioning design under a controlled generator and matched input modalities. Together with the branch-specific controls reported above, they demonstrate the benefits of combining explicit entity-semantic anchors with fine EEG conditioning for multi-entity reconstruction.

Method2-way-I ↑2-way-V ↑ViFi ↑FVD ↓ECE ↓
Text Alignment0.744 ± 0.030.820 ± 0.020.184 ± 0.022910.88 ± 88.870.855 ± 0.03
EEG2Video + Tora0.755 ± 0.020.820 ± 0.030.221 ± 0.032604.46 ± 144.870.810 ± 0.03
MindCine + Tora0.784 ± 0.020.838 ± 0.020.258 ± 0.032718.26 ± 153.090.768 ± 0.04
DynaMind + Tora0.747 ± 0.010.782 ± 0.010.100 ± 0.012875.93 ± 51.910.809 ± 0.03
Coarse+Fine (ours, EEG-only)0.824 ± 0.030.910 ± 0.020.328 ± 0.022219.83 ± 77.630.561 ± 0.03

Reference:
[1] Girdhar R, Ramanan D. Cater: A diagnostic dataset for compositional actions and temporal reasoning[J]. arXiv preprint arXiv:1910.04744, 2019.
[2] Yi K, Gan C, Li Y, et al. Clevrer: Collision events for video representation and reasoning[J]. arXiv preprint arXiv:1910.01442, 2019.

We appreciate your thoughtful review and the opportunity to discuss our work. With the discussion period nearing its end, we would welcome your thoughts on our response and any remaining concerns you would like us to address.

Reviewer_qgQ1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EntiNeuro, a dataset of 624 multi-entity videos with synchronized EEG and eye-tracking recordings from 20 participants across three sessions, and EntiBench for entity recognition and video reconstruction. Its reconstruction framework, EntiRecon, combines coarse entity semantics with eye-guided neural conditioning for a frozen video diffusion model. Experiments report improvements over existing EEG-to-video methods and include component ablations and three-entity generalization tests.

Critical strengths
  1. Useful dataset and problem formulation. Synchronized EEG and eye tracking, repeated sessions, and controlled entity compositions provide a valuable resource for studying multi-entity decoding beyond a single dominant category.
  2. Promising reconstruction framework. The coarse-to-fine conditioning design is well motivated. Within-backbone ablations and human evaluations provide encouraging evidence for its utility, particularly in preserving entity composition.
Critical weaknesses
  1. Performance gains are insufficiently isolated. EntiRecon uses Tora, while the baseline adaptations do not establish a controlled comparison with the same generation backbone and checkpoint. The full model also receives eye tracking unavailable to the EEG-only baselines. Consequently, the main comparison cannot clearly separate methodological gains from generative priors and additional inputs.

  2. Limited stimulus diversity and generalization evidence. The main benchmark contains only six categories and six two-entity compositions, with 402 training and 66 test videos. Repeated recordings increase neural observations but not independent stimulus diversity. Evaluation remains restricted to synthetic videos; cross-session recognition reuses the same stimuli, and three-entity testing covers only two compositions. The small reconstruction test set also warrants uncertainty estimates, particularly for FVD.

  3. The exact-set chance baseline overlooks composition priors. Table 1 reports 6.67% chance accuracy over all 15 possible pairs, although only six balanced compositions occur. Uniform guessing among these six achieves 16.67% without EEG, compared with the proposed encoder’s 17.3% session-average and 16.6% cross-session accuracy. Other metrics provide some positive evidence, but composition-aware baselines and permutation tests are needed to substantiate reliable joint-entity decoding.

  4. Stimulus-specific reconstruction is not convincingly isolated. Coarse-only conditioning already achieves 0.907 in 2-way-V versus 0.915 for the full model. Random distractors and global EEG shuffling do not distinguish coarse category recovery from fine-grained reconstruction. Moreover, Warp-LPIPS measures temporal self-consistency rather than agreement with ground-truth motion, limiting conclusions about recovered dynamics.

Questions for authors
  1. Which exact generation checkpoints were used for each baseline? Can the advantage be demonstrated with a shared Tora backbone and matched input modalities?
  2. How do recognition results compare against composition-aware guessing and label-permutation controls?
  3. Does fine conditioning improve retrieval among videos with identical entity compositions, and does that improvement disappear when EEG is shuffled within the same composition while keeping the coarse prompt fixed?
  4. How stable are reconstruction results across test splits and generation seeds, particularly FVD and the smaller ablation gains?
7 comments
AuthorsOpenReview ↗

(1/3) Response to Reviewer qgQ1

Dear Reviewer qgQ1,

We sincerely thank you for your devoted review. We have listed your concerns point by point and provided our responses. If there is anything we have not addressed sufficiently, please let us know, we would be very happy to have a further discussion with you.

W1/Q1. Fair comparison under a shared generation backbone and matched input modalities.

In our original experiments, EEG2Video, MindCine, and DynaMind used Tune-A-Video initialized from Stable Diffusion v1.4. EntiRecon used the released CogVideoX-5B-based Tora T2V checkpoint, loaded through the SAT implementation from tora/t2v/mp_rank_00_model_states.pt.

To address this concern, we additionally adapted all three baselines to the same Tora checkpoint used by EntiRecon. We compare them with our EEG-only Coarse+Fine variant, excluding eye-tracking inputs from this comparison. We report mean ± standard deviation across three seeded runs, after averaging participant-level scores within each run.

Method2-way-I ↑2-way-V ↑ViFi ↑FVD ↓ECE ↓
EEG2Video + Tora0.755 ± 0.020.820 ± 0.030.221 ± 0.032604.46 ± 144.870.810 ± 0.03
MindCine + Tora0.784 ± 0.020.838 ± 0.020.258 ± 0.032718.26 ± 153.090.768 ± 0.04
DynaMind + Tora0.747 ± 0.010.782 ± 0.010.100 ± 0.012875.93 ± 51.910.809 ± 0.03
Coarse+Fine (ours, EEG-only)0.824 ± 0.030.910 ± 0.020.328 ± 0.022219.83 ± 77.630.561 ± 0.03

With the shared backbone and matched input modalities, Coarse+Fine achieves the best mean performance. These results show that the advantages persist when the generation checkpoint is shared and eye-tracking inputs are excluded. We have added the complete results to Section 4.3.1 of the revised manuscript.

W2. Stimulus design and generalization.

  • Dataset purpose and category selection. Our dataset is designed to investigate whether multiple entity categories can be jointly decoded from EEG and how the decoded information can support video reconstruction. Prior SEED-DV[1] results highlight the difficulty of fine-grained video-category decoding, while our pre-collection experiments on the full 40-category dataset further showed uneven decoding performance across categories. These findings guided our selection of six categories, organized into six balanced two-entity compositions, to systematically examine whether information about co-occurring entities can be recovered from EEG. We hope this benchmark will support future research on EEG-based visual decoding of scenes with more entities and more complex interactions.
  • Controlled video stimuli. Synthetic videos have been used to construct controlled stimuli for testing specific hypotheses in vision research[2,3]. In our benchmark, synthetic generation allows us to specify entity compositions, duration, and intended interactions while varying appearances and scenes. Human screening verifies that the selected videos meet these design requirements. Our current evaluation focuses on controlled synthetic videos; extending it to natural videos with more complex multi-entity interactions is included among the future directions in Section 5.
  • Repeated recordings and generalization. Repeated recordings enable session averaging to improve the reliability of stimulus-related EEG responses, consistent with repetition averaging in THINGS-EEG2[4]. Session-average recognition uses video-disjoint cross-validation, and reconstruction uses a separate video-disjoint train/test split to assess generalization to unseen videos, while cross-session recognition assesses robustness across recording sessions using the same stimuli.
  • Transfer to three entities. Our main study systematically investigates EEG-based decoding and reconstruction of two-entity scenes. The three-entity experiment explores whether these findings extend to scenes containing more entities. We evaluate models trained exclusively on two-entity videos on three-entity scenes without fine-tuning. Further increasing the number of entities and studying more complex interactions are directions for future work.
AuthorsOpenReview ↗

(2/3) Response to Reviewer qgQ1

W3/Q2. Composition-aware evaluation and label-permutation controls.

Unconstrained Top-2 decoding. Top-2 decoding selects the two highest-scoring entities, allowing all 15 pairs. We construct label-shuffle controls by permuting video-level entity-pair labels and retraining the encoder. Using the original EEG–label correspondence yields higher Set Accuracy than these controls in both session-average and cross-session evaluation.

Composition-aware evaluation. Using the same multi-label encoder without retraining, we select the highest-scoring pair among the six valid pairs. Under the same constraint, label-shuffle controls yield 16.6%/16.7% Set Accuracy, close to the six-composition chance level of 16.67%. Our encoder achieves 22.7%/22.2%, significantly exceeding this chance level in both protocols (one-sided one-sample t-tests, p<0.05p<0.05).

These results support our main conclusion that multi-entity visual information can be decoded from EEG, with performance exceeding composition-aware guessing even when the six-composition prior is explicitly incorporated.

MethodSettingSession-average: Set Accuracy ↑Session-average: Jaccard ↑Cross-session: Set Accuracy ↑Cross-session: Jaccard ↑
Uniform guessingAll 15 combinations0.0670.2440.0670.244
Label-shuffleUnconstrained Top-2 decoding0.140 ± 0.0050.252 ± 0.0030.137 ± 0.0070.250 ± 0.003
Our encoderUnconstrained Top-2 decoding0.173 ± 0.037$^{*\dagger}$0.311 ± 0.031$^{*\dagger}$0.166 ± 0.025$^{*\dagger}$0.300 ± 0.021$^{*\dagger}$
Uniform guessingSix valid combinations0.1670.2780.1670.278
Label-shuffleSix-combination constrained decoding0.166 ± 0.0030.278 ± 0.0030.167 ± 0.0020.278 ± 0.002
Our encoderSix-combination constrained decoding0.227 ± 0.028$^{*\dagger}$0.344 ± 0.037$^{*\dagger}$0.222 ± 0.029$^{*\dagger}$0.338 ± 0.034$^{*\dagger}$

∗^{*} and †^{\dagger} indicate performance above the uniform-guessing baseline (one-sample t-test) and label-shuffle control (permutation test), respectively, matched to each setting and evaluation protocol. Both tests are one-sided with p<0.05p<0.05.

W4/Q3. Fine conditioning beyond entity composition.

The 2-way metrics assess class-level semantic consistency but may not fully capture differences between videos sharing the same entity composition. To test whether fine conditioning contributes stimulus-specific information beyond the coarse prompt, we shuffle fine EEG globally or within the same composition at inference. We compare these controls with matched fine EEG and coarse-only conditioning, evaluating both reconstruction quality and within-composition retrieval. In the fine-shuffle comparisons, the target coarse prompt is kept fixed

Coarse EEGFine EEG2-way-I ↑2-way-V ↑ViFi ↑FVD ↓Entity F1 ↑
MatchedNone0.8150.9070.3172774.270.427
MatchedMatched0.8240.9100.3282219.830.454
MatchedGlobal shuffle0.8000.8910.2902511.770.415
Global shuffleMatched0.7790.8680.2692613.180.353
Global shuffleGlobal shuffle0.7350.8370.2212300.050.305
MatchedWithin-composition shuffle0.8100.9050.3052397.710.430
Coarse EEGFine EEGWithin-composition R@1 (%) ↑Within-composition R@5 (%) ↑
MatchedNone10.6152.53
MatchedMatched24.2468.69
MatchedGlobally shuffled8.0847.47
MatchedShuffled within composition9.6350.50

These results support the contribution of fine conditioning to recovering stimulus-specific information beyond the entity composition supplied by the coarse branch.

AuthorsOpenReview ↗

(3/3) Response to Reviewer qgQ1

Q4. Reconstruction stability across splits and seeds.

We clarify that the training–test split is regenerated for each of the three seeds. Each split is composition-balanced at the video level, with 402 training and 66 test videos. Models are retrained for each split, with all methods using identical training and test video IDs within each run. Metrics are averaged over all participants.

The table reports paired gains averaged over 20 participants within each run, summarized as mean ± SD across three split–seed runs. Fine EEG conditioning improves all five metrics over Coarse Only in every run, with higher Entity F1 and lower ECE indicating better multi-entity fidelity. Eye-tracking yields further consistent gains.

Comparison2-way-V gain ↑ViFi gain ↑Entity F1 gain ↑ECE reduction ↑FVD reduction ↑
Coarse+Fine vs. Coarse Only+0.0040 ± 0.0017+0.0113 ± 0.0015+0.0267 ± 0.0100+0.0467 ± 0.0216+554.44 ± 53.57
Full vs. Coarse+Fine+0.0057 ± 0.0025+0.0110 ± 0.0017+0.0150 ± 0.0020+0.0313 ± 0.0225+213.84 ± 97.62

We assessed FVD uncertainty using paired bootstrap resampling of test videos within one seeded run. FVD was recomputed for each participant in every resample, then averaged across participants before calculating the reductions. Both 95% confidence intervals lie above zero, supporting positive average FVD reductions from fine EEG conditioning and eye-tracking guidance.

ComparisonMean FVD reduction95% paired bootstrap CI
Coarse Only → Coarse+Fine494.54[157.45, 949.94]
Coarse+Fine → Full304.23[114.79, 776.89]

W4: Interpretation of Warp-LPIPS.

We clarify that Warp-LPIPS is used to assess temporal coherence. Appendix D.2.2 explicitly defines it as self-consistency within each reconstructed video, without reference to the corresponding ground-truth video. We do not interpret it as evidence of recovering ground-truth motion. Section 5 already acknowledges that stimulus-specific motion is not explicitly modeled; our focus is multi-entity semantic decoding and reconstruction from EEG.

Reference:

[1] Liu X H, Liu Y K, Wang Y, et al. Eeg2video: Towards decoding dynamic visual perception from eeg signals[J]. Advances in Neural Information Processing Systems, 2024, 37: 72245-72273.
[2] Greco A, Siegel M. A spatiotemporal style transfer algorithm for dynamic visual stimulus generation[J]. Nature Computational Science, 2025, 5(2): 155-169.
[3] Davodi C E, Barrington S, Farid H, et al. Perceptual Judgments of Video Authenticity: An Examination of Viewing Duration, Confidence, Content, and Strategies[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 10679-10686.
[4] Gifford, A. T., Dwivedi, K., Roig, G., & Cichy, R. M. A large and rich EEG dataset for modeling human visual object recognition. NeuroImage, 264, 119754, 2022.

Reviewer_xt2YOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EntiNeuro, synchronized EEG and eye-tracking recordings from 20 participants viewing 624 generated videos across three sessions. The main EntiBench tasks use 468 two-entity videos from six predefined compositions; 156 three-entity videos support an extension. EntiRecon decodes entity probabilities from repeated EEG responses to select an entity-level prompt and combines coarse semantic and fine neural conditioning for a frozen video diffusion model, with eye tracking used as auxiliary input in the full system. Entity recognition exceeds the reported random-reference levels. Importantly, the no-eye Coarse + Fine reconstruction already outperforms the three EEG baselines on all six shared metrics in Tables 2-3, and shuffling EEG in that no-eye system reduces semantic and entity fidelity. These results support an EEG contribution within the controlled setting.

Critical strengths
  1. A valuable controlled multimodal resource. EntiNeuro records 62-channel EEG and 250-Hz eye tracking from 20 participants viewing 624 videos over three sessions. The combination of repeated recordings and explicit multi-entity labels offers a useful resource independently of the reconstruction method.
  2. Meaningful evidence for EEG-conditioned reconstruction. The controls support dependence on stimulus-linked EEG and materially strengthen the paper beyond a full multimodal comparison alone.
Critical weaknesses
  1. The incremental fine EEG contribution is not yet isolated. Coarse Only is already an EEG-decoded prompt-only control, reaching ViFi 0.317 and 2-way-V 0.907, versus 0.328 and 0.910 for no-eye Coarse + Fine.
  2. The decoding reference and uncertainty reporting need stronger calibration. A composition-aware empirical null would clarify the claim. The five-rater human study discloses method blinding and randomized order, but omits the number and sampling of rated cases, rater coverage, and aggregation across subject-video cases.
Questions for authors
  1. How many subject-video reconstructions were included in the human evaluation, how were they sampled and assigned to each rater, and were all methods rated on matched cases? Please explain aggregation over cases and raters, and report paired uncertainty and, if available, rater agreement.
  2. Can you report the complete no-eye Coarse + Fine results alongside the EEG baselines, and clarify whether EEG correspondence was shuffled during training or inference, across or within compositions, and jointly across repeated sessions?
  3. I suggest citing these two papers, "Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding" and “PRE-MAP: Personalized Reinforced Eye-tracking Multimodal LLM for High-Resolution Multi-Attribute Point Prediction“, and analyzing the connection among these three works.
3 comments
AuthorsOpenReview ↗

Response to Reviewer xt2Y

Dear Reviewer xt2Y,

We sincerely thank you for your devoted review. We have listed your concerns point by point and provided our responses. If there is anything we have not addressed sufficiently, please let us know, we would be very happy to have a further discussion with you.

Q1: Details of human evaluation.

Human evaluation included 594 reconstructions per method: 66 test videos × three participants (subjects 01–03, selected by ID) × three seeded runs. Five raters were randomly assigned two of the nine participant–seed groups each, ensuring coverage of all groups. Each rater evaluated all methods on the same participant–video–seed cases. Model identities were hidden and presentation order randomized. Each trial showed the ground-truth video alongside one reconstruction. Scores were averaged within each rater and then equally across the five raters.

For the two raters with overlapping assignments, we additionally report paired differences (EntiRecon − MindCine) using all 264 matched rating pairs per criterion and the same averaging procedure. The 95% CIs use 10,000 bootstrap resamples of video IDs, retaining all associated ratings; they quantify video-sampling uncertainty within this subset. Linear-weighted Cohen’s κ is computed separately for each method on the 66 reconstructions rated by both raters (subject 01, seed 2025). The positive κ\kappa estimates suggest the agreement between the two raters.

Criterion (EntiRecon-MindCine)Paired difference ↑95% bootstrap CI
Entity Consistency+1.008[0.890, 1.129]
Temporal Coherence+0.659[0.530, 0.788]
Overall Similarity+1.117[1.023, 1.216]
MethodEntity Consistency κTemporal Coherence κOverall Similarity κ
MindCine0.2010.0570.219
EntiRecon0.1970.0770.242

W1/Q2: Fine EEG contribution.

The complete no-eye Coarse+Fine results alongside the EEG baselines are provided in Table 3 of the revised manuscript. To isolate the contribution of fine EEG, we include coarse-only conditioning, global shuffling of either or both EEG branches, and within-composition fine-EEG shuffling with the coarse prompt fixed. Reconstruction metrics are summarized below.

Training uses matched EEG–video correspondence; shuffling is applied only at inference. Repeated sessions are encoded independently, and their representations are averaged per video before permutation, so sessions are not shuffled independently. When both branches are shuffled, they use the same video permutation.

Coarse EEGFine EEG2-way-I ↑2-way-V ↑ViFi ↑FVD ↓Entity F1 ↑
MatchedNone0.8150.9070.3172774.270.427
MatchedMatched0.8240.9100.3282219.830.454
MatchedGlobal shuffle0.8000.8910.2902511.770.415
Global shuffleMatched0.7790.8680.2692613.180.353
Global shuffleGlobal shuffle0.7350.8370.2212300.050.305
MatchedWithin-composition shuffle0.8100.9050.3052397.710.430

Across participants and seeds, these controlled comparisons support an additional contribution from correctly paired fine EEG beyond the EEG-decoded coarse prompt.

W2: Composition-aware empirical null

We additionally provide a composition-aware label-permutation control in App. E.3. Under six-combination constrained decoding, these controls achieve 16.6%/16.7% Set Accuracy (session-average/cross-session), close to the 16.67% uniform-guessing baseline. With the same decoding constraint applied at inference and no retraining, our multi-label encoder significantly exceeds this chance level in both protocols (one-sided one-sample t-tests, p<0.05p<0.05).

Q3: Relation to prior work.

Thank you for highlighting these relevant studies. Both connect video content and viewer attributes with human responses and attention. HMLLM uses EEG- and eye-tracking-derived indicators to supervise audience-response prediction, while PRE-MAP uses fixation annotations to supervise gaze prediction. Our work studies the complementary direction: decoding multi-entity visual semantics from recorded EEG and reconstructing the viewed video with auxiliary guidance from measured eye tracking. Thus, the recorded signals serve as decoding inputs in our setting.

We have cited both works and added a dedicated subsection to Appendix A (Related Work), discussing them alongside closely related studies and clarifying our focus on controlled multi-entity neural decoding and video reconstruction.

The authors have fully addressed my comments. I have no further comments.

Thank you for your careful review and constructive suggestions. We are pleased that our responses have fully addressed your concerns, and we appreciate your time and engagement throughout the discussion.

Comments on the paper

5 comments
AuthorsOpenReview ↗

A Step Forward in Multi-Entity Visual Decoding from EEG

We thank all the reviewers for your proficient and valuable comments and suggestions. We are cheerful to find that all the reviewers have reached the consensus that our dataset are valuable to the research community. Moreover, we're also glad to see that our problem formulation and benchmark(qgQ1,JSRf), the coarse-to-fine reconstruction design (qgQ1) , the evidence for EEG-conditioned reconstruction (xt2Y) are well recognized, and the paper is thought to be clearly presented (qgQ1,xt2Y).

Here, we sincerely invite all the reviewers to read this general response before diving into the detailed responses to your individual concerns. To help readers to logically understand our research, we explain our thinking on what the actual problem is, how we address it step by step, and what we contribute, which we hope will help resolve concerns about our dataset design and the contribution of our reconstruction framework.

  1. The motivation for multi-entity visual decoding. Standing at the intersection of neuroscience and AI, we pursue the long-term goal of recovering what people see from neural signals, with the hope of contributing to future assistive communication technologies. Toward this goal, our study uses EEG for its higher temporal resolution and lower latency than other brain signals like fMRI, which make it suitable for studying rapidly changing visual experiences. The everyday visual experiences we seek to recover often involve multiple entities and their interactions within the same scene. Progress toward this goal therefore requires addressing a basic question: can semantic information about multiple co-occurring entities be recovered from the same EEG response? However, existing resources, SEED-DV and CineBrain, were not specifically designed for multi-entity decoding and do not provide a dedicated experimental setting for this question. (By the way, this adds to the value of our dataset and we're glad to see that most reviewers can recognize it). Systematically investigating it therefore requires stimuli with explicitly defined entity compositions and corresponding neural recordings. EntiNeuro fills this gap by providing a dedicated experimental basis for examining whether multi-entity semantics can be decoded from EEG and how the decoded information can support reconstruction of the viewed videos.
  2. The construction of EntiNeuro. With this question in mind, we first had to decide which categories to study. Prior SEED-DV findings and our reproduction experiments on its full 40-category dataset showed that some categories were more decodable than others. This gave us a starting point: we chose six relatively decodable categories to examine whether their semantics could still be recovered when they appeared together. This helps us avoid concluding that multiple entities cannot be decoded together from EEG simply because a particular category is difficult to decode. We organized these categories into eight compositions: six two-entity compositions for the main study and two three-entity compositions for an exploratory transfer test. The videos needed to contain the specified entities, show plausible interactions, have consistent durations within each setting, and provide balanced examples across compositions. Finding enough natural clips that met all these requirements was difficult, so we generated the videos synthetically. We consider this an appropriate experimental choice, as researchers often use controlled stimuli to study visual perception. This approach also has precedents in visual benchmarks such as CATER and CLEVRER, which use synthetic scenes to study object movements, compositions, and interactions.
  3. The design of EntiRecon. Having established EntiBench to evaluate entity recognition and video reconstruction, we next ask how the multi-entity information decoded from EEG can support reconstruction of the viewed videos. Existing methods such as EEG2Video and MindCine construct semantic generation conditions by predicting continuous text embeddings from EEG. Given the low SNR ratio of EEG and limited paired data, learning this high-dimensional mapping can increase the risk of overfitting. We therefore take a coarse-to-fine approach. The coarse branch first decodes entity probabilities and constructs an entity-level prompt and soft semantic context. The fine branch then refines EEG tokens using this coarse context and injects them into intermediate diffusion features, supplying additional neural information beyond the entity-level prompt. The design connects explicit entity semantics with finer EEG information for reconstruction. Our experiments test this design directly: the advantage persists with a shared Tora checkpoint and matched EEG-only inputs, while shuffling fine EEG within the same composition reduces within-composition retrieval performance. These results support the contribution of stimulus-specific EEG conditioning to reconstruction.
AuthorsOpenReview ↗

A Step Forward in Multi-Entity Visual Decoding from EEG

Now we have stated the coherent logic among our contributions in collecting the dataset, building the benchmarks, and designing the EntiRecon framework. Meanwhile, we still value all the suggestions from the reviewers which definitely help we improve the paper quality. Therefore, we make the following modifications to the paper accordingly:

  1. Added shared-backbone comparisons and fine EEG ablations (Sections 4.3.1–4.3.2).
  2. Expanded the related-work discussion (Appendix A.3).
  3. Expanded human-evaluation details and statistical analyses (Appendix D.3).
  4. Added composition-aware baselines and label-permutation controls (Appendix E.3).
  5. Added reconstruction stability and uncertainty analyses (Appendix F).
  6. We will add the above discussion in the publicly available version, or a link to this page to help the readers better understand our EntiNeuro.

Finally, we would like to express our sincere appreciation for your recognition of the value of our dataset and benchmark to the research community. We see this work as a starting point for extending EEG-based visual decoding to scenes containing multiple entities. Our work provides resources and empirical evidence for understanding whether multi-entity semantic information can be decoded from EEG and used to reconstruct viewed videos. We hold the deepest belief that with our dataset, our benchmarks, and our framework, the publication of EntiNeuro would open new possibilities to push the area moving forward.

AuthorsOpenReview ↗

(1/3) Summary to Area Chairs

Dear AC, SAC, and PC,

We sincerely thank you and all the reviewers for your careful, thorough, and insightful evaluation of our paper, as well as for the many valuable comments and suggestions provided. Below, we present a summary of the core contributions of EntiNeuro, together with our detailed responses to the reviewers’ questions.

Summary of EntiNeuro’s Core Contributions

EEG-based visual decoding has shown promise in recognizing visual categories and reconstructing viewed videos. Everyday visual scenes, however, often involve multiple entities and their interactions. Investigating whether multi-entity semantic information can be decoded from EEG signals and how the decoded information can support video reconstruction is therefore an important step toward decoding more complex visual scenes.

Our work advances this direction through the following key contributions, which have been recognized by the reviewers:

(1) Extending EEG-Based Visual Decoding to Multi-Entity Scenes. (Recognized by Reviewers qgQ1 and JSRf)

  • We investigate whether multi-entity semantic information can be decoded from EEG signals (RQ1). Recovering what a person sees requires accounting for multiple entities within the same scene, making this question an important step beyond recognizing a single dominant category.
  • We further examine whether the decoded information can support video reconstruction (RQ2), connecting the study of multi-entity information in EEG with the goal of reconstructing more complex visual scenes.

(2) EntiNeuro: A Dedicated Multimodal Dataset for Multi-Entity Visual Decoding. (Recognized by Reviewers qgQ1, xt2Y, and JSRf)

  • We built EntiNeuro, a dedicated data resource for systematically studying multi-entity visual decoding. It provides a common dataset for researchers to develop and evaluate subsequent methods.
  • It contains 624 videos spanning six entity categories, with synchronized EEG and eye-tracking recordings from 20 participants across three sessions.
  • Six balanced two-entity compositions support the main study, while two three-entity compositions enable evaluation of transfer from two-entity to three-entity scenes.

(3) EntiBench: A Benchmark for Multi-Entity Recognition and Reconstruction. (Recognized by Reviewer JSRf)

  • We establish EntiBench to connect entity recognition and video reconstruction, providing evaluation protocols and metrics for both tasks.
  • Entity recognition evaluates whether multi-entity semantic information can be decoded from EEG, under both session-average and cross-session settings.
  • Video reconstruction evaluates how faithfully the reconstructed videos reflect the original visual stimuli, assessing semantic, frame, video, and multi-entity fidelity, complemented by human evaluation.

(4) EntiRecon: A Coarse-to-Fine Video Reconstruction Framework. (Recognized by Reviewers qgQ1 and xt2Y)

  • The coarse branch decodes entity probabilities to construct an entity-level prompt and soft semantic context.
  • The fine branch uses this context to refine EEG tokens and inject stimulus-specific information into intermediate diffusion features, with eye tracking providing auxiliary attentional guidance.

(5) Empirical Findings on Multi-Entity Semantic Information in EEG. (Recognized by Reviewer JSRf)

  • Multi-entity semantic information can be decoded from EEG under both session-average and cross-session settings.
  • Temporal analysis identifies informative viewing periods:
    • Cumulative decoding performance rises rapidly within the first second, then gradually saturates.
    • Among 200-ms local windows, decoding peaks at 400–600 ms.
  • Spectral analysis reveals a theta/alpha advantage:
    • Theta or alpha achieves the highest AP for each of the six entity categories.
    • Participant-level results show the same overall pattern.

(6) Strong Results in Multi-Entity Video Reconstruction. (Recognized by Reviewers qgQ1 and xt2Y)

  • Improved reconstruction fidelity. EntiRecon achieves higher semantic and multi-entity fidelity and lower FVD than the compared baselines. These advantages persist with the same Tora checkpoint and matched EEG-only inputs.
  • Ablation studies. Original and supplementary ablations support the effectiveness of our framework’s key components.
  • Support from human and qualitative evaluations. Human ratings favor EntiRecon in entity consistency, temporal coherence, and overall similarity, while qualitative comparisons show better preservation of multi-entity composition across frames.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.