acceptodds
Under review as a conference paper at ICLR 2027

Shadow Brain: Biomimetic Learning for Decoding Visual Neural Signals

Abstract

Visual neural decoding aims to infer the visual information perceived by an individual from brain activity and is key to connecting biological perception and artificial intelligence in brain–computer interface research. Visual information undergoes parallel processing, selective convergence, and hierarchical integration along biological visual pathways, giving rise to neural responses that vary across individuals. This variability, together with the low signal-to-noise ratio of electroencephalography (EEG), makes accurate alignment between visual information and EEG responses challenging. We propose Shadow Brain, a biomimetic visual–EEG learning framework built on the information-processing principles of the visual pathway from the retina through the lateral geniculate nucleus to the visual cortex. The visual branch draws on the functional sequence from retinal photoreception and signal processing to ganglion cell output. It constructs feature representations under five visual conditions and selectively integrates them using subject-specific learnable weights shared across stimuli within each subject. Inspired by population-level information integration in the visual cortex, the EEG branch employs NeuroWeft as a unified module to jointly model spatial relationships and temporal structure in multichannel EEG. Through contrastive learning, visual fusion, NeuroWeft, and cross-modal projectors are jointly optimized in a shared semantic space, adapting visual information integration to individual EEG signals. Within-subject 200-way image retrieval experiments using trial-averaged EEG from ten subjects in THINGS-EEG achieve mean Top-1 and Top-5 accuracies of 68.4% and 90.9%, respectively. Shadow Brain incorporates biological visual processing principles into visual–EEG correspondence learning, shifting from direct visual–EEG alignment to personalized visual–neural alignment constrained by biological visual processing principles.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Visual neural decoding, Biomimetic learning, Visual–EEG alignment, Contrastive learning, Brain–computer interfaces

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UpC1OpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2(−1)
Presentation
2(−1)
Soundness
2
Read review
Paper summary
  1. This paper proposes Shadow Brain, a biologically inspired framework for EEG-based visual decoding, motivated by information processing along the retina–LGN–visual cortex pathway.
  2. The framework introduces two main components: subject-specific selective fusion of five visual representations and NeuroWeft, an EEG encoder integrating signal refinement, channel interactions, and spatiotemporal modeling. Both components are jointly optimized through cross-modal contrastive learning.
  3. Experiments on THINGS-EEG demonstrate 68.4% Top-1 accuracy in within-subject retrieval and 20.4% in cross-subject retrieval. Additional experiments on THINGS-MEG and ablation studies further evaluate the framework's effectiveness.
Critical strengths
  1. The paper explores an interesting direction by incorporating visual processing principles into cross-modal representation learning. Its subject-specific visual fusion mechanism provides an interpretable approach to accommodating individual differences in neural responses.
  2. Shadow Brain improves retrieval accuracy over the reported baselines on both EEG and MEG benchmarks. The component ablations and signal-level analyses provide additional evidence supporting the effectiveness of visual fusion and NeuroWeft.
Critical weaknesses
  1. Although the framework is motivated by retinal and cortical processing, its implementation mainly relies on conventional image transformations, weighted feature fusion, and neural signal-processing operations. The correspondence between these computational components and actual biological mechanisms remains conceptual. The current experiments do not establish whether the improvements originate from biologically meaningful inductive biases rather than conventional multi-view representation learning.

  2. The proposed visual fusion employs only five globally learned coefficients per subject, while NeuroWeft combines several established operations. Although the overall ablations demonstrate their effectiveness, the paper does not sufficiently distinguish these designs from existing feature-fusion and EEG-encoding strategies. In particular, the main factorial ablation is conducted on a single subject, limiting its ability to establish the consistency of individual component contributions.

  3. The reported improvements are primarily established against published results from methods using different visual representations, neural encoders, and training configurations. Without comparisons under matched visual backbones, input preprocessing, and computational budgets, it is difficult to determine whether the improvements are attributable to the proposed learning mechanisms.

  4. The learned fusion coefficients exhibit broadly similar distributions across subjects, with relatively small deviations from uniform weighting. Furthermore, the cross-subject evaluation uses a fusion vector shared among training subjects rather than subject-specific adaptation. These observations leave uncertainty about whether the proposed fusion captures meaningful individual neural characteristics or simply learns a generally favorable combination of image transformations.

  5. Although the method achieves competitive within-subject retrieval, performance drops substantially under leave-one-subject-out evaluation. The improvements over the strongest reported baselines are also relatively small in this setting. Additionally, the experiments rely on repetition-averaged neural signals, and the paper does not report statistical significance tests for the principal retrieval comparisons, limiting confidence in generalization and robustness.

Questions for authors

My rating can be reconsidered if the authors can convincingly address these concerns and provide stronger supporting evidence during the rebuttal.

  1. Can the authors provide quantitative evidence that the five visual conditions and NeuroWeft capture meaningful biological processing characteristics rather than functioning as conventional data transformations and neural network components?

  2. What fundamentally distinguishes the proposed subject-specific visual fusion from existing learnable multi-view feature aggregation? Can the authors compare against alternative fusion strategies with matched model capacity?

  3. Can the authors provide more detailed ablations of NeuroWeft's internal components, including wavelet-based refinement, channel interactions, and spatiotemporal encoding, across all participants rather than relying primarily on the full-module comparison?

  4. Can the authors evaluate the effects of exchanging learned fusion coefficients between subjects or using a globally shared fusion vector? Such comparisons would help establish whether subject-specific fusion provides meaningful benefits beyond ordinary feature aggregation.

  5. Can the authors provide controlled comparisons using identical visual backbones, EEG preprocessing, and training budgets to better isolate the contributions of Shadow Brain?

  6. Can the authors report confidence intervals or significance tests across independent runs, particularly for the relatively small cross-subject improvements? How does performance change when fewer neural repetitions are available?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your time and detailed comments! We have marked the corresponding changes in orange in the revised manuscript for clarity. Below are our point-by-point responses to your comments.

[W1, Q1] Quantitative evidence of biological relevance

Drawing on the excellent approach used in AVDE [1] to analyze the correspondence between visual representations and regional EEG signals, we added similarity analyses of the fused representations from the five visual conditions and the NeuroWeft representations, providing strong quantitative support for their biological relevance.

  • Quantitative analysis of the visual branch. The image-side processing considers how changes in information at the retinal front end relate to downstream visual processing. Retinal function can affect early cortical visual evoked responses [2]. We therefore used early visual EEG as a neural reference driven by retinal input [3] for representational similarity analysis. Comparing visual features at each layer with early EEG signals revealed distinct regional patterns: correspondence in the occipital region was already high in the shallow encoder layers, peaked at Layer2, and then decreased to 0.043 at the final layer; correspondence in the parieto-occipital and parietal regions increased through the first two layers and declined markedly after Layer2; correspondence in the frontal and central regions was weaker overall.

  • Quantitative analysis of NeuroWeft. We quantified the representational similarity between NeuroWeft and channel-level EEG, and calculated changes in similarity between adjacent modules to assess how each stage affects neural representations. The results revealed clear temporal and spatial patterns: the occipital region showed the largest increase in similarity after early EEG was incorporated, consistent with its role in early visual processing; correspondence in the parieto-occipital and parietal regions also increased progressively, suggesting an association between NeuroWeft representations and posterior visual networks.

These findings show that the fused representations from the five visual conditions correspond to regional features of early visual EEG. NeuroWeft outputs also exhibit regional and temporal structure related to visual processing, providing quantitative support for the framework's biological relevance. The detailed analyses and results are presented in the new Figure 6 in the main text.


[W2, Q2, Q3] Contributions of view fusion and NeuroWeft components

W2: Our design jointly optimizes participant-level five-way visual fusion and EEG encoding through a contrastive learning signal from paired EEG, enabling EEG-driven adaptation of visual targets at the participant level. Each participant learns five fusion weights shared across images; NeuroWeft sequentially connects detail refinement guided by the low-frequency subband, channel self-attention, and spatiotemporal encoding. The overall ablation results across all ten participants are shown below. When visual fusion is disabled, we use original-image features; when NeuroWeft is disabled, we use EEGProject.

Visual fusionNeuroWeftTop-1 (%)Top-5 (%)
×\times×\times34.5566.50
×\times✓\checkmark41.7072.85
✓\checkmark×\times59.9584.57
✓\checkmark✓\checkmark68.4090.90
AuthorsOpenReview ↗

Official Comment by Authors

W2 & Q2: Visual fusion strategies. Compared with gating that predicts weights for each image, our approach adapts at the participant level, using five trainable parameters to adjust the contributions of different visual conditions. The table below compares alternative fusion strategies. All changes are relative to the Top-1 accuracy of 68.40% and Top-5 accuracy of 90.90% reported in the original paper.

Fusion strategyTrainable fusion parametersΔ\Delta Top-1Δ\Delta Top-5
Softmax fusion (ours)500
Equal-weight fusion0−7.50−1.48
Normalized Sigmoid fusion5−4.87−1.05

W2 & Q3: Internal ablations of NeuroWeft. We added internal ablation experiments across all participants, using the same configuration as in the original paper. The table below reports relative changes: change=ablated model−full model\text{change} = \text{ablated model} - \text{full model}, in percentage points (pp); "++" indicates an increase and "−-" a decrease.

Additional experimental configurationΔ\Delta Top-1 %Δ\Delta Top-5 %
Remove wavelet refinement−5.22−1.18
Remove channel self-attention−7.45−2.48
Replace the spatiotemporal convolutional block−37.82−26.22

Replacing the spatiotemporal convolutional block caused the largest decrease in average performance, reducing Top-1 and Top-5 accuracy by 37.82 and 26.22 percentage points, respectively. Removing channel self-attention or wavelet refinement also reduced average recognition performance, with the larger decrease observed for channel self-attention. These results support each component's contribution to average performance across all participants.

The channel self-attention ablation retains other cross-channel operations, so its result reflects the contribution of this specific module; the spatiotemporal ablation reflects the contribution of the entire convolutional block relative to the simplified replacement used. We will include these analyses and show the ablation results for all participants in the revised manuscript.


[W3, Q5] Module replacement comparisons under matched conditions

We added bidirectional module replacement experiments using the same frozen CLIP RN50, EEG input preprocessing, and training budget. Starting from the respective configurations of NeuroBridge and ShadowBrain, we replaced the visual module, the EEG encoder, or both to assess each component's contribution.

The original NeuroBridge configuration is "four-way equal-weight fusion + EEGProject," and the original ShadowBrain configuration is "five-way selective fusion + NeuroWeft." In the replacement columns, ✓ means using the corresponding module from the other method, and ✗ means retaining the original module. Top-1 and Top-5 are reported as accuracy (%), with changes relative to each method's original configuration in parentheses

Starting configurationReplace visualReplace EEGTop-1 (%)Top-5 (%)
NeuroBridge×\times×\times56.8 (0.0)83.3 (0.0)
NeuroBridge✓\checkmark×\times59.2 (+2.4)84.3 (+1.0)
NeuroBridge×\times✓\checkmark57.8 (+1.0)83.6 (+0.3)
NeuroBridge✓\checkmark✓\checkmark67.3 (+10.5)90.2 (+6.9)
ShadowBrain×\times×\times68.4 (0.0)90.9 (0.0)
ShadowBrain✓\checkmark×\times58.9 (−9.5)84.1 (−6.8)
ShadowBrain×\times✓\checkmark59.9 (−8.5)84.6 (−6.3)
ShadowBrain✓\checkmark✓\checkmark58.4 (−10.0)83.7 (−7.2)

In the NeuroBridge configuration, introducing either our visual fusion or NeuroWeft improved both Top-1 and Top-5 accuracy, with the largest gains when both were introduced together. In the reverse replacements, removing either component reduced accuracy, and replacing both reduced performance further. Both sets of replacement experiments support the contributions of the two modules and the overall framework under matched settings.

AuthorsOpenReview ↗

Official Comment by Authors

[W4, Q4] Role of participant-specific fusion weights

We completed weight-swapping and shared-training controls across all participants. In the weight-swapping experiments, we kept the trained models fixed and replaced the fusion weights. For the shared-training experiment, we retrained the models with all participants jointly learning one set of fusion weights, and compared this with learning weights independently for each participant.

ConditionΔ Top-1 (%)Δ Top-5 (%)
Other participants' average weights−2.50−0.50
Other participants' weights, swapped individually−3.82−0.42
Shared training−2.63−0.33

Both controls support the role of participant-specific fusion in the current task: retaining separate weights for each participant yields better average retrieval performance than using other participants' weights or sharing one set across all participants. Interestingly, we also observed limited performance gains under LOSO, which may be related to the use of a fusion vector shared by the training participants. We will test this possibility in future work and explore individualized adaptation for unseen participants. The weight-swapping and shared-training results are presented in Figure 3f of the revised main text.


[W5, Q6] Statistics across runs and number of test repetitions

  • Statistics across independent runs. We agree that model performance is relatively close under LOSO and that further statistical analysis is needed to assess the reliability of cross-participant results. Using the original settings, we added means, standard deviations (SD), and 95% confidence intervals for within-participant and cross-participant results to quantify average performance and variability across runs.
SettingMetricMean ±\pm SD95% confidence interval
Intra-SubjectTop-168.4068.40\\% \pm 1.80\\%[66.17%, 70.63%]
Intra-SubjectTop-590.9090.90\\% \pm 0.72\\%[90.01%, 91.79%]
Inter-SubjectTop-120.4020.40\\% \pm 0.87\\%[19.32%, 21.48%]
Inter-SubjectTop-546.3046.30\\% \pm 0.31\\%[45.92%, 46.68%]

The 95% confidence intervals in the table use t-interval estimates; among the methods compared in the original paper (Table 1), Shadow Brain continues to achieve relatively high average retrieval accuracy. These additional data strengthen confidence in the model's improvements in cross-participant performance.

  • Fewer test repetitions in the within-participant setting. For within-participant 200-way retrieval, we use the results from averaging 80 repetitions, with Top-1 / Top-5 accuracies of 68.40% / 90.90%, as the reference and report the proportion of accuracy retained, defined as the accuracy in each condition divided by its corresponding reference accuracy.
Test repetitionsTop-1 retentionTop-5 retention
80100.00%100.00%
4087.85%96.96%
1676.48%91.51%
461.93%81.97%
1 (single trial)44.87%68.24%

Accuracy decreases as the number of test repetitions is reduced; single-trial Top-1 / Top-5 accuracy still reaches 30.69% / 62.03%, showing that the model can still retrieve images effectively without averaging test repetitions.


References

  • [1] Dai, Sicheng, et al. "Autoregressive Visual Decoding from EEG Signals." International Conference on Learning Representations. 2026.

  • [2] Negishi, Chisako, et al. "Visual evoked potentials in relation to visual acuity in macular disease." Acta Ophthalmologica Scandinavica 79.3 (2001): 271–276.

  • [3] Di Russo, Francesco, et al. "Cortical sources of the early components of the visual evoked potential." Human Brain Mapping 15.2 (2002): 95–111.

Reviewer_LnSrOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies zero-shot EEG-to-image retrieval using trial-averaged EEG. The authors propose Shadow Brain, which combines frozen CLIP RN50 features from five visual conditions through subject-specific selective fusion. The EEG branch uses NeuroWeft to encode multichannel EEG. The framework is evaluated on THINGS-EEG and THINGS-MEG, with ablation experiments examining visual fusion and NeuroWeft.

Critical strengths
  1. The retrieval performance on THINGS-EEG is strong, achieving a mean Top-1 accuracy of 68.4%, exceeding NeuroBridge by 5.2 percentage points, with improvements across all ten subjects.

  2. Table 2 evaluates the contributions of visual fusion and NeuroWeft, while Figure 5a,b compares different contributions across ten subjects. These experiments support the effectiveness of the two main components.

Critical weaknesses
  1. The visual branch closely resembles existing methods. NeuroBridge already averages augmented CLIP features for cross-modal contrastive learning. The main changes here are adding original condition features and replacing equal weight averaging with subject-specific selective fusion. I therefore think the visual branch as an incremental improvement over NeuroBridge.
  2. Table 1 shows mean improvements of 1.4 percentage points in Top-1 accuracy and approximately 0.4 percentage points in Top-5 accuracy. However, the Top-5 improvement is sensitive to individual subjects sub-09 improves by 14 percentage points, while the mean difference across the other nine subjects is approximately −1.1 percentage points. Sub-06 decreases by 9.2 percentage points. These results show individual variability across subjects, particularly in Top-5 accuracy.
Questions for authors
  1. In Figure 2, which two dimensions are the 2-D Haar transform applied? How are spatial relationships between EEG channels represented during low-band-guided detail refinement?

  2. Are the pooled EEG features in Figure 8 obtained from a single model or from independently trained models? If they come from different models, how are the representations aligned before being pooled for t-SNE?

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your careful review and comments. We respond to each point below, and the revisions will be marked in purple in the manuscript.

[W1] The visual branch is similar to NeuroBridge, and the distinctiveness of the improvements needs to be explained.

Our motivation comes from clinical brain-computer interface practice, with a focus on bringing insights from biological visual processing into decoding models. Inspired by parallel retinal coding and nonlinear integration mechanisms [1,2], we constructed five visual conditions: original images, blur, mosaic, low resolution, and noise. Table 1 reports the performance comparison with NeuroBridge, and Table 2 examines the roles of visual fusion and EEG encoding. The new representational similarity analysis shows stronger correspondence between visual features and early EEG in the occipital, parieto-occipital, and parietal regions. Correspondence in posterior brain regions also increases across the stages of NeuroWeft, providing quantitative support for the biological relevance of the design (new Figure 6).


[W2] Cross-participant Top-5 gains are sensitive to individual participants, and performance varies across participants.

For sub-09, UBP [3] and Shadow Brain (ours) achieve Top-5 accuracies of 40.5% and 50.5%, respectively, both ranking second within their respective models, while NeuroBridge achieves 36.5% [4]. The relative performance of different models varies across participants.

We compared relative EEG variability before and after model processing for each participant. The extent of change differs across participants, which may reflect differences in Shadow Brain's sensitivity to their EEG features and may in turn be associated with uneven retrieval gains. Under LOSO, the model is trained on data from the other nine participants and tested on the participant excluded from training; individual differences between the training and test participants may bring out these differences in sensitivity.

Proportion of variability after processing (%).

Participantsub-01sub-02sub-03sub-04sub-05sub-06sub-07sub-08sub-09sub-10
Proportion after processing18.7014.4119.4017.2518.8012.2915.5714.5421.1919.89

For each model, we also considered Top-1 and Top-5 separately and excluded its highest-scoring 1, 3, and 5 participants to examine how the means changed after removing high values. The participants retained may differ across models. In this model-by-model analysis, Shadow Brain (ours) still has the highest mean for both metrics in the table.

Cross-participant results after excluding high values for each model: each cell reports Top-1 / Top-5 (%).

ModelAll 10 participantsExcluding top 1 participantExcluding top 3 participantsExcluding top 5 participants
BraVL1.8 / 7.01.79 / 6.871.66 / 6.531.54 / 6.18
NICE6.2 / 21.45.93 / 20.685.64 / 20.145.44 / 19.36
UBP12.4 / 33.412.02 / 32.2611.03 / 29.9710.40 / 28.74
NeuroBridge19.0 / 45.918.13 / 44.7416.43 / 42.1914.74 / 40.14
Shadow Brain(ours)20.4 / 46.319.60 / 45.3117.77 / 43.8616.28 / 41.96

We will include the variability analysis and exclusion results above in the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

[Q1] Along which two dimensions is the two-dimensional Haar transform applied, and how are spatial relationships between channels represented during refinement?

The two-dimensional Haar transform is applied along the channel and time axes of the preprocessed EEG. Channel relationships during refinement are represented through the Haar pairings defined by the fixed channel order in Appendix E.1 and the convolutional neighborhoods on the subband grids. The LL subband generates gating weights through a 7×77 \times 7 convolution to modulate the three detail subbands. Refinement then proceeds through selective 3×33 \times 3 smoothing, the inverse Haar transform, and residual gating.

We performed a channel-wise perturbation analysis across 10 participants: applying a 1% amplitude perturbation to one input channel also changed the refinement residuals of other output channels. Cross-channel responses account for 50.10% of the responses across all channels, indicating cross-channel interactions during refinement. The table below reports the effect of randomly permuting channels within the refinement module on retrieval accuracy.

ParticipantΔ\Delta Top-1 (%)Δ\Delta Top-5 (%)
sub-01+0.72+0.03
sub-02+0.23+0.70
sub-03+0.470.00
sub-04−0.25+0.38
sub-05−0.47+0.02
sub-06+0.58−0.17
sub-07+1.33+0.48
sub-08+0.10+0.13
sub-09+1.00+0.68
sub-10+0.02+0.50

Removing the entire refinement module and permuting channels within it examine different factors. The former evaluates the contribution of the full refinement process while retaining the other modules; the latter tests sensitivity to the internal channel order while retaining that process. The channel influence matrix is shown in the new Figure 12 in the revised manuscript.


[Q2] Do the pooled features in Figure 8 come from the same model or independently trained models, and how are they aligned before pooling?

The original Appendix Figure 8 directly pooled the outputs of ten independently trained participant models. To improve the comparability of cross-participant visualizations, we added individual visualizations for nine participants in the new figure panels and used an alignment method anchored by common training images. With sub-01 as the fixed reference, we mapped the 512-dimensional features into a common coordinate system, then pooled them for t-SNE to regenerate the pooled visualizations for all participants. The revised results are shown in the new Figure 9: a–r show individual visualizations for the nine participants other than sub-07, whose visualization appears in the main text; s and t show the aligned, pooled results for all ten participants.


References

  • [1] Hsiang, Jen-Chun, et al. "Distributed feature representations of natural stimuli across parallel retinal pathways." Nature Communications 15 (2024): 1920.

  • [2] Karamanlis, Dimokratis, et al. "Nonlinear receptive fields evoke redundant retinal coding of natural scenes." Nature 637 (2025): 394–401.

  • [3] Wu, Haitao, et al. "Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Prior." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

  • [4] Zhang, Wenjiang, et al. "NeuroBridge: Bio-Inspired Self-Supervised EEG-to-Image Decoding via Cognitive Priors and Bidirectional Semantic Alignment." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 21. 2026.

Reviewer_NPDcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Neural decoding for visual information is an important topic while meeting a lot of challenges. This paper proposes a new framework, ShadowBrain, driven by biological visual principles. They construct five apralle visual representations with different informational properties and propose a subject-specific fusion mechanisim. They also popose NeuroWeft to capture multi-channel EEG representation and achieve high decoding performance for Things-EEG datasets.

Critical strengths
  1. It's very interesting to design the overall decoding model based on biological visual processing.
  2. The visual-condition retrieval results provide good beginning for extract different levels of visual features.
Critical weaknesses
  1. Further evidence is necessary to show the use of different level of visual features. How did you set up the five visual conditions? Is that enough to get all those information from EEG signals?
  2. It would be useful to check the data splite for model training and evalution.
Questions for authors
  1. Could you give more explanation of the model setting, how you chose the visual condition, and which part showed a significant effect?
  2. Could you give the overall results of 10 subjects instead of just use on subject for demonstration.
  3. Further evaluation with more datasets would strength the evidence.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your careful review and constructive comments. The corresponding changes in the revised manuscript are marked in light blue. We respond to each point below.

[W1, Q1] Rationale, settings, and contributions of the five visual conditions

Inspired by parallel retinal processing [1] and selective integration in the visual system, we retain the visual detail of the original image as a reference for natural visual input and use the low-resolution condition to reflect limited spatial sampling [2]. Blur and mosaic use neighborhood smoothing and local block formation, respectively, drawing on spatial integration in the bipolar-to-ganglion-cell pathway [3]. The noise condition examines changes in input reliability, motivated by the downstream effects of photoreceptor noise [4]. These correspondences provide functional inspiration and do not directly simulate the responses of specific cells.

Parameter settings (Section 3.2): Gaussian blur uses a fixed 15×1515 \times 15 kernel; Gaussian noise has a mean of 0 and a pixel-space standard deviation of 25; the low-resolution condition reduces both width and height to 10% of their original sizes, followed by bilinear upsampling; mosaic uses level 5 nearest-neighbor block reconstruction. A frozen CLIP RN50 extracts 1024-dimensional features for each condition. These features are fused with learnable participant-specific weights and aligned with the EEG representations extracted by NeuroWeft.

Following the analysis of correspondence between visual representations and regional EEG in AVDE [5], we use early visual EEG as a reference to compare representational similarity between visual features at each layer and EEG from different brain regions. Correspondence in the occipital region peaks at Layer2 and then falls to 0.043 at the final layer. Correspondence in the parieto-occipital and parietal regions increases across the first two layers and declines after Layer2, while correspondence in the frontal and central regions is generally weaker. Analysis across NeuroWeft stages also shows that the occipital region has the largest increase in similarity after early EEG is incorporated, with correspondence in the parieto-occipital and parietal regions increasing progressively. Correspondence between the fused representations of the five visual conditions and early EEG is more pronounced in the occipital, parieto-occipital, and parietal regions. NeuroWeft also exhibits regional and temporal features associated with visual processing. Details are provided in the new Figure 6.

Retrieval results for each of the five individual conditions (%).

Visual conditionMean Top-1 (%)Mean Top-5 (%)
Original image41.7072.85
Gaussian blur49.7581.90
Gaussian noise51.4082.40
Low resolution45.2079.05
Mosaic52.0084.05

All five conditions support EEG-based image retrieval. Across the 10 participants, blur, noise, and mosaic each achieve higher Top-1 accuracy than the original-image condition. Together, the retrieval results and regional correspondence support the ability of these conditions to represent visual information in EEG and their effectiveness as decoding targets.

To assess each visual condition's contribution to retrieval, we added experiments that remove one view at a time across all ten participants. The accuracy changes are reported below; asterisks indicate p<0.05p < 0.05 after Holm correction:

ConfigurationΔ\Delta Top-1 (%)Δ\Delta Top-5 (%)
Remove original image−3.95*−0.33
Remove Gaussian blur−5.12*−1.23*
Remove Gaussian noise−6.45*−2.22
Remove low resolution−6.88*−2.30*
Remove mosaic−8.93*−3.47*

Top-1 decreases significantly in all five removal configurations. Top-5 decreases significantly only when Gaussian blur, low resolution, or mosaic is removed. Removing mosaic produces the largest observed decrease, with Top-1 and Top-5 falling by 8.93 and 3.47 percentage points, respectively. We will add the table to the revised manuscript; the results of removing individual visual conditions are shown in Figure 3e of the revised main text.

AuthorsOpenReview ↗

Official Comment by Authors

[W2] Data splits and evaluation protocol

Our evaluation follows the benchmarks and protocols used in EEG/MEG decoding studies such as NICE, ATM, CognitionCapturer, and UBP: 200-way zero-shot retrieval on THINGS-EEG and THINGS-MEG with non-overlapping training and test concepts, consistent with the evaluation settings of prior work. The original manuscript states in §4.1 and Appendix E that the training set contains 1,654 concepts and the test set contains another 200 concepts. We also checked the concept directories and confirmed that the training and evaluation splits follow this protocol. Within-participant evaluation uses the training and test partitions of the same participant. Cross-participant evaluation trains the model on training data from 9 participants and tests it on the held-out 10th participant.


[Q2] Overall results across all ten participants

The overall evaluation covers all ten participants, with results reported in Table 1 of the original manuscript. Mean retrieval accuracies for the four configurations are listed below.

Visual fusionNeuroWeftMean Top-1 (%)Mean Top-5 (%)
×\times×\times34.5566.50
×\times✓\checkmark41.7072.85
✓\checkmark×\times59.9584.57
✓\checkmark✓\checkmark68.490.9

When visual fusion is disabled, we use original-image features; when NeuroWeft is disabled, we use EEGProject. Table 2 of the original manuscript evaluates the contributions of visual fusion and NeuroWeft, and Figures 5a, b further compare three EEG encoders across the ten participants.

The participant-level analysis also covers all ten participants, showing the fusion weights and their learning trajectories. The weights and corresponding selected epochs are listed below.

ParticipantSelected epochOriginal image (%)Blur (%)Noise (%)Low resolution (%)Mosaic (%)
sub-013317.817.122.617.924.7
sub-024717.116.623.118.624.6
sub-034518.316.523.017.324.9
sub-043721.017.823.916.021.3
sub-053817.915.922.517.726.0
sub-063416.616.722.519.025.2
sub-074118.516.623.517.523.9
sub-082918.217.822.317.923.7
sub-094818.516.623.417.823.7
sub-103818.117.623.217.723.5

Figures 9s, t in Appendix C of the revised manuscript show the semantic category structure of image and EEG features pooled across all participants after alignment.


[Q3] Could you evaluate the model on additional datasets?

We added an evaluation on EEG-ImageNet with eight participants. Shadow Brain achieves mean Top-1/Top-5 accuracies of 22.10%/48.34%, exceeding NeuroBridge by 2.22/4.79 percentage points based on the reported means. The new results will be included in the appendix of the revised manuscript.

MethodMean Top-1 (%)Mean Top-5 (%)
NICE13.0337.93
UBP13.6440.93
CogCapPro17.0241.42
NeuroBridge19.8843.55
Shadow Brain (ours)22.1048.34

Table 3 in Appendix B reports comparative results on THINGS-MEG. Mean Top-1 and Top-5 retrieval accuracies for each method are summarized below.

MethodWithin-participant Top-1 (%)Within-participant Top-5 (%)Leave-one-participant-out Top-1 (%)Leave-one-participant-out Top-5 (%)
NICE12.836.0——
UBP26.755.22.210.4
CogCapPro31.864.6——
NeuroBridge32.260.83.412.8
Shadow Brain (ours)42.468.05.713.1

References

  • [1] Hsiang, Jen-Chun, et al. "Distributed feature representations of natural stimuli across parallel retinal pathways." Nature Communications 15 (2024): 1920.

  • [2] Rossi, Ethan A., and Austin Roorda. "The relationship between visual resolution and cone spacing in the human fovea." Nature Neuroscience 13.2 (2010): 156–157.

  • [3] Schreyer, Helene Marianne, and Tim Gollisch. "Nonlinear spatial integration in retinal bipolar cells shapes the encoding of artificial and natural stimuli." Neuron 109.10 (2021): 1692–1706.e8.

  • [4] Ala-Laurila, Petri, et al. "Cone photoreceptor contributions to noise and correlations in the retinal output." Nature Neuroscience 14.10 (2011): 1309–1316.

  • [5] Dai, Sicheng, et al. "Autoregressive Visual Decoding from EEG Signals." International Conference on Learning Representations. 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Review and Reviewer-Author Discussion Summary

Dear Area Chair and Reviewers,

Thank you for your careful review and constructive feedback. In response to the main concerns, we have conducted the following additional experiments and analyses:

  1. Quantitative evidence of biological relevance. We added representational similarity analyses of the fused visual features and NeuroWeft representations. The results show stronger correspondence between fused visual features and early EEG in posterior brain regions, alongside regional and temporal patterns in NeuroWeft associated with visual processing (new Figure 6).

  2. Component contributions and participant-specific fusion. We added ablations across all ten participants, comparisons of alternative fusion strategies, and weight-swapping and shared-weight controls. Bidirectional module substitutions with NeuroBridge under the same visual backbone, EEG preprocessing, and training budget further assess the contributions of visual fusion and NeuroWeft.

  3. Evaluation on an additional dataset. We added an evaluation on EEG-ImageNet with eight participants. Shadow Brain achieves average Top-1 and Top-5 retrieval accuracies of 22.10% and 48.34%, respectively.

  4. Statistical analysis and evaluation with fewer repetitions. We reported standard deviations and 95% confidence intervals across runs, and examined cross-participant variability and sensitivity to high-performing participants. We also evaluated fewer test repetitions; single-trial, within-participant 200-way retrieval achieves 30.69% Top-1 and 62.03% Top-5 accuracy.

We also clarified the data splits, Haar operations and cross-channel interactions, and added aligned visualizations of pooled participant features. Detailed experimental settings and results are provided in our point-by-point responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.