- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces, in its own words, an atlas of EEG-VIT alignment based on 200-way retrieval accuracy metric. The atlas explores this alignment over electrode positions, epoch time, network depth, transformer components, token readout, and training objective of the VIT model. The experiments show: (i) posterior electrodes can predict visual embeddings more accurately and based on earlier time-points in the EEG segment. Also later network layers use later EEG time-points for prediction; (ii) Retrieval decoding accuracy follows an inverted-U shape across layer depth. The inverted-U shape also exists for sub-layer components, i.e., attention/FFN output and residual stream; (iii) EEG signals contain more information about the low-level visual features compared to semantic features. This is supported both by a sweep over layer depths as well as comparing vision encoders trained with reconstruction v.s. CLIP loss. Finally, the paper puts together all the findings and shows that it can beat state-of-the-art performance achieving 97.47 top-1@200 retrieval accuracy.
Critical strengths
I believe the paper explores interesting factors from the input brain signals as well as the vision models and how they affect the accuracy of visual retrieval from EEG signals. For example, investigating how single electrodes predict visual targets or comparing decoding accuracy across different network depths with different objectives provide interesting insights on how visual decoders from EEG signals should be built.
Critical weaknesses
- The text of the paper is difficult to follow and uses unfamiliar and somehow strange terms (decoding currency, celebrated peak, the writing/coasting era, etc.). I understand the meaning of these words but, to my knowledge, they are not common in the context for which they have been used. I tried my best to follow and understand the text, however, there are still parts that I cannot comprehend (e.g. section 5.i, plus many sentences that I only guessed their meaning from the context or from the figures). I think, the paper is not ready for publication in its current format.
- The paper introduces its contributions as exploring alignment between EEG and ViT models. As the paper suggests itself, alignment is commonly defined as an encoding problem (e.g. Schrimpf et al. 2018). And, I agree with the authors that it is not guaranteed to get the best decoding performance from the findings of the best-performing encoding models, by definition. It is also expected that if we optimize every choices in the design based on our measured metric (here, top-1 retrieval accuracy), we get better results on that metric. So, this paper is at best optimizing the framework for visual decoding from EEG (and only one dataset: THINGS-EEG2) to maximize retrieval accuracy. In my opinion all claims about alignment of different layers or different timepoints or different feature levels should be clarified and limited to the one dataset used in the paper.
- The paper's claims are based on one EEG dataset with a very limited number of test samples. I believe the claims of the paper should be at least validated on one other dataset such as THINGS-MEG (that other related works used),.
Questions for authors
I hardly think I suggest acceptance because of how the paper is written. Nevertheless, I have some suggestions for authors to improve the paper (besides how the paper is written):
- Please include more metrics than only top-1 retrieval accuracy on 200 samples. Even if you want to evaluate the effect on decoding and not encoding, use metrics such as cosine similarity (which is not a rank metric) that allow you to directly compare the representations rather than ranking a limited-set samples.
- Please include more datasets such as THINGS-MEG, or THINGS-EEG1. Evaluation on only one dataset leaves the generalizability of the model as an open question.
- You suggest EEG contains more low-level information about the images (section 6.ii). However, this is only implicitly concluded based on the best performing network depth and training objective. You could also investigate this more by perturbation experiments such as decoloring the input image, removing the edges, and adding noise to the image, compute the feature vectors from these perturbed images, and use them as prediction targets for EEG.