acceptodds
Under review as a conference paper at ICLR 2027

Where, When, and What Aligns: A Retrieval-Grounded Atlas of EEG–ViT Alignment

Abstract

The alignment between human brain activity and pretrained vision models has become both a window into biological vision and the engine of EEG-to-image decoding. In this work, we draw the first systematic atlas of the EEG–ViT interface measured in the deployment metric itself, end-to-end 200-way retrieval, resolved jointly over scalp position, epoch time, network depth, sub-layer computation, token readout, and training objective. Analyzing 21 vision models spanning contrastive, supervised, self-distillation, and reconstruction training, we evaluate approximately 21,000 probe cells on all ten THINGS-EEG2 subjects at three trial-count regimes ( averaged test repetitions), with every claim supported by a per-subject sign test. Our analysis reveals three key findings: (1) where and when: the network's depth axis carries a brain timestamp; each block aligns best with a distinct, progressively later moment of the EEG epoch (177 to 201 ms across blocks 0 to 11, in 10/10 subjects), co-registered with the scalp's posterior-to-anterior latency gradient; (2) where in the network: the well-documented mid-depth alignment peak has a mechanism; early blocks write EEG-visible content into the residual stream, the mid-network stream integrates those writes and overtakes every computation, and late computations fade while the stream coasts on what was already written; (3) what: what EEG shares with a vision transformer is low-level visual structure at coarse spatial grain, not fine-grained semantics: alignment is carried by electrodes over early visual cortex, peaks at one-third to one-half of network depth, deeper still under masked autoencoding, and fades exactly where representations turn semantic, in the late layers, unless the training target itself is pixel reconstruction; spatially, a coarse to pooling of the token grid outperforms the field's global readouts by 6 to 22 points and any single patch by far more, the grain expected of a 63-electrode montage reading through the scalp. Building on these insights, the atlas configures decoders by measurement alone: it distills a five-model fusion pipeline to eight electrodes and a single backbone that surpass the published state of the art, and composes a four-state fusion chain that reaches 97.5% final-epoch accuracy together with the first single-trial accuracy above 22%. Our results suggest that the brain–transformer interface is a lawful and largely unexplored territory, and that deployment-currency measurement offers a new instrument for studying biological vision through transformer computations and vision models through the organization of the brain.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG decoding, vision transformers, neural alignment, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AFALOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces, in its own words, an atlas of EEG-VIT alignment based on 200-way retrieval accuracy metric. The atlas explores this alignment over electrode positions, epoch time, network depth, transformer components, token readout, and training objective of the VIT model. The experiments show: (i) posterior electrodes can predict visual embeddings more accurately and based on earlier time-points in the EEG segment. Also later network layers use later EEG time-points for prediction; (ii) Retrieval decoding accuracy follows an inverted-U shape across layer depth. The inverted-U shape also exists for sub-layer components, i.e., attention/FFN output and residual stream; (iii) EEG signals contain more information about the low-level visual features compared to semantic features. This is supported both by a sweep over layer depths as well as comparing vision encoders trained with reconstruction v.s. CLIP loss. Finally, the paper puts together all the findings and shows that it can beat state-of-the-art performance achieving 97.47 top-1@200 retrieval accuracy.

Critical strengths

I believe the paper explores interesting factors from the input brain signals as well as the vision models and how they affect the accuracy of visual retrieval from EEG signals. For example, investigating how single electrodes predict visual targets or comparing decoding accuracy across different network depths with different objectives provide interesting insights on how visual decoders from EEG signals should be built.

Critical weaknesses
  1. The text of the paper is difficult to follow and uses unfamiliar and somehow strange terms (decoding currency, celebrated peak, the writing/coasting era, etc.). I understand the meaning of these words but, to my knowledge, they are not common in the context for which they have been used. I tried my best to follow and understand the text, however, there are still parts that I cannot comprehend (e.g. section 5.i, plus many sentences that I only guessed their meaning from the context or from the figures). I think, the paper is not ready for publication in its current format.
  2. The paper introduces its contributions as exploring alignment between EEG and ViT models. As the paper suggests itself, alignment is commonly defined as an encoding problem (e.g. Schrimpf et al. 2018). And, I agree with the authors that it is not guaranteed to get the best decoding performance from the findings of the best-performing encoding models, by definition. It is also expected that if we optimize every choices in the design based on our measured metric (here, top-1 retrieval accuracy), we get better results on that metric. So, this paper is at best optimizing the framework for visual decoding from EEG (and only one dataset: THINGS-EEG2) to maximize retrieval accuracy. In my opinion all claims about alignment of different layers or different timepoints or different feature levels should be clarified and limited to the one dataset used in the paper.
  3. The paper's claims are based on one EEG dataset with a very limited number of test samples. I believe the claims of the paper should be at least validated on one other dataset such as THINGS-MEG (that other related works used),.
Questions for authors

I hardly think I suggest acceptance because of how the paper is written. Nevertheless, I have some suggestions for authors to improve the paper (besides how the paper is written):

  1. Please include more metrics than only top-1 retrieval accuracy on 200 samples. Even if you want to evaluate the effect on decoding and not encoding, use metrics such as cosine similarity (which is not a rank metric) that allow you to directly compare the representations rather than ranking a limited-set samples.
  2. Please include more datasets such as THINGS-MEG, or THINGS-EEG1. Evaluation on only one dataset leaves the generalizability of the model as an open question.
  3. You suggest EEG contains more low-level information about the images (section 6.ii). However, this is only implicitly concluded based on the best performing network depth and training objective. You could also investigate this more by perturbation experiments such as decoloring the input image, removing the edges, and adding noise to the image, compute the feature vectors from these perturbed images, and use them as prediction targets for EEG.
Reviewer_3TEGOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how EEG responses align with vision transformers, using 200-way zero-shot image retrieval on THINGS-EEG2 as the measurement. A "probe cell" trains a small contrastive decoder on a slice of the EEG (a channel group and a time window) against a slice of a vision model (a model, a depth, a component and a token readout). The authors evaluate about 21,000 such cells over 21 vision models, ten subjects and three trial-count regimes, with per-subject sign tests. They report that deeper blocks align with later EEG latencies (177 to 201 ms), that the mid-depth alignment peak comes from the residual stream accumulating "writes" from early blocks, and that EEG shares low-level, coarse-grained structure with the ViT, so coarse pooling of the token grid beats class-token and global readouts. Based on this, they configure decoders (eight occipital electrodes and one backbone, and a four-state fusion chain) that reach 91.67% and 97.47% final-epoch top-1, higher than the published numbers they compare with.

Critical strengths
  1. The study is large and has many controls: 21 models, all ten subjects, three trial regimes, paired comparisons with identical seeds, matched random-channel controls, random-init controls and per-subject sign tests instead of single-cell comparisons. As far as I can tell, this makes the empirical regularities (e.g., intermediate depth peaks, coarse pooling beats global readouts) more credible than a typical one-off analysis.
  2. The findings are turned into concrete design choices (which depth, which readout, which electrodes, which models to fuse) that give large gains, which is useful for people building EEG-to-image decoders.
Critical weaknesses

As a non-expert on EEG decoding, I cannot judge the neuroscience side of this paper, so my comments are limited to presentation and what I could check in the text.

  1. The paper is difficult to read for a non-expert. Section 3 defines the central instrument (the probe, the "currencies", the paired protocol), but it starts directly with "Why not encoding fit" and gives no overview of the setup before the details.Terms such as "currency", "reservoir state" and "territory" are used without a simple definition, and many supporting results sit in the appendices (G, I, J, K). A short overview at the beginning of Section 3, ideally with a figure, would help a lot.
  2. Some claims look stronger than the evidence. The paper calls its findings "laws" and "firsts", but everything is on one dataset with one fixed host decoder, and the conclusion admits this ("one benchmark, one montage family"). Section 4 also says that the depth-latency rise "reflects the ensemble re-weighting electrodes of different intrinsic latencies" and does not appear at single electrodes, which weakens the headline claim that network depth carries a brain timestamp. For the comparison with the state of the art in Section 7.2 and Figure 5, the published numbers are taken from other papers under other protocols, while the authors' own configuration is chosen after thousands of probe runs, so I am not sure this is an equal-effort comparison.

Overall, the empirical work looks careful and useful, but the paper is hard to follow and I am not able to verify the neuroscience claims, so I lean slightly toward acceptance.

Questions for authors
  • Can you add a short overview at the start of Section 3 that defines a probe cell, the three trial regimes and the train/validation/test splits used for each type of claim?
  • How are multiple comparisons handled, given the very large number of cells and sign tests? How many of the headline comparisons were chosen after looking at the results?
  • The depth-latency effect is 24 ms (177 to 201 ms). What is the temporal resolution of the windows, and what is the per-subject uncertainty of the peak latency?
  • Can the comparison in Section 7.2 be made at equal effort, e.g., by running a strong published decoder with the same configuration search under your final-epoch protocol?
Reviewer_6EvTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a detailed analysis of the correspondence between EEG brain activity and the activation of vision models, in response to the same natural images. For this, an MLP-based contrastive decoder is trained to align vision model activations (targeting a specific structure like the Queries or Keys of a Transformer, at a specific depth) with EEG (at specific channels and time windows). Experiments are presented on the THINGS-EEG2 dataset at three test-time repetitions regimes (1, 4, 80), and suggest that (1) model depth correlates strongly with the brain activity latency, (2) the inverted U-shaped decoding curve can be explained in terms of three phases where EEG-decodable information is integrated into the residual stream, and (3) different model vision families (reconstruction, contrastive, supervised) have different decodability profiles across depth, showing the optimal layer can vary substantially across models. Leveraging these results, the paper presents a fusion pipeline that outperforms existing EEG-to-image baselines.

Critical strengths

The paper presents novel results, with actionable findings. It probes multiple dimensions of the correspondence between EEG brain activity and vision models that were so far untapped, with most previous work focusing on more readily available outputs of the vision models. This deeper dive provides concrete ways to optimize brain-to-image pipelines for performance (e.g. picking a model depth given the vision model objective family), but also provides interesting insights into the temporal and spatial correspondence of vision models with the brain.

Critical weaknesses
  1. The paper mentions that an equivalent encoding analysis was carried but dropped as the similarity curve was shaped differently (Section 3), however no mechanistic explanation is proposed to explain this. As encoding analyses are frequently used for studying the alignment between brain activity and model activations, e.g. as in Chen & Sivakumar (2026) cited by the paper, this gap limits comparability and interpretation.
  2. Terminology and framing are overloaded (e.g., “atlas”, “coasting”, “eras”, “deployment currency”), and several interpretations appear stronger than the evidence (e.g. conclusions on K>Q and coasting, see Questions). This hurts clarity and makes it harder to follow some parts of the manuscript, notably Section 7.
Questions for authors
  1. Why are the encoding results mentioned in Section 3 so different from the decoding results? Under what conditions can we expect encoding and decoding results to agree, and do the presented results weaken existing encoding-based claims?
  2. The analysis of Section 5 concludes that EEG tracks “content” by comparing Key activations (K) and Query activations (Q). However, content in self-attention is commonly understood to be carried by the Value activations (V). Does this result hold up if focusing the contrasts on V instead?
  3. In the “writing, integration, coasting” framework presented in Section 5, can you clarify how the “coasting” phase was identified, e.g. whether a quantifiable criteria was used? For instance, Figure 3b shows that decoding performance decreases a lot over the last layers; this suggests degradation rather than coasting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.