acceptodds
Under review as a conference paper at ICLR 2027

BEAM: Auditing the Selection-Specificity Gap in EEG and Vision Encoders

Abstract

EEG–vision alignment connects brain recordings to visual concepts through learned representations. Interpreting these representations requires determining which coordinates support concept recognition and influence model predictions. The value of selecting concept-responsive coordinates depends on how their decoding and intervention outcomes compare with equally sized random subsets. We address this problem with the Brain Encoder Audit with Matched Controls (BEAM). Our contributions are threefold. i) We introduce a matched-control audit that measures the benefit of coordinate selection for both linear decoding and downstream intervention, using shared activation sites, inputs, and subset sizes. ii) We identify a selection-specificity gap across the audited EEG and vision settings on THINGS-EEG2: selection benefits intervention more than decoding. A representative across-subject EEG comparison yields 10.74 times the random-control intervention effect and 1.25 times its probe accuracy. iii) We characterize how the value of selection depends on the operation through removal–replacement asymmetry in a CLIP case study and steering across subsets in EEG, and develop a concentration-and-recoverability account of the gap. BEAM helps EEG–vision researchers distinguish coordinates that carry decodable concept information from those whose selection disproportionately affects model predictions, and test whether targeted steering benefits from concept-based coordinate selection.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
EEG–vision alignment, brain decoding, interpretability, concept representations, linear probing, activation interventions, activation steering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GH65OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks a question about how we interpret EEG-vision alignment models: when researchers select "concept-responsive" coordinates and show that a probe can decode concepts from them, or that zeroing them hurts the model, how much of that result comes from the selection rule itself versus from any equally large random subset? BEAM answers this with a matched- control audit: the same encoder site, inputs, and coordinate budget are used to compare a contrastive-response-selected top-k set against a uniformly sampled random set, on ridge-probe readout accuracy and downstream accuracy drop after intervention (zeroing for EEG, donor substitution for the CLIP vision tower). Selection specificity is the ratio of the selected effect to the mean random effect.

On THINGS-EEG2 (NICE-EEG encoder, CLIP ViT-L/14), all 14 conditions above the reporting threshold show a selection-specificity gap: the intervention ratio exceeds the readout ratio everywhere, reaching 10.74x versus 1.25x for across-subject EEG at a 5% budget. Two companion studies qualify the finding: in CLIP, zeroing the selected coordinates barely moves predictions while donor substitution redirects them (removal-replacement asymmetry); in EEG steering, random coordinate subsets achieve the highest accuracy.

Critical strengths
  1. The matched-control design is the right way to ask this question, and the paper executes it carefully: same activation site, same inputs, matched subset sizes, and a disjoint selection/evaluation concept pool so the selection statistic is never evaluated on the concepts that defined it. This turns a vague suspicion about coordinate-selection analyses into a number you can compute for any encoder, which has real methodological value for this subfield.

  2. The reporting is honest in ways this literature often is not. Negative results are included rather than buried (random subsets win the steering study; the 50% budget cell dilutes the headline ratio to 3.45), the authors mark below-threshold cells explicitly instead of silently dropping them, and the conclusion states upfront that the evidence spans one dataset and one main backbone.

Critical weaknesses
  1. The headline number is a ratio of two near-zero effects, and its stability is not established. The 10.74x intervention specificity at across-subject 5% comes from an absolute drop of 0.69 percentage points against a random-control mean of 0.06 points (Table 1). Ratios at this scale are extremely sensitive to how the random control is sampled and averaged, yet the paper reports no variance or confidence intervals for either the denominator or the ratio, and the 0.05-point reporting threshold is lenient enough to admit effects that are arguably noise.

  2. The generality of the gap is asserted more than shown. One dataset, one EEG alignment method (NICE-EEG), one vision model. The gap is a claim about how decodability and causal influence diverge, which is exactly the kind of claim that could flip with a different alignment objective, yet no second encoder family or dataset is audited on the main grid. The concentration-and-recoverability account is also left verbal; nothing in the paper tests its mechanism, and the authors themselves frame it as future work.

Questions for authors
  1. Could you report the variability of the random-control baseline (across resampled random sets) for each audited cell, with confidence intervals on the specificity ratios? without it, the magnitude of the headline gap is hard to trust even if its direction is plausible.

  2. Does the gap replicate with a different EEG-vision alignment method (e.g., a CLIP-supervised encoder or another contrastive objective) or on a second dataset? A single confirmation on an independently trained encoder would substantially raise confidence that the gap is a property of the setting rather than of NICE-EEG and THINGS-EEG2.

1 comment

Thanks for the reply. My issue has been resolved, I'll keep my rating.

Reviewer_3w1oOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces BEAM (Brain Encoder Audit with Matched controls) to assess the contributions of concept-selective compared to randomly selected dimensions in EEG-to-Image decoding pipeline. Concept-selective dimensions are the ones with highest activation variance across different concepts. The random dimensions are selected from the same layer, same input, and with matched size to the concept-selective ones. BEAM is used for linear probing and intervention tasks to answer the question of whether using concept-selective dimensions instead of random ones benefits any of these tasks. The findings show that concept-selective dimensions are more important for intervention compared to decoding (probing). Finally, the authors show the value of concept-selective dimensions is dependent on the task (operation) they are used for.

Critical strengths

I appreciate the authors effort to motivate their experiments and observations with mathematical explanations.

Critical weaknesses

The main problem of the paper is the text and how it has been written. The paper is written in a convoluted language which is hard to follow and understand. While the words and sentences are grammatically correct, there are lots of new and similar terms spread across the paper and used in abstract convoluted sentences. Even with significant scientific contribution, this alone affects the clarity and usability of the paper for the scientific community.

From the paper, I could not understand how the findings can be applied to scientific problems such as EEG-to-Image decoding. The authors mention in the abstract "BEAM helps EEG– vision researchers distinguish coordinates that carry decodable concept information from those whose selection disproportionately affects model predictions, and test whether targeted steering benefits from concept-based coordinate selection". As far as I understood, BEAM only suggests one selection strategy and evaluates whatever that strategy gives. It does not provide a way to find more informative dimensions. Also, the second motivation (steering benefits...) is rejected in the paper.

The claims of the papers are limited to one dataset of one task and with one type of encoder. So, generalizability of the claims to other settings is not tested.

The numbers reported in the paper are single values without any standard deviation. Given that many of the values are very close, it is hard to judge whether the differences are meaningful.

Questions for authors
  1. Can the authors explain why BEAM is limited to EEG-to-Image task? As I mentioned in the weaknesses, the current findings are weak given the limited evaluation.

  2. Do the authors have any explanation of why across-subject decoding is more sensitive to concept-selective dimensions? Have they investigated the differences between across- and within-subject dimensions?

  3. I believe table 1 should also provide absolute values for σread\sigma_\text{read} (the probe accuracies for concept-selective and random dimensions). Without those values it's hard to say whether the two conditions are equally good or equally bad.

Reviewer_xhfCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the BEAM framework to investigate the relationship between the decodability of features within EEG and visual encoders and their actual impact on model predictions. By isolating "concept-sensitive" feature dimensions and comparing the outcomes of linear decoding and feature intervention against those of random dimensions, the authors discover a "Selection-Specificity Gap": while concept-sensitive dimensions offer only a marginal advantage in terms of decodability, they exert a significantly greater influence on model predictions. Furthermore, the study reveals that feature deletion and feature replacement yield different effects; in fact, random dimensions can sometimes outperform selected dimensions when used for feature steering. Ultimately, the paper concludes that the decodability of a feature, the extent to which a model relies on it, and the degree to which it can be controlled are not equivalent concepts.

Critical strengths
  1. The research problem is innovative. The paper goes beyond merely examining the decodability of information from internal model features by further investigating their practical impact on model predictions. By uncovering the "Selection-Specificity Gap," it offers a fresh perspective for understanding the internal representations within both EEG data and visual encoders.

  2. The methodology is rigorous and versatile. The proposed BEAM framework effectively quantifies the practical value of feature selection strategies by employing randomized control groups—ensuring comparisons of decoding and intervention effects are made under consistent conditions regarding the number of features, model locations, and experimental settings. Furthermore, the framework holds the potential for extension to other models and modalities.

  3. The experimental analysis is comprehensive, and the conclusions are thought-provoking. The study encompasses both cross-subject and within-subject EEG data as well as visual encoders, while exploring various operations such as feature deletion, replacement, and "steering." By integrating theoretical explanations—specifically concerning the concentration of contributions and the recoverability of information—the paper provides valuable evidence for distinguishing between feature decodability, model dependency, and controllability.

Critical weaknesses
  1. Limited scope of experimental data and models. The paper relies primarily on a single dataset, THINGS-EEG2, and focuses on the NICE-EEG encoder, with only limited supplementary experiments conducted on other architectures. The lack of validation across multiple datasets and a broader range of model architectures makes it difficult to definitively establish the universality of the "Selection-Specificity Gap."

  2. Lack of rigor in cross-modal experimental comparisons. The interventions applied to EEG and visual encoders—"Zeroing" and "Substitution," respectively—differ, as do the downstream tasks. Consequently, the observed specificity gap may be influenced by these distinct intervention methods and task configurations, complicating direct comparisons between the modalities.

  3. Weak practical effect size for some experimental results. Although the intervention specificity for cross-subject EEG reaches a factor of 10.74, the corresponding decline in accuracy is only 0.69 percentage points. The substantial relative improvement may be an artifact of an excessively small effect in the random control group; thus, the statistical robustness and practical significance of these findings require further verification.

  4. A disconnect between theoretical explanations and experimental validation. While the paper proposes "contribution concentration" and "information recoverability" to account for the Selection-Specificity Gap, these concepts remain largely at the level of theoretical derivation and indirect experimental support. The absence of direct, quantitative analysis regarding feature redundancy, the geometric structure of the representation space, and the dependencies underlying model predictions means the proposed mechanisms have not been sufficiently validated.

Questions for authors

Please refer to the Critical Weaknesses above (1-4)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.