- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper asks a question about how we interpret EEG-vision alignment models: when researchers select "concept-responsive" coordinates and show that a probe can decode concepts from them, or that zeroing them hurts the model, how much of that result comes from the selection rule itself versus from any equally large random subset? BEAM answers this with a matched- control audit: the same encoder site, inputs, and coordinate budget are used to compare a contrastive-response-selected top-k set against a uniformly sampled random set, on ridge-probe readout accuracy and downstream accuracy drop after intervention (zeroing for EEG, donor substitution for the CLIP vision tower). Selection specificity is the ratio of the selected effect to the mean random effect.
On THINGS-EEG2 (NICE-EEG encoder, CLIP ViT-L/14), all 14 conditions above the reporting threshold show a selection-specificity gap: the intervention ratio exceeds the readout ratio everywhere, reaching 10.74x versus 1.25x for across-subject EEG at a 5% budget. Two companion studies qualify the finding: in CLIP, zeroing the selected coordinates barely moves predictions while donor substitution redirects them (removal-replacement asymmetry); in EEG steering, random coordinate subsets achieve the highest accuracy.
Critical strengths
-
The matched-control design is the right way to ask this question, and the paper executes it carefully: same activation site, same inputs, matched subset sizes, and a disjoint selection/evaluation concept pool so the selection statistic is never evaluated on the concepts that defined it. This turns a vague suspicion about coordinate-selection analyses into a number you can compute for any encoder, which has real methodological value for this subfield.
-
The reporting is honest in ways this literature often is not. Negative results are included rather than buried (random subsets win the steering study; the 50% budget cell dilutes the headline ratio to 3.45), the authors mark below-threshold cells explicitly instead of silently dropping them, and the conclusion states upfront that the evidence spans one dataset and one main backbone.
Critical weaknesses
-
The headline number is a ratio of two near-zero effects, and its stability is not established. The 10.74x intervention specificity at across-subject 5% comes from an absolute drop of 0.69 percentage points against a random-control mean of 0.06 points (Table 1). Ratios at this scale are extremely sensitive to how the random control is sampled and averaged, yet the paper reports no variance or confidence intervals for either the denominator or the ratio, and the 0.05-point reporting threshold is lenient enough to admit effects that are arguably noise.
-
The generality of the gap is asserted more than shown. One dataset, one EEG alignment method (NICE-EEG), one vision model. The gap is a claim about how decodability and causal influence diverge, which is exactly the kind of claim that could flip with a different alignment objective, yet no second encoder family or dataset is audited on the main grid. The concentration-and-recoverability account is also left verbal; nothing in the paper tests its mechanism, and the authors themselves frame it as future work.
Questions for authors
-
Could you report the variability of the random-control baseline (across resampled random sets) for each audited cell, with confidence intervals on the specificity ratios? without it, the magnitude of the headline gap is hard to trust even if its direction is plausible.
-
Does the gap replicate with a different EEG-vision alignment method (e.g., a CLIP-supervised encoder or another contrastive objective) or on a second dataset? A single confirmation on an independently trained encoder would substantially raise confidence that the gap is a property of the setting rather than of NICE-EEG and THINGS-EEG2.
Thanks for the reply. My issue has been resolved, I'll keep my rating.