BEAM: Auditing the Selection-Specificity Gap in EEG and Vision Encoders

EEG–vision alignment connects brain recordings to visual concepts through learned representations. Interpreting these representations requires determining which coordinates support concept recognition and influence model predictions. The value of selecting concept-responsive coordinates depends on how their decoding and intervention outcomes compare with equally sized random subsets. We address this problem with the Brain Encoder Audit with Matched Controls (BEAM). Our contributions are threefold. i) We introduce a matched-control audit that measures the benefit of coordinate selection for both linear decoding and downstream intervention, using shared activation sites, inputs, and subset sizes. ii) We identify a selection-specificity gap across the audited EEG and vision settings on THINGS-EEG2: selection benefits intervention more than decoding. A representative across-subject EEG comparison yields 10.74 times the random-control intervention effect and 1.25 times its probe accuracy. iii) We characterize how the value of selection depends on the operation through removal–replacement asymmetry in a CLIP case study and steering across subsets in EEG, and develop a concentration-and-recoverability account of the gap. BEAM helps EEG–vision researchers distinguish coordinates that carry decodable concept information from those whose selection disproportionately affects model predictions, and test whether targeted steering benefits from concept-based coordinate selection.