acceptodds
Under review as a conference paper at ICLR 2027

Variance Is Not Information: Why Priors over EEG Cannot Close the Trial-Averaging Gap

11 views

Abstract

Every headline number on THINGS-EEG2, the standard benchmark for visual decoding from EEG, is computed on responses averaged over 80 repetitions of the same stimulus. The same frozen encoder scores at 80 repetitions and at one, and two reproduced 2026 baselines keep and of their headline number at one repetition. We ask whether that gap can be closed algorithmically and answer no, in a strong sense: classical MMSE estimators recover none of it, and the provably optimal diffusion prior over EEG, which is available in closed form and hence immune to a training-quality objection, recovers none of it and at actively hurts. One measurement explains why. Ordering the response's principal directions by variance, the 32 leading directions carry of the estimated signal energy, retrieve at chance in isolation, and deleting them is free; keep- accuracy does not saturate until . The discriminative signal is high-rank and anti-aligned with variance, so any prior, estimator or likelihood-maximisation step that concentrates its estimate where the variance is allocates capacity where the information is not. Consistently, the estimator closer to the high-SNR response in NMSE loses four points of Top-1. Both findings replicate on THINGS-MEG. The same principle governs the image side: multi-view target augmentation is worth , yet an operator fitted to reproduce that target reaches cosine on held-out images while delivering of its benefit, making it nonlinear adds Top-1, and training it on the retrieval loss instead gives a family whose optimum is the identity. Reproducing a target and reproducing its value are different problems. Progress on this benchmark is measurement-bound rather than model-bound.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG decoding, visual decoding, brain-computer interface, diffusion priors, contrastive representation learning, CLIP, benchmark analysis, negative results

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5WLgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper focuses on an important issue in neural decoding, the gap between average and trial, which is also the gap between offline and online decoding. However, it's clear that the paper is not ready for submission. Comprehensive analysis, plotting, and writing are necessary to enhance the manuscript.

Critical strengths

The research topic is interesting.

Critical weaknesses

The paper is not complete.

Questions for authors

Please further improve the work.

Reviewer_mZEhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the gap between decoding performed with averaged EEG and with single trials on THINGS-EEG2. It shows that for the same frozen encoder, Top-1 retrieval drops from 0.841 with 80 repetitions to 0.158 with one, and two reproduced 2026 baselines show the same pattern. The paper then studies whether this gap can be closed on the EEG side, and finds that everything it tries fails: classical estimators, a closed-form diffusion prior, learned denoisers, retraining and distillation. To explain this, it proposes a mechanism, namely that the highest-variance directions of the EEG are not the ones that carry stimulus information (variance is not information), and it replicates this on MEG. On the target side, a blurred 24-view image target helps a lot (+0.278), but a learned operator that tries to reproduce it recovers only about 17% of that gain. The final findings of this paper show that pooling across subjects beats adding repetitions, but this cannot be distilled into a single-subject model, and the paper recommends reporting full trials-to-accuracy curves instead of single numbers.

Critical strengths

The main strength of this paper is how carefully it analyzes what the THINGS-EEG2 benchmark actually measures. The repetition curve (0.841 with 80 repetitions vs 0.158 with one) is shown on three encoders, two of which are independent reproductions within one point of what was published. This is especially relevant as “report the curve, not the point” is a standard the field could actually adopt, and it changes how any single-number result on this dataset should be read. I also appreciate the supplementary material: every number can be traced back to a raw result file, the tables I checked reproduce exactly, and the code runs as shipped.

The paper also stands out for how systematic and honest its negative results are, which is not that common. Classical estimators, a closed-form prior, learned denoisers, retraining and distillation are all tested, with careful controls and a replication on MEG. The authors even report results that go against their own account, such as the learned operator recovering 58% of the target-side gain on MEG but only 17% on EEG. This candor is precisely what makes the negative results informative, and not just a list of things that didn't work.

Critical weaknesses

A major flaw in the paper resides in its strongest negative result, the diffusion “upper bound”. The paper calls this prior “provably optimal” and an upper bound, but the closed form it uses is only optimal over a fixed reference set, and here that set only contains training responses, while the test images are new. In practice the prior pulls the EEG of a new image toward the responses to other images, so it behaves like a nearest-neighbour average and cannot know anything about the test image. The paper’s own Table 11 shows this pretty clearly: with a full denoising step, accuracy is at chance for every noise level, and the best setting is the one that keeps 90% of the raw input.

Because of this, the “cannot” in the title rests on a prior that is built to fail on new images, which doesn't say much about priors in general. I would recommend renaming it (e.g., a nearest-neighbour denoiser), adding a learned diffusion prior trained on the training responses and tested on held-out images, and scoping the title and abstract to the methods that were actually tested.

A second major issue is that the EEG-side evidence for the mechanism has a few measurement confounds. First, the Wiener filter and the “full” learned denoisers only work in the top 512 of about 15,750 PCA dimensions and throw the rest away (this is visible in the supplementary code), while the plain average keeps everything. The projection alone already drops accuracy from 0.156 to 0.115, which is exactly what the “full” linear denoiser scores, so it isn't the denoiser itself that fails. Second, everything is read through a frozen encoder that is sensitive to any change in its input: even shrinking the true 80-repetition response drops accuracy from 0.841 to 0.247. So an accuracy drop cannot separate lost information from input that just looks unfamiliar. Third, Table 4 does not really show “anti-alignment”, since the top 512 directions keep about 68% of the accuracy and only the first ~32 are uninformative. Finally, the “signal energy” estimate assumes the noise is uncorrelated across features, which is not true for EEG, so the top variance directions might be mostly noise to begin with.

This matters because the mechanism is the main new contribution of the paper and gives it its title. The main gap in Table 1 is not affected, but its explanation is. I would recommend refitting the denoisers at full rank (or cleaning only the top subspace and keeping the rest), training a fresh linear decoder on each modified input, and ordering directions by signal-to-noise, with a noise covariance estimated from the repetitions, instead of by variance.

Other minor points: the text says no classical estimator beats the plain mean, but Table 12 shows two that do (by tiny margins). The EEG row of Table 10 uses medians while the text uses means, which creates an apparent outlier (8.1%) that disappears with means. The Table 13 caption says the “golden” rows shrink toward the origin, but the code shrinks them toward the average training response. The “canonical-correlation basis” is in fact a PCA basis, since no CCA is computed. There are also some factual slips: NeuroBridge reports 63.2% (not 62.3%), THINGS-MEG has 272 gradiometers (not 271 magnetometers), and appendix sections B and E are empty.

Questions for authors
  1. [Clarification] Diffusion prior. Would the authors be open to renaming the closed-form prior and adding a learned prior, or an oracle reference set built from held-out repetitions of the test images? If a prior that actually generalizes also fails, I would consider the closure claim substantiated.

  2. [Clarification] Novelty. The paper says that no published method reports this curve, but NICE (which is cited) compares 5 to 80 repetitions, and Banville et al. (2025) cover 1 to 80 repetitions within and across subjects on THINGS-EEG2. Single-trial and cross-subject results also exist (e.g., Benchetrit et al., ICLR 2024). How does this contribution relate to them?

  3. [Clarification] Full-rank denoisers. Do the conclusions of Section 5 still hold with a denoiser that keeps all dimensions, and with a readout retrained on each modified input? If so, that would remove my main concern about that section.

  4. [Clarification] Table 6 fusion. In the code, Table 6 averages the similarity matrices of 10 separately trained encoders, which is basically an ensemble of models. With embedding averaging instead, 10 subjects × 1 repetition (0.622) actually falls below 1 subject × 10 repetitions (0.646). Could the authors state the fusion rule in the paper and add a same-subject ensemble control?

  5. [Clarification] Table 4 subjects. Table 4 is captioned “10 subjects”, but every value in its result file is a multiple of 1/3000, which 10 subjects × 5 draws × 200 images cannot produce. How many subjects and draws were actually used?

  6. [Clarification] Encoder. The main encoder is never described. The code says it builds on the UBP codebase (CVPR 2025), which is not cited, and no checkpoint is included even though Table 1 mentions a released one. Could the authors describe the encoder and cite its origin?

  7. [Clarification] Target vs gallery. Each arm’s test gallery is built from its own target, so the +0.278 of the blurred target changes both training and the gallery. Reporting the 2 × 2 (training target × gallery target) would help separate the two effects.

Reviewer_UMfiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This study aims to test whether the decoding of images from EEG activity reflects single trial decoding, and whether the performance obtained by averaging multiple trials together can be recoverable. they Authors argues that denoisers and priors cannot recover some information.

Critical strengths
  • Replicate EEG analyses on MEG dataset (although with quite a distinct results, as 58% of the performance seems to be recovered there)
  • I agree with the authors that the field should focus on single trial decoding, which are easier to compare across devices, studies, etc; and are also more relevant for subsequent BCI use.
Critical weaknesses
  • The trial-averaging gap was already systematically evaluated, e.g. Banville et al. (2025) (including multiple EEG, MEG, and fMRI datasets). The contribution of the present paper is narrower that what is being claimed: i.e. EEG priors and denoisers cannot close it on THINGS-EEG2.

  • the authors note: "classical MMSE estimators recover none of it, and the provably optimal diffusion prior over EEG, which is available in closed form and hence immune to a training-quality objection, recovers none of it". This claim is not justified. These analysis assume repetitions are noisy copies of one fixed response. This assumption does not separate stimulus-related variance from session-, electrode-, and brain-state (e.g. evoked versus induced) variance which could explain why variance-based denoising fails. Consequently, the main conclusion relies on the fact that single trial performance is lower than trial-average, which is known and expected, while the upper bound for single trial remain undetermined.

Minor:

  • The paper really read like a Claude-written style, which I must say is getting a bit tiring.
  • There is little empirical report: no decoding curve, no EEG or MEG topographies
Questions for authors
  • Why call the diffusion prior “provably optimal”? It is only the best fit to the memorized training responses, not to EEG responses in general?
  • How much of the large EEG variance comes from the stimulus, and how much from session, electrodes, or brain state?
  • Can you show decoding over time and EEG/MEG topographies? are there specific time points / electrodes that are more or less subject to this variance issue?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.