acceptodds
Under review as a conference paper at ICLR 2027

A Capacity-Matched Test of Observed EEG Amplitude for Motor-Imagery Error Ranking

Abstract

Selective prediction lets a motor imagery system defer uncertain electroencephalography (EEG) trials. We ask whether an observed-amplitude score helps rank decoder errors beyond class probabilities and available channels. We test this on a task built from participants imagining movements of the left or right hand in the PhysioNet EEG Motor Movement/Imagery Database. We measure the error rate among accepted trials with normalized partial area under the risk-coverage curve (pAURC), from accepting half of all trials to accepting every trial. Lower values mean fewer accepted-trial errors. In a follow-up comparison, a permutation sham had the same number of model coefficients as the amplitude selector. Under synthetic electromyographic (EMG) contamination, the full selector minus sham pAURC difference was 0.021% lower than under clean trials. Its participant-level interval included zero. Refitting decoders on separate source participants also left the result unresolved. The earlier comparison with a smaller selector did not meet its prespecified criterion and changed model size, so it cannot isolate the amplitude score. The evidence does not resolve whether the score improves error ranking.

Primary area:
applications to neuroscience & cognitive science
Keywords:
selective prediction, risk-coverage analysis, motor-imagery EEG, error ranking, permutation shams, observed amplitude, cross-subject evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EPpFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the observed EEG amplitude during MI tells us anything about decoding errors that the decoder’s confidence, class probabilities and channel availability don’t already show. The idea is to see whether an amplitude-based quality score can improve error ranking in selective prediction.

The authors compare three logistic selectors on top of a fixed decoder: a baseline selector, a full selector that adds the amplitude score, and a permutation-based sham with the same number of coefficients as the full one. The sham was added after the originally planned comparison, against a smaller selector, failed its criterion. Experiments use PhysioNet MI with 86 subjects, and they report pAURC over 50-100% coverage for clean data, synthetic EMG and EOG, and channel loss, plus several sensitivity analyses.

For full vs. sham, the change in the pAURC difference from clean to EMG trials is -0.021 pp, and the 95% CI includes zero. The authors say that, with the current evidence, they can’t tell whether EEG amplitude helps rank decoding errors.

Critical strengths

S1. A well-designed experiment that is both motivating and carefully controlled.

  • The sham shares decoder predictions, targets, perturbations, and coefficient count with the full selector, which separates the amplitude values from the extra fitted capacity. Although it was introduced after the original comparison, it offers a reusable template for testing signal-quality features.

S2. Transparent reporting.

  • The paper clearly separates prespecified, follow-up, and exploratory analyses, openly reports the failed original criterion, and avoids overclaiming. Multiple sensitivity analyses make the limits of the evidence easy to assess.
Critical weaknesses

W1. The primary result is inconclusive, and its informational value is limited.

  • The primary interaction and both direct effects have intervals that include zero, and without a prespecified minimum effect, neither a benefit nor equivalence can be claimed. The disjoint-source refit, which corrects the in-sample selector labels, widens the interval, and the sham intervals condition on a single permutation draw. The contrasts that do exclude zero are post hoc and unadjusted.

W2. Limited novelty and narrow scope.

  • No new method is introduced, and the evidence comes from a single acquisition source, a single decoder, a single amplitude score, and synthetic EMG with an altered time scale, evaluated offline only. The Cho2017 results cannot be pooled with the main analysis, and the released code cannot reproduce the reported estimates.
Questions for authors

Q1. Beyond the unconditional sham and the jackknife, could the authors report the interaction across multiple independently drawn shams and different regularization strengths?

Q2. Why was 50-100% coverage chosen? Is the effect at 10-40% coverage practically meaningful, or could it largely reflect the variance that comes with having few accepted trials?

Q3. If the main contribution is the matched-sham protocol, what effect size would be practically meaningful, and did the current design have enough precision to detect it?

4 comments

The conditional permutation sham has the same number of fitted coefficients as the amplitude selector, but the reported bootstrap intervals condition on saved source- and target-side permutations. An unconditional permutation is included as a sensitivity check. The paper reports no repeated independent sham constructions or regularization-strength sweep, so variation from those choices remains unmeasured (pp. 4–5, 7–8, 11).

The primary endpoint averages accepted-trial risk from 50% through 100% coverage. The 10–40% profile is exploratory and unadjusted across coverage values; it does not replace the designated endpoint (pp. 5, 8–9, 12, 15). The paper does not specify an intended application, an action-cost target, or a minimum meaningful effect, so the current analysis does not identify a practical effect threshold.

Thank you for the additional experiments and detailed responses. I appreciate the authors' efforts to address my primary methodological concerns by incorporating independent permutation-based sham experiments and a sensitivity analysis regarding regularization.

However, since all confidence intervals for the primary evaluation metric of 50-100% coverage include 0, the effect of the observed-amplitude quality score remains uncertain. Given this, I will maintain my original rating.

Additionally, I have noted that the current PDF version of the manuscript is 10 pages long, whereas the version I initially reviewed was 15 pages. Could you please clarify whether the version of the manuscript has been changed?

AuthorsOpenReview ↗

Clarification on the revised manuscript PDF

Yes. The current 10-page PDF is a discussion-period author revision and replaces the 15-page initial submission. The earlier PDF included its appendix after the references; the current main-paper PDF ends at page 10, and a separate supplementary ZIP is attached. OpenReview records two revisions on 18 November at 22:30 and 22:41 EST; the later one is current. It adds the five independently drawn quality-permutation shams and regularization sensitivity analysis (Section 4.5, Table 5, p. 8), and states the narrower scope: the test evaluates one added feature block, including fitted capacity.

The primary 50–100% interval still includes zero. These analyses do not establish a positive amplitude-score effect or practical equivalence; we report the result as unresolved.

Thank you for the clarification regarding the manuscript revision.

Reviewer_PpNfOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether an observed EEG amplitude score improves error ranking in a selective prediction (reject-option) framework for motor imagery brain-computer interfaces. The core question is: does an amplitude-derived quality score (alone) help rank which trials to defer?

The authors construct three logistic selectors: (1) a matched selector (baseline features only), (2) a full selector (adds amplitude score + confidence interaction), and (3) a sham selector (same capacity as full, but with permuted scores) and then evaluate them on the PhysioNet EEG Motor Imagery dataset across 86–89 participants. The primary endpoint is normalized partial area under the risk-coverage curve (pAURC) over 50–100% coverage.

The main finding is null though: the full-minus-sham pAURC difference is unresolved across primary and all sensitivity analyses

Critical strengths

1- Sham control is an interesting and smart methodological contribution to measure the contribution of amplitude 2- Thorough null reporting and extensive sensitivity analyses

Critical weaknesses

1- The paper never provides a clear argument for why observed amplitude should predict decoder errors. The authors only mention “missing electrodes and muscle activity can alter the recorded signal.” There’s no model of how amplitude relates to classifier uncertainty, no signal processing motivation, and no discussion of when amplitude would and wouldn’t be informative. Without this, the reviewer has difficulty understanding the contributions of this paper. 2- Primary contribution is a negative result with very narrow scope - The amplitude score tested is a simple global mean absolute value (equation 1), not a sophisticated artifact detector, not a learned quality estimator, and not interpretable as electrode impedance or artifact type. So, the null results is not probably informative 3- Lack of machine learning novelty - I understand this is a paper for the application track. however, one would expects some sort of technical/ML novelty or an empirical finding (positive or negative) that substantially changes our understanding.

Note: even though I do not have sufficient expertise to assess this submission from the application side, I do believe the above comments still apply to this study.

Questions for authors

Please see weaknesses

1 comment

The observed-amplitude score summarizes the measured signal. For each trial, the paper computes mean absolute magnitude over available channels, normalizes by the fraction of channels observed, and applies a sigmoid transform. It does not identify whether a signal comes from EMG, electrode contact, or another source (p. 4).

The experiment holds one PhysioNetMI decoder and its target trials fixed. Synthetic EMG is drawn from one public bank, rescaled to the target length, and added to 30% of test trials. The paper identifies this as a synthetic intervention and says it does not represent natural online EMG. The comparison evaluates offline ranking under that setting; it does not test a deployed deferral action or user-facing utility (pp. 3–4, 9).

The paper describes an anonymous repository with code for matched-sham and synthetic checks, but participant-level records and empirical result files are absent from the submission. The participant-level estimates therefore cannot be independently regenerated from the current package.

Reviewer_ktkhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether an observed-amplitude quality score improves error ranking of a frozen MI-EEG decoder beyond decoder confidence and the channel mask, for selective prediction (accept/defer) on a left-vs-right motor-imagery task from PhysioNetMI. Three logistic selectors share [confidence, class probabilities, mask]; the full selector adds [q, c*q] where q is a masked mean-absolute-amplitude score, and a capacity-matched sham replaces q with scores permuted within predicted-class x channel-count strata without correctness labels. The prespecified criterion - the upper 97.5% bootstrap bound of the clean-to-EMG change in the full-minus-sham pAURC difference (50-100% coverage) below zero - failed (-0.020% [-0.129, +0.095]). Follow-up same-coefficient comparisons (86/89 targets, unconditional sham, leave-one-source jackknife, disjoint-source refit) all have intervals spanning zero. Exploratory analyses show the signal concentrated at 10-40% coverage (interaction -2.47% to -0.41%, CIs excluding zero) and in AURC/AUGRC. The authors conclude the evidence does not resolve whether the score helps.

Critical strengths
  1. The capacity-matched permutation sham is a noval and necessary control: it seems to separate the amplitude feature's information from the added fitted capacity, a confound that pervades the EEG quality-metric literature. The stratified, label-free permutation preserves score marginals within strata.

  2. Inference hygiene is exemplary: the participant is the inference unit, conditions are paired, a prespecified failure criterion is reported as failed, bootstrap seed provenance is documented per result family, and every post-hoc element is explicitly disclosed as such.

Critical weaknesses
  1. No minimum effect of interest (MOE) or equivalence bound. The authors state no MOE was prespecified. With pAURC near 40% and intervals of roughly +/-0.1%, "unresolved" cannot distinguish "no effect" from a practically relevant small effect (0.1-0.5% in an accept/defer pipeline is plausible). Without a TOST-style analysis with a justified margin, the central null result is statistically uninformative.

  2. The manuscript reports a study whose outcome is inconclusive, and it stops there. This is unusual for a conference submission. A study of this scope is normally expected to deliver at least a directional conclusion—positive or negative—and, where the evidence remains ambiguous, to specify what would resolve the ambiguity. As presented, the paper seems largely delivered by LLM agents.

  3. Most serious weakness: the study was designed so that it could not answer its own question. The prespecified endpoint (pAURC over 50–100% coverage) averages away exactly the region where the paper's own exploratory data locate the signal: at 10–40% coverage the interaction intervals exclude zero, yet the endpoint window excludes that region entirely, so the study could not have detected the effect even if it exists.

Questions for authors

why do you want to present an almost exemplary controlled null study, whose answer is statistically undecidable in the absence of an equivalence margin? What is the conclusion this work would like to express or deliver? What is the central question, and what is the meaningful prespecified endpoint the exploratory data indicate? The sham protocol is a genuine methodological contribution better suited to a methods/registered-report venue unles.

1 comment

The study did not prespecify a minimum practically important difference or an equivalence margin. Its original success criterion required the upper 97.5% bootstrap bound for the EMG interaction to fall below zero, and that criterion was not met. The follow-up 86-participant full-selector versus same-capacity-sham interaction is −0.021% in normalized pAURC, with a participant-bootstrap 95% interval of [−0.129%, +0.096%]. The separate-source refit gives −0.043% [−0.247%, +0.159%]. These intervals leave the direction and practical magnitude unresolved; they do not show equivalence (pp. 5–7, 11).

The prespecified endpoint averages accepted-trial risk from 50% through 100% coverage. The 10–40% profile is exploratory and unadjusted across coverage values; it does not replace the endpoint selected for the study (pp. 5, 8–9, 12, 15).

The manuscript specifies no minimum meaningful effect or design target for detecting one. The current analysis therefore does not establish a practical detectable effect.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional permutation-sham and low-coverage analyses

Thank you for the concrete questions. We added five independently drawn quality-permutation shams, each evaluated at C = 0.1, 1, and 10, in a confirmation analysis of 89 PhysioNetMI participants (Section 4.5 and Tables 5–6, p. 8). At the prespecified intact-channel EMG-minus-clean endpoint over 50–100% coverage, observed interactions (full minus matched selector) were −0.039% [−0.139%, +0.065%], −0.025% [−0.143%, +0.106%], and −0.029% [−0.149%, +0.099%]. The observed-minus-sham contrasts against the mean of five shams were −0.030% [−0.130%, +0.079%], −0.014% [−0.131%, +0.120%], and −0.016% [−0.137%, +0.117%], using 10,000 hierarchical participant-and-sham bootstrap draws. All intervals include zero, so these checks do not resolve an effect beyond the permutation control. The observed 95% interval widths are 0.204%, 0.249%, and 0.248%. A −0.1% effect remains within each interval, while a 0.5% effect in either direction lies outside them for this endpoint. Without an application-based minimum effect, the intervals do not establish practical equivalence or importance.

The 50–100% range was frozen as primary to assess ranking while retaining at least half of predictions; it was not derived from application-specific deferral costs. In the exploratory 10–40% range, all 12 C-by-coverage observed-minus-sham risk differences are negative (−2.45% to −0.51%), with unadjusted hierarchical intervals below zero. This shows the low-coverage pattern persists across the five shams and participant resampling, but the saved per-participant outputs omit accepted-trial denominators, so we cannot determine whether sparse acceptance explains it. No application cost or minimum meaningful effect was prespecified. We therefore retain 10–40% as exploratory and do not claim practical utility or equivalence; resolving those questions requires count-aware uncertainty and a prospectively specified cost or effect threshold.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.