- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the observed EEG amplitude during MI tells us anything about decoding errors that the decoder’s confidence, class probabilities and channel availability don’t already show. The idea is to see whether an amplitude-based quality score can improve error ranking in selective prediction.
The authors compare three logistic selectors on top of a fixed decoder: a baseline selector, a full selector that adds the amplitude score, and a permutation-based sham with the same number of coefficients as the full one. The sham was added after the originally planned comparison, against a smaller selector, failed its criterion. Experiments use PhysioNet MI with 86 subjects, and they report pAURC over 50-100% coverage for clean data, synthetic EMG and EOG, and channel loss, plus several sensitivity analyses.
For full vs. sham, the change in the pAURC difference from clean to EMG trials is -0.021 pp, and the 95% CI includes zero. The authors say that, with the current evidence, they can’t tell whether EEG amplitude helps rank decoding errors.
Critical strengths
S1. A well-designed experiment that is both motivating and carefully controlled.
- The sham shares decoder predictions, targets, perturbations, and coefficient count with the full selector, which separates the amplitude values from the extra fitted capacity. Although it was introduced after the original comparison, it offers a reusable template for testing signal-quality features.
S2. Transparent reporting.
- The paper clearly separates prespecified, follow-up, and exploratory analyses, openly reports the failed original criterion, and avoids overclaiming. Multiple sensitivity analyses make the limits of the evidence easy to assess.
Critical weaknesses
W1. The primary result is inconclusive, and its informational value is limited.
- The primary interaction and both direct effects have intervals that include zero, and without a prespecified minimum effect, neither a benefit nor equivalence can be claimed. The disjoint-source refit, which corrects the in-sample selector labels, widens the interval, and the sham intervals condition on a single permutation draw. The contrasts that do exclude zero are post hoc and unadjusted.
W2. Limited novelty and narrow scope.
- No new method is introduced, and the evidence comes from a single acquisition source, a single decoder, a single amplitude score, and synthetic EMG with an altered time scale, evaluated offline only. The Cho2017 results cannot be pooled with the main analysis, and the released code cannot reproduce the reported estimates.
Questions for authors
Q1. Beyond the unconditional sham and the jackknife, could the authors report the interaction across multiple independently drawn shams and different regularization strengths?
Q2. Why was 50-100% coverage chosen? Is the effect at 10-40% coverage practically meaningful, or could it largely reflect the variance that comes with having few accepted trials?
Q3. If the main contribution is the matched-sham protocol, what effect size would be practically meaningful, and did the current design have enough precision to detect it?
4 comments
Thank you for the additional experiments and detailed responses. I appreciate the authors' efforts to address my primary methodological concerns by incorporating independent permutation-based sham experiments and a sensitivity analysis regarding regularization.
However, since all confidence intervals for the primary evaluation metric of 50-100% coverage include 0, the effect of the observed-amplitude quality score remains uncertain. Given this, I will maintain my original rating.
Additionally, I have noted that the current PDF version of the manuscript is 10 pages long, whereas the version I initially reviewed was 15 pages. Could you please clarify whether the version of the manuscript has been changed?
Clarification on the revised manuscript PDF
Yes. The current 10-page PDF is a discussion-period author revision and replaces the 15-page initial submission. The earlier PDF included its appendix after the references; the current main-paper PDF ends at page 10, and a separate supplementary ZIP is attached. OpenReview records two revisions on 18 November at 22:30 and 22:41 EST; the later one is current. It adds the five independently drawn quality-permutation shams and regularization sensitivity analysis (Section 4.5, Table 5, p. 8), and states the narrower scope: the test evaluates one added feature block, including fitted capacity.
The primary 50–100% interval still includes zero. These analyses do not establish a positive amplitude-score effect or practical equivalence; we report the result as unresolved.
Thank you for the clarification regarding the manuscript revision.
The conditional permutation sham has the same number of fitted coefficients as the amplitude selector, but the reported bootstrap intervals condition on saved source- and target-side permutations. An unconditional permutation is included as a sensitivity check. The paper reports no repeated independent sham constructions or regularization-strength sweep, so variation from those choices remains unmeasured (pp. 4–5, 7–8, 11).
The primary endpoint averages accepted-trial risk from 50% through 100% coverage. The 10–40% profile is exploratory and unadjusted across coverage values; it does not replace the designated endpoint (pp. 5, 8–9, 12, 15). The paper does not specify an intended application, an action-cost target, or a minimum meaningful effect, so the current analysis does not identify a practical effect threshold.