- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces sEMG-Bench, a standardized, leakage-free benchmark, and audits four representative deep architectures from the literature together with handcrafted feature baselines and a 1D CNN-GRU baseline under shared training and reporting. It states that high offline accuracy in prior work is often misleading because of temporal window leakage from random splits on overlapping frames and because the rest class dominates duration and inflates overall accuracy and macro F1.
Contributions are an open PyTorch pipeline with repetition-grouped two-fold cross-validation and rest-to-repetition indexing; a literature reimplementation study under pooled multi-participant intrapersonal evaluation, per-subject intrapersonal evaluation in the appendix, and interpersonal zero-shot leave-one-subject-out cross-validation; and mechanistic analysis using frozen penultimate linear probes for subject ID and t-SNE of embeddings to support the claim that representations encode biometric subject identity rather than gesture-invariant structure.
Experiments on NinaPro DB5 E1/E2 (10 subjects) use pooled repetition two-fold CV (three seeds) and zero-shot LOSOCV without target calibration, with active macro F1 as the primary metric. LOSOCV yields about 22–25% active macro F1 on E1 versus much higher pooled accuracy; rectification improves pooled end-to-end active macro F1 by roughly six to twenty points. Zero-shot comparisons show no significant difference for the 1D CNN-GRU baseline versus Chen et al. or Cao et al. (p = 0.594, 0.077); subject-ID probes reach about 86% (10% chance). Appendix results on EMG-EPN612 (306 subjects) follow the same evaluation constraints.
Critical strengths
The paper delivers a standardized, leakage-free evaluation framework (repetition-grouped splits, rest indexing, and explicit reporting of active macro F1) and applies it consistently across multiple reimplemented literature models and baselines. This directly addresses two widespread sources of inflated or incomparable sEMG results that the authors document in Table 1 and Section 2. For a benchmark-and-audit submission, this is the main reason the work is useful: it turns fragmented prior numbers into a single, reproducible protocol that future methods can be judged against.
Critical weaknesses
The scientific novelty relative to existing electromyography benchmarking work is limited. The paper itself cites EMGBench and related lines on out-of-distribution and leave-one-subject-out evaluation; the main additions are a NinaPro DB5–focused reimplementation audit, rectification ablations, and rest/leakage messaging that much of the community already treats as evaluation hygiene. The open pipeline is valuable, but the core empirical message ( zero-shot cross-subject active decoding is poor and heavy architectures do not clearly win) is largely confirmatory rather than a new principle or method. That makes the contribution read as an engineering report and literature correction.
The paper’s central claim about latent manifold fragmentation is weaker than the benchmark results. Subject-ID linear probing uses frozen features from gesture models trained on odd repetitions and tested on even repetitions within the same ten subjects, not under LOSOCV where active macro F1 collapses. t-SNE is qualitative and does not include quantitative alignment measures or adaptation experiments that would link the geometry story to zero-shot failure. Because this narrative appears in the title, abstract, and conclusion, it risks overselling what is otherwise a careful audit.
Questions for authors
For LOSOCV results, what input preprocessing was used for every model (signed vs rectified)? If all runs used rectification, please report the zero-shot active macro F1 gap with and without rectification. This affects how we read the architecture comparisons and the abstract’s rectification claim.
For subject-ID linear probing, please state the exact data split: which subjects and repetitions train the gesture model, which train the probe, and which evaluate the probe. In particular, is any held-out LOSO subject included in gesture-model training?
For reimplemented baselines without official code, what validation did you perform that the implementations match the originals?
Paired t-tests are reported across many model pairs and exercises. Did you apply any multiple-comparison correction, and which comparisons are pre-specified versus exploratory?