acceptodds
Under review as a conference paper at ICLR 2027

UNDERSTANDING AND IMPROVING CROSS-SESSION GENERALIZATION IN SURFACE-EMG SILENT SPEECH RECOGNITION

Abstract

Silent-speech decoding from surface electromyography (sEMG) offers a non-invasive communication interface, but performance can deteriorate across recording sessions. We use Mindscape-137, to our knowledge the largest open silent speech sEMG corpus by utterance count, with 25,957 silent utterances across 25 sessions from one speaker spanning 21,632 unique sentences and 2,951 word types, to examine signal variation and evaluate interventions for cross-session recognition. Repeated sentences and electrode reapplication between sessions support content-matched comparisons. Channel amplitudes and relationships vary across sessions, while temporal amplitude-envelope features remain informative about repeated-prompt identity. Imposed gain and mixing perturbations increase decoding errors, so the decoder is sensitive to these transformations. Out-of-span augmentation reduces word error rate from 72.15% to 65.77%. In a session-scaling study with three fixed held-out sessions, increasing training coverage from 5 to 22 sessions reduces fresh-sentence word error rate from 67.22% to 28.30%. Checkpoint averaging and latent covariance alignment provide further improvements in the scaling study, whereas the tested rate correction and labeled calibration do not consistently help.

Primary area:
applications to neuroscience & cognitive science
Keywords:
silent speech, surface electromyography, cross-session generalization, domain adaptation, biosignal decoding, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ddRfOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Mindscape-137, a new sEMG-to-text dataset, and uses it to investigate cross-session variability in sEMG recordings, aiming to identify factors that facilitate or hinder cross-session generalization.

Critical strengths

The main contribution is the dataset that the authors plan to release, which they describe as the largest open silent-speech sEMG corpus in terms of utterance count.

Critical weaknesses

In my opinion, the main limitation lies in the paper’s overall framing. On the one hand, the dataset description lacks the level of detail typically expected from a dataset release paper, particularly regarding the data collection protocol, recording characteristics, and statistics on the recordings and associated labels. On the other hand, the experimental analysis provides limited evidence for generalizable conclusions about cross-session variability and how to address it, as it considers only one dataset and one decoding model. Consequently, the findings may be strongly dependent on the specific experimental setup.

Furthermore, I find that several findings, while reasonable, offer limited novelty or actionable insights for the community. For instance, checkpoint averaging is a well-established technique, and previous work on neural recordings has already demonstrated scaling trends showing the benefits of training on larger datasets.

I also have some concerns about the manuscript’s clarity. Several passages required multiple readings to understand the proposed methods and findings. Some concepts and procedures are not sufficiently explained, including the gain and mixing transformations introduced around line 263 and the covariance alignment procedure. Additionally, Figure 1A appears to contain rendering artefacts in my PDF viewer. More importantly, it is not clear until around line 86 that Mindscape-137 is a dataset introduced by the authors themselves.

Finally, as far as I understand, no data have been made publicly available so far, even in the form of a subset. This makes it difficult to independently assess the dataset’s quality and potential utility.

Questions for authors

I would appreciate the authors’ comments on the points raised above.

Reviewer_QjrEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why silent-speech decoding from facial and neck sEMG degrades across sessions. It uses a single-speaker corpus of 25 sessions over 14 days, including 22.1 hours of speech and sentences repeated in every session. Channel gain and mixing vary more across sessions than within them; simulated changes raise a fixed decoder’s WER by up to 6.70 points. Out-of-span augmentation reduces WER on sessions 14 and 15 from 72.15% to 65.77%. Training on 22 rather than 5 sessions reduces fresh-sentence WER on three held-out sessions from 67.22% to 28.30%, though this comparison also changes data volume. Checkpoint averaging and covariance alignment provide smaller gains as coverage grows. Rate correction and labeled calibration do not consistently help. The contributions are the corpus, drift analysis, and evaluation of existing interventions.

Critical strengths
  • The 22.1-hour corpus is larger than those cited in the paper. Repeated anchor sentences support comparisons across sessions with the content held fixed (A.2). The analysis also describes its probes carefully, treating them as tests of what a readout can access (§2.5).

  • The evaluation uses matched controls and reports negative results. Each correction is compared with a plain checkpoint from the same run across all eight evaluations (Table 6). The two runs’ pooled WERs differ by no more than 0.78 points (Table 5).

Critical weaknesses
  • The session-scaling result does not isolate diversity. Training on 22 rather than 5 sessions also changes data volume, optimizer updates, and how close the training dates are to the test dates (§2.4, §3.3).

  • Generalization is tested on one speaker and three interleaved held-out sessions. The 22-session model uses recordings made after every test session (§2.2, Table 3), so performance on a future recording day remains unknown.

  • Out-of-span augmentation is described too briefly to reproduce (§2.4). It is also evaluated with one run per condition on the same sessions 14 and 15.

  • The conclusion that calibration has “nothing left to fix” (§4) goes beyond the evidence from one recipe, one run per size, and fixed decoder settings. Table 8 reports a 3.48-point calibration gain in a different setup, which further calls for a narrower conclusion.

  • The overall presentation of the paper could be improved.

Questions for authors
  • Does training on 22 sessions still help when data volume and optimizer updates are matched to the 5-session setting?

  • How is the span of between-session variation estimated, and what transformations are sampled outside it? Were the settings tuned on sessions 14 and 15, and does the augmentation help in the session-scaling experiment?

Reviewer_fJhdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper primarily investigates how surface electromyography (sEMG) silent speech recognition models can better adapt to different recording sessions for the same individual. The authors use a longitudinal dataset comprising 25 sessions to analyze how channel amplitudes and inter-channel relationships change after reapplying electrodes, and how these changes affect recognition performance. Based on this analysis, the paper compares several methods including data augmentation, increasing training sessions, checkpoint averaging, and target session adaptation. Experiments show that expanding training coverage significantly improves cross-session recognition, while checkpoint averaging and covariance alignment further enhance performance.

Critical strengths
  1. The problem is highly practical, and the research motivation is clear.

For electromyography devices that require repeated wear, whether the model remains effective upon subsequent use is a practical concern. The article addresses this issue by analyzing how signals change over time and exploring ways to improve recognition performance, with a clear research objective.

  1. The design of longitudinal data and repeated sentences is particularly valuable.

The dataset contains extensive recordings of the same participant across multiple sessions, with each session including both repeated and novel sentences. This design facilitates comparing signal changes while controlling for text content, and allows separate evaluation of the model's performance on familiar versus new sentences, providing a solid foundation for analyzing cross-session variations.

  1. The experiments cover various approaches and also report attempts that did not yield consistent improvements.

From signal perturbation and data augmentation to the number of training sessions, checkpoint averaging, and target session adaptation, the paper compares various approaches. Signal-level analysis and final recognition results can also complement each other. Additionally, the authors report that rate correction and labeled calibration failed to consistently improve performance, observations which are also valuable for reference.

Critical weaknesses
  1. The benefits of increasing training sessions still need to be further differentiated.

The model performance improved significantly when the number of training sessions increased from 5 to 22. However, during this process, the number of training samples, the coverage of sentence content, and the number of training updates also increased simultaneously, and the training data became temporally closer to the test sessions. Therefore, it is currently difficult to determine how much of the improvement stems from the diversity of the sessions themselves.

  1. The conclusion about labeled calibration could be stated more cautiously. The paper states that, once training covers ten or more sessions, labeled calibration has “nothing left to fix.” I find this statement somewhat strong. The calibration experiments use a fixed fine-tuning configuration, with only one training run per training-set size. These results show that the tested configuration provides no sustained benefit for models trained on ten or more sessions, but they do not rule out improvements from other calibration strategies. It would be helpful to limit the conclusion to the tested setting, or examine whether the results hold under different fine-tuning configurations.

  2. The specific implementation of out-of-span data augmentation remains unclear.

This method brings relatively noticeable improvements, but the main body primarily explains that it generates transformations beyond the existing session variation space, and the appendix does not provide a complete construction process. For instance, it remains unclear how session variations are represented, how the corresponding space is estimated, or how directions outside the space and perturbation magnitudes are sampled. Since this is a key result of the paper, we recommend supplementing mathematical definitions, pseudocode, and necessary parameters to help readers understand its distinction from standard channel perturbation augmentation and facilitate reproducibility of experiments.

  1. The data is derived from only one participant, and the generalizability of the conclusion remains limited.

Although the dataset contains numerous recordings and multiple collection sessions, all the data come from the same participant. It is not yet clear whether the observed cross-session variation patterns and the effects of different improvement methods also hold true for other users. The authors have acknowledged this limitation, but it still restricts the generalization of these empirical conclusions. It is recommended to verify the main results on the cross-session data from additional participants, even if the scale is smaller, as this can help determine the consistency of these findings.

Questions for authors

Except for the weakness stated above, the primary concern of the paper is the novelty and data scale. Overall, the main contribution of this article lies in the longitudinal data collection for a single subject. However, it seems that the scale of the data is not particularly diverse because there is only one subject. Additionally, the method is not particularly novel. I need the author to elaborate on the novelty further in order to allay my concerns. Other questions can refer to the above Weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.