acceptodds
Under review as a conference paper at ICLR 2027

Typing Matters: Muscle, Language and Typing Behaviour in Live Keystroke Decoding from sEMG

Abstract

Surface electromyography (sEMG) from the wrists can recover typed text without a keyboard, and is usually posed as a signal-inference problem: characters are read out of the muscle signal and character error rate is reported offline; we argue that this framing removes most of the problem. The decoder is constrained simultaneously by the muscles, by the language being typed, and by typing behaviour such as typos, corrections, runs of backspace, line breaks. A usable system must produce predictions for typed keys live while the hand keeps moving. Existing systems fall short on two counts: (1) they process sEMG signals in fixed windows and carry nothing from one window to the next, discarding accumulated context from the typist over a session, and (2) they lack the understanding of the underlying linguistic structure that drives typing. This means every keystroke is inferred from the muscle signal alone, with little to no ability to leverage any holistic understanding of language. To this end we propose a fully causal, zero-lookahead sEMG typing decoder built on a state space model (SSM), in which a single model can read and decode a whole sEMG session, without ever requiring windowed partitions, and conditioned on what has already been typed. We achieve this through a novel fusion of a traditional typing decoder with an efficient, autoregressive language model, enabling us to instill both sEMG and linguistic understanding in a singular model. Furthermore, to train such a model for typing (per-character prediction) rather than whole-world text, we introduce a retokenization method that rewrites ordinary prose as human-like typing: keystrokes, typos, backspaces and line breaks included. We thus overcome the lack of any large typing corpus by converting the large, text-based WikiText into an accurate, human-like typing dataset. Our model, having never seen a single sEMG-to-character transcript in typing, achieves a state-of-the-art (SOTA) character error rate of % on emg2qwerty's cross-user benchmark - a % improvement over the previous SOTA with fewer parameters - while running strictly causally with no future context at all.

Primary area:
applications to neuroscience & cognitive science
Keywords:
sEMG, Mamba, Bio-Signals

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8KesOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to decode keystrokes live from surface electromyography. This study shifts the focus towards the decoder part instead of the encoder. Further, the proposed method expands the limited context size for better decoding. The proposed method achieves state-of-the-art performance on the emg2qwerty benchmark. An ablation study verifies the effectiveness of the proposed components.

Critical strengths
  1. Performance: the proposed method achieves state-of-the-art performance on the emg2qwerty benchmark in the generic setting and presents a comparison of model sizes.

  2. Ablation: the paper presents ablation studies in several settings to verify the effectiveness of the proposed components.

Critical weaknesses
  1. Experiment setting: the emg2qwerty benchmark includes not only generic but also personalized settings. This study does not perform experiments in the personalized setting.

  2. Incremental engineering effort: it seems the proposed method combines the reported research outcomes for better performance. Yet, the performance gain does not seem significant, given that this model uses extra data (Sec 3.3) and is x8 larger (2.2M vs 16.26M).

  3. User study: the user study would verify the paper's claim on real-time decoding.

Questions for authors
  1. Does your model run in real-time? What is the computational cost in edge computing settings?

  2. Please clarify the CER of your causal encoder, which is the removal of the typing model from your model (causal encoder + typing model) in Table 1.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8Kes

We thank the reviewer for the thoughtful review and for acknowledging our performance and ablation studies. We address the weaknesses and questions below.

W1 — Evaluation on the personalized benchmark

We additionally evaluate our typing prior with released personalized TDS encoders across all eight users. It reduces mean CER from 6.91% with the six-gram prior to 6.53%, improving performance for every user (Table R2).

Table R2: Personalized performance, CER (%).

Model / decoderCER ↓
Tiny Transformer [1]9.70
Small Transformer [1]7.90
Large Transformer [1]6.80
Personalized TDS, greedy11.28
Personalized TDS + six-gram6.91
Personalized TDS + our typing prior6.53
Personalized Mamba + our typing prior6.48

W2(a) — Novelty and significance beyond combining existing components

Our main contribution researches a simple question: if the task is decoding typing, why is typing behavior not modeled?

We test this by varying the prior’s training data (Table 2), comparing random errors with our fitted typing process (Table 3), and removing individual retokenizer components (Appendix A.8, Table 12). We also introduce a new method for turning ordinary text into typing data, including errors and corrections which directly benefits EMG decoding.

Our system achieves state-of-the-art cross-user CER. Our typing prior also improves the previous SOTA's Transformer [1] (Table 1), showing that the benefit extends beyond our encoder. As our decoder is compatible with previous works, we hope future methods can build upon our work as a promising model for typing behavior.

W2(b) — Model size and additional training data

Our full system uses 16.24M parameters, compared with 109M for [1]’s strongest encoder (Table 1). Their best 2.2M model is distilled from a 109M teacher, so the comparison should also mention the larger model used to train it.

Creating the additional typing data is also one of our contributions. We turn ordinary prose into keystroke sequences using an error-and-correction process fitted to real typing. Tables 2–3 examine the benefit of this training data.

As mentioned previously, our typing prior also improves Mehlman et al.’s Transformer [1] (Table 1), showing that it can benefit other encoder architectures.

W3 — Live user study

Like prior work on emg2qwerty, we evaluate on publicly released recordings. We do not have access to the research wristbands needed for a live user study. We additionally test streaming by replaying recorded signals at their original acquisition rate. A live user study remains future work.

Q1(a) — Real-time decoding

We replayed a 20.1-minute recording at its original acquisition rate on one NVIDIA H100 PCIe, running the encoder and typing prior. Below are our statistics.

Table R3: Real time decoding statistics

MeasurementResult
Compute real-time factor ↓0.395
Compute throughput2.53× real time
Chunk completion lag, median158 ms
Chunk completion lag, 95th percentile681 ms
Peak GPU tensor memory408.7 MiB

Q1(b) — Edge computing and memory

Ideally, we would measure runtime on the intended edge device using live wristband input. We do not have access to that setup. We instead measured greedy encoder inference on an H100 as stream length increased. GPU memory plateaus near 71 MiB as the stream grows to 20 minutes, and the SSM state remains fixed at 0.79 MB (Appendix A.2, Table 6). Table R4 shows memory does not grow with how long the user has been typing, thus our model is lightweight and deployable.

Table R4: Streaming memory statistics

Input durationPeak GPU tensor memorySSM state
1 s70.71 MiB0.79 MB
300 s71.27 MiB0.79 MB
1200 s71.27 MiB0.79 MB

Q2 — Encoder-only performance

Our encoder-only performance is 37.58 ± 0.50% CER with greedy CTC decoding across five final encoder seeds. Please see Figure 1, Tables 2–3, Appendix A.5 (Table 9) and Appendix A.6 (Table 10) for the encoder-only comparisons and ablations. The drop in performance illustrates the importance of typing.


[1] Mehlman, N., Gagnon-Audet, J.-C., Shvartsman, M., Niu, K., Miller, A. H., & Sodhani, S. (2025). Scaling and Distilling Transformer Models for sEMG. Transactions on Machine Learning Research. See Appendix C.4.

I sincerely appreciate the author taking time for preparing this response.

I think my concerns on the parameter/computational efficiency and the significance of the technical contribution remain unresolved.

  • NVIDIA H100 PCIe is not suited for practical deployment.
  • The distilled model (2.2M) is also included as a baseline, and the proposed model has 16.24M parameters.
  • The improvement from 6.91 to 6.53 does not seem significant taking the extra data into account.

We thank the reviwer for his comments and take this oppertunity to provide furthur evidence below,

1) Parameter size, distillation and extra data

We agree that a 2.2 million parameter model is better suited to edge deployment than a 16 million parameter model, and we acknowledge that our model has seen more data, but note that we already show in the main paper how this extra data benefits previously published baselines (Table 1, “Mehlman et al. Large + our typing model”), with potential benefits for future models. Mehlman et al. also use distillation to obtain their Tiny model. Following this approach, we extend the comparison by distilling both our encoder and typing prior to 1.91 million parameters combined, and evaluating their Tiny model with the same distilled typing prior. When we distill our system to a similar model size, we still outperform their model, both with and without our typing prior. We would also like to point out that this extra typing data is one of the central contributions of this paper.

ModelTotal parametersCross-user CER (%)Personalized CER (%)
Mehlman Tiny + our distilled typing prior2.98M30.116.74
Mehlman Tiny alone2.20M31.909.70
Our distilled encoder + distilled typing prior1.91M28.176.54

Our model reduces cross-user CER by 1.94 percentage points compared with Mehlman Tiny using the same prior, and by 3.73 percentage points compared with Mehlman Tiny alone.

2) Edge computing and runtime

The reviewer is right that an H100 is not an edge device. We have therefore added streaming encoder measurements on an RTX 2080 Ti and CPU-only hardware. Encoder computation runs faster than real time in all three configurations, including 5.2× real time using a single CPU core.

Table R5: Streaming encoder on non-H100 hardware (1.9 M parameter model)

MeasurementRTX 2080 Ti (11 GB) + 1 CPU coreCPU only, 1 coreCPU only, 4 cores
Compute real-time factor ↓0.0330.1910.121
Compute throughput30.6× real time5.2× real time8.3× real time
Chunk compute latency, median16.7 ms97.8 ms62.0 ms
Chunk compute latency, 95th percentile17.1 ms98.9 ms63.2 ms
Peak memory2.9 GB0.86 GB0.86 GB

The H100 affects the reported latency. The fact that we show peak GPU tensor memory under 1 GB in the earlier table shows there is promise for our work on the edge.

3) Extra data

Creating the extra typing data is one of the central contributions of this paper. We also evaluate its benefit on previous models through our typing prior:

  • Main paper, Table 1: Mehlman’s Large encoder with our typing prior achieves 26.77% cross-user CER, compared with its published 30.5%.
  • Table R2: Released personalized TDS encoders improve from 6.91% to 6.53% CER, a 5.5% relative reduction, with improvements for all eight users.
  • The new distillation comparison above: Mehlman Tiny improves from 31.90% to 30.11% cross-user CER when given our distilled typing prior.

The quoted 6.91% → 6.53% result therefore already measures transfer to an existing encoder. Tables 2–3 further examine the contribution of the typing data and how it is generated.

Reviewer_7ZvQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses sEMG-based typing by proposing an sEMG state-space decoder that processes not only the current window of sEMG signals but also their full history. The authors also introduce a typing language model pretrained on text rendered as keystrokes. To build this, they propose a retokenization method that rewrites prose as keystroke sequences through a simulated error-and-correction process. The model is evaluated on the emg2qwerty cross-user benchmark against transformer- and convolution-based baselines, and it achieves the lowest character error rate.

Critical strengths

Strong motivation: The paper clearly identifies what existing EMG typing interfaces need, and the proposed decoder directly addresses this gap.

Strong evaluation: The method is compared against state-of-the-art methods, with an ablation study for each design decision. The proposed typing model also improves the performance of the other methods, which shows its value beyond this work.

Critical weaknesses

My main concern with this work is the writing and organization. The methods section in particular needs improvement, as key elements are introduced without formal definitions. These include the state transition and projection variables (A, B, Ĉ), the parameters at lines 250-251, and the evaluation metric (character error rate). Similarly, the results section, especially the ablation study, does not clearly explain what each configuration represents. In addition, several appendix sections (A.3, A.4, A.5, A.7, A.8) are never referenced in the main text, which weakens the paper’s organization.

The paper reports strong quantitative results but does not interpret them. It lacks a discussion or conclusion section, as well as a section on limitations and future work. As a result, there is no discussion of the proposed work's performance, under which conditions it might fail, or what its limitations are.

Questions for authors

How much does retokenization contribute to performance? How would the results change if a generic typing model without retokenization were used?

Regarding the weaknesses section above, I suggest addressing the missing definitions, clarifying the ablation configurations, adding discussion, limitations, and future work sections, and fixing the appendix references.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7ZvQ

We thank the reviewer for the constructive feedback on presentation, and for recognizing the motivation and evaluation of our work. We address the weaknesses and questions below.

W1a / Q2 — Formal definitions of state-space variables, parameters and CER

State-space variables: We have defined the roles of Ā, B̄ and C̄ immediately after Eq. (1), at line 215. We have now made the reference to Mamba [1] more explicit for its detailed parameterization.

Learned and fixed parameters: These are explicitly identified at lines 252–253. We have now added furthur definitions in the appendix.

CER: We define CER in §4.1, lines 406–408. We have now also added an explicit formula with an explanation in the appendix

All changes have been added in the revised ICLR pdf, with changes highlighted in green. We thank you for your suggestions, which we believe improve the clarity of the paper.

W1b / Q2 — Clear descriptions of ablation configurations

The configurations for Tables 2–5 are described in Sec. 4.3.1 (lines 444–447), Sec. 4.3.2 (lines 451–454), Sec. 4.3.3 (lines 470–474), and Sec. 4.3.4 (lines 478–483), respectively. For additional clarity, we have added a dedicated definitions section in the appendix explaining the ablation configurations.

W1c / Q2 — Appendix references

We have added references to Appendices A.3, A.4, A.5, A.7 and A.8 in the main text.

W2 — Discussion, limitations, conclusion and future work

We have added dedicated discussion section on limitations, conclusion and future work to the revised manuscript.(Highlighted in green)

Q1 — Contribution of retokenization versus a generic typing model

This is one of the main contributions of our work, which we already investigated and examined through three complementary comparisons:

  • Training source: Table 2 (Sec. 4.3.1, lines 444–447) compares a prior trained directly on real keystrokes, a prose prior, and our retokenized-prose prior.
  • Rewriting process: Table 3 (Sec. 4.3.2, lines 451–454) compares random-error rewriting with our fitted process. Both contain errors and backspaces; the comparison examines how those events are generated.
  • Individual components: Appendix A.8, Table 12 (lines 766–786), removes retokenizer components individually and measures their contribution to matching real typing statistics on held-out typists.

[1] Lahoti et al. (2026). Mamba-3: Improved Sequence Modeling Using State Space Principles. ICLR.

We thank the reviewer for their time and consideration. With less than 24 hours left in the discussion, we would appreciate hearing whether our responses have addressed their concerns. We are happy to clarify any remaining points.

I thank the authors for their responses. After reading the responses and the other reviewers’ comments, I have decided to keep my score.

Reviewer_aupkOpenReview ↗
Rating
2(−1)
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper decodes the keystrokes a person actually types, including typos and backspaces, from wrist sEMG over a full session. It combines a causal Mamba encoder trained with CTC, a Mamba typing model trained on WikiText-103 converted into keystroke sequences, and a prefix beam search that fuses their predictions. On eight unseen users in the official emg2qwerty split, it reports 25.13 CER with 16.24M parameters, compared with 30.5 for the best published system’s 109M encoder. Adding the typing model to that system lowers its CER to 26.77. Other text priors perform worse. The paper also examines state resets, causality, and overlap between training text and test prompts.

Critical strengths
  • A transferable typing prior. With the network and search fixed, the typing prior reaches 25.13 CER, against 33.62 for a prose prior and 28.46 for random errors (Tables 2 and 3). It also improves the encoder of Mehlman et al. from 30.5 to 26.77 (Table 1).

  • Strong and efficient results. The system reaches 25.13 CER with 16.24M parameters, against 30.5 from a 109M encoder (Table 1). Analyses of state resets, causality, and test-prompt overlap support the design (Tables 4, 8, and 11).

Critical weaknesses
  • The paper does not evaluate live output. Although the title, abstract, and Fig. 1 describe live decoding, CER is measured after the session (§2), and predictions may use later frames. There is no commit rule, CER for committed output, or latency result.

  • The prose-prior comparison may be unfair. All priors must score backspace as a token (Eq. 7), but the prose prior has not seen backspaces in training. It is also unclear whether α and β_ins were tuned separately for each prior in Table 2.

  • The 1.64-point gap is attributed too strongly to the window (§4.2). The systems also differ in architecture and size (Table 1), so this comparison cannot isolate the window’s effect.

My expertise in sEMG is limited, so I would give particular weight to reviewers with domain expertise on the field-specific aspects of these concerns.

Questions for authors
  • Lack of discussion about limitation, conclusion, and future work.
  • Some tables in the paper could be arranged better like Table 2-5.
3 comments
AuthorsOpenReview ↗

Response to Reviewer aupk

We thank the reviewer for the careful reading and for recognizing the transferable typing prior and the efficiency of our system. We address the weaknesses and questions below.

W1 — Live output, commitment rule, committed CER and latency

Our claim of live refers to our architecture’s ability to predict incrementally with a persistent state. The previous SOTA architecture [1] processes windows independently but do not demonstrate continuous stateful inference, leaving streaming outside their scope. This is the main motivation of our use of a causal SSM. TDS and Splashnet are causal and live, but as we discussed in our Related Works (lines 148–150) our distinction is a decoder for this benchmark that is causal and never restarts.

To maintain fair comparison, we follow the official emg2qwerty decoded-sequence CER evaluation protocol [2]. We additionally report instant committed-output CER in Table R1. Please note that these values follow a different commitment rule and are therefore not directly comparable with official emg2qwerty CER.

Table R1: Committed CER, mean ± SD across five encoder seeds and 16 sessions. Delay is measured from first top-prefix appearance, rather than physical keypress time.

Commitment delay (ms)CER (%)
038.61 ± 0.35
20036.14 ± 0.41
40033.68 ± 0.51
80030.91 ± 0.49
160028.89 ± 0.38
320027.62 ± 0.36

W2 — Fairness of the prose-prior comparison

We agree that the prose-prior comparison alone could be unfair which is why we already included a comparison against a typing-only prior trained on real keystrokes in Table 2 of the main paper(“keystrokes” row, 34.91% CER). This is the central idea of our paper: both language and typing behavior matter.

We further examine typing behavior separately in Table 3, comparing random-error rewriting with our fitted rewriting process. Please note that both variants are trained with errors and backspaces; what changes is how those typing events are generated.

W2(b) — Separate tuning of fusion parameters

We kept α = 1 and β_ins = 2.5 fixed to keep the fusion rule unchanged while varying the prior. This ablation compares different prior-training sources under a common decoding configuration, rather than comparing each prior at its separately tuned optimum. This tests our central idea that the typing prior matters.

W3 — Attributing the performance gap to windowing

We agree that the attribution to windowing was too strong and we have changed it in the revised ICLR manuscript (highlighted in green) to present it as a possible hypothesis, since the comparison involves different models.

Q1 — Discussion, limitations, conclusion and future work

We have added dedicated discussion section on limitations, conclusion and future work to the revised manuscript.(Highlighted in green)

Q2 — Table organization

We have reorganized Tables 2–5 to make the comparisons and ablation settings clearer.


[1] Mehlman, N., Gagnon-Audet, J.-C., Shvartsman, M., Niu, K., Miller, A. H., & Sodhani, S. (2025). Scaling and Distilling Transformer Models for sEMG. Transactions on Machine Learning Research. See Appendix C.4.

[2] Sivakumar, V., Seely, J., Du, A., Bittner, S. R., Berenzweig, A., Bolarinwa, A., Gramfort, A., & Mandel, M. I. (2024). emg2qwerty: A Large Dataset with Baselines for Touch Typing using Surface Electromyography. Advances in Neural Information Processing Systems, 37.

We thank the reviewer for their time and consideration. With less than 24 hours left in the discussion, we would appreciate hearing whether our responses have addressed their concerns. We are happy to clarify any remaining points.

Thanks for the response and the new committed-output results. The added limitations section, the softened windowing claim, and the reorganized tables address my minor points. However, the committed CER shows a large gap from the reported 25.13: 38.61 at zero delay, worse than the greedy readout, and still 27.62 at 3.2 s. The abstract also suggests that no sEMG-to-character transcripts were used, although the encoder is trained on them. As I noted in my review, a large live-performance drop would lower my rating, and given my limited expertise I also defer to the domain-expert reviewers. I'm lowering my score to weak rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.