Typing Matters: Muscle, Language and Typing Behaviour in Live Keystroke Decoding from sEMG

Surface electromyography (sEMG) from the wrists can recover typed text without a keyboard, and is usually posed as a signal-inference problem: characters are read out of the muscle signal and character error rate is reported offline; we argue that this framing removes most of the problem. The decoder is constrained simultaneously by the muscles, by the language being typed, and by typing behaviour such as typos, corrections, runs of backspace, line breaks. A usable system must produce predictions for typed keys live while the hand keeps moving. Existing systems fall short on two counts: (1) they process sEMG signals in fixed windows and carry nothing from one window to the next, discarding accumulated context from the typist over a session, and (2) they lack the understanding of the underlying linguistic structure that drives typing. This means every keystroke is inferred from the muscle signal alone, with little to no ability to leverage any holistic understanding of language. To this end we propose a fully causal, zero-lookahead sEMG typing decoder built on a state space model (SSM), in which a single model can read and decode a whole sEMG session, without ever requiring windowed partitions, and conditioned on what has already been typed. We achieve this through a novel fusion of a traditional typing decoder with an efficient, autoregressive language model, enabling us to instill both sEMG and linguistic understanding in a singular model. Furthermore, to train such a model for typing (per-character prediction) rather than whole-world text, we introduce a retokenization method that rewrites ordinary prose as human-like typing: keystrokes, typos, backspaces and line breaks included. We thus overcome the lack of any large typing corpus by converting the large, text-based WikiText into an accurate, human-like typing dataset. Our model, having never seen a single sEMG-to-character transcript in typing, achieves a state-of-the-art (SOTA) character error rate of $25.13$% on emg2qwerty's cross-user benchmark - a $5.4$% improvement over the previous SOTA with $6.7\times$ fewer parameters - while running strictly causally with no future context at all.