PI-EMG: Enhancing Surface Electromyographic Decoding using Privileged Information

The _emg2pose_ benchmark facilitates training generic models for surface electromyography (sEMG) hand pose decoding, which can be used for novel intuitive human-computer interaction modalities. This benchmark defines two tasks: tracking, where hand pose is predicted from sEMG starting from a ground-truth initial pose, and regression, where the initial pose is not provided. After stabilizing position training, position decoding reduced tracking errors by 3.5% – 5.1% relative to matched velocity models. On the regression task, we find that initial pose conditioning provides valuable privileged information (PI), with multi-task tracking and regression training reducing regression task error by 5.3% – 8.0%. Other privileged information that is not available at test time, such as metadata, can also be used to enhance training. To take advantage of metadata such as user ID and sample ID, we implemented Metadata-conditioned Layers (McLayers), which stochastically use either metadata-conditioned mappings or generic mappings during training, but default to generic mappings at test-time. Our model, PI-EMG, incorporates architecture and hyperparameter optimization, multi-task training, and train-time metadata-conditioning, to reduce causal pose decoding error by 10.0% – 16.4% for tracking and 10.0% – 15.7% for regression on unseen users relative to the vemg2pose baseline. PI-EMG therefore achieves a new published state-of-the-art for causal sEMG decoding on the _emg2pose_ benchmark. Likewise, for the _emg2qwerty_ benchmark, which evaluates two-handed sEMG typing, adding user-wise McLayers to the prior published state-of-the-art model reduced beam search character error rates by 18% for unseen users.