Muscles Know First: Decoding Hand Motion from Wrist EMG for Latency-Compensated Vision-Based Robot Teleoperation

Vision-based teleoperation has become the most accessible way to drive a dexterous robot hand, but a dramatic delay (150–400 ms) of robot hand motion against the operator’s current pose forces the operator to slow down and wait for the robot, owing to the inherent flaws of the vision-based method. To overcome this limitation, we present Muscles Know First (MKF), a latency-conditioned two-stream forecaster that fuses the fresh surface electromyography (sEMG) stream with the delayed pose stream, predicting the operator’s pose passed onto the robot. Experiments are separately conducted with motion tracking and hand–object contact tasks. For the motion tracking task, MKF recovers 61% of the tracking error caused by latency on unseen users with non-accumulated characteristic, and three control groups attribute the sEMG contribution to its timing rather than to its content. In a MuJoCo Shadow Hand testbed with three contact tasks timed to the operator’s motion, the proposed MKF restores success rate from 57/26/13% to 92/83/67%, respectively, under 300 ms of latency with an inference time of only 8.5 ms. The outcomes of this study reveal a promising way for low-latency and more accurate robot hand teleoperation, potentially promoting dexterous robotic manipulation.