- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to tackle the significant latency of vision-based teleoperation. It proposes to utilize an additional sensor modality, i.e., a surface electromyography (sEMG) wristband, to compensate for the latency. The paper proposes Muscles Know First (MKF), a two-stream framework that effectively fuses the vision and sEMG signals.
Critical strengths
- The idea of using sEMG to compensate for the latency of vision-based robot teleoperation is interesting, reasonable, and effective.
- The method is built on a practical assumption of differential latency between vision-based hand tracking and sEMG signals. The proposed MKF tackles these temporally unaligned modalities with a two-stream framework.
- The paper has performed thorough experiments on the emg2pose benchmark.
Critical weaknesses
The biggest issue is that there is no guarantee that the method can be actually applied to real robot teleoperation. The paper does not provide any real-world demonstrations of robot teleoperation, which is far more complicated. Moreover, the experiments only consider a fixed wrist.
Questions for authors
The current experiments only consider a fixed wrist. In real robot teleoperation, the operator needs to move and rotate the wrist while moving the fingers. Will wrist movement affect the sEMG signals for finger prediction, and how can MKF be extended to 6-DoF wrist and arm motion?
4 comments
Author Response to Reviewer XsUM (Part 2/2: Q1 — Wrist/Arm Motion & 6-DoF Extension)
Q1: Will wrist movement affect sEMG signals for finger prediction, and how can MKF be extended to 6-DoF wrist and arm motion?
Response: We greatly appreciate the insights and valuable two-part question.
(1) Does wrist movement hurt sEMG finger prediction? (New 23-stage test across 20 unseen users)
Physiologically, wrist flexion/extension, deviation, and forearm rotation do affect forearm sEMG through shared finger–wrist muscles, stabilizing co-contraction, and slight electrode shifts. Predicting 20 wrist-local finger angles removes global wrist motion from the output space, though raw sEMG still contains both wrist and finger activity.
While our MuJoCo hand mounts a fixed wrist to isolate finger timing, human operators in emg2pose move their wrists and arms freely. To test directly whether active wrist and arm motion degrades MKF, we broke down our 20-unseen-user, 3-run evaluation (2,172 test files at ) across all 23 emg2pose gesture stages (added in Sec. 4 & Appendix G.5 Table 23):
| Gesture Stage (20 Unseen Users, ) | Wrist & Arm Involvement | Files | Stale | Kin-only | MKF (Ours) | Gain vs. Kin |
|---|---|---|---|---|---|---|
| Near/far reach & fist grasp | Full-arm forward reaching & punch grasp | 70 | 16.54° | 10.77° | 7.70° | +28.5% |
| Doorknob turn & fist grasp | Forearm pronation/supination during grasp | 43 | 15.35° | 9.69° | 7.54° | +22.2% |
| Wrist flexion & abduction | Explicit wrist flexion, extension & deviation | 78 | 7.71° | 5.96° | 4.77° | +20.0% |
| Two-handed freestyle motion | Continuous bimanual free-space motion | 154 | 13.21° | 9.91° | 7.99° | +19.4% |
| Fast table-tennis & throwing swings | Rapid forehand/backhand wrist swings | 51 | 12.50° | 8.02° | 6.59° | +17.8% |
| One-handed freestyle motion | Continuous unimanual free-space motion | 114 | 17.05° | 11.95° | 9.97° | +16.6% |
| All 23 Stages Combined | 23 / 23 stages improved (+2.3% to +28.5%, median +16.0%) | 2,172 | 9.26° | 6.86° | 5.73° | +16.4% |
MKF beats Kin-only in every single stage (23/23): even quiet rest/desk stages improve by +2.3%–4.0% (1.887° → 1.843° on unconstrained rest), while active finger, wrist-rotation, and arm-reaching stages improve by +7.8% to +28.5% (median +16.0%). Because MKF re-anchors every 20 ms to and attends over 1 s of 16-channel sEMG trained across varied arm/wrist postures, it separates finger motion bursts from background wrist activity (while physical testing under external object loads remains an important next step).
(2) How MKF extends to 6-DoF wrist and arm teleoperation
In standard vision-based teleoperation systems (e.g., AnyTeleop, Qin et al., 2023), global 6-DoF wrist/arm control and local 20-DoF finger control are already modular (added with diagram in Appendix G.5 Fig. 9):
-
20-DoF finger & wrist-angle pathway (MKF): MKF plugs directly into the hand controller to predict the 20 wrist-local finger joint angles from delayed vision + fresh 16-channel forearm sEMG. Moreover, because forearm sEMG directly captures wrist flexors, extensors, and forearm rotators (as in Kang & Hong, 2025; Kang et al., 2026), MKF's same two-stream architecture can jointly predict wrist rotation angles alongside the 20 finger joints.
-
Parallel 6-DoF arm/wrist pathway: For 3D wrist translation in workspace (driven by elbow and shoulder muscles above the wristband), a parallel kinematic forecaster (Kin-only trunk) predicts the 6-DoF wrist target from the camera's wrist trajectory for the robot arm's IK controller—and can optionally incorporate upper-arm sEMG (Feleke et al., 2021) or onboard wristband motion sensors when available.
The authors' response has clarified some of my concerns. However, without real-world results, I am not fully convinced of the method's applicability to real-world robot teleoperation. Nevertheless, I will maintain my original rating due to the novel exploration of the problem and well-verified results on the benchmark.
Thank you for your support and positive evaluation
Thank you very much for taking the time to read our response and for maintaining your positive rating! We truly appreciate your recognition of our novel exploration of latency-compensated teleoperation and our benchmark verification, and your constructive suggestions on wrist/arm motion and 6-DoF integration have made the paper much clearer and stronger.
Author Response to Reviewer XsUM (Part 1/2: Overview & W1)
Thank you for the positive assessment (Weak acceptance) and for recognizing both the practicality of pairing delayed vision with a low-latency sEMG wristband and our thorough evaluation on emg2pose. Below, we report our newly completed 23-stage wrist/arm evaluation directly in this comment and address your concern and two-part question in order (anonymized code and MuJoCo suite: https://anonymous.4open.science/r/muscles-know-first-34799/).
W1: Real-world robot teleoperation applicability and the fixed-wrist setup
Response: We appreciate this insightful comment; we apologize for not making two key points clearer in the initial manuscript:
(1) Why we use offline replay (and next-step physical deployment): We use offline replay to isolate open-loop forecasting accuracy from operator adaptation: in live teleoperation, operators facing delay instinctively slow down ("move-and-wait") and never repeat the exact same motion twice, which masks true compensation rates across baselines (No compensation, Kin-only, DM-KF, MKF). Replaying emg2pose (370 h, 193 users) through real-tracker degradations (WiLoR, MediaPipe) and a 100 ms robot execution delay lets us compare algorithms fairly on identical trajectories; closing the loop with human-in-the-loop adaptation on physical hardware is our immediate next step (clarified in Abstract, Sec. 1, and Sec. 6).
(2) Human operators move their wrists and arms freely (only the simulated robot base is fixed): In emg2pose, human operators do not fix their arms on a table—they move their wrists and arms freely in space across diverse heights, postures, wrist rotations, and reaches (MKF beats Kin-only in 23/23 stages, shown in Q1 below). We fixed the simulated Shadow Hand base in MuJoCo solely to isolate 20-DoF finger-timing accuracy from 6-DoF robot-arm IK errors—matching how vision teleoperation systems (AnyTeleop, Qin et al., 2023) decouple local hand control from arm control.