- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MuscleSight, an arm-mounted ultrasound dataset and benchmark for hand decoding, covering 33 participants, 113 sessions, and a range of isolated movements and hand–object interactions. Its central finding is that strong within-session performance does not translate into cross-session or cross-participant generalization. Few-shot session calibration helps substantially, whereas increasing the training population does not consistently improve pose regression under the evaluated setup.
Critical strengths
- I see the evaluation protocol as the strongest contribution. The paper clearly demonstrates how random within-session splits can overestimate progress and makes cross-session and cross-participant generalization explicit benchmark targets.
- The dataset provides a useful combination of repeated sessions, participant diversity, continuous pose labels, and object interactions. This creates opportunities that are difficult to study with the smaller existing ultrasound datasets.
- The negative findings are supported by several model families, constant predictors, representation analysis, and calibration experiments. Using separate trials for calibration and evaluation also makes the adaptation results more credible.
Critical weaknesses
- My main concern is label quality. Pose targets are generated through visual reconstruction and conversion between hand models. The reported conversion residual does not measure the accuracy of the original visual estimates, particularly under object occlusion. This uncertainty matters when interpreting relatively small improvements over constant predictors.
- The participant-scaling conclusion is stronger than the experimental coverage supports. These experiments use one seed and a fixed number of optimization steps while changing both participant diversity and data volume. The results establish a limitation of the evaluated training setup, but do not yet establish that additional participants are generally ineffective.
- Cross-participant evaluation relies on five test participants. Variation across training seeds does not capture uncertainty associated with the choice of held-out participants. Also, tracking is evaluated over one-second windows. The improvement over holding the initial pose is modest, especially on hand–object interactions, so the evidence for sustained transferable tracking remains limited.
Questions for authors
- The paper uses camera-derived pose estimates as supervision and reports residual errors from converting between hand models, however, these residuals do not directly measure the accuracy of the original visual estimates. Could the authors validate a representative subset against an independent reference and report label accuracy separately for isolated movements and occluded hand–object interactions?
- The held-out-participant evaluation includes five test participants, while the reported s.d. mainly describes variation across training seeds. I want to see some additional participant splits or a leave-one-participant-out evaluation to establish whether the generalization gap and calibration benefits are consistent.
- Pose tracking is evaluated over one-second windows and receives the true initial pose. What about longer sequences without reinitialization, and report improvements over holding the initial pose across different motion magnitudes? This would help clarify whether the model learns transferable motion information beyond short-term pose persistence.
5 comments
Author's Response - Thank you for your feedback (Part 2: Q2)
W3 / Q2. Uncertainty over the choice of held-out participants. We agree, and re-evaluated every model on six paired folds that hold out every participant once. The 33 participants are divided into six groups of five or six, with six other participants per fold for validation. Fold k holds out group k in two ways, by a held-out-participant model trained on none of the group's sessions and by a held-out-session model also trained on the group's other sessions, under three constraints:
- Same test set: both models are scored on the same items, the group's held-out sessions.
- Probe site seen in training: every tested recording's probe site appears in training, recorded by other participants for the held-out-participant model and by the same participant for the held-out-session model.
- Equal training size: whole sessions of other participants are removed from the held-out-session training set until it matches the held-out-participant one within 1%.
The random split is drawn six times with different seeds. Held-out number we report in this discussion is a mean over the six folds, and ± s.d. is the standard deviation across folds, so it includes the variation between test participants (Tables R9a, R9b).
Table R9a. Image models on six paired folds: Image2gesture and Image2pose (MuscleSight-Hand, mean ± s.d. over folds or random draws). The six folds hold out every participant once. Bold: best per column (constant predictor not ranked).
| Model | Random split (6 draws) | Held-out session (6 folds) | Held-out participant (6 folds) |
|---|---|---|---|
| Gesture accuracy ↑ | |||
| Constant predictor | 0.217 ± 0.005 | 0.231 ± 0.011 | 0.231 ± 0.011 |
| Lite-CNN | 0.894 ± 0.014 | 0.147 ± 0.050 | 0.164 ± 0.041 |
| Best held-out model (DINOv3, MLP) | 0.550 ± 0.011 | 0.234 ± 0.015 | 0.229 ± 0.021 |
| Pose MAE (degrees) ↓ | |||
| Constant predictor | 18.78 ± 0.04 | 18.74 ± 1.28 | 18.75 ± 1.31 |
| Lite-CNN | 6.50 ± 0.11 | 18.22 ± 0.92 | 18.60 ± 0.97 |
| Lite-CNN + augmentation | 10.04 ± 0.10 | 17.24 ± 0.97 | 17.37 ± 0.77 |
Table R9b. Video models on six paired folds: Video2pose regression and tracking (mean ± s.d. over folds). Seen: windows withheld from the training sessions, scored by the held-out-session network. Bold: best per block and column (constant predictor not ranked).
| Model | Seen | Held-out session (6 folds) | Held-out participant (6 folds) |
|---|---|---|---|
| MuscleSight-Hand, pose regression (MAE, degrees) ↓ | |||
| Constant predictor | 18.79 ± 0.27 | 18.85 ± 1.37 | 18.87 ± 1.42 |
| Lite-CNN | 14.36 ± 2.96 | 17.83 ± 0.98 | 17.72 ± 1.09 |
| Lite-CNN + temporal conv. | 8.56 ± 2.75 | 18.97 ± 1.07 | 18.03 ± 1.36 |
| MuscleSight-Hand, pose tracking (MAE, degrees) ↓ | |||
| Constant predictor (hold the initial pose) | 9.12 ± 0.11 | 9.61 ± 0.56 | 9.61 ± 0.56 |
| Lite-CNN | 9.13 ± 0.09 | 9.70 ± 0.56 | 9.74 ± 0.50 |
| Lite-CNN + temporal conv. | 7.20 ± 0.50 | 8.45 ± 0.43 | 8.50 ± 0.47 |
| MuscleSight-HOI, pose regression (MAE, degrees) ↓ | |||
| Constant predictor | 13.71 ± 0.13 | 13.89 ± 0.46 | 13.89 ± 0.44 |
| Lite-CNN | 12.02 ± 0.63 | 13.99 ± 0.56 | 13.87 ± 0.50 |
| Lite-CNN + temporal conv. | 10.96 ± 1.33 | 14.40 ± 0.57 | 14.57 ± 0.68 |
| MuscleSight-HOI, pose tracking (MAE, degrees) ↓ | |||
| Constant predictor (hold the initial pose) | 7.05 ± 0.10 | 7.41 ± 0.57 | 7.41 ± 0.57 |
| Lite-CNN | 7.08 ± 0.11 | 7.50 ± 0.51 | 7.52 ± 0.57 |
| Lite-CNN + temporal conv. | 6.37 ± 0.19 | 7.00 ± 0.53 | 7.10 ± 0.49 |
Author's Response - Thank you for your feedback (Part 3: Q3 & W2)
W3 (tracking) / Q3. Longer tracking and motion magnitude. We appreciate the suggestion. Rolled out autoregressively without reinitialization, each 1-s window starting from the previous window's prediction, tracking still beats holding the initial pose at 3 s. The gain grows with motion (Table R10). The model therefore tracks motion rather than only persisting the initial pose; its mean gain is diluted by windows in which the hand barely moves.
Table R10. Tracking gain over holding the initial pose (degrees) by motion decile; starts ranked by the constant predictor's MAE, i.e. how far the hand moves from the initial pose, within each fold and horizon (held-out participants, mean over six folds).
| Motion decile (held-out participants) | 1 (least) | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 (most) |
|---|---|---|---|---|---|---|---|---|---|---|
| MuscleSight-Hand | ||||||||||
| Constant predictor MAE, 1 s | 1.4 | 2.8 | 4.0 | 5.0 | 6.1 | 7.5 | 9.3 | 11.8 | 16.4 | 32.0 |
| Tracking gain, 1 s | −0.69 | −0.59 | −0.40 | −0.18 | −0.01 | +0.29 | +0.79 | +1.51 | +3.01 | +7.40 |
| Tracking gain, 3-s rollout | −0.49 | −0.51 | −0.29 | −0.09 | −0.03 | +0.33 | +1.17 | +1.70 | +2.03 | +4.87 |
| MuscleSight-HOI | ||||||||||
| Constant predictor MAE, 1 s | 0.7 | 2.0 | 3.1 | 4.4 | 5.8 | 7.2 | 8.8 | 10.6 | 13.2 | 18.2 |
| Tracking gain, 1 s | −0.15 | −0.32 | −0.34 | −0.27 | −0.14 | −0.00 | +0.24 | +0.53 | +1.08 | +2.49 |
| Tracking gain, 3-s rollout | −0.34 | −0.62 | −0.87 | −0.43 | −0.20 | +0.19 | +0.43 | +0.85 | +1.45 | +2.73 |
W2. Participant scaling. We agree with this reading. We re-ran the experiment on all six participant folds, training every cohort for the same number of epochs (Table R7).
Table R7. Participant scaling with every cohort trained for the same number of epochs (held-out participants; mean ± s.d. over six folds). Calibration: few-shot fine-tuning on half of each test session's calibration pool; all columns are scored on the same test-session items.
| Task (held-out participants) | Smallest cohort | Full pool | Smallest + calibration | Full pool + calibration |
|---|---|---|---|---|
| Image2gesture, balanced accuracy ↑ | 0.073 ± 0.033 | 0.156 ± 0.010 | 0.168 ± 0.021 | 0.230 ± 0.027 |
| Image2pose, MAE ↓ | 18.34 ± 2.28 | 16.32 ± 1.84 | 12.99 ± 1.80 | 12.86 ± 1.84 |
| Video2pose regression, MAE ↓ | 17.67 ± 1.48 | 16.95 ± 2.32 | 13.57 ± 1.77 | 12.84 ± 1.72 |
Author's Response - Thank you for your feedback (Part 1: Q1)
We thank the reviewer for the constructive review and the concrete suggestions, which we have run during the discussion period.
W1 / Q1. Independent mocap reference. We agree that the conversion residual reflects the fidelity of the MANO-to-UmeTrack conversion rather than the accuracy of the labels. To obtain an independent reference, we recorded additional sessions with an electromagnetic data glove (MANUS Quantum Metagloves), a non-visual sensor unaffected by occlusion. We compared the labels with the glove in two ways (Table R6):
Table R6. Labels against an electromagnetic data glove (all 20 joint angles; same participant). Jitter: robust s.d. of the change between consecutive frames at 10 Hz (median over recordings). Same posture, two same-day sessions: s.d. of the same held posture (same task and hand state) in two sessions recorded on the same day. Agreement: postures held still in the same task and hand state, each system averaged over its sessions; r: correlation over the 20 angles.
On MuscleSight-Hand, the labels agree with the glove nearly as closely as either system agrees with itself across sessions, and they vary between sessions about as much as the glove. With objects, agreement drops while remaining more repeatable across sessions than the glove; part of this gap may reflect the glove itself, which can alter the grasp or be affected by metal objects. For the camera-ready version, we will also validate a subset of the labels frame by frame against a calibrated multi-view capture setup, which the data glove does not allow because of the appearance gap between bare and gloved hands.