acceptodds
Under review as a conference paper at ICLR 2027

MuscleSight: A Benchmark for Ultrasound Representations of Muscle State in Manual Tasks

Abstract

Arm-mounted ultrasound images the muscles that move the hand, including deep muscles that surface electromyography cannot isolate. Existing models report strong gesture and pose decoding, but are evaluated mostly within participants and sessions. We introduce MuscleSight, a dataset and benchmark for testing whether ultrasound-based hand decoding generalizes: 37.7 hours from 33 participants across 113 sessions and 37 manual tasks, including isolated motions, static postures, and interactions with 72 distinct objects. Synchronized multi-view videos provide audited hand-pose and gesture labels. Our benchmark evaluates frame- and clip-based gesture classification, pose regression, and pose tracking within sessions and on held-out sessions and participants. Within sessions, a 0.28 M-parameter CNN reaches 89.5% gesture accuracy (majority baseline: 21.0%). On held-out sessions and participants, however, gesture classifiers fail to beat the majority baseline, pose regressors improve over a constant mean pose by at most 1.8°, and only initialized pose tracking transfers reliably. Learned representations strongly encode recording session, revealing substantial session-specific structure. Few-shot calibration on each new session improves every calibrated model, while simply adding training participants does not: a calibrated pose model trained on 2 participants outperforms an uncalibrated model trained on 28. Data, labels, splits, and code will be released.

Primary area:
datasets and benchmarks
Keywords:
Representation learning, hand-object interaction, muscle imaging

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TccdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MuscleSight, an arm-mounted ultrasound dataset and benchmark for hand decoding, covering 33 participants, 113 sessions, and a range of isolated movements and hand–object interactions. Its central finding is that strong within-session performance does not translate into cross-session or cross-participant generalization. Few-shot session calibration helps substantially, whereas increasing the training population does not consistently improve pose regression under the evaluated setup.

Critical strengths
  • I see the evaluation protocol as the strongest contribution. The paper clearly demonstrates how random within-session splits can overestimate progress and makes cross-session and cross-participant generalization explicit benchmark targets.
  • The dataset provides a useful combination of repeated sessions, participant diversity, continuous pose labels, and object interactions. This creates opportunities that are difficult to study with the smaller existing ultrasound datasets.
  • The negative findings are supported by several model families, constant predictors, representation analysis, and calibration experiments. Using separate trials for calibration and evaluation also makes the adaptation results more credible.
Critical weaknesses
  • My main concern is label quality. Pose targets are generated through visual reconstruction and conversion between hand models. The reported conversion residual does not measure the accuracy of the original visual estimates, particularly under object occlusion. This uncertainty matters when interpreting relatively small improvements over constant predictors.
  • The participant-scaling conclusion is stronger than the experimental coverage supports. These experiments use one seed and a fixed number of optimization steps while changing both participant diversity and data volume. The results establish a limitation of the evaluated training setup, but do not yet establish that additional participants are generally ineffective.
  • Cross-participant evaluation relies on five test participants. Variation across training seeds does not capture uncertainty associated with the choice of held-out participants. Also, tracking is evaluated over one-second windows. The improvement over holding the initial pose is modest, especially on hand–object interactions, so the evidence for sustained transferable tracking remains limited.
Questions for authors
  1. The paper uses camera-derived pose estimates as supervision and reports residual errors from converting between hand models, however, these residuals do not directly measure the accuracy of the original visual estimates. Could the authors validate a representative subset against an independent reference and report label accuracy separately for isolated movements and occluded hand–object interactions?
  2. The held-out-participant evaluation includes five test participants, while the reported s.d. mainly describes variation across training seeds. I want to see some additional participant splits or a leave-one-participant-out evaluation to establish whether the generalization gap and calibration benefits are consistent.
  3. Pose tracking is evaluated over one-second windows and receives the true initial pose. What about longer sequences without reinitialization, and report improvements over holding the initial pose across different motion magnitudes? This would help clarify whether the model learns transferable motion information beyond short-term pose persistence.
5 comments
AuthorsOpenReview ↗

Author's Response - Thank you for your feedback (Part 1: Q1)

We thank the reviewer for the constructive review and the concrete suggestions, which we have run during the discussion period.

W1 / Q1. Independent mocap reference. We agree that the conversion residual reflects the fidelity of the MANO-to-UmeTrack conversion rather than the accuracy of the labels. To obtain an independent reference, we recorded additional sessions with an electromagnetic data glove (MANUS Quantum Metagloves), a non-visual sensor unaffected by occlusion. We compared the labels with the glove in two ways (Table R6):

  • Precision. Frame-to-frame jitter is the robust standard deviation of the change between consecutive frames at 10 Hz. Between-session variation is the standard deviation of the same held posture (same task and hand state) in two sessions recorded on the same day.
  • Agreement on held postures. For the participant who wore the glove on the instrumented hand, we paired postures held still in the same task and hand state, averaged each system over its sessions. Comparing each system with itself across two sessions gives the natural variation of the posture as a reference.

Table R6. Labels against an electromagnetic data glove (all 20 joint angles; same participant). Jitter: robust s.d. of the change between consecutive frames at 10 Hz (median over recordings). Same posture, two same-day sessions: s.d. of the same held posture (same task and hand state) in two sessions recorded on the same day. Agreement: postures held still in the same task and hand state, each system averaged over its sessions; r: correlation over the 20 angles.

MuscleSight-HandMuscleSight-HOI
Precision (°)
Frame-to-frame jitter, labels1.11.7
Frame-to-frame jitter, glove0.60.7
Same posture, two same-day sessions (SD), labels6.87.5
Same posture, two same-day sessions (SD), glove6.413.4
Agreement on held postures
Labels vs glover 0.93, slope 1.02r 0.81, slope 0.89
Labels vs labels, two sessionsr 0.96, slope 1.01r 0.95, slope 0.99
Glove vs glove, two sessionsr 0.97, slope 0.98r 0.87, slope 0.99

On MuscleSight-Hand, the labels agree with the glove nearly as closely as either system agrees with itself across sessions, and they vary between sessions about as much as the glove. With objects, agreement drops while remaining more repeatable across sessions than the glove; part of this gap may reflect the glove itself, which can alter the grasp or be affected by metal objects. For the camera-ready version, we will also validate a subset of the labels frame by frame against a calibrated multi-view capture setup, which the data glove does not allow because of the appearance gap between bare and gloved hands.

AuthorsOpenReview ↗

Author's Response - Thank you for your feedback (Part 2: Q2)

W3 / Q2. Uncertainty over the choice of held-out participants. We agree, and re-evaluated every model on six paired folds that hold out every participant once. The 33 participants are divided into six groups of five or six, with six other participants per fold for validation. Fold k holds out group k in two ways, by a held-out-participant model trained on none of the group's sessions and by a held-out-session model also trained on the group's other sessions, under three constraints:

  • Same test set: both models are scored on the same items, the group's held-out sessions.
  • Probe site seen in training: every tested recording's probe site appears in training, recorded by other participants for the held-out-participant model and by the same participant for the held-out-session model.
  • Equal training size: whole sessions of other participants are removed from the held-out-session training set until it matches the held-out-participant one within 1%.

The random split is drawn six times with different seeds. Held-out number we report in this discussion is a mean over the six folds, and ± s.d. is the standard deviation across folds, so it includes the variation between test participants (Tables R9a, R9b).

Table R9a. Image models on six paired folds: Image2gesture and Image2pose (MuscleSight-Hand, mean ± s.d. over folds or random draws). The six folds hold out every participant once. Bold: best per column (constant predictor not ranked).

ModelRandom split (6 draws)Held-out session (6 folds)Held-out participant (6 folds)
Gesture accuracy ↑
Constant predictor0.217 ± 0.0050.231 ± 0.0110.231 ± 0.011
Lite-CNN0.894 ± 0.0140.147 ± 0.0500.164 ± 0.041
Best held-out model (DINOv3, MLP)0.550 ± 0.0110.234 ± 0.0150.229 ± 0.021
Pose MAE (degrees) ↓
Constant predictor18.78 ± 0.0418.74 ± 1.2818.75 ± 1.31
Lite-CNN6.50 ± 0.1118.22 ± 0.9218.60 ± 0.97
Lite-CNN + augmentation10.04 ± 0.1017.24 ± 0.9717.37 ± 0.77

Table R9b. Video models on six paired folds: Video2pose regression and tracking (mean ± s.d. over folds). Seen: windows withheld from the training sessions, scored by the held-out-session network. Bold: best per block and column (constant predictor not ranked).

ModelSeenHeld-out session (6 folds)Held-out participant (6 folds)
MuscleSight-Hand, pose regression (MAE, degrees) ↓
Constant predictor18.79 ± 0.2718.85 ± 1.3718.87 ± 1.42
Lite-CNN14.36 ± 2.9617.83 ± 0.9817.72 ± 1.09
Lite-CNN + temporal conv.8.56 ± 2.7518.97 ± 1.0718.03 ± 1.36
MuscleSight-Hand, pose tracking (MAE, degrees) ↓
Constant predictor (hold the initial pose)9.12 ± 0.119.61 ± 0.569.61 ± 0.56
Lite-CNN9.13 ± 0.099.70 ± 0.569.74 ± 0.50
Lite-CNN + temporal conv.7.20 ± 0.508.45 ± 0.438.50 ± 0.47
MuscleSight-HOI, pose regression (MAE, degrees) ↓
Constant predictor13.71 ± 0.1313.89 ± 0.4613.89 ± 0.44
Lite-CNN12.02 ± 0.6313.99 ± 0.5613.87 ± 0.50
Lite-CNN + temporal conv.10.96 ± 1.3314.40 ± 0.5714.57 ± 0.68
MuscleSight-HOI, pose tracking (MAE, degrees) ↓
Constant predictor (hold the initial pose)7.05 ± 0.107.41 ± 0.577.41 ± 0.57
Lite-CNN7.08 ± 0.117.50 ± 0.517.52 ± 0.57
Lite-CNN + temporal conv.6.37 ± 0.197.00 ± 0.537.10 ± 0.49
AuthorsOpenReview ↗

Author's Response - Thank you for your feedback (Part 3: Q3 & W2)

W3 (tracking) / Q3. Longer tracking and motion magnitude. We appreciate the suggestion. Rolled out autoregressively without reinitialization, each 1-s window starting from the previous window's prediction, tracking still beats holding the initial pose at 3 s. The gain grows with motion (Table R10). The model therefore tracks motion rather than only persisting the initial pose; its mean gain is diluted by windows in which the hand barely moves.

Table R10. Tracking gain over holding the initial pose (degrees) by motion decile; starts ranked by the constant predictor's MAE, i.e. how far the hand moves from the initial pose, within each fold and horizon (held-out participants, mean over six folds).

Motion decile (held-out participants)1 (least)2345678910 (most)
MuscleSight-Hand
Constant predictor MAE, 1 s1.42.84.05.06.17.59.311.816.432.0
Tracking gain, 1 s−0.69−0.59−0.40−0.18−0.01+0.29+0.79+1.51+3.01+7.40
Tracking gain, 3-s rollout−0.49−0.51−0.29−0.09−0.03+0.33+1.17+1.70+2.03+4.87
MuscleSight-HOI
Constant predictor MAE, 1 s0.72.03.14.45.87.28.810.613.218.2
Tracking gain, 1 s−0.15−0.32−0.34−0.27−0.14−0.00+0.24+0.53+1.08+2.49
Tracking gain, 3-s rollout−0.34−0.62−0.87−0.43−0.20+0.19+0.43+0.85+1.45+2.73

W2. Participant scaling. We agree with this reading. We re-ran the experiment on all six participant folds, training every cohort for the same number of epochs (Table R7).

Table R7. Participant scaling with every cohort trained for the same number of epochs (held-out participants; mean ± s.d. over six folds). Calibration: few-shot fine-tuning on half of each test session's calibration pool; all columns are scored on the same test-session items.

Task (held-out participants)Smallest cohortFull poolSmallest + calibrationFull pool + calibration
Image2gesture, balanced accuracy ↑0.073 ± 0.0330.156 ± 0.0100.168 ± 0.0210.230 ± 0.027
Image2pose, MAE ↓18.34 ± 2.2816.32 ± 1.8412.99 ± 1.8012.86 ± 1.84
Video2pose regression, MAE ↓17.67 ± 1.4816.95 ± 2.3213.57 ± 1.7712.84 ± 1.72
Reviewer_DyEkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MuscleSight, a dataset and benchmark for decoding hand gestures and articulated hand poses from arm-mounted B-mode ultrasound. The dataset comprises 37.7 hours of recordings from 33 participants across 113 acquisition sessions, covering 37 manual tasks and interactions with 72 objects. It contains two subsets: MuscleSight-Hand, focused on isolated movements and static hand postures, and MuscleSight-HOI, covering hand–object interaction tasks. Ultrasound recordings are synchronized with camera observations, which are processed to obtain hand-pose labels expressed as 20 UmeTrack finger joint angles. The authors also provide gesture and movement-phase annotations for MuscleSight-Hand. The benchmark investigates four tasks: single-frame gesture classification, single-frame pose regression, temporal pose regression, and initialized pose tracking. Crucially, these tasks are evaluated under within-session, held-out-session, and held-out-participant protocols. The paper argues that future ultrasound-based hand-decoding research should prioritize cross-session generalization and efficient calibration.

Critical strengths
  • The dataset is a genuine resource: ~5× the participants of prior downloadable forearm-ultrasound datasets, the first with hand–object interaction, and with audited labels. The dataset includes 33 participants, 113 sessions, multiple probe locations, and diverse manual tasks, including object manipulation.
  • The paper identifies a limitation in the evaluation of ultrasound-based gesture and pose decoding: much of the existing literature emphasizes within-participant or within-session accuracy. The proposed benchmark explicitly separates these conditions from generalization to unseen sessions and participants. This is a valuable contribution because deployment of wearable sensing interfaces inherently involves variations in sensor placement, recording conditions, and users.
  • The representation analysis investigates whether learned embeddings encode participant, session, or movement information. The few-shot calibration experiments further assess whether limited target-session supervision can recover predictive performance. The participant-scaling analysis provides an additional perspective on the relationship between dataset size, transfer, and personalization.
Critical weaknesses
  • Ground truth is WiLoR → MANO → IK → UmeTrack, with a 2.9 mm median conversion error and no independent mocap reference. The audit reports only a flag rate (5.8% Hand, 4.7% HOI), not the error distribution on accepted labels, and a human watching a trajectory catches gross failures rather than systematic per-joint bias. Since model differences are a few tenths of a degree (0.187 vs 18.3 on held-out participant), the label ceiling may account for a meaningful share of the reported gap.
  • Features encoding session identity (section 5.2) is useful, but session is confounded with probe placement, anatomy, gain, task mix, and label quality. The held-out-session split keeps participants and probe sites seen in training, so re-attachment and participant effects are entangled by construction. With six sites covering 96.9% of recordings, a site-controlled or re-attachment analysis is feasible and would raise the benchmark's diagnostic value; a session-ID probe would also help.
  • Fixed optimization steps while varying cohort size changes exposures per example, so the "more training participants do not improve pose regression" result is confounded by under-optimization. It is also single-seed and non-monotonic, and the authors note the largest cohort does not reproduce the §5.3 scores.
  • Nothing motivated by cross-domain generalization is tested: no session-invariant training, no domain adaptation, no probe-location conditioning, no ultrasound-specific SSL. EchoCLIP is a cardiac model, so its weakness says little about ultrasound representations generally.
Questions for authors
  1. Can the authors distinguish session-specific acquisition effects from participant-specific anatomy and differences in task distributions?
  2. How sensitive are the results to synchronization offsets between ultrasound frames and camera-derived pose annotations?
  3. Tracking is initialized with the true pose. Does it remain effective when the initial pose is estimated from the first frame, as it would be at deployment?
  4. How does performance vary across the 13 probe sites, and how much of the held-out-session gap is attributable specifically to probe re-attachment versus anatomy?
6 comments
AuthorsOpenReview ↗

Author's Response - Thank you for your feedback (Part 1: Q1 & Q2)

We thank the reviewer for the detailed review and the concrete suggestions, which we have run during the discussion period.

Q1. Session-specific acquisition effects vs. anatomy and task distribution. We appreciate the comments and have separated them with two controlled comparisons.

(i) Paired folds. We re-evaluated all models on six paired folds. The 33 participants are divided into six groups of five or six, and each group is held out twice: by a held-out-session model that also trains on the same people's other sessions, and by a held-out-participant model that sees none of their sessions. Both models are scored on the same test items, train on equally many frames (within 1%), and have seen every tested probe site in training. Participants, probe sites and tasks evaluated are therefore identical; the held-out-session model has, in addition, seen these participants' anatomy at these sites in other sessions. Nearly all of the gap is present as soon as the session is held out (Table R1).

Table R1. MuscleSight-Hand; mean ± s.d. over six folds, or over six random draws for the random split. Held-out session and held-out participant are scored on the same items with equal training size. Bold: best per task and column (constant predictor not ranked).

Task / model (MuscleSight-Hand)Random split / SeenHeld-out sessionHeld-out participant
Image2gesture, accuracy ↑
Constant predictor0.217 ± 0.0050.231 ± 0.0110.231 ± 0.011
Lite-CNN0.894 ± 0.0140.147 ± 0.0500.164 ± 0.041
Best held-out model (DINOv3, MLP)0.550 ± 0.0110.234 ± 0.0150.229 ± 0.021
Image2pose, MAE (degrees) ↓
Constant predictor18.78 ± 0.0418.74 ± 1.2818.75 ± 1.31
Lite-CNN6.50 ± 0.1118.22 ± 0.9218.60 ± 0.97
Lite-CNN + augmentation10.04 ± 0.1017.24 ± 0.9717.37 ± 0.77
Video2pose regression, MAE (degrees) ↓
Constant predictor18.79 ± 0.2718.85 ± 1.3718.87 ± 1.42
Lite-CNN14.36 ± 2.9617.83 ± 0.9817.72 ± 1.09
Lite-CNN + temporal conv.8.56 ± 2.7518.97 ± 1.0718.03 ± 1.36
Video2pose tracking, MAE (degrees) ↓
Constant predictor (hold the initial pose)9.12 ± 0.119.61 ± 0.569.61 ± 0.56
Lite-CNN + temporal conv.7.20 ± 0.508.45 ± 0.438.50 ± 0.47

(ii) Dual-probe recordings. In dual-probe sessions two probes image two sites at the same instants, so participant, session, task and pose label are shared and only the site, i.e. the anatomy imaged, differs (Table R2). On held-out data the two sites of a pair differ by at most 2.0°, small against the 6–10° by which every site's error rises from the random split to the held-out splits.

Table R2. Two probe sites imaged at the same instants (Image2pose MAE in degrees at the first / second site, Lite-CNN + augmentation; mean ± s.d. over the draws or folds that test the pair).

Sites (first / second)Random splitHeld-out sessionHeld-out participant
FA-A-M / FA-A-P9.43 ± 0.13 / 9.87 ± 0.2516.26 ± 1.59 / 16.93 ± 2.0315.96 ± 1.11 / 17.52 ± 1.98
FA-P-D / FA-P-P10.71 ± 0.54 / 10.58 ± 0.4120.22 ± 5.11 / 18.24 ± 1.8820.07 ± 5.62 / 18.48 ± 3.02
FA-A-D / FA-P-M10.23 ± 0.25 / 10.43 ± 0.6419.68 ± 3.05 / 18.47 ± 0.0519.16 ± 3.48 / 20.20 ± 0.20

Q2. Synchronization sensitivity. To measure sensitivity to synchronization, we re-paired every test pose with the ultrasound frame nearest t − δ, for offsets up to ±800 ms, without retraining (Table R3). Within the 50-ms pairing tolerance no held-out score moves by more than 0.02°. Even at ±800 ms, the held-out Image2pose error rises by at most 0.87°, while the random-split model loses up to 6.98°; for scale, the pose itself changes by 14.1° over 800 ms.

Table R3. Change in Image2pose MAE (degrees; Lite-CNN on MuscleSight-Hand, fold 1) when each test pose is paired with the ultrasound frame nearest t − δ, without retraining.

Offset δ (ms)Random splitHeld-out sessionHeld-out participantPose change over δ
0 (MAE as paired)6.4418.8920.050
−800+6.26+0.76+0.4714.1
−200+0.50+0.09+0.035.0
−50−0.05−0.01−0.001.5
+50+0.23+0.02+0.021.5
+200+1.03+0.16+0.115.0
+800+6.98+0.87+0.6214.1
AuthorsOpenReview ↗

Author's Response - Thank you for your feedback (Part 2: Q3)

Q3. Tracking with an estimated initial pose. We thank the reviewer for this question (Table R4). An estimated initial pose arises in two ways. (a) It can be predicted from the window's first frame by a separate model (Image2pose or Video2pose regression). (b) In an autoregressive rollout, only the first window starts from the ground-truth pose; every later window starts from the model's own last prediction. In every setting, dataset and held-out split, tracking is more accurate than the constant predictor that holds the same initial pose for every frame.

Deployment. We follow the tracking formulation of emg2pose [1], which also initializes from the ground-truth pose and motivates this with deployment scenarios in which the initial pose comes from outside the sensor: the user is prompted to match a given pose before tracking starts; visual feedback lets the user correct an erroneous initial prediction; or the pose is available intermittently, for example from a camera whenever the hand is not occluded.

Table R4. Video2pose tracking from ground-truth and estimated initial poses (MAE in degrees; mean ± s.d. over six folds; Sess./Part.: held-out session/participant). The constant predictor holds the same initial pose as tracking. Beyond 1 s, each window is initialized with the previous window's prediction. Bold: the lower error of each pair.

Initialization / predictorHand, Sess.Hand, Part.HOI, Sess.HOI, Part.
Ground-truth initialization
1 s
Constant predictor9.61 ± 0.569.61 ± 0.567.41 ± 0.577.41 ± 0.57
Tracking8.45 ± 0.438.50 ± 0.477.00 ± 0.537.10 ± 0.49
2 s, autoregressive rollout
Constant predictor12.88 ± 0.6912.88 ± 0.698.55 ± 0.618.55 ± 0.61
Tracking11.30 ± 0.6911.54 ± 0.688.11 ± 0.548.20 ± 0.51
3 s, autoregressive rollout
Constant predictor12.87 ± 0.7312.87 ± 0.739.07 ± 0.619.07 ± 0.61
Tracking11.86 ± 0.6912.00 ± 0.598.67 ± 0.498.76 ± 0.49
Estimated initialization, 1 s
Initial pose from Image2pose
Constant predictor18.08 ± 0.8518.26 ± 0.5018.39 ± 1.1418.44 ± 1.23
Tracking17.92 ± 0.8618.15 ± 0.4917.94 ± 1.2817.94 ± 1.26
Initial pose from Video2pose regression
Constant predictor19.38 ± 1.0718.21 ± 1.3714.23 ± 0.5214.34 ± 0.56
Tracking19.15 ± 0.8818.12 ± 1.3014.18 ± 0.5714.26 ± 0.60
Reference: direct pose regression, no initial pose, 1 s
Video2pose regression18.98 ± 1.0618.05 ± 1.3614.43 ± 0.5814.60 ± 0.70
Image2pose, every frame17.24 ± 0.9417.39 ± 0.6918.26 ± 1.1718.28 ± 1.18

[1] S. Salter et al. emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation. NeurIPS Datasets and Benchmarks Track, 2024.

AuthorsOpenReview ↗

Author's Response - Thank you for your feedback (Part 3: Q4 & W2)

W2 / Q4. Site-controlled analysis. We appreciate the comment and agree that a controlled analysis would raise the benchmark's diagnostic value. Table R5 splits the six-fold Image2pose results by the site at which each test frame was imaged. It covers the six forearm sites that carry 96.9% of recordings. Every fold's training set contains all six sites. Under the random split the error is similar at every site; with the session or participant held out it rises by around 6-10° at every site.

Table R5. Image2pose MAE (degrees) by probe site for the six sites with 96.9% of recordings (Lite-CNN + augmentation; mean ± s.d. over the folds that test the site; random split: over the six draws).

SiteConstantRandom splitHeld-out sessionHeld-out participant
FA-A-M18.68 ± 1.439.56 ± 0.1116.17 ± 1.6615.91 ± 1.17
FA-A-P18.78 ± 1.359.96 ± 0.0717.17 ± 1.6317.71 ± 1.68
FA-A-D19.46 ± 1.229.88 ± 0.2218.32 ± 3.0617.63 ± 3.45
FA-P-D18.41 ± 2.0310.51 ± 0.3320.23 ± 5.1120.17 ± 5.59
FA-P-M18.52 ± 1.6310.60 ± 0.2119.43 ± 3.4221.82 ± 4.11
FA-P-P18.40 ± 2.0210.53 ± 0.3418.39 ± 1.9818.60 ± 3.14
Reviewer_9TNwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduce a new dataset called MuscleSight. It collects around 40 hours of arm-mounted B-mode ultrasound, together with synchronized video, from 33 participants. Paper builds benchmarks and baseline numbers on cross-session and within-session generalization performance, showing the challenge of ultrasound hand decoding.

Critical strengths

The dataset represents a valuable contribution to the community and reflects substantial data collection efforts. It includes a sizable participant cohort, extensive recording hours, a broad range of tasks, and rich multimodal measurements. Making the dataset publicly available would significantly enhance its value to the research community.

The study also shows overall comprehensive and carefully designed baseline methods and fair evaluation protocols.

Critical weaknesses

The conclusion 'Performance does not transfer across sessions or participants' is too strong - To draw such a strong conclusion, I'd like to see the authors conduct a lot more experiments. For example: Have the authors exhaust ways to perform baselining and pre-processing on the ultrasound images? Sufficient amount of pre-training on similar datasets (does not have to be arm-mounted ultrasound, but does other ultrasound help)? For cross modality modeling, has the authors exhaust all methodologies (e.g. CLIP, JEPA, and other biosignal or time series specific cross-modal modeling techniques)? Authors should clearly state what they have done, and only draw conclusions based on their observation such as "cross-session or cross-subject generalization is challenging" instead of making the claim.

Questions for authors

Line 203 reference is broken

3 comments
AuthorsOpenReview ↗

Author's Response - Thank you for your feedback

We thank the reviewer for the positive assessment of the dataset and the baselines.

W1. Scope of the conclusion. We agree, appreciate the suggestion, and will adopt the proposed framing: we will state that cross-session and cross-participant generalization is challenging for the models and training recipes we evaluated. During the discussion period we re-evaluated every model on six paired folds and also tested several additional directions mentioned in comments:

  • Additional pretrained ultrasound foundation model: USFM [1], an ultrasound foundation model pretrained on general ultrasound images, as a frozen encoder. It is the best frozen encoder on the random split (8.67° against 10.05° for EchoCLIP), but on held-out participants it is level with the natural-image DINOv3 (17.98° against 17.90°; Table R8b).

Table R8b. USFM against Lite-CNN and the other frozen encoders, all probed with the same linear and MLP heads (MuscleSight-Hand; mean ± s.d. over six folds, or six random draws for the random split). Bold: best per task and column (constant predictor not ranked).

ModelRandom splitHeld-out sessionHeld-out participant
Image2gesture, accuracy ↑
Constant predictor0.217 ± 0.0050.231 ± 0.0110.231 ± 0.011
Lite-CNN0.894 ± 0.0140.147 ± 0.0500.164 ± 0.041
Lite-CNN + augmentation0.653 ± 0.0250.204 ± 0.0320.200 ± 0.020
DINOv3 ViT-B/16, linear0.253 ± 0.0080.119 ± 0.0180.127 ± 0.022
DINOv3 ViT-B/16, MLP0.550 ± 0.0110.234 ± 0.0150.229 ± 0.021
EchoCLIP, linear0.281 ± 0.0070.133 ± 0.0260.128 ± 0.025
EchoCLIP, MLP0.688 ± 0.0050.219 ± 0.0280.198 ± 0.027
USFM, linear0.377 ± 0.0070.150 ± 0.0340.144 ± 0.019
USFM, MLP0.777 ± 0.0130.204 ± 0.0320.222 ± 0.028
Image2pose, MAE (degrees) ↓
Constant predictor18.78 ± 0.0418.74 ± 1.2818.75 ± 1.31
Lite-CNN6.50 ± 0.1118.22 ± 0.9218.60 ± 0.97
Lite-CNN + augmentation10.04 ± 0.1017.24 ± 0.9717.37 ± 0.77
DINOv3 ViT-B/16, linear16.36 ± 0.0217.69 ± 1.0717.90 ± 0.98
DINOv3 ViT-B/16, MLP11.72 ± 0.0518.25 ± 0.9118.38 ± 1.31
EchoCLIP, linear16.19 ± 0.0218.01 ± 0.6918.10 ± 0.85
EchoCLIP, MLP10.05 ± 0.0418.72 ± 1.1718.77 ± 1.49
USFM, linear15.37 ± 0.0717.84 ± 1.3617.98 ± 1.44
USFM, MLP8.67 ± 0.0618.12 ± 1.4418.57 ± 1.56
  • Domain generalization and test-time adaptation: MixStyle [2], GroupDRO [3], DANN [4], AdaBN [5], TENT [6] and per-recording standardization (Table R8a). None closes the gap at the evaluated settings.

Table R8a. Domain generalization and test-time adaptation added to Lite-CNN, one setting per method (MuscleSight-Hand; mean ± s.d. over six folds). Pose: Image2pose MAE in degrees. Bold: best per column (constant predictor not ranked).

MethodGesture acc. ↑, Sess.Gesture acc. ↑, Part.Pose MAE ↓, Sess.Pose MAE ↓, Part.
Constant predictor0.231 ± 0.0110.231 ± 0.01118.74 ± 1.2818.75 ± 1.31
Lite-CNN, no augmentation0.147 ± 0.0500.164 ± 0.04118.22 ± 0.9218.60 ± 0.97
Lite-CNN + augmentation0.204 ± 0.0320.200 ± 0.02017.24 ± 0.9717.37 ± 0.77
Domain generalization (training recordings only)
+ MixStyle0.139 ± 0.0210.145 ± 0.02018.67 ± 0.8418.60 ± 0.88
+ GroupDRO0.121 ± 0.0140.134 ± 0.01319.02 ± 0.8618.75 ± 1.12
+ DANN (recording classifier)0.098 ± 0.0700.101 ± 0.03418.46 ± 0.8218.60 ± 0.94
Test-time adaptation (test recording's frames, never its labels)
+ per-recording standardization0.157 ± 0.0530.178 ± 0.03418.14 ± 1.0118.30 ± 1.17
+ AdaBN0.180 ± 0.0260.183 ± 0.02418.23 ± 0.8918.46 ± 1.14
+ TENT (gesture only)0.186 ± 0.0340.190 ± 0.022——

On cross-modal modeling: in our benchmark, models take ultrasound alone as input, and the cameras provide only the pose labels. Objectives that align ultrasound with the paired video or pose, such as contrastive (CLIP-style) or predictive (JEPA-style) pretraining, are a promising direction that the synchronized recordings in MuscleSight make possible, and we will include it in future work, together with in-domain self-supervised pretraining.

Line 203. Thank you for bringing this to our attention; the reference should point to App. A.4 (Composition), and we will fix it.

[1] Jiao et al., USFM, Med. Image Anal. 2024. [2] Zhou et al., MixStyle, ICLR 2021. [3] Sagawa et al., GroupDRO, ICLR 2020. [4] Ganin et al., DANN, JMLR 2016. [5] Li et al., AdaBN, ICLR Workshop 2017. [6] Wang et al., TENT, ICLR 2021.

Hi Reviewer 9TNw,

Thank you again for your positive and encouraging review. Your suggestions helped us broaden the evaluation, and we have posted our responses with the new experiments. We would be very grateful for any further feedback when you have time, and we will carefully incorporate it into the revision.

Thank you, I think additional experiments help with the contribution of the paper. I will remain my score

Comments on the paper

1 comment
AuthorsOpenReview ↗

Main Updates for the Rebuttal

We thank all reviewers (9TNw, DyEk, Tccd) for their time and constructive feedback, and are encouraged that they find:

  • The dataset a genuine resource: ~5× the participants of prior downloadable forearm-ultrasound datasets, the first with hand–object interaction (DyEk), with a sizable cohort, extensive recording hours and rich multimodal measurements (9TNw, Tccd).
  • The evaluation protocol the strongest contribution: separating within-session, held-out-session and held-out-participant performance shows how random within-session splits overestimate progress (Tccd, DyEk).
  • The baselines and analyses careful and credible: several model families, constant predictors, representation analysis and calibration on separate trials (Tccd, 9TNw, DyEk).

We address each reviewer's concerns in the individual responses, which also report the new results.

During the discussion period we ran the following experiments and analyses; each reviewer's thread reports the results with tables:

  • Six paired cross-validation folds for the held-out-session and held-out-participant evaluations: every participant is held out once, both evaluations score the same items with matched training size, and ± s.d. is over folds (Tccd, DyEk)
  • Controlled comparisons separating session, anatomy, probe site and task: paired folds, dual-probe recordings and per-site results (DyEk)
  • Label quality: precision against a data glove and agreement with it (DyEk, Tccd)
  • Participant scaling with matched epochs, with and without few-shot calibration (DyEk, Tccd)
  • Domain-generalization and test-time adaptation methods, and an ultrasound foundation model (USFM) as a frozen encoder (DyEk, 9TNw)
  • Sensitivity to synchronization offsets up to ±800 ms (DyEk)
  • Pose tracking from estimated initial poses, over 3-s rollouts and by motion magnitude (DyEk, Tccd)

We will scope our conclusions to the evaluated models and training recipes, and incorporate these results in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.