- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents the IMUHand Benchmark, a dataset and benchmark for continuous 3D hand motion reconstruction from sparse IMU inputs. The authors designed a dual-hand IMU data acquisition device, with six IMU sensors on each hand—placed on five fingertips and the back of the hand. Combined with an optical motion capture system and a first-person stereo camera, they achieved synchronized collection of IMU signals, RGB video, and 3D hand motion annotations.
The dataset covers scenarios including bimanual coordinated movements, basic hand gestures, and hand-object interactions, with per-frame MANO annotations. In addition, the authors propose an IMU-only reconstruction method based on Canonicalizer, Pose Estimator, and Root Bypass. By introducing secondary calibration and independent wrist orientation estimation, this method improves hand reconstruction accuracy under sparse IMU conditions. Experimental results show that the approach achieves competitive performance under a unified evaluation protocol.
Critical strengths
-
The paper introduces a dataset that combines continuous bimanual IMU measurements, synchronized egocentric RGB videos, and per-frame 3D MANO annotations. Compared with existing datasets focusing on visual observations or single-hand motions, this dataset provides valuable multimodal information for studying 3D hand motion reconstruction.
-
Accurate 3D hand motion data is important for understanding dexterous manipulation and human-object interactions. The proposed dataset could support future research on human motion understanding, cross-modal learning, dexterous robotic manipulation, and human-to-robot motion transfer.
-
The authors develop a dedicated IMU glove system and provide a relatively complete pipeline covering sensor calibration, multimodal synchronization, MANO annotation generation, and standardized evaluation protocols. These efforts improve the practical value of the dataset and facilitate future research.
-
The paper compares the proposed baseline with several IMU-based and vision-based methods. The ablation studies also demonstrate the effectiveness of the proposed components, particularly the Canonicalizer, which significantly improves reconstruction accuracy.
Critical weaknesses
-
Although the paper provides detailed descriptions of the hardware and data collection process, the visualization of the collected IMU signals is limited. In particular, the paper lacks intuitive visualizations of raw acceleration and orientation signals under different hand motions. Such visualizations would help readers better understand the characteristics and quality of the dataset.
-
While the paper includes wrist orientation estimation and describes global translation in its MANO annotations, the proposed reconstruction baseline does not recover global wrist translation or relative bimanual positions. It would be helpful to clarify whether complete 6DoF wrist poses, including global positions and orientations, will be provided in the released dataset. This information is particularly important for robotic manipulation and bimanual coordination tasks.
-
The dataset includes only 10 subjects, and the paper mentions that hand-object interaction data involves only two subjects. This relatively limited participant diversity may restrict the dataset's coverage of different hand shapes, wearing conditions, and manipulation behaviors.
-
The current training and testing sets are split within action categories, and the paper does not provide a dedicated evaluation on unseen subjects. A subject-independent evaluation would help demonstrate the robustness of the proposed method to variations in hand shapes, sensor placement, and individual motion patterns.
Questions for authors
-
Does the dataset provide complete 6DoF wrist pose annotations? Specifically, are the global wrist translations and orientations available for both hands? If so, can the relative spatial positions between the two hands be recovered from the released annotations?
-
Could the authors provide more visualizations of the recorded IMU signals? For example, acceleration and orientation trajectories corresponding to different hand motions, ideally synchronized with the MANO annotations and egocentric video. Such visualizations would help demonstrate the quality and characteristics of the collected IMU data.
-
How well does the proposed method generalize to unseen subjects? Since the current train/test split is performed within action categories, could the authors additionally report cross-subject evaluation results?
-
What is the planned release scope of the dataset? Will the authors release raw IMU measurements, calibration parameters, synchronized videos, complete MANO annotations, and the tools required to reproduce the benchmark?
4 comments
Response to Reviewer DzgX — Part 2/3
W3 — Participant Diversity
W3: The dataset includes only 10 subjects, and the paper mentions that hand-object interaction data involves only two subjects. This relatively limited participant diversity may restrict the dataset's coverage of different hand shapes, wearing conditions, and manipulation behaviors.
Response:
Thank you for highlighting participant diversity. We have clarified hand-size variation and manipulation coverage in Section 4.4 and Appendix G, and added four-method evaluations on unseen participants and donnings across all six pose metrics.
1. Hand shapes and donning conditions.
The ten participants span hand lengths of 16.1–19.7 cm and bony palm widths of 5.5–7.0 cm, with per-template MANO shape annotations. The 302 experimental recordings cover 17 participant–donning groups: eight participants contribute one donning each, and two contribute six and three, providing varied hand sizes and repeated glove placements.
2. Manipulation coverage.
The 23 action categories include six object-manipulation categories (B01–B06), basic gestures, and bimanual coordination. The manipulation subset within the experimental data contains 51 recordings and 94 single-hand sequences, with synchronized bimanual IMU, egocentric RGB, and per-frame MANO annotations. We corrected the original two-participant description: three participants contribute 14, 35, and two recordings, with 49/51 from the two principal contributors.
3. Added cross-user and cross-donning evidence.
We hold out all 77 recordings of four participants and two complete donnings from two other participants (21 recordings), excluding both test groups from training, validation, and normalization. Each of the four methods is retrained from scratch and evaluated on the same scored frames. Table R24 reports all six metrics, including the six-category manipulation subset of one unseen participant.
Table R24. Four-method comparison. Root/Local: degrees; geometric errors: mm.
| Method | Root | Local | MPJPE | PA-MPJPE | MPVPE | PA-MPVPE |
|---|---|---|---|---|---|---|
| Unseen participants (77 recordings) | ||||||
| VIFNet-S | ||||||
| MGDHand | ||||||
| RT-HandTrack | N/A | |||||
| Ours | ||||||
| Unseen donnings (21 recordings) | ||||||
| VIFNet-S | ||||||
| MGDHand | ||||||
| RT-HandTrack | N/A | |||||
| Ours | ||||||
| Unseen participant: manipulation (six categories; 57,791 hand-frames) | ||||||
| VIFNet-S | ||||||
| MGDHand | ||||||
| RT-HandTrack | N/A | |||||
| Ours |
Ours has the lowest mean on all six metrics for unseen donnings, and on Root and all four geometric metrics for unseen participants and the manipulation subset. MGDHand has slightly lower Local error in the latter two groups. Section 4.4 clarifies dataset coverage; Appendix F.2 details the held-out evaluation.
Continued in Part 3/3.
Response to Reviewer DzgX — Part 3/3
W4 / Q3 — Cross-Subject Evaluation
W4: The current training and testing sets are split within action categories, and the paper does not provide a dedicated evaluation on unseen subjects. A subject-independent evaluation would help demonstrate the robustness of the proposed method to variations in hand shapes, sensor placement, and individual motion patterns.
Q3: How well does the proposed method generalize to unseen subjects? Since the current train/test split is performed within action categories, could the authors additionally report cross-subject evaluation results?
Response:
Thank you for this suggestion. To directly assess performance on new users, we added a cross-subject evaluation with four participants held out entirely. On these participants, Ours achieves mm MPJPE, mm PA-MPJPE, and Root error.
Protocol.
All 77 recordings of these four participants are excluded from training, validation, and normalization. The joint split uses 173 training and 31 validation recordings, and separately holds out two complete donnings from otherwise seen participants (21 recordings). The test participants are entirely unseen, while all test action categories are represented in training. Each of the four methods is retrained from scratch using its benchmark architecture and training procedure. Normalization uses only training data, and model selection uses only validation data. All methods are evaluated on the same scored frames and metrics.
Table R25. Cross-participant results on 77 held-out recordings (359,746 scored hand-frames). Lower is better. Root/Local: degrees; geometric errors: mm.
| Method | Root | Local | MPJPE | PA-MPJPE | MPVPE | PA-MPVPE |
|---|---|---|---|---|---|---|
| Unseen participants (359,746 scored hand-frames) | ||||||
| Ours | ||||||
| MGDHand | ||||||
| VIFNet-S | ||||||
| RT-HandTrack | N/A |
Results.
Compared with MGDHand, Ours has lower Root and both PA errors; MPJPE and MPVPE are close, while Local is slightly higher. Compared with VIFNet-S and RT-HandTrack, every comparable mean error is lower. Across the four unseen participants, Ours’ MPJPE ranges from 8.98 to 10.99 mm. On the two complete held-out donnings (21 recordings), MPJPE is mm versus MGDHand’s mm.
Manuscript revisions.
Appendix F.2 presents the protocol and four-method comparison. Appendix I consolidates the evaluation limitations and future work, including the scope of the held-out evaluation.
Q4 — Scope of the release
Q4: What is the planned release scope of the dataset? Will the authors release raw IMU measurements, calibration parameters, synchronized videos, complete MANO annotations, and the tools required to reproduce the benchmark?
Response:
The release will include temporally aligned IMU measurements, calibration parameters, synchronized egocentric videos, complete MANO annotations, camera calibration parameters (intrinsics and fixed extrinsics), frame-aligned camera poses, and the tools required to reproduce the benchmark.
Thank you for the response. It addressed my concerns. I believe this work is meaningful to the community. I have maintained my weak accept and raised the scores for Soundness and Confidence. Looking forward to the open-source release and hoping this work gets accepted.
Response to Reviewer DzgX — Part 1/3
Thank you for your careful review and helpful suggestions. We have organized our replies into 3 parts using your W/Q labels to make them easier to follow. In connection with your request for more visualizations, please also see our project page (https://imuhand.pages.dev/) for visual examples of the recorded motions and annotations, which we refer to in the responses below.
W1 / Q2 — IMU signal visualization
Response:
We have added a figure in Appendix H showing recorded three-axis acceleration and orientation quaternions from the index-finger IMU during finger flexion and extension, finger waves, and typing. We will also provide synchronized videos in the “Inside the dataset” section of our project page (IMUHand), showing these signals together with egocentric video and the corresponding MANO annotations.
W2 / Q1 — Global Wrist Annotations
Response:
Yes. The annotations retain global wrist rotation and translation for both hands, together with local MANO rotations and per-template shape parameters. Applying the supplied coordinate transforms places the synchronized hands in the same ChingMu world frame, so their relative spatial positions can be recovered from the annotations. The baseline predicts wrist orientation and local articulation, but not global translation; fixed-spacing visualizations therefore do not depict the annotated relative hand positions.
Table R23.
Continued in Part 2/3.