acceptodds
Under review as a conference paper at ICLR 2027

IMUHand Benchmark:A Dataset and Baseline for 3D Hand Motion Reconstruction from Sparse IMUs

Abstract

Continuous 3D hand motion reconstruction is fundamental to natural human-computer interaction and robotic manipulation. Vision-based methods exploit rich appearance and scene information but are limited by occlusion and restricted fields of view during hand-to-hand contact and hand-object interaction. Hand-mounted inertial measurement units (IMUs) do not require optical visibility, yet sparse inertial observations are insufficient to directly recover the full articulated hand pose and are affected by orientation drift and variations in sensor placement. Existing inertial hand datasets differ in acquisition settings and annotation formats. The lack of large-scale public datasets and standardized evaluation protocols for continuous IMU-only bimanual reconstruction further hinders fair comparison. To address this gap, we develop a pair of IMU gloves for synchronized data acquisition and real-time reconstruction. Using this system, we construct Benchmark, which provides long, continuous bimanual motion sequences, per-frame MANO ground truth, synchronized egocentric video, and a unified IMU-only evaluation protocol. We further propose a reconstruction method that combines learned secondary calibration with independent root orientation estimation to recover wrist orientation and the full articulated hand pose from IMU sequences alone. Under the unified protocol, our method achieves the highest reconstruction accuracy among the inertial methods evaluated.

Primary area:
datasets and benchmarks
Keywords:
sparse IMUs, 3D hand motion reconstruction, inertial hand benchmark, wearable sensing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DzgXOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents the IMUHand Benchmark, a dataset and benchmark for continuous 3D hand motion reconstruction from sparse IMU inputs. The authors designed a dual-hand IMU data acquisition device, with six IMU sensors on each hand—placed on five fingertips and the back of the hand. Combined with an optical motion capture system and a first-person stereo camera, they achieved synchronized collection of IMU signals, RGB video, and 3D hand motion annotations.

The dataset covers scenarios including bimanual coordinated movements, basic hand gestures, and hand-object interactions, with per-frame MANO annotations. In addition, the authors propose an IMU-only reconstruction method based on Canonicalizer, Pose Estimator, and Root Bypass. By introducing secondary calibration and independent wrist orientation estimation, this method improves hand reconstruction accuracy under sparse IMU conditions. Experimental results show that the approach achieves competitive performance under a unified evaluation protocol.

Critical strengths
  1. The paper introduces a dataset that combines continuous bimanual IMU measurements, synchronized egocentric RGB videos, and per-frame 3D MANO annotations. Compared with existing datasets focusing on visual observations or single-hand motions, this dataset provides valuable multimodal information for studying 3D hand motion reconstruction.

  2. Accurate 3D hand motion data is important for understanding dexterous manipulation and human-object interactions. The proposed dataset could support future research on human motion understanding, cross-modal learning, dexterous robotic manipulation, and human-to-robot motion transfer.

  3. The authors develop a dedicated IMU glove system and provide a relatively complete pipeline covering sensor calibration, multimodal synchronization, MANO annotation generation, and standardized evaluation protocols. These efforts improve the practical value of the dataset and facilitate future research.

  4. The paper compares the proposed baseline with several IMU-based and vision-based methods. The ablation studies also demonstrate the effectiveness of the proposed components, particularly the Canonicalizer, which significantly improves reconstruction accuracy.

Critical weaknesses
  1. Although the paper provides detailed descriptions of the hardware and data collection process, the visualization of the collected IMU signals is limited. In particular, the paper lacks intuitive visualizations of raw acceleration and orientation signals under different hand motions. Such visualizations would help readers better understand the characteristics and quality of the dataset.

  2. While the paper includes wrist orientation estimation and describes global translation in its MANO annotations, the proposed reconstruction baseline does not recover global wrist translation or relative bimanual positions. It would be helpful to clarify whether complete 6DoF wrist poses, including global positions and orientations, will be provided in the released dataset. This information is particularly important for robotic manipulation and bimanual coordination tasks.

  3. The dataset includes only 10 subjects, and the paper mentions that hand-object interaction data involves only two subjects. This relatively limited participant diversity may restrict the dataset's coverage of different hand shapes, wearing conditions, and manipulation behaviors.

  4. The current training and testing sets are split within action categories, and the paper does not provide a dedicated evaluation on unseen subjects. A subject-independent evaluation would help demonstrate the robustness of the proposed method to variations in hand shapes, sensor placement, and individual motion patterns.

Questions for authors
  1. Does the dataset provide complete 6DoF wrist pose annotations? Specifically, are the global wrist translations and orientations available for both hands? If so, can the relative spatial positions between the two hands be recovered from the released annotations?

  2. Could the authors provide more visualizations of the recorded IMU signals? For example, acceleration and orientation trajectories corresponding to different hand motions, ideally synchronized with the MANO annotations and egocentric video. Such visualizations would help demonstrate the quality and characteristics of the collected IMU data.

  3. How well does the proposed method generalize to unseen subjects? Since the current train/test split is performed within action categories, could the authors additionally report cross-subject evaluation results?

  4. What is the planned release scope of the dataset? Will the authors release raw IMU measurements, calibration parameters, synchronized videos, complete MANO annotations, and the tools required to reproduce the benchmark?

4 comments
AuthorsOpenReview ↗

Response to Reviewer DzgX — Part 1/3

Thank you for your careful review and helpful suggestions. We have organized our replies into 3 parts using your W/Q labels to make them easier to follow. In connection with your request for more visualizations, please also see our project page (https://imuhand.pages.dev/) for visual examples of the recorded motions and annotations, which we refer to in the responses below.

W1 / Q2 — IMU signal visualization

W1: Although the paper provides detailed descriptions of the hardware and data collection process, the visualization of the collected IMU signals is limited. In particular, the paper lacks intuitive visualizations of raw acceleration and orientation signals under different hand motions. Such visualizations would help readers better understand the characteristics and quality of the dataset.

Q2: Could the authors provide more visualizations of the recorded IMU signals? For example, acceleration and orientation trajectories corresponding to different hand motions, ideally synchronized with the MANO annotations and egocentric video. Such visualizations would help demonstrate the quality and characteristics of the collected IMU data.

Response:

We have added a figure in Appendix H showing recorded three-axis acceleration and orientation quaternions from the index-finger IMU during finger flexion and extension, finger waves, and typing. We will also provide synchronized videos in the “Inside the dataset” section of our project page (IMUHand), showing these signals together with egocentric video and the corresponding MANO annotations.

W2 / Q1 — Global Wrist Annotations

W2: While the paper includes wrist orientation estimation and describes global translation in its MANO annotations, the proposed reconstruction baseline does not recover global wrist translation or relative bimanual positions. It would be helpful to clarify whether complete 6DoF wrist poses, including global positions and orientations, will be provided in the released dataset. This information is particularly important for robotic manipulation and bimanual coordination tasks.

Q1: Does the dataset provide complete 6DoF wrist pose annotations? Specifically, are the global wrist translations and orientations available for both hands? If so, can the relative spatial positions between the two hands be recovered from the released annotations?

Response:

Yes. The annotations retain global wrist rotation and translation for both hands, together with local MANO rotations and per-template shape parameters. Applying the supplied coordinate transforms places the synchronized hands in the same ChingMu world frame, so their relative spatial positions can be recovered from the annotations. The baseline predicts wrist orientation and local articulation, but not global translation; fixed-spacing visualizations therefore do not depict the annotated relative hand positions.

Table R23.

QuantityDataset annotationsBaseline prediction
Global wrist orientationBoth handsBoth hands
Global wrist translationBoth handsNot predicted
Relative hand placementRecoverable in shared world frameNot reconstructed
Local MANO articulationBoth handsBoth hands
Hand shapePer-template parametersMean shape for evaluation

Continued in Part 2/3.

AuthorsOpenReview ↗

Response to Reviewer DzgX — Part 2/3

W3 — Participant Diversity

W3: The dataset includes only 10 subjects, and the paper mentions that hand-object interaction data involves only two subjects. This relatively limited participant diversity may restrict the dataset's coverage of different hand shapes, wearing conditions, and manipulation behaviors.

Response:

Thank you for highlighting participant diversity. We have clarified hand-size variation and manipulation coverage in Section 4.4 and Appendix G, and added four-method evaluations on unseen participants and donnings across all six pose metrics.

1. Hand shapes and donning conditions.

The ten participants span hand lengths of 16.1–19.7 cm and bony palm widths of 5.5–7.0 cm, with per-template MANO shape annotations. The 302 experimental recordings cover 17 participant–donning groups: eight participants contribute one donning each, and two contribute six and three, providing varied hand sizes and repeated glove placements.

2. Manipulation coverage.

The 23 action categories include six object-manipulation categories (B01–B06), basic gestures, and bimanual coordination. The manipulation subset within the experimental data contains 51 recordings and 94 single-hand sequences, with synchronized bimanual IMU, egocentric RGB, and per-frame MANO annotations. We corrected the original two-participant description: three participants contribute 14, 35, and two recordings, with 49/51 from the two principal contributors.

3. Added cross-user and cross-donning evidence.

We hold out all 77 recordings of four participants and two complete donnings from two other participants (21 recordings), excluding both test groups from training, validation, and normalization. Each of the four methods is retrained from scratch and evaluated on the same scored frames. Table R24 reports all six metrics, including the six-category manipulation subset of one unseen participant.

Table R24. Four-method comparison. Root/Local: degrees; geometric errors: mm.

MethodRootLocalMPJPEPA-MPJPEMPVPEPA-MPVPE
Unseen participants (77 recordings)
VIFNet-S14.8214.8214.2114.2110.7210.728.098.099.909.907.727.72
MGDHand14.1214.1213.70\boldsymbol{13.70}10.2810.287.647.649.499.497.327.32
RT-HandTrackN/A18.0218.0214.7014.7010.9610.9613.5513.5510.9510.95
Ours11.54\boldsymbol{11.54}13.8213.8210.19\boldsymbol{10.19}7.31\boldsymbol{7.31}9.41\boldsymbol{9.41}6.99\boldsymbol{6.99}
Unseen donnings (21 recordings)
VIFNet-S16.5316.5312.7012.708.808.806.906.908.188.186.616.61
MGDHand16.4716.4711.9611.968.218.216.376.377.647.646.106.10
RT-HandTrackN/A17.8617.8613.0713.079.889.8812.1112.119.869.86
Ours14.61\boldsymbol{14.61}11.52\boldsymbol{11.52}7.58\boldsymbol{7.58}5.80\boldsymbol{5.80}7.03\boldsymbol{7.03}5.58\boldsymbol{5.58}
Unseen participant: manipulation (six categories; 57,791 hand-frames)
VIFNet-S15.1615.1613.7013.709.849.848.018.019.119.117.557.55
MGDHand13.0513.0513.11\boldsymbol{13.11}9.439.437.677.678.748.747.257.25
RT-HandTrackN/A17.1817.1813.7013.709.969.9612.6912.6910.2610.26
Ours7.40\boldsymbol{7.40}13.2913.299.17\boldsymbol{9.17}7.35\boldsymbol{7.35}8.53\boldsymbol{8.53}7.00\boldsymbol{7.00}

Ours has the lowest mean on all six metrics for unseen donnings, and on Root and all four geometric metrics for unseen participants and the manipulation subset. MGDHand has slightly lower Local error in the latter two groups. Section 4.4 clarifies dataset coverage; Appendix F.2 details the held-out evaluation.

Continued in Part 3/3.

AuthorsOpenReview ↗

Response to Reviewer DzgX — Part 3/3

W4 / Q3 — Cross-Subject Evaluation

W4: The current training and testing sets are split within action categories, and the paper does not provide a dedicated evaluation on unseen subjects. A subject-independent evaluation would help demonstrate the robustness of the proposed method to variations in hand shapes, sensor placement, and individual motion patterns.

Q3: How well does the proposed method generalize to unseen subjects? Since the current train/test split is performed within action categories, could the authors additionally report cross-subject evaluation results?

Response:

Thank you for this suggestion. To directly assess performance on new users, we added a cross-subject evaluation with four participants held out entirely. On these participants, Ours achieves 10.1910.19 mm MPJPE, 7.317.31 mm PA-MPJPE, and 11.54∘11.54^\circ Root error.

Protocol.

All 77 recordings of these four participants are excluded from training, validation, and normalization. The joint split uses 173 training and 31 validation recordings, and separately holds out two complete donnings from otherwise seen participants (21 recordings). The test participants are entirely unseen, while all test action categories are represented in training. Each of the four methods is retrained from scratch using its benchmark architecture and training procedure. Normalization uses only training data, and model selection uses only validation data. All methods are evaluated on the same scored frames and metrics.

Table R25. Cross-participant results on 77 held-out recordings (359,746 scored hand-frames). Lower is better. Root/Local: degrees; geometric errors: mm.

MethodRootLocalMPJPEPA-MPJPEMPVPEPA-MPVPE
Unseen participants (359,746 scored hand-frames)
Ours11.54\boldsymbol{11.54}13.8213.8210.19\boldsymbol{10.19}7.31\boldsymbol{7.31}9.41\boldsymbol{9.41}6.99\boldsymbol{6.99}
MGDHand14.1214.1213.70\boldsymbol{13.70}10.2810.287.647.649.499.497.327.32
VIFNet-S14.8214.8214.2114.2110.7210.728.098.099.909.907.727.72
RT-HandTrackN/A18.0218.0214.7014.7010.9610.9613.5513.5510.9510.95

Results.

Compared with MGDHand, Ours has lower Root and both PA errors; MPJPE and MPVPE are close, while Local is slightly higher. Compared with VIFNet-S and RT-HandTrack, every comparable mean error is lower. Across the four unseen participants, Ours’ MPJPE ranges from 8.98 to 10.99 mm. On the two complete held-out donnings (21 recordings), MPJPE is 7.587.58 mm versus MGDHand’s 8.218.21 mm.

Manuscript revisions.

Appendix F.2 presents the protocol and four-method comparison. Appendix I consolidates the evaluation limitations and future work, including the scope of the held-out evaluation.

Q4 — Scope of the release

Q4: What is the planned release scope of the dataset? Will the authors release raw IMU measurements, calibration parameters, synchronized videos, complete MANO annotations, and the tools required to reproduce the benchmark?

Response:

The release will include temporally aligned IMU measurements, calibration parameters, synchronized egocentric videos, complete MANO annotations, camera calibration parameters (intrinsics and fixed extrinsics), frame-aligned camera poses, and the tools required to reproduce the benchmark.

Thank you for the response. It addressed my concerns. I believe this work is meaningful to the community. I have maintained my weak accept and raised the scores for Soundness and Confidence. Looking forward to the open-source release and hoping this work gets accepted.

Reviewer_fHrFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper, IMUHand Benchmark: A Dataset and Baseline for 3D Hand Motion Reconstruction from Sparse IMUs, presents a capture system, a dataset, and an inertial hand reconstruction baseline. Each glove contains six IMUs, located on the five distal phalanges and the back of the hand. Measurements are synchronized with egocentric stereo video and an independent optical motion-capture system, whose skeletons are fitted to MANO. The authors report 630 single-hand sequences from 10 participants and over two million valid synchronized frames. The reconstruction task estimates wrist orientation and local joint rotations; hand shape, global translation, and relative bimanual positions are outside its scope.

Method. The baseline combines a learned Canonicalizer with slow and fast correction branches, a graph/temporal pose estimator, and a Root Bypass that retains information from the original back-of-hand observations. Three-stage training uses virtual IMUs synthesized from training-set MANO annotations, followed by calibration training and joint adaptation with measured signals.

Validation. On 291,283 test hand-frames, the method achieves 7.117 mm MPJPE versus 7.496 mm for adapted MGDHand, a 5.1% reduction, and 8.248 degrees wrist orientation error versus 10.114 degrees. MGDHand remains better on local rotation error, at 9.038 versus 9.228 degrees. The geometric metrics fix shape, translation, and root orientation, so they measure normalized local articulation. The paper also reports comparisons with vision methods, component ablations, and describes online reconstruction and Linker Hand O6 teleoperation.

Critical strengths

S1. A useful acquisition and benchmarking contribution for an important sensing regime. The combination of sparse finger-mounted IMUs, paired bimanual recordings, egocentric video, and independently acquired optical reference data is relevant to hand tracking during occlusion and manipulation (Sections 3-4). Fixing the sensor configuration and MANO evaluation protocol can make comparisons more interpretable than comparisons across incompatible glove layouts and output representations. If its quality and accessibility are established, this dataset could be valuable independently of the baseline's architectural novelty.

S2. A sensible calibration architecture with informative component ablations. Separating observation correction from pose inference, while retaining a separate path for wrist orientation, addresses a practical difficulty caused by mounting variation and glove movement. Table 4 provides useful evidence on the reported split: removing the Canonicalizer raises MPJPE from 7.117 to 8.100 mm and wrist error from 8.248 to 13.520 degrees. Removing the Root Bypass raises wrist error to 9.296 degrees while barely changing local geometry, consistent with its intended role. These results support further investigation of the approach and are more informative than an aggregate accuracy comparison alone.

Critical weaknesses

W1. The reliability and generalization properties of the benchmark, the paper's central contribution, are not sufficiently established.

  • The split does not establish generalization to new users or donning sessions. Section 6.1 specifies an approximately 80/20 sequence split within action categories and appropriately keeps both hands and modalities together. However, it does not state whether participants or donning sessions overlap between training and test sets. This matters specifically for a method intended to correct mounting and repeated-donning errors: performance on another sequence from an already observed session does not establish robustness to a new placement. Hand-object interactions involve only two participants (p. 8), further limiting the evidence for that setting. This is a missing generalization test, not evidence that leakage occurred.
  • The reference annotations and synchronization lack quantitative validation. Section 4.3 and Appendix B describe gap repair, skeleton reconstruction, MANO fitting, and validity filtering, but provide no fitting-residual distributions, repair/rejection rates, or quantitative annotation audit. Appendix A.3 explains clock mapping without reporting residual synchronization accuracy. Optical references are a sensible starting point, but the subsequent processing needs validation, particularly during contact and rapid motion. This matters both for the dataset's usefulness and for interpreting the 0.379 mm MPJPE advantage over MGDHand; the paper does not establish the reference pipeline's uncertainty or sensitivity.
  • The dataset cannot yet be adequately characterized or inspected from the supplied material. Section 4.4 does not give sequence-duration distributions, counts of independent paired recordings and donning sessions, or a detailed action/object breakdown. Table 1 marks the dataset as publicly released, whereas the reproducibility statement describes a future release after quality checks and review (p. 10); the supplied manuscript contains no dataset/code access link. These omissions leave the proposed benchmark less substantiated than its headline frame count suggests.

W2. The evidence does not yet isolate or reproducibly establish the claimed calibration and accuracy benefits.

  • The main comparison is incompletely specified. Section 6.2.1 says the baselines are adapted with only necessary interface changes, but does not explain their training schedules, output mappings, or which teacher/distillation and filtering components are retained. This is consequential for methods whose learning mechanisms differ. In particular, Section 2.3 groups MGDHand with methods requiring visual teachers, although its teacher uses MANO and IMU sequences; the distinction matters because the proposed method also uses MANO-derived supervision (MGDHand, Section 3.1). The proposed method's own L_pose and L_can are not fully defined, and the value of lambda_can and perturbation distributions are absent. Table 2 gives no uncertainty estimates, while Section 6.1 reports a single seed. Consequently, the robustness of the relatively small MPJPE gain is unclear. The overall pipeline is understandable, but the input-coordinate conventions, training losses, and baseline adaptations are insufficiently specified. For example, Eq. (4) uses MANO-frame measurements while Appendix A.1 describes root-relative normalization in Eq. (15), without clearly identifying the tensors passed to each branch. The unresolved figure reference in Section 4.4 is a minor presentation issue.
  • The methodological contribution and training strategy need a more specific control. Combining established GNN, GRU, TCN, and LSTM components can yield a useful method; their reuse alone does not establish limited novelty. Table 4 already evaluates removal of the Canonicalizer, Root Bypass, slow branch, and fast branch, and these results support the roles of those modules. The remaining gap is that the three-stage training recipe is not compared with an end-to-end alternative using the same architecture and comparable optimization budgets. This makes it difficult to determine how much of the improvement depends on the proposed pretraining and calibration stages, and limits the evidence for the specific methodological advance claimed.
  • Average pose errors do not establish the proposed correction mechanism's behavior over time or across placements. Table 4 shows that removing modules degrades accuracy, but does not measure corrected-sensor errors, errors versus elapsed time since calibration, or performance under documented changes of donning. Nor is the learned root estimate compared with simply using the statically calibrated back-of-hand orientation from Eq. (11). Without these checks, it is difficult to separate improved pose prediction from the specific benefit of secondary calibration, or to assess the long-duration motivation. Hidden-state resets and calibration resets during evaluation are also unspecified.
  • The sensor mode needs clarification before the drift results can be interpreted. Appendix D names the BNO085's “Rotation Vector” output, while Section 7 discusses six-axis sensing without an absolute heading reference. The manufacturer's documentation distinguishes magnetometer-assisted Rotation Vector from magnetometer-free Game Rotation Vector (CEVA BNO08X datasheet, Sections 2.2.2 and 2.2.4). This could be a terminology issue, but the actual report mode, magnetic input, and reference-frame alignment materially affect what the wrist-orientation and drift claims mean.

Overall, the dataset and acquisition system are useful engineering contributions. My recommendation reflects both the incomplete validation of the benchmark and the limited evidence that the proposed calibration and training design provides a distinct, reproducible methodological benefit. The concern about methodological advance rests on these missing controls and the uncertain robustness of the reported improvement, rather than on the use of established network components. Global hand translation and relative bimanual positioning are explicitly outside the task defined in Section 5.1; this limits the system's applicability to full spatial bimanual tracking but is not an independent reason for rejection. The remaining concerns appear addressable through clarification and targeted analyses of the existing data.

Questions for authors
  1. Are training and test sets disjoint by participant, donning session, or only recording sequence? Are windows constructed after the sequence split? Please provide a split manifest and counts/durations by participant, session, and action, and clarify how the 630 single-hand sequences correspond to paired recordings.
  2. Which dataset and implementation assets are currently available for anonymous review, and how should Table 1's release status be interpreted?
  3. What are the MANO fitting residuals against the optical observations, the proportions of repaired/rejected frames, and the residual IMU-to-mocap timing error? Please report these by motion/contact condition where possible and describe the validity thresholds. How was annotation quality checked beyond the quantities directly optimized by the fitting objective?
  4. What are the complete definitions and weights of L_pose and L_can, the synthetic perturbation distributions, and the validation/model-selection procedure?
  5. For each adapted baseline, which original training components and output mappings were retained or changed, especially MGDHand's MANO-IMU teacher and VIFNet-S's distillation? Were equivalent training annotations and comparable tuning opportunities available?
  6. Which BNO085 report type/ID was recorded, and was the magnetometer used? Please reconcile Appendix D with Section 7.
  7. Please specify the input frames before and after Eq. (15), including what reaches the Root Bypass. How is the calibrated IMU frame aligned with the mocap root reference, and what calibration information is available during deployment?

Suggestions that could change my assessment (additional experiments)

  1. Using the existing recordings, report a held-out-session evaluation, and subject-held-out results where the data permit, together with sequence/session-level uncertainty and training-seed variability for the comparison with MGDHand. This would test whether the calibration benefits generalize beyond the current split and whether the accuracy improvement is robust.
  2. Compare the full three-stage training procedure with an end-to-end alternative using the same architecture and comparable optimization budgets. This would isolate the contribution of the training strategy beyond the component ablations already reported in Table 4.
  3. For the full model and calibration ablations, report error versus time since calibration, state/reset handling, and a direct calibrated-root-orientation baseline to isolate the proposed calibration's contribution over time.

Future work (does not affect my assessment)

  1. Quantifying contact consistency and downstream teleoperation task success would help establish the reconstruction system's practical value for dexterous manipulation.
16 comments
AuthorsOpenReview ↗

Response to Reviewer fHrF — Part 1/15

Thank you for the time and care you have put into this detailed review. To address each point and provide the requested supporting details, we have organized our replies into 15 parts within the per-comment character limit. We appreciate your patience with the length of our response and have retained your W/Q labels for ease of reference. For your questions about dataset inspection and practical use, please also see our project page (https://imuhand.pages.dev/), which brings together the motion examples and reconstruction and teleoperation demonstrations discussed below.

W1.1 / Q1 — Split Boundary, Manifests, and Overlap

W1.1: The split does not establish generalization to new users or donning sessions. Section 6.1 specifies an approximately 80/20 sequence split within action categories and appropriately keeps both hands and modalities together. However, it does not state whether participants or donning sessions overlap between training and test sets. This matters specifically for a method intended to correct mounting and repeated-donning errors: performance on another sequence from an already observed session does not establish robustness to a new placement. Hand-object interactions involve only two participants (p. 8), further limiting the evidence for that setting. This is a missing generalization test, not evidence that leakage occurred.

Q1: Are training and test sets disjoint by participant, donning session, or only recording sequence? Are windows constructed after the sequence split? Please provide a split manifest and counts/durations by participant, session, and action, and clarify how the 630 single-hand sequences correspond to paired recordings.

Response:

Thank you for requesting these details. The original "approximately 80/20" wording was imprecise. The explicit train/validation/test manifest used for the supplementary evaluations is reported below. Action categories are strata: complete recordings within each category are assigned to partitions, rather than assigning whole action categories to different partitions. Actions with fewer than three recordings are assigned to training only. The standard split is disjoint by complete recording, and windows are constructed after splitting. Both hands and all modalities remain together. Participants and donnings overlap: all eight test participants and 14 test donning groups also occur in training. The additional unseen-participant and unseen-donning evaluation is reported in Q8 and Appendix F.2.

Continued in Part 2/15.

AuthorsOpenReview ↗

Response to Reviewer fHrF — Part 2/15

W1.1 / Q1 — Split Boundary, Manifests, and Overlap (continued)

Response (continued from the preceding part):

The 630 single-hand sequences correspond to 317 recordings (313 with both hands and four with one hand). The experimental subset contains 572 hand sequences from 302 recordings. The requested split summary, participant/donning and action statistics, and complete compact manifest are provided below. Durations are in minutes over valid intervals, counting overlapping left/right observations once.

Table R6. Standard split: complete recordings, single-hand sequences, and recording duration in minutes.

PartitionRecordingsHand sequencesMinutes
Train214403351.49
Val478776.71
Test418266.46

Table R7. Participant/donning counts and durations. Cells give recordings / hand sequences / recording minutes; multi-donning participants also have a total row.

Participant–donningTrainValidationTest
S01–0112/20/20.583/5/5.233/6/5.35
S02–0111/22/19.232/4/3.282/4/3.37
S02–028/16/10.291/2/1.516/12/6.63
S02–038/16/13.962/4/3.523/6/5.32
S02–041/2/1.771/2/1.671/2/1.75
S02–067/14/11.601/2/1.560/0/0.00
S02–0725/47/39.934/8/6.366/12/9.96
S02 total60/117/96.7811/22/17.8918/36/27.03
S03–0219/37/32.814/8/6.891/2/1.79
S04–017/13/11.671/1/1.300/0/0.00
S05–0316/27/25.514/7/6.474/8/6.90
S06–0215/30/25.886/12/10.236/12/10.29
S07–016/10/8.952/3/3.052/4/3.28
S08–0115/26/24.413/6/5.031/2/1.66
S09–0110/19/16.454/8/6.730/0/0.00
S10–0112/24/20.022/3/3.202/4/3.46
S10–025/10/8.290/0/0.001/2/1.75
S10–0337/70/60.137/12/10.683/6/4.96
S10 total54/104/88.459/15/13.886/12/10.17

Table R8. Action-level counts and durations, in the same format.

ActionTrainValidationTest
A0116/30/26.024/6/6.503/6/4.95
A0215/29/24.433/6/5.103/6/5.11
A0325/49/41.685/10/8.685/10/8.58
A0412/24/20.403/6/5.093/6/3.85
A0510/18/16.162/4/3.202/4/3.09
A0613/21/21.933/5/4.753/6/3.67
A0711/21/17.212/4/3.472/4/3.46
A082/4/3.450/0/0.000/0/0.00
A091/2/1.530/0/0.000/0/0.00
A102/4/3.570/0/0.000/0/0.00
A1111/22/18.432/4/3.552/4/3.57
A129/17/15.552/4/3.212/4/3.17
A139/17/15.492/4/3.302/4/3.19
A148/16/13.482/4/3.341/2/1.78
A158/16/13.812/4/3.262/4/3.31
A1616/30/26.004/6/6.593/6/5.18
A1710/17/15.122/4/3.022/4/3.27
B015/8/7.061/2/1.431/2/1.59
B026/12/9.422/3/2.941/2/1.70
B037/14/11.852/4/3.251/2/1.78
B046/11/10.041/2/1.731/2/1.77
B056/10/8.981/2/1.681/2/1.71
B066/11/9.882/3/2.611/2/1.75

Reading the manifest. S01–S10 identify participants; the following code identifies donning. Prefix 0912 with A01: 06 means recording 0912-06, action A01. Suffix ranges are inclusive; Tr/Va/Te denote training/validation/testing.

Continued in Part 3/15.

AuthorsOpenReview ↗

Response to Reviewer fHrF — Part 3/15

W1.1 / Q1 — Split Boundary, Manifests, and Overlap (continued)

Response (continued from the preceding part):

Table R9. Compact split manifest for all 302 experimental recordings.

Participant–donningPrefixSplitAction: recording suffixes
S01–010913TrA01: 15; A02: 02; A03: 03,17; A04: 05; A06: 07; A07: 08; A12: 10; A13: 11; A14: 12; A15: 13; A16: 14
S01–010913VaA01: 01; A02: 16; A04: 18
S01–010913TeA04: 04; A05: 06; A11: 09
S02–010912TrA01: 70; A03: 59; A04: 60; A06: 62; A07: 63; A11: 64; A12: 65; A13: 66; A14: 67; A15: 68; A16: 69
S02–010912VaA01: 57; A05: 61
S02–010912TeA02: 58,71
S02–020913TrA02: 20,33; A03: 21; A04: 34; A07: 25; A11: 26; A14: 29; A15: 30
S02–020913VaA12: 27
S02–020913TeA01: 19; A04: 22; A05: 23; A06: 24; A13: 28; A16: 31
S02–030914TrA02: 02; A03: 03; A04: 04; A05: 05; A06: 06; A12: 09; A13: 10; A15: 12
S02–030914VaA01: 01; A16: 13
S02–030914TeA07: 07; A11: 08; A14: 11
S02–040914TrA03: 16
S02–040914VaA02: 15
S02–040914TeA01: 14
S02–060920TrA01: 15; A11: 10; A12: 11; A13: 12; A14: 13; A15: 14; A16: 08
S02–060920VaA17: 09
S02–070920TrA01: 31,48; A02: 16,32; A03: 33; A04: 17,34; A05: 19,35; A06: 20,37; A07: 21; A11: 22,42; A12: 43; A13: 24,44–45; A14: 25,46; A16: 28,50; A17: 40–41,51
S02–070920VaA07: 36; A15: 47; A16: 39; A17: 30
S02–070920TeA03: 18; A12: 23; A15: 26; A16: 27,38; A17: 29
S03–020912TrA01: 33,47,55; A02: 48,56; A03: 49; A04: 36,50; A05: 37,51; A06: 38,52; A07: 39; A08: 40; A10: 41; A12: 43; A13: 45; A15: 46; A16: 54
S03–020912VaA02: 34; A07: 53; A11: 42; A14: 44
S03–020912TeA03: 35
S04–010921TrA01: 01; A02: 02; A03: 03; A04: 04; A05: 05; A07: 07; A16: 08
S04–010921VaA01: 10
S05–030922TrA03: 52; A06: 54,56–57; A16: 37–38; A17: 40; B01: 35,45; B02: 46; B03: 41,47; B05: 33; B06: 34,44,50
S05–030922VaA03: 51; B02: 36; B04: 42; B05: 49
S05–030922TeA03: 53; A17: 39; B04: 48; B05: 43
S06–020912TrA01: 06,31; A02: 07,23; A03: 08; A05: 11,26; A06: 12; A07: 28; A08: 14; A09: 15; A10: 16; A11: 10; A12: 18; A15: 21
S06–020912VaA03: 24,32; A04: 09; A11: 17; A14: 20; B02: 30
S06–020912TeA01: 22; A04: 25; A06: 27; A07: 13; A13: 19; B03: 29
S07–010921TrA01: 35; A04: 34; A05: 36; A07: 38; A11: 39; A16: 40
S07–010921VaA06: 37; A16: 41
S07–010921TeA03: 33; A12: 42
S08–010921TrA01: 12; A02: 11,31; A03: 14; A04: 15; A06: 17; A07: 18; A11: 22,27; A12: 23,28; A13: 29; A14: 25; A15: 26; A16: 21
S08–010921VaA05: 16; A13: 24; A15: 20
S08–010921TeA15: 19
S09–010921TrA01: 46; A02: 43; A03: 44; A05: 47; A07: 49; A11: 53; A14: 56; A16: 50–51; A17: 52
S09–010921VaA04: 45; A06: 48; A12: 54; A13: 55
S10–010922TrA01: 64,66; A03: 59–60,62,68,71; A04: 63; A07: 67; A11: 70; B03: 72; B04: 73
S10–010922VaA03: 61; A06: 69
S10–010922TeA02: 58; A06: 65
S10–020922TrA03: 79; B01: 74; B02: 75; B03: 78; B05: 76
S10–020922TeB06: 77
S10–030923TrA02: 46; A03: 27–29,32,47,49; A06: 48; A16: 01–03; A17: 04–07,51; B01: 23–24; B02: 20,22,37,43; B03: 09–10,39; B04: 11–13,35,40; B05: 14,16,38,44; B06: 18–19,41
S10–030923VaA03: 31; A16: 50; B01: 25; B03: 08,33; B06: 17,34
S10–030923TeA03: 26; B01: 36; B02: 21

Continued in Part 4/15.

Reviewer_kVE7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IMUHand Benchmark, a dataset and IMU-only baseline for 3D hand motion reconstruction. The main contribution of this work is providing high-quality dataset containing synchronized egocentric video, MANO annotation, and IMU signals, constructing an IMU-only evaluation protocol. The performance of the proposed method was validated by comparison with existing methods and ablation studies.

Critical strengths
  1. Practical value. The proposed baseline could be useful for downstream tasks such as robot hand motion collection via teleoperation in resource-limited situations since this method requires only six IMU sensors and shows real-time capability.
Critical weaknesses
  1. Potentially overstated claims. Table 1 shows the ‘bimanual’ and ‘hand-object interaction’ columns. The competitive advantage of bimanual reconstruction over unimanual reconstruction is mainly from recovering the relative bimanual positions, which is not considered in this work. Also, especially in bimanual cases, the hand-object interaction reconstruction often requires a world frame shared by both hands. However, as shown in the supplementary video, the global translation is fixed and the hands are distant from each other during manipulation, implying the proposed method could not fully reflect hand-object interaction.

  2. Lack of justification for not using vision information during training. The proposed baseline uses IMU signals in both training and inference without validating why vision information should not be used for training. According to L129-134, the previous works utilize visual teachers during training while the proposed method uses only IMUs even with vision information from motion capture system and stereo camera. Unfortunately, the paper did not include ablation study on using vision information during training, which could explain why the authors used only IMU signals.

Questions for authors

In Table 3, could you explain why the coverage of HaMeR in unoccluded condition is lower than occluded condition? Do the vision-based models perform poorly regardless of whether hands are occluded or not due to domain gap (i.e. those models are trained to take bare-hand images but gloved-hand images are taken for the comparison as shown in Figure 3)?

5 comments
AuthorsOpenReview ↗

Response to Reviewer kVE7 — Part 1/4

Thank you for your careful review and helpful comments. To make our replies easier to follow, we have organized them into 4 parts using your W/Q labels. As a visual reference for our discussion of bimanual reconstruction, please also see our project page (https://imuhand.pages.dev/), which includes the motion and reconstruction examples discussed below.

W1 — Scope of Bimanual and Hand--Object Interaction Claims

W1: Potentially overstated claims. Table 1 shows the ‘bimanual’ and ‘hand-object interaction’ columns. The competitive advantage of bimanual reconstruction over unimanual reconstruction is mainly from recovering the relative bimanual positions, which is not considered in this work. Also, especially in bimanual cases, the hand-object interaction reconstruction often requires a world frame shared by both hands. However, as shown in the supplementary video, the global translation is fixed and the hands are distant from each other during manipulation, implying the proposed method could not fully reflect hand-object interaction.

Response:

Thank you for this comment. We agree that our method does not recover relative hand placement or complete hand–object interactions. It predicts wrist orientation and local articulation, without global translation. The fixed-spacing supplementary visualization shows individual hand poses and does not represent their actual spatial relationship. We did not explain this adequately, and some wording extended beyond the capability evaluated in our experiments.

Bimanual capture and hand–object interaction refer to synchronized recordings of both hands and recordings of object manipulation, respectively; these describe the recorded scenarios. The dataset does retain this spatial information: bilateral annotations include global rotation and translation and can be transformed into the same ChingMu world frame. Table R1 summarizes the distinction between the spatial information stored in the dataset and the outputs predicted by our IMU-only baseline.

Table R1. Distinction between spatial quantities retained in the IMUHand dataset annotations and the outputs predicted by our IMU-only baseline (Ours).

AspectIMUHand Dataset & AnnotationsBaseline Model Outputs (Ours)
Bimanual captureSynchronized recordings of both hands (IMU, egocentric stereo RGB, and 12-camera ChingMu optical mocap)Independent per-hand inference from 6 IMUs per hand; no cross-hand positional constraint
Wrist rotation & posePer-frame global wrist rotation Rtroot∈SO(3)\mathbf R^{\mathrm{root}}_t \in \mathrm{SO}(3) and 15 local MANO joint rotations θt\boldsymbol\theta_tPredicts R^troot∈SO(3)\widehat{\mathbf R}^{\mathrm{root}}_t \in \mathrm{SO}(3) and 15 local MANO joint rotations θ^t∈SO(3)15\widehat{\boldsymbol\theta}_t \in \mathrm{SO}(3)^{15}
Global translation & shared world framePer-frame translation tt∈R3\mathbf t_t \in \mathbb R^3 for both hands; coordinate transforms place both meshes in the shared ChingMu world frame, preserving relative hand positionsNot predicted (proot=0\mathbf p^{\mathrm{root}}=\mathbf 0 per wrist); supplementary video uses a fixed display spacing to show local articulation
Hand shape & object manipulationPer-template shape β∈R10\boldsymbol\beta \in \mathbb R^{10} for each hand; synchronized recordings of everyday object manipulationDecoded at mean shape (β=0\boldsymbol\beta=\mathbf 0); object 6D pose and contact states are not reconstructed

We have made the following revisions to distinguish the dataset annotations from the method outputs:

  • Section 4.3, Ground-Truth Annotation: After the MANO parameter definitions, we explain the bilateral global rotations, translations, and shared world frame, making explicit that the annotations preserve relative hand placement throughout the motion.

  • Section 4.4, Dataset Composition: We replace "full bimanual pose reconstruction" with evaluation of wrist orientation and articulated hand pose for both hands, matching the model outputs.

Continued in Part 2/4.

AuthorsOpenReview ↗

Response to Reviewer kVE7 — Part 2/4

W2 — Motivation for IMU-Only Training and Role of RGB

W2: Lack of justification for not using vision information during training. The proposed baseline uses IMU signals in both training and inference without validating why vision information should not be used for training. According to L129-134, the previous works utilize visual teachers during training while the proposed method uses only IMUs even with vision information from motion capture system and stereo camera. Unfortunately, the paper did not include ablation study on using vision information during training, which could explain why the authors used only IMU signals.

Response:

Thank you for pointing out that the research motivation was insufficiently explained. Our objective is to study IMU-only hand reconstruction for applications in which visual observations are unavailable or visual recording is impractical. We therefore establish an inertial reconstruction baseline and a common evaluation protocol, using sparse hand-mounted IMUs as sensory inputs and motion-capture-derived MANO annotations as supervision.

We also use inertial inputs during training to provide a reference baseline for this research setting without additional visual information. Synchronized RGB video is included in the dataset to support comparisons with vision-based methods and future multimodal research; it is not a required input to the baseline. The potential benefit of using visual information during training has not been evaluated in this work, and we do not claim that IMU-only training outperforms vision-assisted training. Table R2 summarizes the role of each recorded modality in our benchmark protocol.

Table R2. Roles of the synchronized modalities in IMUHand across dataset construction, IMU-track training, and online deployment.

ModalityDataset ConstructionIMU-Track TrainingOnline Deployment
Hand-mounted IMUs (6 per hand)Recorded synchronously with timestamps and static flat-hand calibrationSole sensory input to the baseline and adapted inertial methodsSole sensory input (with 10 s per-donning flat-hand calibration)
ChingMu optical mocap (12 cameras)Reconstructs 3D skeletons fitted to per-frame MANO annotationsSupplies MANO pose supervision and offline virtual-IMU targetsNot used
Egocentric stereo RGB (ZED, 30 Hz)Synchronized visual reference for visibility labeling and multimodal tasksNot used in the IMU-only baseline track (used only to fine-tune visual baselines in Section 6.2.2)Not used

We have made the following revisions to state this objective explicitly:

  • End of the first Introduction paragraph (Section 1): We specify hand-pose reconstruction with sparse hand-mounted IMUs as the sole sensory input and explain the role of motion-capture-derived MANO supervision.

  • End of the dataset-comparison paragraph and the second contribution (Section 1): We explain that the benchmark supports comparison of inertial methods under a common sensor configuration, output definition, and evaluation protocol, while synchronized RGB is provided as an additional modality.

  • Related Work (Sections 2.2–2.3): We state the intended applications with limited visual observations or constraints on visual recording and remove wording that presented vision-assisted training as a shortcoming of prior methods.

Continued in Part 3/4.

AuthorsOpenReview ↗

Response to Reviewer kVE7 — Part 3/4

Q1 — HaMeR Coverage Reversal and Gloved-Hand Domain Adaptation

Q1: In Table 3, could you explain why the coverage of HaMeR in unoccluded condition is lower than occluded condition? Do the vision-based models perform poorly regardless of whether hands are occluded or not due to domain gap (i.e. those models are trained to take bare-hand images but gloved-hand images are taken for the comparison as shown in Figure 3)?

Response:

Thank you for these constructive suggestions. Guided by them, we redesigned the vision comparison with a more appropriate setup and grouping: per-hand occlusion and truncation labels are now computed from GT geometry and camera poses, and the visual baselines are fine-tuned on our gloved-hand training data. The revised comparison directly addresses both the coverage reversal and the domain-gap concern.

Coverage reversal.

The original labels described whole sequences: "occluded" sequences could include visible hands, while "unoccluded" sequences could include out-of-frame hands. The 35.2% versus 39.1% coverage therefore did not compare actual hand visibility. We now label each hand-frame from GT meshes and camera geometry: OO is the fraction of its in-image silhouette hidden by the other hand, and TT is the fraction of its full silhouette outside the image. With T=0T=0, HaMeR coverage falls from 98.2% to 40.3% and 23.2% as occlusion increases (Table R5).

Gloved-hand domain gap.

We fine-tuned HaMeR and ACE-Ego-Hand on training-split gloved-hand RGB, selecting checkpoints only on validation data. With identical inputs and validity masks before/after fine-tuning, clear-view MPJPE falls from 14.76 to 5.37 mm for HaMeR and from 14.70 to 5.33 mm for ACE (Table R3). Thus, the adapted visual models perform well in clear views.

Table R3. Domain adaptation: MPJPE (mm) before/after fine-tuning. All-test N=210,106N=210{,}106; clear-view N=24,739N=24{,}739 (O=T=0O=T=0). Each method uses its valid outputs.

MethodAll test / PretrainedAll test / Fine-tunedClear view / PretrainedClear view / Fine-tuned
HaMeR15.436.0214.765.37
ACE15.955.9214.705.33

Results after adaptation.

Tables R4 and R5 reproduce the full revised manuscript Table 3. Across the three high-occlusion groups, Ours has MPJPE 7.77–7.85 mm versus 16.61–18.17 mm for HaMeR and 10.44–10.82 mm for ACE, with 100% versus 7.9–23.2% HaMeR coverage. Fully out of frame, Ours achieves 9.45 mm versus ACE’s 12.87 mm; HaMeR returns only 2/620 outputs. Visual models are more accurate in clear views, and ACE remains competitive under partial truncation without inter-hand occlusion.

Continued in Part 4/4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.