acceptodds
Withdrawn from ICLR 2027

GroundHand: Rethinking Egocentric Manipulation Data Collection with High-Precision Hand Annotation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Egocentric dexterous manipulation data are increasingly used for robot manipulation learning, yet existing data collection pipelines often cannot provide high-precision hand state annotations with consistent absolute physical scale. Common approaches first estimate hand poses from egocentric videos, recover ego-camera poses separately, and then transform the estimated hand states into a world coordinate system. Errors can accumulate throughout this pipeline, rendering the resulting annotations unreliable for fine-grained, dexterous manipulation. In this work, we propose GroundHand, a low-cost capture-and-annotation system that captures synchronized ego-view and wrist-view manipulation videos and produces high-precision MANO hand states directly in a unified workspace coordinate system. GroundHand uses a calibrated multi-view RGB-D setup with an exo-view annotation hardware cost of approximately $3,000, substantially lower than Vicon-based MoCap systems. Built upon this system, we develop a multi-source hand annotation model to estimate fine-grained hand pose and global translation. To continuously improve automatic annotation quality, we further introduce iterative human-in-the-loop model refinement, where sparse manual corrections are fed back during training to progressively refine the model. Experiments on existing benchmarks and our manually validated data show that GroundHand improves hand state accuracy and annotation efficiency over baseline methods.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Low-Cost Hand Motion Capture, Egocentric Manipulation Dataset, 3D Hand Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jJQaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GroundHand, a capture-and-annotation system for obtaining absolute-scale hand-state annotations from egocentric dexterous manipulation data. The system combines synchronized egocentric and wrist-mounted cameras with a calibrated multi-view RGB-D setup. On top of this hardware system, the paper proposes a multi-source hand reconstruction model that combines camera-aware image features, fused point-cloud features, a subject-specific MANO shape prior, and temporal decoders for hand pose and global translation. The authors further introduce an iterative human-in-the-loop refinement procedure in which sparse 2D corrections are used to optimize MANO annotations and fine-tune the reconstruction model.

The method is evaluated on DexYCB and on a newly captured evaluation subset. The reported results include MPJPE and MPVPE on DexYCB, point-cloud/mesh consistency metrics on the captured data, and reductions in human correction ratio and annotation time after iterative refinement. The paper argues that separating egocentric observation collection from exocentric geometric annotation can provide more accurate hand states at a substantially lower hardware cost than marker-based motion-capture systems.

Critical strengths

The paper addresses an important practical bottleneck in learning from egocentric dexterous manipulation videos: obtaining hand states with consistent global scale under severe occlusion. The combination of calibrated multi-view RGB-D observations, wrist-mounted views, point-cloud features, and MANO-based reconstruction is a reasonable system design for this problem.

The paper also evaluates the system from multiple perspectives, including benchmark hand reconstruction, geometric consistency on captured data, component ablations, and the reduction of manual correction effort through iterative refinement. This provides a useful initial assessment of the proposed data-collection pipeline and its practical potential.

Critical weaknesses

1. The methodological contribution is not sufficiently differentiated from existing techniques.

GroundHand combines several established components, including calibrated multi-view reconstruction, RGB-D point-cloud features, Plücker-ray embeddings, MANO shape priors, temporal attention, and decoupled prediction heads. The paper does not clearly identify a new learning objective, reconstruction principle, or architectural mechanism that constitutes the central algorithmic contribution. As a result, the main novelty appears to lie primarily in the integration of known components into a particular capture system. The results on DexYCB show that GroundHand is competitive with existing methods, but the improvement in absolute MPJPE and MPVPE over the strongest listed multi-view baseline is relatively limited. On the captured data, the larger gains may also be substantially influenced by the use of seven calibrated exocentric RGB-D cameras and two wrist-mounted cameras. The paper therefore needs stronger evidence to separate improvements caused by the sensing setup from improvements caused by the proposed reconstruction model. In its current form, the contribution is closer to a system integration study than to a sufficiently distinct learning contribution for ICLR.

2. The experimental evidence does not fully support the claims of high-precision absolute-scale annotation.

The captured-data evaluation primarily uses PC2Mesh and Mesh2PC distances between reconstructed MANO meshes and visible RGB-D points. These metrics are useful for measuring geometric agreement, but they do not directly establish the accuracy of the full hand pose and global translation, especially for occluded surfaces. The paper also compares against a human-corrected optimization result, which is useful as a practical reference but does not provide an independent ground-truth measurement of the hand state. In addition, the compared pipelines receive substantially different inputs. GroundHand uses calibrated multi-view RGB-D observations and wrist-mounted cameras, whereas the ego-view baselines mainly use monocular images and the device-based baselines rely on their own tracking systems. This makes the comparison informative at the pipeline level, but it does not isolate the contribution of the proposed model. The experimental section would also benefit from clearer information about the number of subjects, sequences, tasks, and frames; the train/test split; the evaluation protocol for the pretrained and refined variants; and more quantitative ablations of the major components and camera configurations. Without these additions, the current experiments do not fully justify the broad claims regarding precision, scalability, and generality.

Questions for authors

1、How are the “GroundHand (pretrained)” and “GroundHand (ours)” variants trained and evaluated on DexYCB? In particular, does the latter use data from the newly captured sequences before evaluation on DexYCB, and are subjects and sequences strictly separated across training and testing?

2、For the captured-data evaluation, what is the reference annotation used for each metric? Are the reported metrics computed on held-out sequences that were not used for manual correction or model refinement? Can the authors provide a direct evaluation of global translation and hand pose against an independent reference?

3、Can the authors provide controlled comparisons that disentangle the effects of the sensing setup from the effects of the reconstruction model, such as matched-input baselines or systematic ablations of exocentric views, wrist views, point-cloud features, the MANO shape prior, and temporal decoding? Reporting results across subjects and manipulation tasks would also help assess the scope of the claimed improvements.

Reviewer_GgXUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GroundHand is a low‑cost capture-and-annotation system for egocentric manipulation data. The core idea is to split data capture and high‑quality hand annotation: egocentric and wrist cameras collect footage for robot learning, while calibrated external multi‑view RGB‑D cameras provide geometry to reconstruct MANO‑format hand states in consistent world coordinates. Iterative human‑in‑the‑loop refinement are then used to improve automatic hand annotation.

Critical strengths
  1. Metric‑scale egocentric hand reconstruction.
  2. Human‑in‑the‑loop refinement with sparse human corrections.
  3. Low-cost data collection system.
Critical weaknesses
  1. No tactile sensing or contact‑related measurements that are popular in nowadays manipulation topic.
  2. Geometric reconstruction quality only. NO task labels/QA pairs for assessing tool‑use cognition (training VLMs).
  3. No public dataset to check the data quality or implementatio details.
  4. How these hardware are synchronized? What's the synchronized time gap between sensors?
Questions for authors

Justify the mentioned weakness.

Reviewer_BBKFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GroundHand, a data collection and annotation system for egocentric dexterous manipulation. The system uses multiple calibrated RGB-D cameras and wrist cameras to reconstruct hand poses in a unified coordinate system. It further develops a multi-source hand reconstruction model and an iterative human-in-the-loop refinement process to improve annotation accuracy. Experiments on DexYCB and the authors' collected data show improvements in hand reconstruction accuracy and annotation efficiency.

Critical strengths

The paper addresses an important problem in egocentric manipulation data collection, particularly the difficulty of obtaining accurate hand poses with consistent physical scale.

The proposed capture system is practical and relatively low cost compared with traditional motion capture systems.

The combination of multi-view RGB-D observations, wrist cameras, and subject-specific hand models is reasonable for improving hand reconstruction accuracy.

The iterative human-in-the-loop refinement provides a practical approach to improving annotation quality while reducing manual annotation effort.

The experiments include comparisons with existing hand reconstruction methods and commercial tracking devices. The ablation studies also demonstrate the contributions of different model components and camera configurations.

Critical weaknesses

The evaluation of hand annotation accuracy is particularly concerning. On the authors' collected data, the proposed model uses RGB-D point clouds as inputs, while annotation accuracy is evaluated using the distance between reconstructed hand meshes and the same point clouds. This evaluation is not independent and cannot establish the actual accuracy of hand poses, particularly for occluded joints and finger articulations. Although independent ground truth evaluation is provided on DexYCB, the improvements are relatively small. Since high-precision hand annotation is the central contribution of this paper, independent ground truth validation on the collected data is essential.

The experimental comparisons are not entirely fair. GroundHand uses seven calibrated external RGB-D cameras and additional wrist cameras, while several baselines rely on egocentric observations or headset tracking. The substantial differences in available sensor information make it difficult to determine how much improvement comes from the proposed reconstruction method rather than the additional cameras. More comparisons under matched input conditions are needed.

The novelty of the proposed reconstruction method is somewhat limited. Most components, including multi-view feature fusion, point cloud encoding, temporal attention, and subject-specific MANO priors, have been investigated in previous studies. The main contribution appears to be system integration rather than a fundamentally new reconstruction method.

The scalability of the proposed data collection system is insufficiently demonstrated. The system requires a fixed multi-camera setup, camera calibration, subject-specific hand calibration, and manual refinement. The paper does not clearly report the total scale and diversity of the collected dataset, including the number of subjects, manipulation tasks, and recorded sequences. These details are important for evaluating its practical scalability.

The paper motivates the proposed system through its potential benefits for robot manipulation learning, but no downstream robot learning experiments are provided. It remains unclear whether the improved hand annotations lead to better imitation learning or manipulation performance. Such experiments would help establish the practical value of the collected data beyond hand reconstruction accuracy.

Questions for authors

The model uses RGB-D point clouds as inputs and evaluates hand pose accuracy against point clouds from the same capture setup, making the evaluation insufficiently independent to validate the claimed high precision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.