acceptodds
Withdrawn from ICLR 2027

Interaction-Centric Human-to-Dexterous Demonstration Capture System and Dataset

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large-scale human manipulation demonstrations offer a low-cost source of diverse data for dexterous robot manipulation, but the substantial embodiment gap between humans and robots limits their direct use for robot learning. We present an interaction-centric hand-object data capture pipeline that converts human demonstrations into dexterous manipulation data through cost-efficient acquisition and annotation. Our key insight is to leverage spatio-temporal hand-object interaction as an embodiment-transferable representation, using it to constrain motion estimation and preserve interaction relationships during human-to-robot transfer. Specifically, we introduce interaction-guided object motion estimation that exploits spatio-temporal contact plausibility to prune implausible pose hypotheses, complementing limited visual evidence under sparse-view capture and reducing the pose search space for efficient annotation. We further preserve hand-object contact relationships during simulation-based motion retargeting, transforming human demonstrations into physically plausible robot manipulation data. Experiments demonstrate several-fold improvements in object motion annotation efficiency over existing methods and sub-centimeter object pose estimation under sparse-view capture, while physics-based simulation validates interaction-preserving cross-embodiment transfer. Building on this pipeline, we construct H2D, comprising 5.6 hours of human demonstrations and 2,200 dexterous manipulation trajectories with structured 3D hand-object states.

Primary area:
datasets and benchmarks
Keywords:
Hand-Object Interaction, Dexterous Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Pi6ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an interaction-aware framework that leverages embodiment-transferable hand-object interaction priors to recover structured human manipulation states and transfer them to dexterous robots. The work also introduces an HOI capture system and H2D, a dataset comprising 5.6 hours of human manipulation demonstrations and 2,200 dexterous manipulation trajectories with structured 3D hand-object states.

Critical strengths
  1. The captured HOI dataset is large in size, has good annotations, and will be beneficial to relevant research communities. H2D combines multiview observations, structured hand-object annotations, and robot trajectory generation across unilateral and bimanual tasks.
  2. The proposed HOI pose estimation method shows better results compared to baselines HO-CAP and AlignPose. On DexYCB, the method reduces object center error from 17.92 mm for HO-CAP to 7.86 mm, while reducing reported processing time from 18.26 to 0.99 seconds per frame.
  3. The paper also evaluates reconstructed trajectories through simulation with SPIDER-like trajectory optimization, demonstrating benefits to downstream robot learning tasks.
Critical weaknesses
  1. Missing isolated ablation studies. Although through quantitative results on DexYCB HOI reconstruction the paper demonstrates improvement over baselines (HO-CAP and AlignPose), the paper lacks critical isolated ablation studies about separately evaluating semantic contact correspondence, temporal consistency, SDF constraints, hypothesis pruning, and final interaction-based scoring. It's unclear how much each proposed module actually helps the whole pipeline.
  2. Too loose simulation success thresholds. In table 4, 'success' uses object trajectory errors below 0.1 m and 0.3 rad, while these thresholds can tolerate substantial deviations (especially position errors) for dexterous manipulation. Task-specific completion metrics and threshold sensitivity analysis would make the results more convincing.
  3. Unintuitive and limited methodologic improvement. The HOI pose estimation improvement proposed in this work seems limited, since it's simply doing some heuristic-based specific optimization, without fundamental changes to the overall pipeline. Moreover, the added contact cost and consistency don't always hold true: for example, when contact changes like in scenarios like regrasping, in-hand reorientation, and handover, the penalty terms seem to encourage all hands and fingers to get close to the object, instead of encouraging only relevant parts. The authors should add more details and make it more clear that how contact-region masking is implemented and how contact sets are maintained.
  4. Dexterity in the dataset. From the paper figures, the demonstrated trajectories in the dataset are mainly pick-and-place, without truly dexterous behaviors such as in-hand reorientation. The paper also lacks proper details on the task composition of the dataset, while table 1 is not sufficient.
  5. Unscalable capture. Capture requires 3–5 synchronized RGB-D cameras, scanned object models, and semi-automatic segmentation. This is a standard lab setting and seems not very scalable to in-the-wild data collection.
Questions for authors
  1. Please conduct isolated, controlled ablation experiments to verify the effectiveness of each proposed modules.
  2. Please elaborate more on how contact-region masking is implemented and how contact sets are maintained.
  3. Please tighten the success threshold for SPIDER simulation results to sanity check whether the retargeted robot trajectories actually preserve the manipulation goal.
  4. Please provide more details on the dataset compositions and dexterous behaviors.
Reviewer_yMV1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed a system which estimate hand object interation and retarget the tarjectory to simulated robot. The core idea is using interaction(or contact) as constraint to get better trajectory. The paper also provide a HOI dataset called H2D.

Critical strengths
  1. This work propose a capture system and a dataset.
Critical weaknesses
  1. The paper fails to clearly describe the system architecture and the dataset. From the pipeline design down to the implementation, the presentation is disorganized and omits many important details. The experiments are also insufficient: aside from the dataset introduced in the paper itself, the authors compare only one or two methods on a single dataset using a partial set of metrics, which demonstrates almost nothing. This may be a writing problem, but it raises serious doubts about whether the paper was carefully checked and polished.
  2. Novelty and value of the proposed method and dataset. The novelty is quite limited. To a large extent, the paper stitches together existing methods rather than reorganizing the problem from first principles, so the pipeline itself offers little that is new. The experiments also do not compare against recent methods, and the metrics reported on DexYCB are not particularly strong. In addition, the dataset is limited in scale, size, and diversity, and the paper does not analyze it in sufficient detail.
Questions for authors
  1. Do H2D trajectories improve a fixed VLA/BC policy vs (a) kinematic retargeting without interaction preservation and (b) an equal number of teleop robot demos on at least one real dexterous task?
  2. How does annotation accuracy scale when reducing from 4 views to 2 or 1, and does interaction pruning remain necessary with denser cameras?
  3. What is object/category diversity beyond the 16 objects, and can the pipeline handle deformables or transparent items?
  4. Against recent automatic HOI datasets in Table 1 (HOT3D, GigaHands, etc.), what unique learning benefit do H2D’s robot trajectories provide beyond having R labels?
  5. Please report confidence intervals / sequence-level variance for Table 4 and failure cases where interaction priors hurt.
Reviewer_xLZKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a sparse multi-view RGB-D capture pipeline that reconstructs human hand-object trajectories and converts them into dexterous robot demonstrations. It augments FoundationPose with semantic contact consistency and SDF-based hypothesis filtering, and incorporates the resulting interaction representation into SPIDER for physics-based retargeting.

Critical strengths

Strengths:

  1. The work addresses a practical data acquisition problem and connects human motion annotation with robot demonstration generation through a shared interaction representation.
  2. reported results show improvements in annotation accuracy and computational cost over the evaluated pipelines, while simulation experiments provide an additional check across different hand embodiments.
Critical weaknesses

W1. The methodological contribution is insufficiently established.

The proposed pipeline builds on USTHand, FoundationPose [1], and SPIDER [2], but lacks controlled ablations isolating semantic contact correspondence, temporal consistency, and SDF constraints. In particular, there is no comparison between unmodified SPIDER and the proposed modification using identical input trajectories and computational budgets. The simulation comparison changes the upstream object-motion estimates, so it cannot isolate the benefit of the new interaction representation for retargeting. This distinction matters because SPIDER already includes contact-guided trajectory generation.

W2. The dataset’s value for downstream robot learning remains unverified.

This paper does not evaluate policies trained on generated dataset and explicitly leaves policy learning to future work. Annotation accuracy and simulated trajectory tracking provide useful intermediate evidence, but do not establish whether the generated demonstrations improve learning efficiency, task success, or generalization. Related work such as ManipTrans [3] demonstrates how generated manipulation data can be evaluated through policy learning. A matched learning experiment is needed to substantiate H2D’s proposed role as a resource for robot learning.

W3. The interaction objective contains unresolved formulation issues.

As written, the SDF objective sums over all recovered hand vertices and uses ϕ(d)=max⁡(−d,0,d−δ)\phi(d)=\max(-d,0,d-\delta). Under the usual positive-outside SDF convention, this penalizes both penetration and distances exceeding the contact margin. Applied to all hand vertices, it would therefore penalize legitimate non-contact regions and bias object-pose selection toward excessive proximity. The manuscript does not specify a contact mask or explain why this attraction should apply to the entire hand. The contact-consistency objective also leaves empty contact sets undefined, although these naturally occur when contact is established or released. These ambiguities affect the central hypothesis-filtering mechanism.

W4. Essential implementation details are missing.

The available supplementary archive contains evaluation CSV files and summary records, but no additional methodological documentation or implementation. Important details remain unspecified, including the construction of contact regions and heatmaps, contact thresholds, interaction weights and the precise integration of semantic contact information into SPIDER. Consequently, the current materials do not allow the proposed modifications to be reproduced or the formulation concerns above to be resolved.

[1] Wen et al. FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[2] Pan et al. SPIDER: Scalable Physics-Informed Dexterous Retargeting. arXiv:2511.09484, 2025.

[3] Li et al. ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

Questions for authors

Question: See weaknesses. This paper requires substantial revision to clarify its methodological contribution and strengthen its experimental validation. I encourage the authors to address these concerns thoroughly before resubmission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.