acceptodds
Withdrawn from ICLR 2027

Hand2Robot: Object-Centric 4D Hand Reconstruction from Human Demonstrations for Robotics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human demonstrations can capture task-relevant manipulation without using the target robot, but the recorded hand motions cannot directly serve as robot action supervision. Hand reconstruction and motion retargeting provide a direct route to such supervision, yet recovering hand motion alone does not ensure physically consistent hand-object interactions. To address this issue, we introduce Hand2Robot, an object-centric 4D hand reconstruction framework for robot policy post-training. Given videos of humans performing the target task, Hand2Robot extracts interaction clips, reconstructs object geometry, and generates robot action supervision through hand-object interaction optimization and motion retargeting. For object tracking, we combine observation confidence with temporal motion constraints to select a globally consistent object pose trajectory from per-frame pose candidates. For hand refinement, we use object geometry and motion as a reference and combine staged 2D alignment with object-local contact constraints to refine hand trajectories and finger joint configurations. Experiments on DexYCB and HO3D-v3 show improved object pose accuracy and hand-object interaction quality. Experiments on three real-robot dexterous manipulation tasks further show that reconstructed and retargeted human demonstrations can effectively supplement limited robot data for target-task policy post-training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Human Demonstrations, 4D Hand Reconstruction, Hand-Object Interaction, Motion Retargeting, Robot Policy Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Yc9wOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed Hand2Robot, an object-centric 4D hand reconstruction framework. The pipeline can be divide into three steps, recon, refine, and policy training/deploy. The proposed method achieve high object pose accuracy and hand-object interaction quality on two common used datasets, and show that the human data can benefit the policy training.

Critical strengths
  1. This work covering the full pipeline from data collection to policy training, and it analyzes and ablates several key factors along the way.
  2. The method achieves strong results. The paper is well written and easy to follow.
Critical weaknesses
  1. The paper proposed a pipeline built from existing models(e.g. SAM3D, foundationpose and WiLoR), with the main contribution lying in pipeline construction and optimization strategies. The pipeline is not particularly novel, foundationpose for 6-dof, WiLoR or HaMeR for hand pose is close to a 'default options' in recent works.

  2. Experiments.The current experiments are done in third-view setting, The absence of experiments from an ego-view, or from other viewpoints, makes me doubt whether the method generalizes across camera viewpoints. The paper show that human demostration can coorporate with real robot demostration to achieve higher success rate, but the number of demostrations quite small. Although the results provide some evidence that human data can be used to train a single-task policy, the scale of the data and the scope of the task is limited, so whether the human demonstrations produced by this paper can train a generalizable policy remains an open question.

Questions for authors

Please refer to weaknesses. The following are some additional questions.

  1. If human videos are recorded from a mismatched viewpoint (or RGB-only without depth), by how much do Tables 3–4 degrade relative to the matched RGB-D setup?
  2. Please ablate which refinement terms (mask / ray-contact / SDF / temporal) are necessary for the policy gains, not only for Table 2 metrics.
  3. How does performance change when retargeting to a second hand morphology (e.g., Allegro/Inspire) under the same post-training recipe?
  4. Would the paper’s conclusions change if the human demonstrations were scaled to the thousands or tens of thousands? And would the human demonstrations be more effective if the training data were retargeted sequences rendered in simulation?
  5. Could the authors provide a straightforward, detailed breakdown of the compute and wall-clock time required for this pipeline to process one hour of video?
  6. The related work has minor inaccuracy, (e.g. RoboWheel focuses on hand–object interaction, it is not a work that emphasizes hand pose estimation).
3 comments
AuthorsOpenReview ↗

Response to Reviewer Yc9w

We thank the reviewer for the careful review and constructive feedback.

Q1:

The paper proposed a pipeline built from existing models(e.g. SAM3D, foundationpose and WiLoR), with the main contribution lying in pipeline construction and optimization strategies. The pipeline is not particularly novel, foundationpose for 6-dof, WiLoR or HaMeR for hand pose is close to a 'default options' in recent works.

A1:

(a) Technical focus

Foundation models provide initial estimates. We select a sequence-level path from object-pose candidates using observation-agreement weights and inter-frame motion costs, then refine global hand poses and finger configurations against this fixed reference using staged 2D alignment, contact constraints, and temporal regularization.

(b) Evidence and contribution scope

Table 1 and Appendix Table 7 assess reconstruction gains; Table 3 evaluates downstream supervision using the same source videos. We will clarify these specific contributions and measured effects in the revision.

Q2:

If human videos are recorded from a mismatched viewpoint (or RGB-only without depth), by how much do Tables 3–4 degrade relative to the matched RGB-D setup?

A2:

(a) Viewpoint

Human and robot demonstrations use the same fixed side-view camera system to evaluate action supervision under controlled observations. We plan to add egocentric human-data experiments.

(b) Depth input

We collect demonstrations specifically for target-task post-training, allowing RGB-D capture. Depth supports object scale and pose estimation; our setting differs from pretraining on existing RGB-only datasets.

Q3:

Please ablate which refinement terms (mask / ray-contact / SDF / temporal) are necessary for the policy gains, not only for Table 2 metrics.

A3:

Following the reconstruction ablations (Appendix Table 7), we are conducting a policy ablation with the same videos, initialization, retargeting, and training settings. We compare the full method with a variant removing both per-frame ray-contact and temporal-contact terms, while retaining 2D alignment, SDF, basic smoothing, and initialization regularization.

Q4:

How does performance change when retargeting to a second hand morphology (e.g., Allegro/Inspire) under the same post-training recipe?

A4:

We reconstruct MANO hand motion (Romero et al., 2017) before target-hand retargeting. We plan to retarget the same trajectories to a second hand and assess contact preservation, object penetration, and kinematic feasibility.

Q5:

Would the paper’s conclusions change if the human demonstrations were scaled to the thousands or tens of thousands? And would the human demonstrations be more effective if the training data were retargeted sequences rendered in simulation?

A5:

(a) Data scale and generalization

We study target-task post-training with limited robot data. Within the tested budgets, additional Hand2Robot human demonstrations improve performance, supporting their complementary value.

(b) Rendering robot observations

Rendering robot observations, as explored in RoboWheel, may narrow human-robot appearance differences but introduce synthetic-to-real gaps in lighting, materials, occlusion boundaries, and camera registration.

Q6:

Could the authors provide a straightforward, detailed breakdown of the compute and wall-clock time required for this pipeline to process one hour of video?

A6:

Hand2Robot generates supervision offline. On an Intel Xeon Gold 6430 CPU and NVIDIA RTX 5880 Ada GPU, the full pipeline takes approximately 20 minutes for a 200-frame 480×640 video, without additional subsampling. Independent clips allow multi-GPU parallelism to reduce wall-clock time. We plan to assess subsampling or keyframe optimization with trajectory interpolation, reporting per-stage runtime, parallel throughput, and reconstruction quality.

Q7:

The related work has minor inaccuracy, (e.g. RoboWheel focuses on hand–object interaction, it is not a work that emphasizes hand pose estimation).

A7:

RoboWheel covers hand-object reconstruction, refinement for physical plausibility, cross-embodiment retargeting, and data augmentation. We will discuss this work in the related work section.

I find the path-selection and refiner components useful, and I am willing to raise my score on that basis. I nevertheless remain unconvinced by the current downstream training strategy. Although simulation introduces an additional domain gap, it is at least (looks-like) scalable: trajectories can be augmented, and more realistic rendered environments are available. The authors have not provided sufficiently persuasive evidence that the proposed data pipeline is better aligned with the more fundamental goals of scene, task, and embodiment generalization, and I am not convinced on this point.

AuthorsOpenReview ↗

Response to Reviewer Yc9w

We thank the reviewer for recognizing the value of the path-selection and refiner components, and we appreciate the reviewer’s constructive critique. We have taken this concern seriously.

Reviewer_nTqmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Hand2Robot turns human RGB-D demonstrations into supervision for dexterous robot learning. It selects object pose trajectories from FoundationPose candidates, refines hand motion with image and contact constraints, and retargets the result to an xHand. Evaluation on DexYCB, HO3D-v3, and three robot tasks shows improved reconstruction and useful policy-training gains. Combining human and robot demonstrations approaches the performance of training with more robot demonstrations.

Critical strengths

The real-robot results are encouraging. Comparing supervision generated from the same source videos is a useful control, and the benefit over robot-only training appears across all three tasks.

The reconstruction ablations expose meaningful trade-offs. Removing image alignment reduces penetration but worsens the wrist trajectory, illustrating why geometric plausibility cannot be judged by penetration alone.

Critical weaknesses

The contribution beyond existing video-to-robot pipelines remains unclear. The main robot baselines are a basic WiLoR pipeline and an adapted MagicHOI module, and the proposed trajectory and contact refinements have not been separately tested in policy learning.

The evaluation is narrow: selected reconstruction subsets and three robot tasks, with 20 trials per condition.

Questions for authors
  1. What is the main advantage over existing video-to-robot pipelines such as RoboWheel, VideoManip, and DexImit? Some of these use RGB videos and report learning without additional robot demonstrations. The settings differ, so their success rates are not directly comparable, but I would like to understand what Hand2Robot gains from its input requirements and where that benefit is demonstrated.

  2. Which proposed refinement accounts for the downstream improvement? The reconstruction ablations are useful, but a small robot ablation with initialization, retargeting, and training held fixed would help establish the contribution of object trajectory selection or contact refinement. Why is Do As I Do compared only at the smaller human-data budget, and how was the effect of its Sharpa-to-xHand adaptation checked?

  3. Why does adding WiLoR or MagicHOI supervision substantially hurt the robot-only policy? Does the shared initialization described for the reconstruction comparison also apply to Table 3, including depth alignment? Please clarify how failed reconstructions are handled and whether all source videos contribute training samples.

  4. What do the robot demonstrations provide that the generated human data lacks? The weak human-only result deserves more discussion. Section 3.4 describes pairing actions with video observations, but are these raw human images or edited images? What state inputs accompany them, and how are the relative-action targets constructed?

  5. What evidence shows that the method recovers correct interactions under occlusion, rather than merely smoother motion? Please define “visible-hand frames” and explain how ambiguous ray intersections and contacts that disappear and reappear are handled beyond the stated outlier rejection. For Open Box, which part is represented by the tracked rigid mesh?

  6. Please provide the evaluation sequence selection criteria, and report any available variation across independent policy-training runs. Most of the advantage over Do As I Do comes from Open Box; the other tasks are tied or differ by one successful trial. How stable is this comparison?

  7. Two evaluation details need clarification. Why does “w/o d,c,” which should select the highest-scoring candidate per frame, differ so much from the FoundationPose baseline?

2 comments
AuthorsOpenReview ↗

Response to Reviewer nTqm (Part 1/2)

We thank the reviewer for the careful review and constructive feedback.

Q1: Relationship to existing video-to-robot pipelines

What is the main advantage over existing video-to-robot pipelines such as RoboWheel, VideoManip, and DexImit? Some of these use RGB videos and report learning without additional robot demonstrations. The settings differ, so their success rates are not directly comparable, but I would like to understand what Hand2Robot gains from its input requirements and where that benefit is demonstrated.

A1:

(a) Technical focus

We acknowledge the complete video-to-robot pipelines of RoboWheel, VideoManip, and DexImit. Our focus is reconstruction before retargeting: observation agreement weights the global selection of multiple object-pose candidates to establish a motion reference. Global hand motion and finger configurations are then refined using 2D observations, contact constraints, and temporal regularization.

(b) Supporting evidence

Table 1 and Appendix Table 7 report object-trajectory and hand-refinement ablations; Table 3 assesses the generated supervision's downstream value using the same source videos.

Q2: Downstream contributions and the Do As I Do comparison

Which proposed refinement accounts for the downstream improvement? The reconstruction ablations are useful, but a small robot ablation with initialization, retargeting, and training held fixed would help establish the contribution of object trajectory selection or contact refinement. Why is Do As I Do compared only at the smaller human-data budget, and how was the effect of its Sharpa-to-xHand adaptation checked?

A2:

(a) Reconstruction ablations and downstream policy results

Table 3 uses common xHand retargeting, coordinate transformation, and policy training. Each mixed condition uses the same 20 robot demonstrations (R) and 100 source human videos (H) per task. Success rates (%):

Training data and methodPlace VaseOpen BoxInsert ToastMean
20R, robot-only45.060.040.048.3
20R+100H, WiLoR25.020.010.018.3
20R+100H, Hand2Robot80.085.075.080.0

Table 3 supports the value of generated supervision but does not isolate individual refinements' policy contributions. We will distinguish these pipeline-level gains from the reconstruction ablations in the revision.

(b) The 50H comparison budget

Do As I Do (code) takes approximately three hours per demonstration in our reproduction setup. We therefore use a smaller budget, excluding failed videos. With the same 20R and 50 source human videos per task, its adapted pipeline and Hand2Robot achieve mean success rates of 45.0% and 55.0%, respectively (Table 4).

(c) xHand adaptation

We adapt Do As I Do outputs to xHand using the same retargeting method as Hand2Robot. Since our tasks involve simple grasp-and-place motions without complex in-hand manipulation, we expect retargeting errors to have a limited impact.

Q3: Baseline degradation, initialization, and failed reconstructions

Why does adding WiLoR or MagicHOI supervision substantially hurt the robot-only policy? Does the shared initialization described for the reconstruction comparison also apply to Table 3, including depth alignment? Please clarify how failed reconstructions are handled and whether all source videos contribute training samples.

A3:

(a) Why WiLoR or MagicHOI supervision can hurt performance

Residual wrist-position, finger-pose, or contact errors may propagate into action supervision and conflict with robot demonstrations. Shared retargeting, coordinate transformation, and relative-action encoding do not automatically remove them.

(b) Initialization and depth alignment in Table 3

For Table 3, WiLoR and MagicHOI follow their original methods with RGB-D depth alignment. Hand2Robot uses depth-aligned WiLoR initialization followed by hand-object optimization.

(c) Failure handling and training-sample inclusion

In our Hand2Robot real-robot experiments, approximately 2% of human videos are excluded for object-grounding failures; all data generated from the remaining videos are used for training.

AuthorsOpenReview ↗

Part 2/2 — Q4–Q7

Q4: The role of robot demonstrations and the policy data interface

What do the robot demonstrations provide that the generated human data lacks? The weak human-only result deserves more discussion. Section 3.4 describes pairing actions with video observations, but are these raw human images or edited images? What state inputs accompany them, and how are the relative-action targets constructed?

A4:

(a) What robot demonstrations provide

Human-only training may be affected by the visual shift from human hands during training to robot hands at deployment, as well as differences in hand morphology and execution characteristics. Human data provide task-relevant motion and interaction supervision; robot demonstrations provide target-platform observation-action pairs and grasping details, including finger configurations and contact patterns.

(b) Visual observations

We use raw human-video images without replacing human hands or rendering robot images. After retargeting and coordinate transformation, actions use the robot-data representation and are paired with the corresponding video observations (Section 3.4).

(c) State inputs and relative-action targets

Actions comprise 3D end-effector translation, 3D axis-angle rotation, and 12 xHand joint angles (18 dimensions). Human states use current-frame retargeted wrist poses and joint angles; robot states use their measured counterparts. Both sources share relative-action encoding referenced to the initial frame of each clip.

Q5: Interaction recovery, visibility, contact matching, and Open Box

What evidence shows that the method recovers correct interactions under occlusion, rather than merely smoother motion? Please define “visible-hand frames” and explain how ambiguous ray intersections and contacts that disappear and reappear are handled beyond the stated outlier rejection. For Open Box, which part is represented by the tracked rigid mesh?

A5:

(a) Evidence beyond temporal smoothness

DexYCB results show lower MRPE and IV (Table 2). Without per-frame ray contact (Appendix Table 7), JM changes only from 2.49 to 2.53, but MRPE increases from 38.49 to 48.70. This supports positional recovery beyond smoothing.

(b) Definition of visible-hand frames

Visible-hand frames are those in which the hand appears in the image. MPJPE and MRPE are computed only on these frames.

(c) Ray intersections and reappearing contact

Multiple ray intersections are resolved by choosing the nearest or farthest valid hit according to the candidate's surface normal and left/right-hand convention. Concave geometries can remain ambiguous. Candidates without valid intersections are excluded from the current frame's ray-contact loss. Reappearing contacts are rematched from current hand and object geometry rather than retaining pre-interruption correspondences.

(d) The rigid component tracked in Open Box

Open Box involves lifting a removable lid by its handle and placing it on the table. The lid and handle are reconstructed and tracked as one rigid mesh with a 6D pose trajectory.

Q6: Evaluation subsets and comparison stability

Please provide the evaluation sequence selection criteria, and report any available variation across independent policy-training runs. Most of the advantage over Do As I Do comes from Open Box; the other tasks are tied or differ by one successful trial. How stable is this comparison?

A6:

(a) Sequence selection

We randomly select 1,000 DexYCB sequences for object-pose evaluation and 120 of these for hand evaluation, covering diverse object categories and interaction patterns. We randomly select 12 HO3D-v3 sequences.

(b) Real-robot evaluation and the observed task differences

We evaluate each trained policy in 20 real-robot trials per task, with initial object positions sampled randomly from the training distribution. This follows evaluation practices in related work (Chi et al., 2024; DexCast, 2026). The gains over Do As I Do are primarily observed on Open Box and Insert Toast, which is related to the lower difficulty of Place Vase.

Q7: Why w/o d,c differs from FoundationPose

Two evaluation details need clarification. Why does “w/o d,c,” which should select the highest-scoring candidate per frame, differ so much from the FoundationPose baseline?

A7:

The FoundationPose baseline uses continuous tracking. All our ablations retain per-frame re-registration; removing temporal cost and observation weight therefore gives per-frame highest-score selection, rather than the continuously tracked baseline.

Reviewer_y2WhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Hand2Robot converts human RGB-D videos into actions for dexterous robot policy post-training via object-centric 4D hand-object reconstruction. It picks a globally consistent object motion sequence from per-frame FoundationPose candidates, then refines the hand motion against that fixed object. On three real-robot tasks, finetuned on 20 robot demos plus 100 processed human videos reaches 80.0% mean success, versus 48.3% with robot data alone and close to the 81.7% from 60 robot demos.

Critical strengths
  • The motivation is clear, and the downstream result proves the gains resulting from the proposed method: adding 100 Hand2Robot-processed human demos lifts mean success from 48.3% to 80.0%.

  • Keeping the object as the reference and selecting the pose trajectory globally makes the method robust under occlusion and symmetry. Table 1 backs this up—dropping either the transition cost or the observation weight blows up rotation error (61.63° / 38.20° vs 18.47° for the full model).

Critical weaknesses
  • There is no runtime analysis. The pipeline does heavy per-frame scoring, rendering, and per-segment optimization, but I wonder how long any of this takes. If this is meant to be a scalable data engine, that is a basic missing piece.

  • There is no failure analysis. The method consists with many foundation models, and any stage can go wrong and poison the process results. Without knowing when and how often reconstruction fails, it is hard to trust the generated data.

  • The comparison to sequence-level hand-reconstruction methods (e.g., HOLD, MagicHOI) is indirect. They are only compared on reconstruction metrics, or as a hand-optimization module plugged into the authors' pipeline—not as full alternatives for producing action supervision. So the case for this design stays incomplete.

Questions for authors

The problem is practical, and 20R+100H → 80.0% vs 48.3% for 20R alone is a solid downstream signal. The object-centric design also looks reasonable. But three gaps are hard to ignore: no runtime for a method sold as scalable data generation, no failure analysis for a long model chain whose errors become training labels, and no direct comparison against sequence-level reconstruction as an action-supervision baseline.

In the rebuttal, I would like to see: (1) a simple runtime table—per-stage time, end-to-end throughput per video, and how long the hand optimization actually takes; (2) a short failure breakdown over failed videos (what broke, how often, which stage, effect on downstream success); and (3) at least one sequence-level method (e.g., HOLD) evaluated as a full action-supervision baseline, not only as a reconstruction or module swap comparison.

1 comment
AuthorsOpenReview ↗

Response to Reviewer y2Wh

We thank the reviewer for recognizing the practical motivation, object-centric design, and real-robot results. We address the three requested clarifications below.

The problem is practical, and 20R+100H → 80.0% vs 48.3% for 20R alone is a solid downstream signal. The object-centric design also looks reasonable. But three gaps are hard to ignore: no runtime for a method sold as scalable data generation, no failure analysis for a long model chain whose errors become training labels, and no direct comparison against sequence-level reconstruction as an action-supervision baseline.

In the rebuttal, I would like to see:

Q1: Runtime and hand-object optimization

(1) a simple runtime table—per-stage time, end-to-end throughput per video, and how long the hand optimization actually takes;

A1:

Hand2Robot generates action supervision offline, and its computational cost should be reported separately from demonstration collection cost. On an Intel Xeon Gold 6430 CPU and an NVIDIA RTX 5880 Ada GPU, processing a 200-frame video at 480×640 resolution takes approximately 20 minutes for the full pipeline. For this 200-frame video, the measured per-stage runtimes under serial execution are: 0.9 seconds for object grounding with Rex-Omni, 33 seconds for object segmentation with SAM3, 33 seconds for object reconstruction with SAM 3D, 265 seconds for object pose processing, 100 seconds for hand reconstruction with WiLoR, 35 seconds for hand alignment, and 12 minutes for hand-object optimization.

Q2: Failure handling and failure modes

(2) a short failure breakdown over failed videos (what broke, how often, which stage, effect on downstream success);

A2:

In our experimental setup, approximately 2% of videos are automatically filtered out due to object grounding failures. The remaining videos are processed through the subsequent stages, and all resulting data are used for policy training. We also identify several potential failure modes in the pipeline. 1、SAM 3D may inaccurately reconstruct the grasping region of an object, particularly narrow structures such as the lid handle in the Open Box task. 2、Occlusion during manipulation can cause abrupt jumps or offsets in the estimated object poses. Our object trajectory optimization helps mitigate these errors. 3、During hand-object optimization, a decrease in the optimization loss does not always imply a physically plausible interaction. Our optimization is regularized toward the initial WiLoR estimates, which helps limit excessive deviations from the initial hand motion.

Q3: Complete reconstruction pipelines as action-supervision baselines

and (3) at least one sequence-level method (e.g., HOLD) evaluated as a full action-supervision baseline, not only as a reconstruction or module swap comparison.

A3:

Table 3 uses the complete MagicHOI (Wang et al., 2025) reconstruction pipeline, followed by depth alignment and the same xHand retargeting, coordinate transformation, and relative-action encoding as Hand2Robot for π0.5 post-training. With the same 20 robot demonstrations and 100 source human videos per task, MagicHOI achieves success rates of 40.0%, 25.0%, and 15.0%, averaging 26.7%. We will correct the manuscript’s inaccurate description of this baseline as a hand-optimization module. We also evaluate Do As I Do (Paliwal et al., 2026), retaining its reconstruction and performing retargeting and dynamics-based post-optimization directly on xHand. With the same 20 robot demonstrations and 50 source human videos per task, this adapted baseline and Hand2Robot achieve mean success rates of 45.0% and 55.0%, respectively (Table 4), providing an additional complete-pipeline comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.