- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposed Hand2Robot, an object-centric 4D hand reconstruction framework. The pipeline can be divide into three steps, recon, refine, and policy training/deploy. The proposed method achieve high object pose accuracy and hand-object interaction quality on two common used datasets, and show that the human data can benefit the policy training.
Critical strengths
- This work covering the full pipeline from data collection to policy training, and it analyzes and ablates several key factors along the way.
- The method achieves strong results. The paper is well written and easy to follow.
Critical weaknesses
-
The paper proposed a pipeline built from existing models(e.g. SAM3D, foundationpose and WiLoR), with the main contribution lying in pipeline construction and optimization strategies. The pipeline is not particularly novel, foundationpose for 6-dof, WiLoR or HaMeR for hand pose is close to a 'default options' in recent works.
-
Experiments.The current experiments are done in third-view setting, The absence of experiments from an ego-view, or from other viewpoints, makes me doubt whether the method generalizes across camera viewpoints. The paper show that human demostration can coorporate with real robot demostration to achieve higher success rate, but the number of demostrations quite small. Although the results provide some evidence that human data can be used to train a single-task policy, the scale of the data and the scope of the task is limited, so whether the human demonstrations produced by this paper can train a generalizable policy remains an open question.
Questions for authors
Please refer to weaknesses. The following are some additional questions.
- If human videos are recorded from a mismatched viewpoint (or RGB-only without depth), by how much do Tables 3–4 degrade relative to the matched RGB-D setup?
- Please ablate which refinement terms (mask / ray-contact / SDF / temporal) are necessary for the policy gains, not only for Table 2 metrics.
- How does performance change when retargeting to a second hand morphology (e.g., Allegro/Inspire) under the same post-training recipe?
- Would the paper’s conclusions change if the human demonstrations were scaled to the thousands or tens of thousands? And would the human demonstrations be more effective if the training data were retargeted sequences rendered in simulation?
- Could the authors provide a straightforward, detailed breakdown of the compute and wall-clock time required for this pipeline to process one hour of video?
- The related work has minor inaccuracy, (e.g. RoboWheel focuses on hand–object interaction, it is not a work that emphasizes hand pose estimation).
3 comments
I find the path-selection and refiner components useful, and I am willing to raise my score on that basis. I nevertheless remain unconvinced by the current downstream training strategy. Although simulation introduces an additional domain gap, it is at least (looks-like) scalable: trajectories can be augmented, and more realistic rendered environments are available. The authors have not provided sufficiently persuasive evidence that the proposed data pipeline is better aligned with the more fundamental goals of scene, task, and embodiment generalization, and I am not convinced on this point.
Response to Reviewer Yc9w
We thank the reviewer for recognizing the value of the path-selection and refiner components, and we appreciate the reviewer’s constructive critique. We have taken this concern seriously.
Response to Reviewer Yc9w
We thank the reviewer for the careful review and constructive feedback.
Q1:
A1:
(a) Technical focus
Foundation models provide initial estimates. We select a sequence-level path from object-pose candidates using observation-agreement weights and inter-frame motion costs, then refine global hand poses and finger configurations against this fixed reference using staged 2D alignment, contact constraints, and temporal regularization.
(b) Evidence and contribution scope
Table 1 and Appendix Table 7 assess reconstruction gains; Table 3 evaluates downstream supervision using the same source videos. We will clarify these specific contributions and measured effects in the revision.
Q2:
A2:
(a) Viewpoint
Human and robot demonstrations use the same fixed side-view camera system to evaluate action supervision under controlled observations. We plan to add egocentric human-data experiments.
(b) Depth input
We collect demonstrations specifically for target-task post-training, allowing RGB-D capture. Depth supports object scale and pose estimation; our setting differs from pretraining on existing RGB-only datasets.
Q3:
A3:
Following the reconstruction ablations (Appendix Table 7), we are conducting a policy ablation with the same videos, initialization, retargeting, and training settings. We compare the full method with a variant removing both per-frame ray-contact and temporal-contact terms, while retaining 2D alignment, SDF, basic smoothing, and initialization regularization.
Q4:
A4:
We reconstruct MANO hand motion (Romero et al., 2017) before target-hand retargeting. We plan to retarget the same trajectories to a second hand and assess contact preservation, object penetration, and kinematic feasibility.
Q5:
A5:
(a) Data scale and generalization
We study target-task post-training with limited robot data. Within the tested budgets, additional Hand2Robot human demonstrations improve performance, supporting their complementary value.
(b) Rendering robot observations
Rendering robot observations, as explored in RoboWheel, may narrow human-robot appearance differences but introduce synthetic-to-real gaps in lighting, materials, occlusion boundaries, and camera registration.
Q6:
A6:
Hand2Robot generates supervision offline. On an Intel Xeon Gold 6430 CPU and NVIDIA RTX 5880 Ada GPU, the full pipeline takes approximately 20 minutes for a 200-frame 480×640 video, without additional subsampling. Independent clips allow multi-GPU parallelism to reduce wall-clock time. We plan to assess subsampling or keyframe optimization with trajectory interpolation, reporting per-stage runtime, parallel throughput, and reconstruction quality.
Q7:
A7:
RoboWheel covers hand-object reconstruction, refinement for physical plausibility, cross-embodiment retargeting, and data augmentation. We will discuss this work in the related work section.