- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents a method that reconstructs a monocular video of hand-object interaction and retargets it to a dexterous hand. The retargeted sequence is treated as an imperfect reference for RL to fix in simulation. During training, the object shapes and scales can be randomized to reduce overfitting to the reconstructed mesh. The results show that policies reach about 95% of ground-truth-reference performance. Training on "siblings" also increase the success rate from 18% to 68% on DexYCB.
Critical strengths
- Although there's no ground breaking idea in this paper, the design of the training pipeline with digital siblings is quite effective. It's a simple practice that does not add engineering or compute overhead. The results are compelling.
- The ablation experiments on geometry are valuable. In particular, the comparisons between scale randomization, retrieved siblings, and generated siblings are very informative.
Critical weaknesses
-
There is no closed-loop policy with perception demonstrated in the real world. All real-world results are open-loop replay. This significantly weaken the claims against Do As I Do, which only solves the reconstruction problem but has demonstrated much wider set of open-loop reply in the real world. Without convincing evidence of closed-loop policy in the real world, it is crucial to show more fair comparisons against Do As I Do to establish the advantages of the proposed method. a. Compare on Do as I Do’s own benchmarks. b. Support the argument that RL is more effective than SPIDER when constructing noisy references.
-
There are a number of relevant works not mentioned in the paper. It is challenging to establish novelty of this work without rigorous comparisons against a collective set of existing knowledge discovered previously.
- Video2Sim2Real
- Human2sim2Robot
- C2Dex
Questions for authors
- For the evaluation, did you measure the error against the ground-truth human trajectory or the reconstructed reference used for training? The 85.7% success on reconstructed meshes versus 67.8% on ground-truth meshes is difficult to interpret as fidelity to the original interaction.
- How do you adjust the reference motion to various digital siblings? Different meshes can have different convention for its coordinate frame. Do you need to do anything to align the siblings? How do you determine their initial placement?
3 comments
Response to Reviewer StAd (2/3): SPIDER-style baseline and ground-truth evaluation
(Part 2 of 3)
W1(b). Support the argument that RL is more effective than SPIDER when constructing noisy references.
Policy learning versus trajectory optimization. We implemented a SPIDER-style sampling-based optimizer in HandMimic's simulator and action space, with the same reconstructed references and retargeting. It adapts Do as I Do's SPIDER-derived optimizer rather than reproducing the official SPIDER. Each version optimizes one action sequence per demonstration, which is then replayed unchanged on every evaluation shape:
- Default: optimized on the reconstructed mesh, with up to 8 optimization iterations per planning step.
- More iterations: the same, with up to 16 iterations per step.
- On HandMimic's training meshes: one plan optimized jointly on the reconstructed mesh and its 12 siblings, so that it must work on all 13 (at fixed scale).
The HandMimic rows are the full method and the two training-geometry ablations from Table 3. All rows use the same per-step criterion and evaluation; the held-out set has 325 shape-demonstration pairs.
| SR (%) | Reconstructed | Real scan | Held-out shapes |
|---|---|---|---|
| SPIDER-style, default | 81.8 | 46.6 | 37.9 |
| SPIDER-style, more iterations | 84.8 | 45.2 | 40.1 |
| SPIDER-style, on HandMimic's training meshes | 49.8 | 46.3 | 33.8 |
| HandMimic (siblings + scale randomization) | 85.7 | 67.8 | 60.3 |
| HandMimic, scale randomization only | 85.6 | 62.1 | 48.6 |
| HandMimic, no randomization | 69.3 | 18.0 | 6.4 |
On the reconstructed mesh, the two approaches are comparable. On held-out shapes, HandMimic is 22.4 points higher (95% confidence interval: +10.9 to +33.3 points) and better on 23 of 28 demonstrations. Neither more iterations nor optimizing on HandMimic's training meshes closes this gap; optimizing on all 13 meshes even lowers success on the reconstructed mesh to 49.8%, suggesting that one fixed plan cannot fit every shape.
Neither method is re-optimized on held-out shapes: the optimizer replays its plan unchanged, while the trained policy adapts by observing each shape's geometry. The optimizer would have to be re-run for every new shape. The advantage therefore does not come from RL alone, and we will revise this claim in the paper.
Q1. For the evaluation, did you measure the error against the ground-truth human trajectory or the reconstructed reference used for training?
Evaluation target. We measure the error against the reconstructed reference, which is the policy's training target. Both 85.7% and 67.8% measure how well the policy tracks the reconstructed object trajectory. They differ only in the simulated object: the reconstructed mesh for 85.7%, and the object's YCB scan, resized to the reconstructed size, for 67.8%. We will rename the second condition "real scan", since it measures sensitivity to geometry rather than fidelity to the original interaction.
Fidelity to the original interaction. We additionally compare simulated object motion with DexYCB ground truth on the reconstructed mesh (600 rollouts per sequence):
| Trajectory | Displacement error vs GT (mm) | Relative-rotation error vs GT (deg) |
|---|---|---|
| HandMimic policy | 42.7 | 15.6 |
| Reconstructed reference | 40.8 | 14.4 |
| Static object | 96.1 | 31.4 |
Each error compares how far the object has moved and rotated since the first frame with the same quantity in ground truth. The motion executed by the policy is almost as close to ground truth as the reconstructed reference itself, with errors only 1.9 mm and 1.2 deg larger, and both are far closer than an object that never moves. We will add this analysis.
Response to Reviewer StAd (3/3): sibling alignment and related work
(Part 3 of 3)
Q2. How do you adjust the reference motion to various digital siblings? Different meshes can have different convention for its coordinate frame. Do you need to do anything to align the siblings? How do you determine their initial placement?
Aligning siblings and reusing the reference. Generated siblings align their reconstructed up direction with the demonstration's world-up; retrieved assets align their intrinsic up direction with the demonstrated object's. We then search rotations about this axis and refine with yaw-only ICP, first searching axis-aligned orientations when the up direction is inconsistent. Each mesh is scaled to the anchor's bounding-box diagonal and expressed in the anchor's coordinate frame, so the object trajectory and retargeted hand motion are reused without retargeting again. We will update Appendix B.3 to reflect this procedure.
Initial placement. Each sibling starts from the reference's initial object pose with a vertical offset chosen so its lowest point rests on the table at the grasp frame. The same offset is applied to the entire object and hand trajectories, preserving their relative motion.
W2: Related work. There are a number of relevant works not mentioned in the paper.
Video2Sim2Real / Human2sim2Robot / C2Dex
We will add these works. Human2Sim2Robot and Video2Sim2Real use RGB-D demonstrations, whereas HandMimic starts from monocular RGB. C2Dex shares the monocular setting and focuses on preserving demonstrated contacts during reconstruction and retargeting, followed by residual RL. Its reconstruction and retargeting code is listed as pending, so a direct comparison is not yet possible. HandMimic's distinct focus is training across sibling geometries from a single video-derived interaction and transferring to unseen shapes. Video2Sim2Real already uses scale randomization; our scale-only comparison isolates what siblings add beyond it (medium/far held-out success rises from 57.3%/21.6% to 70.2%/36.6%).
Following the reviewer's suggestion, we compared against Do as I Do with a shared reconstruction and against trajectory optimization in a shared simulator. HandMimic leads in the first comparison and on held-out shapes in the second.
Response to Reviewer StAd (1/3): real-world scope and Do as I Do comparison
(Part 1 of 3)
We thank the reviewer for recognizing the digital-sibling training and geometry ablations. Below we clarify the scope of the real-world results, extend the Do as I Do comparison, and add a SPIDER-style baseline in our own simulator.
Real-world scope. We agree that the real-world experiments do not demonstrate a closed-loop policy. They are open-loop replays of simulated rollouts, and closed-loop performance is evaluated only in simulation. For each real object, including the held-out ones, the policy is rolled out in simulation with a mesh of that object reconstructed from ZED2 RGB-D, and the resulting trajectory is replayed after FoundationPose registration (Appendix D). We will state this explicitly in the paper.
Path to closed-loop deployment. The current policy takes the object pose from the simulator as input. On the robot, this pose would have to be estimated continuously during manipulation, while the hand occludes the object, so we validated the learned motions by replay. A standard teacher-student scheme removes this dependency: the state-based policy, already trained with geometry, scale, and friction randomization, is distilled into a student that observes object point clouds. We will describe this path in the revision.
As the reviewer suggests, we therefore focus on fairer comparisons with Do as I Do. Because both methods currently deliver a trajectory to the robot, the comparisons below measure how well each method's motion tracks the demonstrated object motion under shared conditions.
Do as I Do's code release includes a whisking example but does not package its DexYCB evaluation clips, so we evaluate on our 28-sequence DexYCB subset rather than replicating its benchmark. We strengthen the Sec. 4.2 comparison by giving Do as I Do's released retargeter our reconstruction (XHand, without its UR3 arm), which separates the effect of reconstruction from that of physical correction:
Both stages contribute. Our reconstruction lifts Do as I Do from 17/28 to 24/28 under its own criterion, and HandMimic's retargeting and policy learning add 3 more; under the per-step criterion, they add 17 (from 7/28 to 24/28). Two differences remain: Do as I Do's preprocessing alters the target motion in two sequences, and the methods use different simulators and success definitions. The next comparison removes these differences by using a single simulator.