acceptodds
Under review as a conference paper at ICLR 2027

Handmimic: Learning Dexterous Manipulations from Videos

10 views

Abstract

While humans naturally learn dexterous manipulation by watching others and practicing, robotic teleoperation of dexterous hands remains prohibitively expensive and non-trivial. We introduce HandMimic, a framework for learning dexterous manipulation policies directly from real-world monocular videos. We first reconstruct and align the observed hand-object interaction in simulation, while treating the resulting motion as an imperfect reference rather than a physically valid robot demonstration. We then practice in simulation using reinforcement learning to recover physically feasible behaviors from the reconstructed reference. To reduce dependence on a single reconstructed object, we further train across digital siblings with varying geometry and scale. Quantitative experiments show that policies learned from reconstructed references approach those trained from ground-truth references, while digital-sibling training improves generalization to unseen geometry and scale variations. We further validate the resulting motions on a real dexterous robot. Together, these results support using reconstructed human interactions as references for physical practice across varying object geometries.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_StAdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a method that reconstructs a monocular video of hand-object interaction and retargets it to a dexterous hand. The retargeted sequence is treated as an imperfect reference for RL to fix in simulation. During training, the object shapes and scales can be randomized to reduce overfitting to the reconstructed mesh. The results show that policies reach about 95% of ground-truth-reference performance. Training on "siblings" also increase the success rate from 18% to 68% on DexYCB.

Critical strengths
  1. Although there's no ground breaking idea in this paper, the design of the training pipeline with digital siblings is quite effective. It's a simple practice that does not add engineering or compute overhead. The results are compelling.
  2. The ablation experiments on geometry are valuable. In particular, the comparisons between scale randomization, retrieved siblings, and generated siblings are very informative.
Critical weaknesses
  1. There is no closed-loop policy with perception demonstrated in the real world. All real-world results are open-loop replay. This significantly weaken the claims against Do As I Do, which only solves the reconstruction problem but has demonstrated much wider set of open-loop reply in the real world. Without convincing evidence of closed-loop policy in the real world, it is crucial to show more fair comparisons against Do As I Do to establish the advantages of the proposed method. a. Compare on Do as I Do’s own benchmarks. b. Support the argument that RL is more effective than SPIDER when constructing noisy references.

  2. There are a number of relevant works not mentioned in the paper. It is challenging to establish novelty of this work without rigorous comparisons against a collective set of existing knowledge discovered previously.

  • Video2Sim2Real
  • Human2sim2Robot
  • C2Dex
Questions for authors
  1. For the evaluation, did you measure the error against the ground-truth human trajectory or the reconstructed reference used for training? The 85.7% success on reconstructed meshes versus 67.8% on ground-truth meshes is difficult to interpret as fidelity to the original interaction.
  2. How do you adjust the reference motion to various digital siblings? Different meshes can have different convention for its coordinate frame. Do you need to do anything to align the siblings? How do you determine their initial placement?
3 comments
AuthorsOpenReview ↗

Response to Reviewer StAd (1/3): real-world scope and Do as I Do comparison

(Part 1 of 3)

We thank the reviewer for recognizing the digital-sibling training and geometry ablations. Below we clarify the scope of the real-world results, extend the Do as I Do comparison, and add a SPIDER-style baseline in our own simulator.

W1. There is no closed-loop policy with perception demonstrated in the real world. All real-world results are open-loop replay.

Real-world scope. We agree that the real-world experiments do not demonstrate a closed-loop policy. They are open-loop replays of simulated rollouts, and closed-loop performance is evaluated only in simulation. For each real object, including the held-out ones, the policy is rolled out in simulation with a mesh of that object reconstructed from ZED2 RGB-D, and the resulting trajectory is replayed after FoundationPose registration (Appendix D). We will state this explicitly in the paper.

Path to closed-loop deployment. The current policy takes the object pose from the simulator as input. On the robot, this pose would have to be estimated continuously during manipulation, while the hand occludes the object, so we validated the learned motions by replay. A standard teacher-student scheme removes this dependency: the state-based policy, already trained with geometry, scale, and friction randomization, is distilled into a student that observes object point clouds. We will describe this path in the revision.

As the reviewer suggests, we therefore focus on fairer comparisons with Do as I Do. Because both methods currently deliver a trajectory to the robot, the comparisons below measure how well each method's motion tracks the demonstrated object motion under shared conditions.

W1(a). Compare on Do as I Do's own benchmarks.

Do as I Do's code release includes a whisking example but does not package its DexYCB evaluation clips, so we evaluate on our 28-sequence DexYCB subset rather than replicating its benchmark. We strengthen the Sec. 4.2 comparison by giving Do as I Do's released retargeter our reconstruction (XHand, without its UR3 arm), which separates the effect of reconstruction from that of physical correction:

Method / reconstruction inputMean error < 10 cm / 0.5 radPer-step error ≤\leq 6 cm / 30 deg
Do as I Do / its reconstruction17/280/28
Do as I Do / our reconstruction24/287/28
HandMimic / our reconstruction (Sec. 4.2, as reported)27/2824/28

Both stages contribute. Our reconstruction lifts Do as I Do from 17/28 to 24/28 under its own criterion, and HandMimic's retargeting and policy learning add 3 more; under the per-step criterion, they add 17 (from 7/28 to 24/28). Two differences remain: Do as I Do's preprocessing alters the target motion in two sequences, and the methods use different simulators and success definitions. The next comparison removes these differences by using a single simulator.

AuthorsOpenReview ↗

Response to Reviewer StAd (2/3): SPIDER-style baseline and ground-truth evaluation

(Part 2 of 3)

W1(b). Support the argument that RL is more effective than SPIDER when constructing noisy references.

Policy learning versus trajectory optimization. We implemented a SPIDER-style sampling-based optimizer in HandMimic's simulator and action space, with the same reconstructed references and retargeting. It adapts Do as I Do's SPIDER-derived optimizer rather than reproducing the official SPIDER. Each version optimizes one action sequence per demonstration, which is then replayed unchanged on every evaluation shape:

  • Default: optimized on the reconstructed mesh, with up to 8 optimization iterations per planning step.
  • More iterations: the same, with up to 16 iterations per step.
  • On HandMimic's training meshes: one plan optimized jointly on the reconstructed mesh and its 12 siblings, so that it must work on all 13 (at fixed scale).

The HandMimic rows are the full method and the two training-geometry ablations from Table 3. All rows use the same per-step criterion and evaluation; the held-out set has 325 shape-demonstration pairs.

SR (%)ReconstructedReal scanHeld-out shapes
SPIDER-style, default81.846.637.9
SPIDER-style, more iterations84.845.240.1
SPIDER-style, on HandMimic's training meshes49.846.333.8
HandMimic (siblings + scale randomization)85.767.860.3
HandMimic, scale randomization only85.662.148.6
HandMimic, no randomization69.318.06.4

On the reconstructed mesh, the two approaches are comparable. On held-out shapes, HandMimic is 22.4 points higher (95% confidence interval: +10.9 to +33.3 points) and better on 23 of 28 demonstrations. Neither more iterations nor optimizing on HandMimic's training meshes closes this gap; optimizing on all 13 meshes even lowers success on the reconstructed mesh to 49.8%, suggesting that one fixed plan cannot fit every shape.

Neither method is re-optimized on held-out shapes: the optimizer replays its plan unchanged, while the trained policy adapts by observing each shape's geometry. The optimizer would have to be re-run for every new shape. The advantage therefore does not come from RL alone, and we will revise this claim in the paper.

Q1. For the evaluation, did you measure the error against the ground-truth human trajectory or the reconstructed reference used for training?

Evaluation target. We measure the error against the reconstructed reference, which is the policy's training target. Both 85.7% and 67.8% measure how well the policy tracks the reconstructed object trajectory. They differ only in the simulated object: the reconstructed mesh for 85.7%, and the object's YCB scan, resized to the reconstructed size, for 67.8%. We will rename the second condition "real scan", since it measures sensitivity to geometry rather than fidelity to the original interaction.

Fidelity to the original interaction. We additionally compare simulated object motion with DexYCB ground truth on the reconstructed mesh (600 rollouts per sequence):

TrajectoryDisplacement error vs GT (mm)Relative-rotation error vs GT (deg)
HandMimic policy42.715.6
Reconstructed reference40.814.4
Static object96.131.4

Each error compares how far the object has moved and rotated since the first frame with the same quantity in ground truth. The motion executed by the policy is almost as close to ground truth as the reconstructed reference itself, with errors only 1.9 mm and 1.2 deg larger, and both are far closer than an object that never moves. We will add this analysis.

AuthorsOpenReview ↗

Response to Reviewer StAd (3/3): sibling alignment and related work

(Part 3 of 3)

Q2. How do you adjust the reference motion to various digital siblings? Different meshes can have different convention for its coordinate frame. Do you need to do anything to align the siblings? How do you determine their initial placement?

Aligning siblings and reusing the reference. Generated siblings align their reconstructed up direction with the demonstration's world-up; retrieved assets align their intrinsic up direction with the demonstrated object's. We then search rotations about this axis and refine with yaw-only ICP, first searching axis-aligned orientations when the up direction is inconsistent. Each mesh is scaled to the anchor's bounding-box diagonal and expressed in the anchor's coordinate frame, so the object trajectory and retargeted hand motion are reused without retargeting again. We will update Appendix B.3 to reflect this procedure.

Initial placement. Each sibling starts from the reference's initial object pose with a vertical offset chosen so its lowest point rests on the table at the grasp frame. The same offset is applied to the entire object and hand trajectories, preserving their relative motion.

W2: Related work. There are a number of relevant works not mentioned in the paper.

Video2Sim2Real / Human2sim2Robot / C2Dex

We will add these works. Human2Sim2Robot and Video2Sim2Real use RGB-D demonstrations, whereas HandMimic starts from monocular RGB. C2Dex shares the monocular setting and focuses on preserving demonstrated contacts during reconstruction and retargeting, followed by residual RL. Its reconstruction and retargeting code is listed as pending, so a direct comparison is not yet possible. HandMimic's distinct focus is training across sibling geometries from a single video-derived interaction and transferring to unseen shapes. Video2Sim2Real already uses scale randomization; our scale-only comparison isolates what siblings add beyond it (medium/far held-out success rises from 57.3%/21.6% to 70.2%/36.6%).

Following the reviewer's suggestion, we compared against Do as I Do with a shared reconstruction and against trajectory optimization in a shared simulator. HandMimic leads in the first comparison and on held-out shapes in the second.

Reviewer_8P8ZOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper reconstructs hands, objects, and their motions from monocular videos of human manipulation, retargets the hand motions to a robot, and subsequently derives physically executable actions through reinforcement learning in simulation. The authors validate the method using six real-world video sequences, and experiments on a physical robot. When using reconstructed references, the success rate in simulation approaches that achieved with GT annotations.

Critical strengths
  • The overall experimental workflow is comprehensive, covering from video-based reconstruction and motion retargeting to simulation training and real-world execution.
  • The experimental results appear promising, rather than requiring the robot to precisely mimic human hands, the approach focuses on the motion of the objects, utilizing reinforcement learning to adjust contact strategies. Policies trained on reconstructed references achieve a success rate close to that of policies trained on ground-truth references.
Critical weaknesses

The method seems to combine existing approaches, and I'm not sure why each design choice is needed. The reconstruction relies heavily on off-the-shelf tools, while the policy learning follows ManipTrans with changes to the rewards and training setup. The paper does not test these changes separately, so it's hard to tell how much comes from increasing the object reward weight or removing early termination based on hand tracking. Table 3 also shows that scale randomization alone raises the success rate on real-world geometry from 18.0% to 62.1%. Adding generated “siblings” brings it to 67.8%. Given that most of the gain comes from scale randomization, I'd like to see clearer evidence of what the added components offer over simpler alternatives.

Questions for authors

Please see the weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8P8Z: reward and termination ablations, benefit beyond scale

We thank the reviewer for the positive assessment. The new ablations below test the two training changes separately, and we clarify what siblings add beyond scale randomization.

W1: Design choices. The paper does not test these changes separately, so it's hard to tell how much comes from increasing the object reward weight or removing early termination based on hand tracking.

Reward and termination. We revert each change to ManipTrans's version within HandMimic, one at a time, on all 28 sequences. As the reference, we retrain unmodified HandMimic under the same conditions.

Success rate (%)HandMimicwith ManipTrans's rewardwith ManipTrans's termination
Reconstructed mesh85.562.657.1
Real scan64.255.150.0

Each change matters on its own. ManipTrans’s reward uses an object-orientation weight of 1 instead of our 5, and applies the fingertip-force term at every frame rather than only after grasp. Under this reward, 7 sequences that HandMimic solves on the reconstructed mesh now fail, and the failures are telling: the object stays within 4 mm of its reference position, but its orientation is off by 34 deg. ManipTrans’s termination rule makes 8 solved sequences fail for a different reason. It ends an episode whenever the hand strays from its reference after grasp, yet keeping the object on track often requires exactly that. Unmodified HandMimic crosses ManipTrans’s thresholds in 57% of training episodes.

Reconstruction alignment. Starting from the same perception outputs on the 28 sequences, scale calibration reduces object-size error from 11.5% to 6.9%. Together with translation alignment, it reduces the hand–object error at the grasp frame from 86.3 mm to 18.8 mm. Both corrections matter in simulation, where the policy practices with the object at its estimated scale and position relative to the hand.

W2: Benefit beyond scale. Given that most of the gain comes from scale randomization, I'd like to see clearer evidence of what the added components offer over simpler alternatives.

Beyond scale. We agree that scale randomization explains most of the real-scan gain: 44.1 points, versus 5.7 from siblings. The real scan is the same object as the reconstruction, resized to the reconstruction's bounding-box diagonal, so it differs only by small errors in proportions and local shape. Varying the object's size during training already makes the policy robust to such errors.

Siblings target a different case: objects whose shape differs from the demonstrated one. With the same training budget, they raise success on 325 held-out DexYCB shape–demonstration pairs from 48.6% to 60.3%. The gain grows with shape difference: from 72.5% to 80.3% on close shapes, 57.3% to 70.2% on medium shapes, and 21.6% to 36.6% on far shapes. On held-out in-the-wild objects, success rises from 30.7% to 43.3%.

Retrieved siblings give similar gains on DexYCB, so the main ingredient is practicing across multiple geometries. Generation needs no asset library containing the object's category, and it does better on the six in-the-wild videos, reaching 43.3% versus 36.9% for retrieval, where matching assets are likely scarcer.

Reviewer_RSXNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

HandMimic learns dexterous manipulation policies from monocular human videos. It addresses the challenge of learning physically feasible manipulation from inaccurate reconstructions by treating reconstructed interactions as imperfect kinematic references and using object-focused reinforcement learning across varied object geometries to recover executable behaviors. Specifically, It combines pretrained reconstruction models with object-scale calibration and hand–object alignment, then retargets the recovered motion into a kinematic reference. Reinforcement learning prioritizes object motion while allowing deviations from the hand reference. Training across object shape and scale variations further reduces dependence on a single reconstructed mesh. Experiments on a selected DexYCB subset and in-the-wild videos evaluate reconstruction, manipulation success, and geometry generalization, with additional validation through open-loop execution on a physical robot.

Critical strengths

Simple and effective improvements to an existing framework. Object-scale calibration, hand–object alignment, and geometry augmentation form a practical approach to improving manipulation learning from imperfect video reconstructions. Under the reported reconstructed-demonstration protocol with ground-truth meshes, success increases from 6.0% for ManipTrans to 67.8% for HandMimic; additional shape variation also improves generalization beyond scale augmentation alone. These results support the practical value of the combined system.

Critical weaknesses

The methodological novelty is limited and insufficiently isolated. The reconstruction pipeline combines existing estimators with incremental alignment refinements. Learning physically feasible behavior from imperfect references is already implemented through residual learning in ManipTrans and addressed explicitly for noisy video reconstructions in DexMan. Digital-sibling training is closely related to geometry domain randomization, with much of the reported improvement coming from scale augmentation. Without targeted component ablations and a direct comparison with DexMan, the paper does not clearly establish a substantive methodological advance beyond an effective combination of existing techniques and training adaptations. This limits the significance of its three claimed contributions.

Questions for authors

1, Could the authors clarify the specific methodological relationships to prior work, particularly ManipTrans and DexMan, and distinguish the proposed contributions from existing reconstruction, reference-based policy learning, and geometry augmentation techniques?

  1. What accounts for the substantial performance improvement over ManipTrans? A more detailed explanation or targeted ablations separating the effects of reward weighting, termination criteria, training initialization, and geometry augmentation would help identify the sources of this improvement.

  2. Could the authors clarify the selection criteria for the 28-sequence DexYCB evaluation subset, including the initial sequence selection before visibility filtering? How representative is this subset, and what supports its adequacy for validating the claimed performance improvements?

3 comments
AuthorsOpenReview ↗

Response to Reviewer RSXN (1/3): relation to ManipTrans and DexMan

(Part 1 of 3)

We thank the reviewer for the detailed feedback. Below, we decompose the gain over ManipTrans, ablate the reward and termination changes, and detail the subset selection, with reconstruction results on all 80 sampled sequences.

Q1 / W1. Could the authors clarify the specific methodological relationships to prior work, particularly ManipTrans and DexMan, and distinguish the proposed contributions from existing techniques?

Contribution. Our results show where learning from video references breaks down: a policy trained on the single reconstructed mesh overfits to it, reaching 69.3% on that mesh but only 18.0% on the real object (Table 3). HandMimic counters this by training, from a single video-derived interaction, across varied geometries. It reaches 67.8% on the real object, close to the 71.4% obtained from ground-truth references (Table 2), and we evaluate transfer on held-out real scans. Neither ManipTrans nor DexMan reports training across varied geometries or evaluating on held-out shapes. HandMimic builds on established reference-based RL; its contribution is a full pipeline from monocular video to policy, together with training across digital siblings and scales, which counters overfitting to the reconstructed mesh. We will make this positioning explicit in the contribution statements.

Relation to ManipTrans. ManipTrans assumes accurate hand–object references from MoCap. It first pretrains a policy to imitate human hand motion, then adds a residual policy for object interaction. HandMimic instead starts from monocular RGB video and covers the full pipeline from reconstruction to policy, so its references carry errors in pose, alignment, scale, and geometry that MoCap data largely avoid. ManipTrans is sensitive to such errors: its success drops from 75.7% with ground-truth references to 35.4% with our reconstructed ones (Table 2). HandMimic's design targets these errors. It trains the policy from scratch across digital siblings and scales, weights object tracking above hand tracking, and removes hand-tracking termination after grasp (Sec. 3.2–3.4).

Relation to DexMan. DexMan also learns from noisy monocular references, using a contact-prior reward that attracts hand keypoints to designated object-surface regions. HandMimic uses the demonstrated contacts only for alignment and retargeting. After grasp, its reward encourages fingertip contact force without specifying where to touch, so contacts can adapt to each geometry. DexMan's public repository contains only a README, so our comparison is methodological rather than numerical.

AuthorsOpenReview ↗

Response to Reviewer RSXN (2/3): sources of the gain and alignment ablation

(Part 2 of 3)

Q2 / W1. What accounts for the substantial performance improvement over ManipTrans? ... targeted ablations separating the effects of reward weighting, termination criteria, training initialization, and geometry augmentation ...

Where the gain comes from. Tables 2 and 3 share the same evaluation protocol and together decompose the gain:

SR (%)ManipTransHandMimic, single mesh+ scale randomization+ generated siblings (full)
Reconstructed mesh35.469.385.685.7
Real scan6.018.062.167.8
Held-out shapes–6.448.660.3

On the reconstructed mesh, most of the gain comes from the learner changes. On the real scan, where the reconstructed geometry is inaccurate, most comes from geometry augmentation. On held-out shapes, siblings add 11.7 points over scale randomization alone.

Reward and termination. HandMimic differs from ManipTrans in two reward terms and one termination rule. Our reward weights object orientation at 5 instead of 1 and applies the fingertip-force term only after grasp (weight 2, instead of 1 at every frame); after grasp, our termination keeps only the object checks and drops ManipTrans's fingertip-deviation and finger-contact checks. We revert each choice to ManipTrans's exact version within HandMimic, one at a time, on all 28 sequences, keeping everything else fixed. As the reference, we retrain unmodified HandMimic under the same conditions (85.5% / 64.2%; paper: 85.7% / 67.8%).

SR (%), 28 sequencesHandMimic (retrained)ManipTrans rewardManipTrans post-grasp termination
Reconstructed mesh85.562.657.1
Real scan64.255.150.0

On the reconstructed mesh, ManipTrans's reward turns 7 solved sequences into failures, and its termination rule turns 8. The failures differ in kind. With ManipTrans's reward, failing policies keep the object within 4 mm of the reference position but miss its orientation by 34 deg. With ManipTrans's termination, they lose the object's position: keeping the object on track often requires the hand to depart from the reconstructed reference, and in 57% of unmodified HandMimic's training episodes, the hand exceeds ManipTrans's post-grasp fingertip-deviation thresholds. On the real scan, both changes also lower success.

W1: Reconstruction alignment. The reconstruction pipeline combines existing estimators with incremental alignment refinements.

Contribution of alignment. From identical perception outputs on the 28 sequences, scale calibration reduces mean object-size error from 11.5% to 6.9%. Grasp-frame hand–object placement error, measured against ground truth without alignment, falls from 86.3 mm (raw) to 33.7 mm after scale calibration and 18.8 mm after translation alignment. These corrections matter in simulation, where the policy practices with the object at its estimated scale and position relative to the hand. We will add this ablation to the paper.

AuthorsOpenReview ↗

Response to Reviewer RSXN (3/3): subset selection and results on all 80 sequences

(Part 3 of 3)

Q3. Could the authors clarify the selection criteria for the 28-sequence DexYCB evaluation subset, including the initial sequence selection before visibility filtering? How representative is this subset, and what supports its adequacy for validating the claimed performance improvements?

Selection. We start from all 501 right-hand DexYCB sequences. Using the dataset annotations, we keep the 255 sequences in which the object is segmented and fully inside the image in every frame, and the hand is visible once it enters. From these, we randomly sample four sequences per object, giving 80 sequences across all 20 objects. To limit reconstruction and policy-training cost, we evaluate eight of these objects: master chef can, sugar box, tomato soup can, mustard bottle, pudding box, potted meat can, bowl, and power drill. The visibility filter in Sec. 4 (mean visible-object fraction ≥0.65\geq 0.65) removes one bowl and three power-drill sequences, leaving 28.

All 80 sampled sequences. To test whether this choice favours HandMimic, we ran both methods on the 48 sequences of the remaining 12 objects, so the results now cover all 20 DexYCB objects. The sequence list was fixed by the sample before any run, and we used Table 1's code and evaluation, with no tuning or per-sequence intervention. Below are the means without visibility filtering:

SequencesMethodPA-MPJPE ↓MRPE ↓F5 ↑F10 ↑CD ↓JM ↓
Paper's 8 objects (32)Do as I Do3.6979.713.527.226.43.89
HandMimic3.9659.020.640.220.40.64
Remaining 12 objects (48)Do as I Do4.2390.413.025.428.33.90
HandMimic4.4982.117.232.824.70.73
All 80Do as I Do4.0186.113.226.127.63.90
HandMimic4.2872.918.635.723.00.69

Every Table 1 ranking holds on the new objects: HandMimic is better on MRPE, F5, F10, CD, and JM. Do as I Do could not be scored on one new sequence. Policy evaluation remains on the 28 sequences. We will add the selection protocol and the per-sequence results for all 80 sequences.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. We are glad that Reviewer RSXN recognizes the practical value of the combined system, Reviewer 8P8Z highlights the comprehensive workflow and promising results, and Reviewer StAd finds digital-sibling training effective and the geometry ablations informative.

New evidence. We add (1) reward and termination ablations on all 28 sequences, and a reconstruction-alignment ablation (RSXN, 8P8Z); (2) a SPIDER-style trajectory-optimization baseline in our simulator, an extended Do as I Do comparison, and evaluation against ground-truth object motion (StAd); and (3) reconstruction results on all 80 sampled sequences, covering all 20 DexYCB objects (RSXN).

Main finding. The new and existing results point to one conclusion. A policy trained on the single reconstructed mesh overfits to it: it reaches 69.3% on that mesh but only 18.0% on the real object and 6.4% on held-out shapes (Table 3). Practicing across varied geometries counters this overfitting. HandMimic reaches 67.8% on the real object, close to the 71.4% obtained from ground-truth references (Table 2), and 60.3% on 325 held-out shape–demonstration pairs, compared with 48.6% with scale randomization alone and 37.9% for the SPIDER-style optimizer.

Revision. We will add the new results, protocol details, and related work to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.