- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors proposed to use a generative model to do physics-based dexterous retargeting from human demonstrations, which they call GNR. Specifically, they use a conditional flow matching model to learn the residual correction control sequence delta u conditioned on the reference state trajectory and object geometry. This model is trained on MPC generated data, then proven to work to some degree on cases where MPC fails. It is capable to do test time scaling by sampling from the generative model many times. They developed a data engine using GNR and collected a physics-based dexterous manipulation dataset.
Critical strengths
- The authors focus on scaling up dexterous manipulation data, which is an important problem.
Critical weaknesses
- It is unclear whether GNR generated data can be beneficial for learning-based algorithms. In figure 17, the curves are too noisy to judge visually which dataset is significantly better. Although the best checkpoint SR shows GNR and GNR+MPC are stronger, the overall trends of the curves are too noisy to make this result convincing. Perhaps it is helpful to re-evaluate figure 17 using a larger number of test episodes to reduce noise and reveal a clearer trend. Also, this experiment should be in the main paper. As a paper that proposes a data generation method and data engine, it is very important to experimentally prove that the generated data (quantity plus quality) is beneficial for downstream learning methods while comparing against existing datasets.
- No real world experiments are provided.
- Experimental comparisons are only against analytic methods. The authors mentioned RL-based methods but never compared against them experimentally.
Questions for authors
- In the supp material coffee_pot folder, the human was grasping the body of the cup, while robot was grasping the handle of the cup, why is there an inconsistency?
- Could the authors provide insights into why GNR is able to generalize to OOD objects while only trained on 23 objects?
- In the comparisons of efficiency in Table 2, is GNR also evaluated using only CPU? Or is it allowed to use GPU? Does the MPC baseline only use CPU?
- What are the differences between OOD object and OOD geometry? The authors claim a dataset spanning 3.2k object geometries in the abstract, could they also provide the span of the number of distinct objects?
3 comments
The authors have revised and added experiments regarding the effectiveness of generated data on downstream learning methods, which experimentally verified that their neural retargeted data is beneficial towards robotic manipulation tasks. The concern regarding generalization from 23 source objects has also been explained using shared local geometries, augmentation, and test time scaling. Therefore, I believe this paper can be a valid reference for the dexterous manipulation community on scaling up data from humans, and have adjusted my overall evaluation accordingly.
Response to Reviewer xGTo
We are delighted to hear that our responses and revised draft have addressed your concerns, and we thank you for recommending acceptance!
Thank you for the thoughtful and constructive suggestions, and for acknowledging the importance of our research question of scaling up dexterous data and the value of GNR to the dexterous manipulation community.
To summarize, we added experiments demonstrating successful sim-to-real transfer with GNR-generated data on a highly challenging high-precision, long-horizon dexterous manipulation task, and showing the effectiveness of GNR for policy learning. Following your suggestion, we also provided insights into GNR's generalization: it results from both data scaling and test-time scaling, and objects with different overall geometries can share similar local geometry at contact points, which benefits from our real2sim pipeline that devlelops geometry data augmentation. We also added RL baseline experiments and GPU-accelerated results based on your comments.
Your feedback substantially strengthened both the completeness and the clarity of the paper. We truly appreciate your insight and time!
Response to Reviewer xGTo
We thank the reviewer for acknowledging the importance of scaling up dexterous demonstrations and for the constructive suggestions! We added several new experiments and clarifications to address the concerns. Real-robot and simulation demos are also included on the anonymous webpage for review: https://gnr-review.github.io/.
W2 — Real-robot experiments
We added real-robot experiments to Sec. 5.4 based on the suggestion. We are glad to show that a visuomotor policy trained with GNR data achieves zero-shot sim-to-real transfer on a highly challenging dexterous task. To our knowledge, this is also the first result in retargeting studies that achieves millimeter-precision policy sim-to-real transfer over long horizons for five-fingered dexterous manipulation. Dexterous retargeting and sim2real is generally very challenging, and state-of-the-art methods primarily retarget trajectories for tasks with 50–150 mm precision over a few seconds (e.g., CHORD, SPIDER, DexTrack).
W1 — Can GNR-generated data benefit downstream learning?
GNR can benefit downstream learning with matched or improved data quality while reducing dataset generation cost by 4x-11.2x. As suggested by the reviewer, we moved this experiment from the appendix to the main paper Sec. 5.4 alongside our new sim-to-real results. We added 3-seed mean/std results to Fig. 6 to compare policies trained on GNR, MPC, and GNR+MPC datasets. Each checkpoint on the curve is evaluated on 500 held-out episodes. At 200k training steps, policies trained on GNR and GNR+MPC data achieve 53.4% and 61.3% SR, respectively, compared with 55.2% for MPC. GNR yields similar policy success, and *GNR+MPC improves success by 6.1 pp, while GNR reduces data-generation cost by 11.2× and GNR+MPC by about 4×.
W3 — Comparison with RL baselines
We added two RL baselines, DexTrack and ManipTrans, to App. B.2. We adapt both to our simulator following their implementations. GNR uses 256 samples (13k environment steps) in open loop without policy optimization and achieves 56.1% SR. With 3,000 training rollouts (162k steps) per demonstration, DexTrack and ManipTrans reach 53.3% and 51.7%, at 32× and 75× higher CPU cost than GNR.
Q1 — Human and robot video demos in the supplementary material
We revised the video demos in App. K.3 and on the anonymous website so the human video and robot rollouts show the aligned motion.
Q2 — Why does GNR generalize when trained on 23 objects?
Following your suggestion, we added a discussion of GNR's generalization to App. B.4. This generalization benefits from both data and test-time scaling. Because the egocentric videos in HOT3D cover a limited set of objects, our real-to-sim engine broadens data coverage by stretching and deforming object meshes (see Fig. 12) and varying motions and physical parameters. Objects with different overall geometries can share similar local geometries at contact points, so the model can reuse contact patterns learned from the training objects. Test-time scaling complements data scaling by increasing the likelihood of generating a feasible trajectory with more samples. On OOD objects, GNR's SR increases from 12.81% to 44.33% as we increase the sample budget from 1 to 256.
Q3 — Can GNR use a GPU? Does MPC use a CPU?
GNR performance gains are agnostic to the selection of GPU and CPU. Both GNR and MPC use CPU for the retargeting comparison in Tab. 2 because SuperDex is a CPU-bound simulator to enable high-precision physics. GNR can also run with GPU-based simulation. We added clarifications in App. I where we compare GNR with SPIDER using GPU-accelerated MuJoCo Warp. GNR achieves 95% SR with 384 samples, compared with 96% for SPIDER with 16,384–32,768 samples.
Q4 — What is the difference between OOD objects and OOD geometries?
HOT3D has 33 distinct source objects (Fig. 11). OOD-object means the source object is unseen during training. OOD-geometry means the source object is seen during training and the test uses an unseen mesh variant (Fig. 12). We augment the original objects through methods including stretching and deformation (Fig. 12). The final dataset contains successful trajectories for 3.3k geometries across the 33 source objects (Tab. 16).
[1] Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration
[2] SPIDER Scalable Physics-Informed DExterous Retargeting
[3] DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References